Related Work on Occlusion Detection in Front Projection Environments

The problem of occlusion detection in front projection environments has been addressed in the context of various applications, such as shadow removal, occluder light suppression, as well as hand detection and tracking for gesture recognition. Current occlusion detection techniques can be divided into two groups, namely direct and indirect occlusion detection. The former approach locates the occluding object directly in the scene, while the latter detects an occlusion indirectly by locating its more easily discernible shadow. Previous occlusion detection techniques and their target applications are discussed below.

Indirect Occlusion Detection

Shadow Removal (or Virtual Rear Projection)

While the occurence of shadows can be reduced by mounting the projector off-axis from the display surface, recent research has addressed the problem directly using Virtual Rear Projection (VRP), where multiple projectors redundantly illuminate the display (the term Virtual Rear Projection was introduced by Summet et al. [Summet03StudyVRP] to encompass such techniques).

Passive VRP (PVRP) simulates rear projection with two overlapping projectors that each illuminate the full display at half-intensity; occluding one projector results in 'half-shadows' where output is still visible albeit at a lower contrast level [Summet03StudyVRP].

Active VRP (AVRP) involves the use of a camera to track and remove shadows dynamically while the display is in use. When one projector is occluded, detected shadow are filled in selectively by an unoccluded projector [Jaynes01] [Sukthankar02Shadow] [Flagg03Switched] [Jaynes04]. This contrasts with PVRP in that visual feedback from the camera allows for adaptive control of projector output in shadow regions to attain the desired display intensity.

Sukthankar et al. [Sukthankar02Shadow] and Jaynes et al. [Jaynes01] detect occluded regions by performing a pixel-wise comparison between predicted and captured camera images. In later work, Jaynes et al. propose a new, more efficient region-based shadow removal approach [Jaynes04].

The shadow removal systems introduced by Sukthankar et al. and Jaynes et al. differ in that the former cannot support dynamic displays. Any image that is to be projected when the display is in use must be made available during system initialization for pre-generation of predicted camera images. The shadow removal systems described in references [Cham03Suppression] and [Flagg03Switched] are based on the same framework. Alternatively, Jaynes et al. use geometric and color calibration data to warp and color-correct the current projector framebuffer image at run-time, in order to dynamically synthesize predicted camera images for arbitrary projected displays.

Occluder Light Suppression

Based on the detected shadow position, Cham et al. [Cham03Suppression] infer the location of the user who is occluding the display. Then, in addition to removing shadows by selectively intensifying the output of a redundant projector, corresponding pixels in the occluded projector are also turned off to simultaneously prevent the projection of distracting light onto the user.

For applications in which only suppression of projected light on the user is required, Tan and Pausch present an alternative technique that detects shadows in infrared camera images [Tan02].

Identifying the Occluded Projector in AVRP Displays

A subproblem of occlusion detection pertaining to AVRP systems involves identifying which projector is being occluded. The dual-projector shadow removal system of Sukthankar et al. [Sukthankar02Shadow] does not perform this task. When compensating for shadows, all projectors are instructed to increase their output intensity simultaneously in the appropriate display regions. This invariably results not only in shadows being filled in by the unoccluded projector, but also in additional light being projected unnecessarily onto the user by the occluded one.

This problem can be solved by probing each projector in the event of occlusion: each projector is serially instructed to vary its output intensity by a small amount; if no subsequent change is detected in the camera image, then the current projector is determined to be occluded. This allows for the simultaneous removal of shadow and suppression of occluder light [Cham03Suppression].

The disadvantage of cyclical probing is that additional rendering iterations are required to identify the occluded projector. Flagg et al. eliminate the need for probing using a dual-projector binary switching display system for fast shadow elimination and occluder light suppression [Flagg03Switched].

Motivation for the Adopted Approach to AVRP

For our research on shadow detection and removal in the Shared Reality Environment, we adopted an occlusion detection approach based on camera-projector geometric and color calibration, similar to that of Jaynes et al. [Jaynes01], as it inherently provides the required support for dynamic projected content. For shadow removal, we used an XOR display configuration similar to that proposed by Flagg et al. [Flagg03Switched], where each display pixel is always assigned to exactly one projector, to remove the need for projector probing and enable a faster system response to occlusions.

Direct Occlusion Detection

Hand Detection and Tracking

Various direct occlusion detection techniques have been proposed to support gestural interfacing with front-projected displays. Some standard detection techniques used in computer vision have shown a certain degree of success, such as those relying on frame differencing [Pinhanez01Touchscreens] or background subtraction [vonHardenberg01]. However, these approaches are not robust in the case of significant or sudden changes in projected content.

A simple way to deal with dynamic projection is to force user interaction to occur only in designated control areas outside the projected display region [Wellner93] [vonHardenberg01] [Takao03], although imposing constraints on user movement is often undesirable. Sato et al. use of a thermal infrared camera to detect hands by sensing objects within body temperature range [Sato00]. Camera-projector synchronization can also be used to detect occlusions by capturing images of the scene when projection is momentarily turned off. However, the latter two approaches require expensive specialized equipment.

Alternatively, an occlusion detection technique based on camera-projector geometric and color calibration enables the dynamic synthesis of predicted camera views through geometric and color correction of projector images. This allows for improved background segmentation without the need for specialized hardware and facilitates detection despite unpredictable changes in projected background and illumination. Indeed, while our implemented calibration-based detection system was intended for dynamic shadow detection and removal, early results suggested that it could well find application for other HCI tasks relevant to interaction in immersive environments, for example, hand detection for gesture recognition or person detection and tracking [Hilario04]. Similar independent research conducted by Licsar and Sziranyi also adopts a calibration-based approach to dynamic background generation and subtraction [Licsar04].


Last update: 22 June 2005