DETAILS


COLUMNS


CONTRIBUTIONS

a a a

VISFILES

Vol.32 No.1 February 1998
ACM SIGGRAPH



Augmented Reality: "The Future's So Bright, I Gotta Wear (See-through) Shades"



T.Todd Elvins
San Diego Supercomputer Center


February 98 Columns
Entertaining the Future Images and Reversals


T.Todd Elvins
Previous article by T.Todd Elvins Next article by T.Todd Elvins

Introduction

The fledgling field of wearable computers, where augmented reality (AR) technologies will play a critical role, recently attracted 400 enthusiasts to MIT for the Wearables Computer Symposium. (Editor's note: See Joan Truckenbrod's symposium review.) The well-established ACM UIST Conference, held the following week, gathered only half that number. I was intrigued by this phenomenon and decided to learn more about AR. The most up-to-date reference is the August 1997 issue of the MIT Press journal Presence: Teleoperators and Virtual Reality [4] which is dedicated to describing state-of-the-art AR (abstracts are under the "Science and Technology" category at the MIT Press web site). Distilled from the August issue of Presence and a few other sources, the following article gives a quick introduction to AR.

AR, sometimes called mediated reality, has an interesting past and a bright future. It will probably surprise no one that Ivan Sutherland, the father of interactive computer graphics, is also credited with inventing AR. In 1968, Sutherland built a head-mounted display (HMD) with monochrome monitors and mirrors. The half-silvered mirrors enabled the observer to see through the computer graphics display into the real world. That was 30 years ago. It seems likely that 30 years (or less) from now some people will choose to be constantly on-line via lightweight non-obtrusive glasses capable of dynamically compositing or blending high-resolution stereo 3D graphics with a view of the real world.

Blending Reality

Real world imagery and computer graphics can be mixed in combinations from 100 percent real to 100 percent virtual, and all percentages in between. "AR displays are those in which the image is of a primarily real environment, which is enhanced, or augmented, with [3D] computer-generated imagery." [2] As opposed to virtual reality (VR), AR displays supplement reality rather than replace it [2]. For example, a view of the space shuttle cockpit could be dynamically overlaid with computer-generated wiring diagrams to assist a technician during maintenance procedures.

In Azuma's recent AR survey [1], the author writes that AR is any system that is characterized by: (1) a combination of real and virtual, (2) interaction in real time, and (3) real and virtual scenes registered in three dimensions. AR has the potential to enhance an observer's perception of the real world giving the observer superman-like vision. Augmentations might display information not readily detectable by human senses, such as radiation emissions or the interior of a laser printer [3]. Another mixed reality blending ratio, called augmented virtual reality, supplements a virtual environment (VE) with views from the real world. This type of augmentation is useful when a VE visitor needs to retain access to and awareness of real world objects and views. For example, a worker on a virtual assembly line might wish to see a video view of an actual assembled part further down the conveyor belt.

Applications

Numerous AR applications have been proposed, discussed and attempted. Few, if any however, are in use today. Perhaps the most mature AR applications are visualizations of threat envelopes displayed on heads up displays for military pilots, and AR systems for aircraft manufacturing where 3D overlays of wire bundles have been demonstrated. Also under development are medical and medical training systems that will allow physicians to see 3D MRI, CT or ultrasound data in place within a patient. AR may someday also be used as a surgical assistant or as an in-view display for procedure guides.

Tuceryan, et al, have demonstrated a prototype AR system called GRASP which annotates and describes complex machinery such as automobile engines [7]. This capability could be helpful in a number of assembly, maintenance and repair scenarios. Other applications of AR include data and information visualization, hazardous environment operations, navigation, entertainment and urban planning. An urban planning scenario might involve studying many alternative uses for a valuable downtown lot. By donning a mobile HMD and walking around the site, models of the proposed construction outcomes could be seen in place of the current buildings.

AR Versus VR

There are still many unsolved problems in both AR and VR and several interesting comparisons in their respective utility. Where VR users do not have visual access to the real world, AR users can concentrate on physical reality but have quick access to 3D computer overlays if needed. At the other end of the blending spectrum, AR users can immerse themselves in the 3D computer model but still have situational awareness of their physical surroundings. In terms of safety, some AR displays degrade gracefully when sensors or displays fail, defaulting to 100 percent real world. When a VR system fails, observers can be temporarily frozen in place, or sightless. In terms of rendering requirements, a VR system must render all of the pixels on the display while an AR system may need to render only a portion of the pixels. One final AR advantage is that for some applications, a low-resolution, even monochrome monitor, is adequate.

Because AR does not provide a synthetic environment, AR HMDs must be worn in the actual physical work environment to be useful. For AR systems to come into common use, they will have to become robust mobile computing devices with a means for monitoring the exact location and orientation of the HMD. AR has several years of R&D ahead before it will reach the current technological maturity of VR. While AR has yet to be deployed to the workplace, hundreds, perhaps thousands, of commercially built immersive VR systems have been purchased and installed, and are now used on a daily basis.

While both VR and AR make use of six degree of freedom (DOF) tracking devices, tracker latency is not as critical in VR because the scene refresh can be delayed until the update from the device arrives. This is not the case for AR systems which require tracker updates to keep up with observer head and hand motion and possibly with other moving objects in the physical environment. If the tracker(s) fall behind, or cause delays in overlay rendering, observer disorientation and even motion sickness may result [2]. Fast tracking is also required to ensure that virtual and physical objects do not interpenetrate in the 3D space. Unfortunately, the update rate and accuracy of today's commercially produced trackers are insufficient for AR purposes. Although video cameras and laser range finders can be used to track and/or verify the locations of objects in the physical world via computer vision techniques, these algorithms are currently too CPU intensive to be helpful.

Another VR advantage is that tracked real world objects (such as the observer's hand) do not have to be precisely registered since real world objects and virtual objects are all rendered in the same image. AR requires precise registration because overlays off by even one pixel are immediately noticeable. For an AR overlay to be exactly registered, the virtual camera parameters must exactly match the real world camera parameters. Cameras attached to moving objects present an additional challenge as the camera must remain calibrated at all times.

Optical Versus Video

There are currently two AR display technologies, optical see-through head-mounted displays (OSTHMD), and video see-through head-mounted displays (VSTHMD). OSTHMDs use half-silvered mirror devices in front of the eyes and currently impose a relatively narrow field of view. Light from the real world passes through the half-silvered mirror into the observer's eyes much as light passes through a pair of sunglasses. Light from a small computer monitor is reflected off the mirror and also passes into the observer's eyes. If the computer graphics are correctly registered, 3D rendered objects appear in reasonable places in the real world scene. One difficult issue with this type of display is determining the proper ratio of real world light to monitor-generated light. Another issue is that graphics displayed via a half-silvered mirror do not fully obscure real world objects. The CG images appear somewhat translucent, which may or may not be a problem depending on the application. Yet another issue is that OSTHMDs cannot delay the real world image even a single refresh to give the CG images a chance to keep up. Finally, since OSTHMDs do not capture video, there is no source of ancillary data by which to aid in tracking scene objects and the HMD itself.

VSTHMDs do not actually let the observer see-through, but instead blend computer graphics with video captured with cameras mounted just above the observer's eyes. The resulting composited images are displayed in the HMD video monitors. By ignoring the video camera streams and displaying only virtual content, the VSTHMD becomes a VR system. As with optical HMDs, video HMDs have some problems. First, digitizing the frames of video from the two video cameras takes a non-zero amount of time that must be added to the time required to composite the graphics onto the digitized images and send the result to the displays. Additionally, the digitized images should be corrected for camera distortion before the compositing step. Another issue is that, while optical HMDs allow the observer to see the real world, video HMDs reduce the world to the resolution of the video camera, or the resolution of the HMD display, whichever is less. If the HMD cameras and monitors each resolve 640x480 pixels, then the observer's view of the world will be limited to this resolution. There are also many issues of matching the focus and brightness of the digitized video and graphics overlays. Finally, VSTHMDs do not fail gracefully. If the system crashes, the observer is sightless until they remove the HMD.

Registration

Registration errors are differences between the observed 3D position of real world objects and their corresponding computer-generated representation [1]. There are many sources of static and dynamic registration errors in AR. Static errors are those that appear when the observer and the environment are stationary, and dynamic errors are those that appear when things are moving. Static errors are caused by factors such as optical distortion, tracking inaccuracy, misalignments in OSTHMD optical components and incorrect viewing parameters. Dynamic errors are caused by system delays, or lags, and are the main contributor of registration errors. Lag is the time between a real world movement, and the point at which the updated image corresponding to that movement is displayed in the HMD. Lag can be reduced in a number of ways such as optimizing system performance, taking advantage of the VSTHMD video digitizing delay, labeling real world objects with fiducials, incorporating range finder data and applying clever techniques such as prediction and image deflection. Dynamic errors are an especially important and active research problem because, as discussed earlier, graphics that trail behind their corresponding real world objects can cause observer dizziness and nausea [2].

Developing techniques for calibrating the real world camera parameters (the observer's eye for OSTHMDs) and the synthetic world camera is another important research problem. For a thorough discussion of image, camera, pointer, tracker and object calibration, see [7].

T.Todd Elvins is a Staff Scientist at San Diego Supercomputer Center. He recently finished his Computer Engineering Ph.D. at the University of California, San Diego, and his research interests include perceptually based user interface design, data and information visualization, Web-based imaging and computer graphics. He can be contacted at:




T.Todd Elvins
San Diego Supercomputer Center
University of California
San Diego
MC 0505
La Jolla, CA
92093-0505, USA

Web site

The copyright of articles and images printed remains with the author unless otherwise indicated.

Future Work

It is clear that AR has a potentially large number of exciting applications. However, a number of technologies must mature before AR can attain this bright future: HMDs must offer higher resolution monitors, greater comfort and become less conspicuous (some interesting alternatives to head-worn displays are hand-held AR [5] and monitor-based AR [4]). Time critical rendering algorithms, such as just-in-time incorporation of tracker measurements, are needed, as are methods for predictive and hybrid tracking. Research into eye tracking technologies has the potential to yield registration improvements, and further research is needed into techniques for system delay reduction. Finally, issues of portability must be resolved so that observers can wear these new improved systems at any location where there is work or play to be done.

References

  1. Azuma, Ronald T. "A survey of augmented reality," Presence: Teleoperators and Virtual Reality, 6(4), August 1997, pp. 355-385.
  2. Drascic, David and Paul Milgram. "Perceptual issues in augmented reality," Stereoscopic Displays and Virtual Reality Systems III, Proceedings of the SPIE, volume 2653, 1996, pp. 123-134.
  3. Feiner, Steven, Blair MacIntyre and Doree Seligmann, "Knowledge-based augmented reality," Communications of the ACM, 36(7), July 1993, pp. 52-62.
  4. Klinker, Gudrun J., Klaus H. Ahlers, David E. Breen, Pierre-Yves Chevalier, Chris Crampton, Douglas S. Greer, Dieter Koller, Andre Kramer, Eric Rose, Mihran Tuceryan and Ross T. Whitaker. "Confluence of computer vision and interactive graphics for augmented reality," Presence: Teleoperators and Virtual Reality, 6(4), August 1997, pp. 433-451.
  5. Rekimoto, Jun. "NaviCam: a magnifying glass approach to augmented reality," Presence: Teleoperators and Virtual Reality, 6(4), August 1997, pp. 399-412.
  6. Sutherland, Ivan. "A head mounted three dimensional display," Proceedings of the 1968 Fall Joint Computer Conference, AFIPS, 33, 1968, pp. 757-764.
  7. Tuceryan, Mihran, Douglas S. Greer, Ross T. Whitaker, David E. Breen, Chris Crampton, Eric Rose and Klaus H. Ahlers. "Calibration requirements and procedures for a monitor-based augmented reality system," IEEE Transactions on Visualization and Computer Graphics, 1(3), September, 1995, pp. 255-273.