Interaction in 3D Graphics
Vol.32 No.4 November 1998
Sketching in 3D
Based mainly on a series of studies the author conducted at the University of Toronto, this article reviews the usability of various six degrees of freedom (6 DOF) input devices for 3D user interfaces. The following issues are covered in the article: the multiple aspects of input device usability (performance measures), mouse based 6 DOF interaction, mouse modifications for 3D interfaces, free-moving isotonic 6 DOF devices, desktop isometric and elastic 6 DOF devices, armature-based 6 DOF devices, position vs. rate control and the form factors of 6 DOF control handle. These issues are treated at an introductory and practical level, with pointers to more technical and theoretical references.
As three-dimensional (3D) graphics moves to the core of many mainstream computer systems and applications, the search for usable input devices for 3D object manipulation becomes both an academic inquiry and a practical concern. In the case of the 2D graphical user interface (GUI), the computer mouse established itself very early and quickly replaced the light pen as the de facto standard input device (see  for a review of mouse history). In the case of 3D interfaces, however, there is still not an obvious winner suitable for all applications. Primarily based on the author’s own research, this article offers a few perspectives on the usability of various input devices for 3D interface. This article does not intend to present a comprehensive literature review or a series of experimental studies in a methodical manner. Rather, it intends to be introductory and practical. Interested readers are encouraged to examine more technical details in the papers referenced.
In order to be able to manipulate 3D objects, one generally needs at least six degrees of freedom (6 DOF), three for X, Y and Z translation and three for 3D rotation. The difficulty in establishing a standard 6 DOF device is twofold. First, there are engineering challenges in terms of sensor technologies, manufacturing cost and designer’s creativity. It is highly likely that the most elegant 6 DOF device has still not been designed. Second, and perhaps more importantly, even if we could easily make any device we like to, there is only a very limited knowledge about what properties a good 6 DOF controller should have. Given the long history of human factors study on input control devices, dating back to World War II , “One would expect the relationship of the hand to the controlled element, being at the one time both an input and output, to be a fruitful area for research,” but the reality is that little is well understood . Burrows pointed out that the reluctance to conduct research in this area is understandable in view of the immensity of the possible interactions among the many dimensions of control feel.
This is not to say that there isn’t any intellectual guidance to 6 DOF input device design. Motivated by the manual control problems in vehicles, air crafts and other dynamic complex machines, the topic of “manual control and tracking” has been extensively studied (see  for a summary). However, system dynamics resulted from mass, spring, viscosity, transmission delay, etc. in these systems soon dominated the area. The study of input control device properties (e.g.  ) quickly gave way to mathematical control theory modeling of man-machine systems. The more general body of knowledge on human motor control and learning (see  for example), while offering many insights, rarely provides direct design guidelines. One recent review of the scattered literature related to input device design is provided in .
Performance Measures of 6 DOF Input Devices
There are many choices in designing or selecting a 6 DOF input device. The choice on every design dimension may have implications on users’ performance. Aside from application specific requirements, there are at least six aspects to the usability of a 6 DOF input device, namely:
Figure 1: A mouse with a roller for 3D input .
Figure 2: A mouse with two additional degrees of freedom .
Figure 3: A sample of free moving 6 DOF devices — “flying mice.” (a) The “Bat,” designed by C. Ware ; (b) The Cricket™, manufactured by Digital Image Design Inc., New York, NY, USA; (c) The MITS Glove, designed by the author, consists of a Bird™ tracker and a clutch.
Figure 4: Trial completion time in a 6 DOF docking task. The free moving device was faster, particularly in early stage .
Figure 5: Coordination as measured by inefficiency in free moving position control vs. in elastic rate control. Significantly longer movement was “wasted” with the free moving device. See  for details.
Figure 6: The Fingerball. This device uses the same sensor as in the glove in Figure 3c. However, the different form factor made it possible to manipulate it with fingers, in addition to wrist and arm, leading to better performance  See page 102 for color image.
Figure 7: Performance difference between the Fingerball and the Glove in a 6 DOF docking task .
Figure 8: A sample of desktop 6 DOF input devices. A sample of input devices for 6 DOF manipulation: (a) The Spaceball™ is an isometric device manufactured by Spaceball Technologies Inc., Boston, MA, U.S.A. (b) The SpaceMaster™ is an elastic device with a small range of movement (5 mm in translation and 15° in rotation), manufactured by BASYS GmbH, Erlangen, Germany. (c) The Space Mouse™ is an elastic device with slight movement (5 mm in translation and 4° in rotation). It was initially designed by DLR, the German aerospace research establishment, manufactured by Space Control Company, Malching, Germany and marketed by Logitech, Fremont, CA, U.S.A. br>
Figure 9: Two input device design dimensions: transfer function vs. controller resistance.
Figure 10: Isotonic rate control and isometric position control tend to produce poor performance [24, 21].
Figure 11: Coordination as quantified by movement inefficiency. The free moving position control device “wasted” more movement. Adapted from Figure 10 in .
Figure 12: A prototype of a 6 DOF Elastic General-purpose Grip (EGG) [21, 20].
Figure 13: Time performance difference between an elastic and an isometric rate controller in a 6 DOF docking task [21, 20].
Figure 14: Tracking error with an elastic vs. isometric 6 DOF controller task [21, 24].
Figure 15: A single arm multi-DOF armature input device .
The first four aspects are common to all input devices and their meanings are obvious. The fifth aspect, coordination, is unique to multiple degrees of freedom input control. There are many ways of measuring the degree of coordination. One effective way of quantifying it is based on the ratio between the length of actual trajectory and that of the most efficient trajectory in the coordination spaces, including translation space, rotation space and the 2D space between translation and rotation . By such a measure, in order to produce the most coordinated path, one has to simultaneously move all degrees of freedom involved at the same pace towards their respective goal states.
The sixth aspect of input device usability is the ease of device acquisition. This is often an overlooked aspect of input device usability. Although a mouse is less dexterous than a pen-like input device (a stylus), the fact that a mouse can be more easily acquired is one important reason that made it the dominant 2 DOF input device. Many factors, such as the distance to the computer keyboard home row (ASDFGHJKL keys), contribute to the ease of device acquisition. One of them is the device location persistence when released. With a mouse or a trackball, when released by the hand (in order to type something, for example) it stays in position. This is not true with a stylus.
With these measures in mind, the remainder of the paper examines a few common classes of 6 DOF input devices.
Mouse Based 6 DOF Input
The simplest implementation of 6 DOF manipulation can be six graphical sliders on the computer screen. Each can be dragged with a standard 2 DOF input device, such as a mouse. There are two fundamental usability problems to such a 6 DOF interaction technique. First, people cannot mentally decompose orientation into separate rotational axes . Second, since one has to time-multiplex between the separate degrees of freedom, it is not possible to form a coordinated movement in the 6 DOF space.
Researchers soon moved away from the six slider implementation to more complex mapping techniques. One such technique is enclosing the manipulated 3D object with a virtual sphere . Some of these techniques in fact have been widely used in 3D graphics software such as VRML browsers and CAD packages. A recent study  found the mouse mapping techniques still inferior to integrated 6 DOF devices.
Mice Modified for 3D Operation
Efforts have been made to add more physical degrees of freedom to the mouse for 3D interfaces. One example is the roller mouse that had an additional degree of freedom by means of a roller (, see Figure 1). A user could rotate the roller to move a 3D cursor in the depth dimension (translation only). Another example is the rockin’ mouse with two additional degrees of freedom by means of rocking motions on a tablet  (see Figure 2).
Since dedicated physical degrees of freedom are provided for the third dimension, these modified mice should outperform a conventional mouse that operates by means of simple mode switching. (One common mouse switching technique, for example, is that pressing and holding a mouse button down switches vertical mouse cursor motion to motion in the depth dimension). It was shown that in a 3D positioning task, the rockin’ mouse was 30 percent faster in comparison to a standard mouse tablet . On the other hand, since the depth dimension is operated by a behavior and muscle groups different from those of the x-y mouse motion, it can be difficult to produce simultaneous, coordinated motion with either the roller mouse or the rockin’ mouse.
Given the success of the mouse in 2D interfaces, it is natural to attempt to make a flying mouse — a mouse that can be moved and rotated in the air for 3D object manipulation. Indeed many such devices have been made. Theoretically, a mouse is a free-moving, i.e. isotonic device. When using such a device, the displacement of the device is typically mapped to a cursor displacement. This type of mapping (transfer function) is also called position control. Figure 3 shows a few examples of 6 DOF isotonic position control devices. Most of these devices are instrumented with a magnetic tracker for 6 DOF sensing.
The advantages of these “flying mice” devices are:
However, there are many disadvantages to this class of devices:
Form Factor Matters
Of course, there are many variations within each class of devices. One such variation is the form factor of the control handle (shape and size). As shown in Figure 3, there has been a variety of shape and size used in constructing free moving 6 DOF input devices. Some of these devices, such as the glove implementation (Figure 3c) require manipulation with wrist and arms, but exclude the fingers. Figure 6 shows an alternative design, based on the very same sensor with a different shape and size that do allow the participation of the fingers that have higher dexterity. It was shown that significant performance advantage could be gained with such a design . See figure 7.
The alternative to free moving input devices, are devices that are mounted on a stationary surface. We may call these desktop devices. Figure 8 shows some examples of 6 DOF “joystick” devices that fall into this class. Common to such devices is a self-centering mechanism. They are either isometric devices that do not move by a significantly perceptible magnitude or elastic devices that are spring-loaded. When tension is released from the handle, the handle returns to a null position. Typically these devices work in rate control mode, i.e. the input variable, either force or displacement, is mapped onto the velocity of the cursor. In other words, the cursor position is the integration of input variable over time. Hence the term first order control. In contrast, in the case of the free moving (isotonic) devices, input variable (device displacement) is scaled to position (location and orientation) itself. Hence the term zero-order control.
Compatibility Between Resistance and Transfer Function
With the proper clutching mechanism, it is conceivable to implement an isometric device in position control mode or an isotonic device in rate control mode (see Figure 9) Interestingly, these two combinations tend to produce poor user performance (see Figure 10). The reason is quite simple: the self-centering mechanism in an isometric device facilitates the “start, speed-up, maintain speed, slow-down and stop” cycle in rate control. The later half of the cycle is somewhat automatic with the self-centering mechanism in isometric devices. With a free moving device, one has to deliberately return to the null position.
The Pros and Cons of Desktop 6 DOF Devices
When used in rate control, an isometric device offers the following advantages:
On the other hand, isometric rate control devices may have the following disadvantages:
An elastic device moves proportionally to the force applied to it and yet maintains the self-centering mechanism needed for rate control. Figure 12 shows a prototype of a 6 DOF elastic rate control input device, dubbed EGG (elastic general-purpose grip). A similar suspension structure had been used for different purposes . Since a user can feel both the pressure and the displacement with an elastic device, a stronger feedback is provided (see  Chapter 3 for detailed neuro-physiological analysis). However, the use of spring loading is not without tradeoff. The more displacement for the same force is provided, the stronger the kinesthetic feedback is provided, but the weaker the self-centering mechanism is for rate control compatibility (See  Chapter 3 for this two-factor analysis of elastic devices).
When optimized, the appropriate amount of elasticity does improve a user’s performance. This is particularly true at the early learning stages (see Figures 13 & 14).
Another class of multi DOF input devices are mechanical armatures. Some of these are specialized and geometrically compatible with the graphical object being controlled so as to enable the posing of computer graphics characters more-or-less using puppeteering techniques. This is a technique that has been used by ILM, for example . Digital Image Design Incorporated’s “Monkey”  as seen in the cover of this special issue and the armatures from Puppet Works  are commercially available examples of this class of technology.
While strong in their intended, specialized purpose, their use is limited. They can do an excellent job for some types of character posing, but their applicability to interaction in general is limited, at least in this “puppet” type configuration.
One option that can bring armatures much closer to the generality seen with the other devices discussed occurs if they are configured as a single arm. This can be done with either the DID or Puppet Works devices, and is the standard configuration for other armatures, such as Immersion Corp.’s MicroScribe illustrated in Figure 15 .
In this configuration, the armature is actually a hybrid between a flying-mouse type of device and a desktop device. It is like a desktop device in that it is typically mounted on the desktop, and consumes a small footprint. On the other hand, it is like a flying mouse in that manipulating the end point of the arm in space provides the desired 6 DOF position data. Conceptually, these are near isotonic — with exceptional singularity positions — position control device like a flying mouse and thus share many of the pros and cons of isotonic position control discussed earlier. In addition, this approach has the following particular advantages:
On the other hand, they also have some particular drawbacks:
This class of devices can also be equipped with force feedback, as with SensAble Technology’s Phantom Device . While showing some promise and with novel user experiences, force feedback devices have yet to demonstrate their real benefit in the lab or in practice for mainstream applications. It is, of course, very interesting as a research area.
From this brief and incomplete guided tour, one can conclude that the complexity of 6 DOF input is far from being solved. None of the existing devices fulfills all aspects of usability requirement for 3D manipulation. A six degree of freedom device that suits all, or the majority of, 3D applications has yet to be developed. However, the research conducted so far has offered implications for potential improvement — with many insights into the characteristics, pros and cons of various designs. These insights should help the selection of various types of devices for different tasks. For example, when speed and short learning is a primary concern (such as in video games), free moving devices are most suitable. When fatigue, control trajectory quality and coordination are more important, such as in teleoperation, isometric or elastic rate control devices should be selected. This article has only covered a few aspects of 6 DOF input device design. Many other areas, such as the cooperation of two hands in 3D input  and those covered by other articles in this issue, are equally important. Further research, both developmental and experimental, is certainly needed.
I thank Paul Milgram and Bill Buxton for their longstanding mentorship. Some of the studies cited in this paper were conducted at the University of Toronto with funding support from ITRC, NSERC and DCIEM. Bill Buxton reviewed the article and co-authored the Multi-DOF Armatures section.
Shumin Zhai is a Research Staff Member at the IBM Almaden Research Center where he conducts research and innovative development in input devices and interaction techniques, theoretical modeling of human computer interaction (HCI), advanced graphical user interfaces and computer vision-based next generation multi-modal interaction techniques. He received his Ph.D. degree from the University of Toronto where he worked on 3D interfaces and six degrees of freedom input control.