May 99 Columns
First, thanks to Todd Elvins for asking me to take over the VisFiles column. He set a high standard to live up to. I'll start my VisFiles columns by expressing a vision of the future of visualization and discussing the difficult problems that have to be solved to reach that future. This is inspired by Jim Blinn's list of 10 unsolved computer graphics problems - my favorite moment at SIGGRAPH 98. Your comments and thoughts on this column are welcome, as are your suggestions for future columns. (Editor's note: you can review Jim Blinn's SIGGRAPH 98 comments in Computer Graphics 33(1) February 1999, pps. 43-47.)
Change = Fun
Computer science and technology are constantly changing, which is what makes them so much fun. The two most important areas of change are:
- The interface between computers and people - that's where visualization comes in
- The interface between people and other people created by computer networking.
These changes are leading to the following vision:
A virtual world shared by everyone, integrated naturally with the physical world, where all electronic information is accessible, and accessed by people directly via their eyes, gestures, ears and voices.
I believe that this vision will become reality by the time today's students retire. What are the visualization problems that must be solved to reach this vision?
The first two problems involve the visual quality of visualizations.
- Make the spatial and temporal resolution of visual displays indistinguishable from physical reality. Display and geometry resolution, and response times to user interaction, must all be brought to human perceptual limits. This is primarily a hardware problem that will be solved if Moore's law holds true for 20 or 30 more years. High performance graphics systems already employ parallel algorithms, which will continue to be important for solving this problem.
- Integrate virtual reality with physical reality. This means eliminating the need for special helmets, glasses, gloves and wands, and embedding displays as part of the physical environment. This can be accomplished partly by artificial vision systems that sense the positions of users' heads, eyes, hands and bodies, as well as by electronics that will be part of most peoples' clothes and jewelry. It will also require novel display technologies. Most current displays are two-dimensional projections (as are images on human retinas), but high quality virtual reality may require true three-dimensional displays in order to accommodate multiple viewers and viewers' quick head and eye movements. It is currently hard to imagine how virtual reality can create a convincing illusion of reality, but technology has a habit of doing the impossible.
The second pair of problems involve the integration of visualization with other computer technologies.
- Integrate visualization with networking, voice, artificial vision, computation and data storage. Visualization must be integrated with networking so remote users can share virtual spaces, integrated with speech recognition so the system can understand statements like "no, the thing next to it," integrated with sonification so sound and images can be synchronized and complementary, integrated with artificial vision to implement less obtrusive user interaction techniques and to reconstruct realistic images of remote collaborators integrated with virtual information displays (i.e., lifelike avatars), and integrated with computation and data storage to create better user interfaces to these resources. And of course, developers of the shared virtual world will need visualizations of its design and implementation to help them with debugging and tuning.
- Optimize physical resources used to perform visual interactions. Even though it will eventually be easy to render at the limits of human perception, users will always need to visually interact with data sets whose sizes challenge the capacity of their equipment. Furthermore, Moore's law will not reduce speed-of-light network latencies. Thus algorithm efficiency and resource optimization will always be important. Solutions will include: 1) parallel algorithms for common operations, 2) strategies for moving and replicating data in the memory hierarchy (i.e., cache, RAM, disk) based on visual access patterns and 3) strategies for moving and replicating data on the network based on access patterns for visualizations and interactions shared by multiple users.
The third pair of problems involve the information depicted in visualizations.
- Find effective ways to visualize numerical information of high dimension. Visualization has been successful because so much computer data is produced that describe the four-dimensional space-time world that our eyes and brains evolved to see. Examples include weather, chemical and aerodynamic simulations. However, it is a lot harder to design effective visualizations for numerical data that doesn't fit four-dimensional space-time. The shared virtual world will include lots of it, including statistics and financial data. There are a number of existing techniques, such as parallel coordinates, interactive 3D hyperslices, interactive projections to 3D, overloading multiple data dimension onto display dimensions (a.k.a. "worlds within worlds") and arrays of scatter diagrams for various pairs of data dimensions. Each technique reveals some information but no technique reveals all information. Hence system level solutions that allow users to interactively select and combine different techniques and allow users to integrate data analysis calculations with visualization techniques are important.
- Find effective ways to visualize non-numerical information. Common non-numerical information includes large networks, like the Web and the phone system, and statements in natural languages, in programming languages, and recently in the genetic sequence language. All displays involve numbers (i.e., 2D or 3D pixel coordinates, color values and times) and exist in space and time. Hence non-numerical information must be transformed into numbers and embedded in four-dimensional space-time for display. There are generally an extremely large number of ways of choosing numbers and embeddings for non-numerical information (e.g., there are many layouts for a graph, and many choices of subgraphs for graphs too large to be depicted in one image) and these choices must be determined by algorithms specific to the information being visualized and the goals of the visualization. Furthermore, visualizations may be animated and interactive (e.g., visually interacting with the execution of a program) where basic visualization and interaction techniques may be combined in limitless ways. The problem of visualizing non-numerical information really divides into a large number of problems, specific to different types of information and user goals. The closest thing to a general solution is a system that provides users with a variety of basic visualization techniques and allows them to interactively combine and interact with those techniques.
The fourth pair of problems involve user interactions with visualizations.
- Find effective visual idioms for direct manipulation user interactions with visualizations. Direct manipulation will be essential for users immersed in the shared virtual world. Keyboards and mice will be largely replaced by voice recognition and gestures. User interface design will evolve to algorithms for parsing natural human gestures in the way that speech recognition tries to parse natural human speech. Gesture and speech parsing will be integrated, for example, to recognize a curve drawn by a finger in space as the user says "a function like this." There will be enormous complexity in the way that users manipulate visualization with their hands, moving data from one place to another, bending and smoothing curves and surfaces, rearranging network topologies and so on. It is worth keeping in mind that user interfaces that seem natural from the programmer's point of view feel unnatural to users, and those that feel natural to users seem peculiar and full of special cases to programmers.
- Find effective visual idioms for collaborative interactions among multiple users. There is a long list of issues here: how to share control of visualization and computations, how to share part of your environment and keep part private, how to share different parts with different people simultaneously, how to manage sharing among large groups, how to form ad hoc subgroups, how to find people and communicate with them wherever they are and how to adapt to whatever environment they have where they are: at work, at home, driving or riding in a car, on an airplane, walking or in a restaurant (non-VR seating will be available). As one specific example, imagine a teleconference where participants interact with lifelike images of each other: how can the system allow participants to make eye contact?
Space Science and Engineering Center
1225 W. Dayton Street
Madison, WI 53706
The copyright of articles and images printed remains with the author unless otherwise indicated.
The final pair of problems involve abstractions of the visualization process.
- Define effective abstractions for the visualization and user interaction process. Examples include the relational and field data models for information being visualized, the data flow model of the visualization process, mathematical models of human visual perception, mathematical models of interaction, models of users and their tasks and more general models of computing and distributed computing. Some of these models incorporate object-oriented programming ideas. Building a system around an abstract model gives it generality, but it also leads to inefficiency and difficult user interfaces. Building a system in response to immediate needs and without an abstraction makes it difficult to adapt to new uses. Effective abstractions have generality where it is needed but also make limiting assumptions that permit efficient and usable implementations. In addition to its practical consequences, this is the foundations problem for visualization.
- Present abstractions to users in ways that reconcile expressiveness and ease-of-use. Some users want to visualize simple information with minimal effort, while others want to customize and experiment with the entire visualization and interaction process in order to understand complex information. Advanced users want clear presentations of complex choices, whereas basic users want the system to supply reasonable defaults so they can ignore those choices. Even advanced users want to focus on certain choices without needing to learn about all possible choices. Systems succeed or fail largely on how well they solve this problem.
Plea for Uniform Network Environment
Finally, building the shared virtual world will be difficult enough without doing it on top of the current "Tower of Babel" of incompatible systems on the Internet. We need a uniform network computing environment and Java is currently the only technically and politically credible candidate. Sun has done a wonderful service by creating Java and deserves to profit. However, Sun needs to share control of the standard and Microsoft needs to lose its fear of platform independence, open standards and open source. We as users and developers have the power to demand an open and uniform network computing environment.
Bill Hibbard's research interests are interaction techniques, data models and distributed architectures for numerical visualization. He leads the SSEC Visualization Project and is primary author of the Vis5D and VisAD systems. He has degrees in mathematics and computer science from the University of Wisconsin - Madison.