Authors:

In the eighties when supercomputers mass-produced data, the need arose for effective tools that aid human cognition for data exploration, hypothesis building, and reasoning. In 1987, the U.S. National Science Foundation Report “Visualization in Scientific Computing” [6] was published, stating the new challenges and proposing large-scale funding for scientific visualization. The field of visualization quickly started to evolve. The goal was (and still is) to generate images that show what is inside the data. Early attempts tried to establish a mapping between the data and optical properties. For example, volume visualization voxels were assigned color and transparency, and the simulation of light transport generates images following photorealistic principles. Visualization also focused on realistic light transport, as photorealism was an unquestioned paradigm of computer graphics.

Later, when visualization was already a well-established field, people were questioning the sense of the simulation of light transport for the purpose of visualization. Photorealism in many cases prohibited the effective depiction of features of interest. Since then, non-photorealistic rendering (NPR) models were adopted and used in visualization. NPR techniques are commonly inspired by artistic styles and techniques that do not focus on a realistic depiction of scenes and objects. They go beyond photorealism and express features that cannot be shown using physically correct light transport.

Scientific visualization, however, does not have the same degree of freedom in the depiction as visual art does. Visualization is bound to the depiction of the underlying data and cannot make use of all types of NPR methods. Scientific visualization needs the artistic freedom to depict features in an expressive way, while simultaneously providing insight into the underlying data. These requirements are met by a discipline that is much older than visualization - scientific illustration. Traditional illustration developed a toolbox with rendering techniques that depict knowledge in an effective way. Scientific illustrators carefully measure objects to correctly convey information about, and relations between features of interest. Abstraction is the key that enables expressive rendering techniques that go beyond reality, and are of immense value for the correct interpretation of the phenomena and the according data. Information visualization never had the option to use photorealistic techniques due to the abstract nature of the data, and therefore explored techniques like focus+context earlier. Flow- and especially volume visualization, only recently started to explore the potential of illustration techniques. Illustration is a novel approach for the presentation of knowledge in scientific visualization. Subsequently, a vivid subfield (i.e., illustrative visualization) evolved, and algorithms were developed that automate the process of generating imagery-using techniques from traditional illustration. At the beginning, low-level abstractions, i.e., “how” to render features of interest, were explored. More recently, illustrative visualization deals with high-level abstractions or the question “what” to render.

All techniques commonly utilize well-known methods that were carefully developed by traditional illustrators, and have shown to be very effective in communication. Traditional illustration techniques are a valuable source for new visualization methods. Illustrative visualization is the implementation and automation of elaborate drawing skills that were developed to improve the depiction of information, and is therefore not a new technology.

Both illustration and visualization aim to amplify cognition using visual representations of underlying data. Therefore, the term illustrative visualization is a tautology. We believe that the current field of illustrative visualization will have a considerable impact on how visualization is done in the future. In this article, we first briefly review what has happened so far in illustrative visualization, and later give a taste of what is to come.

What has happened so far – Some Examples

Visual abstraction, one of the key components of illustrative visualization, follows the abstraction techniques of traditional illustration. We see similar abstraction concepts in earlier visualization approaches. Before illustrative visualization was recognized as a new direction in visualization research, focus+context techniques were used for visual abstractions. In fact, the visual abstraction techniques of traditional illustrations have used the focus+context concept. For example, the term uneven distribution of visual resources [9] can mean in illustration sketchy contour drawing for contextual information combined with a detailed watercolor style for the object in focus.

Focus+context techniques have been extensively researched in information visualization. Here, the levels of transparency, saturation, sharpness, or dedicated screen-space are typical examples of visual resources distribution. In semantic depth of field [14], the sharpness of 2D/3D objects has been used for guiding the observer’s focus to the most relevant features, whereas context objects are blurry (Figure 1 left). Another technique utilizing the focus+context concept is the magic lens metaphor, which is somewhat related to close-up views from illustrations. Magic lens have been applied to a broad set of different data, such as map exploration [12] (Figure 1 right), volume data exploration [26], or abstract graph interaction [4]. The resource distribution can be, for example, controlled by relevant information, e.g., a degree-of-interest (DOI) function [10].

A controlling mechanism in the spirit of the focus+context concept is one key component of illustrative visualization. The second component is the set of visual abstraction techniques originating from illustrations. The illustrator’s toolbox contains drawing styles like pencil, brush, or watercolor styles. In illustrative visualization, we refer to algorithms that are concerned with visual styles as low-level visual abstractions.

Line drawings are one of the most effective and difficult-to-master visual abstractions used in traditional illustration. In computer graphics, several techniques have been developed to represent mesh or volume data with feature lines to mimic hand drawn lines. These techniques are often based on local data properties such as first and second order derivatives. Typical representatives are ridge-and-valley lines [13], or view-dependent lines such as contours [18, 19], suggestive contours [7] (Figure 2), or apparent ridges [11]. In addition to line drawing, handcrafted shading techniques such as stippling [17], hatching, or toon shading [8,16] have been simulated with computerized techniques. Nowadays, computer-generated stylized depictions provide good results that are similar to but still distinguishable from handcrafted illustrations [18].

Low-level visual abstractions, such as those mentioned above, have been the primary focus of the NPR research. In addition to these low-level techniques, the illustrator works with expressive techniques that change the layout or deform features to increase the communicative intent of the illustration. Expressive techniques such as cutaways, breakaways, close-ups, or exploded views relate to the focus+context concept. We refer to illustration-inspired focus+context techniques as high-level visual abstraction techniques. They map knowledge about the features to specific depictions, such that features are emphasized or suppressed according to their importance and the intent of the illustration.

High-level visual abstractions have been outside the main NPR research direction. These techniques usually require inherent relevance information (e.g., importance or DOI) to be defined on the data. They have been the main scope of illustrative visualization research as a continuation of the focus+context techniques. An important function has been, for example, the steering mechanism for defining view-dependent interactive cutaways [24] (Figure 3 left). Other more explicit ways of focus definition have been used for interactive cutaways [3,15], close-ups [1,23], exploded views [2] (Figure 3 right), or peel-aways [5] for volume data. Here the focus is defined as a geometric region in the data (e.g., sphere or cube) or by a segmentation mask. These techniques can be seen as interactive focus+context methods, which have their static predecessor in traditional illustration. In literature these “what” to render approaches are also referred as to smart visibility techniques [25].

A Taste of What is to Come

Today we are at a point where many illustration techniques are available for visualization, and the question arises how to effectively use them. The goal of illustrative visualization is the development of software that enables domain experts and scientists to make illustrations of their work. In the future, experts should be able to illustrate their thinking and reasoning, and should be able to effectively communicate their derived knowledge. Systems that allow experts to render high quality illustrations without specific illustration skills can substantially support the reasoning process and enable more effective communication.

A system that allows easily derived high quality illustrations is not even close to becoming available and it will take more research to realize it. The future of illustrative visualization research goes in two directions. On one hand, the integration of illustration techniques into the reasoning process of scientists has to be researched. On the other hand, the seamless integration of illustrative visualization methods into the workflow of traditional illustrators should be dealt with. In the following we give a preview on both future research directions.

The Illustrative Visualization Paradigm Shift

So far, illustrative visualization is mainly seen as a tool for the effective communication of knowledge. On the top of Figure 4, we show today’s place of illustrative visualization in the science pipeline of knowledge gain and knowledge communication. The domain experts acquire data and explore it, using visualization methods of their choice, or in some cases, simply the visualization methods that are easily available. After examination of a subset of data, the experts formulate a hypothesis and do additional analysis on it. Again, visualization methods are used. If the hypothesis cannot be rejected, a model of the underlying phenomenon is formed. It is again used for further analysis and for the presentation of gained knowledge. For presentation purposes, the visualization is made as effective as possible to communicate the knowledge.

Presently, illustrative visualization primarily aims to mimic or enhance a wide variety of traditional illustration techniques. The used techniques have one aspect in common: the knowledge of meaningful entities is used to derive visual abstractions. Once a feature of an object is known it can be abstracted. Traditional illustration utilizes visual abstractions for the communication of knowledge. Features are identified and assigned a degree of interest for the communicative intent. For example, specific regions of a volume or flow data, that are known to be irrelevant for the phenomenon in question, can be drawn in a sketchy way to provide context. In this example abstraction is used to derive semantics like “irrelevant data” and “sketchy drawing.” The derivation of semantics is inherent to the knowledge-gain process, and can be used in visualization software to create visual abstractions. An example is annotation and labeling. Labels are an abstract visual representation of the derived semantics that are able to directly relate features of interest and their meaning.

Along these lines, it is natural that illustrative visualization was mainly used in the last step of the knowledge gain and communication pipeline, where knowledge about the data is given and can be used to derive visual abstractions. The motivation of illustrative visualization research is commonly the communication of knowledge to a nonexpert, such as for patient communication. The communication among experts was far less an issue for illustrative visualization. However, expert knowledge can be introduced earlier into the knowledge gain pipeline. Many aspects of the underlying phenomena are often already known before measurements are taken. The experts often do have knowledge that can explicitly be used to aid the exploration and analysis process. Furthermore, machine-learning algorithms can be used to imitate the knowledge-gain process and can automatically derive hypothesis and models. These intermediate representations of knowledge can be used in visualization software to derive visual abstractions. As shown on the bottom of Figure 4, the experts will explore their data using automatically derived, as well as explicitly given, knowledge. The evaluation of their hypothesis and the derived models are automatically illustrated using knowledge-assisted visualization methods. The use of illustrative visualization methods will enhance their reasoning process and also illustrate the reasoning process itself. Experts can use the illustrations for more effective communication between them. We believe that the introduction of illustrative visualization to early stages of the knowledge gain pipeline is a paradigm shift and has the potential to make expert reasoning more effective and more comprehensible. The derived semantics can be stored as metadata that is used by illustrators for the presentation of knowledge to a broader audience.

Won’t Somebody Think of the Illustrators?

The profession of an illustrator has undergone drastic changes since the beginning of the information age. Currently, 3D modeling and rendering software is increasingly replacing pen and paper. Non-photorealistic rendering techniques have allowed illustrators to smoothly handle this transition: traditional rendering can be simulated while still taking advantage of the benefits of 3D software. However, one important aspect is missing from the workflow: data. Typically, illustrators will rely on external sources as well as their pre-existing domain knowledge to build a mental model of the subject to be depicted. However, visualization aims to provide exactly this integration – the creation of expressive images based on measured or simulated data in an interactive environment. So why is it that illustrators don't take advantage of the many visualization tools that have evolved from the past 20+ years of research? One problem is that visualization software is frequently designed specifically for domain experts, such as research scientists or engineers, and utmost care is taken to accurately present the original data. An illustration, on the other hand, deliberately simplifies or distorts the data according to its communicative intent. This difference in focus often leaves too little flexibility for the illustrator. Moreover, user interfaces employ the language of domain experts while illustrators are more used to terms and concepts used in graphical arts.

One direction in illustrative visualization research is to provide illustrators with sufficient artistic freedom while still enabling them to naturally interact with data from various sources. A single software package that integrates state-of-the art rendering and modeling with the visualization pipeline is still not available. However, is this just an intermediate goal? Assuming that we can solve the vast amount of technical and research challenges involved in illustrative visualization, will that not just make the profession of an illustrator redundant? If domain experts can create illustrations themselves, what is the task of an illustrator?

We do not believe that solving the open challenges in illustrative visualization will eliminate illustrators. Instead, we envision a shift in focus (which is already in progress) to one of the most important aspects of illustration: visual communication. While it is plausible that we can mimic the abstraction techniques used by illustrators to a sufficient degree, the choice and composition of these techniques is difficult to automate. Essentially, this task boils down to the simulation of human cognitive processes – something that is notoriously hard. Domain experts may be able to create illustrations to communicate aspects of their analysis to other people, which share similar mental models, i.e., other domain experts. Illustrators, on the other hand, have the expertise to grasp the essentials of a complex finding and transform them into a visualization aimed at a specific target group, which might have a very different background. Even if all semantic properties of the data could be derived automatically, many aspects of this transformation heavily rely on the expert knowledge of the illustrator. Thus, illustrators are domain experts themselves, skilled in the art and science of visual communication.

This realization makes the illustrative visualization pipeline shown in Figure 4 directly applicable to the illustrator. In this case, the input data includes the findings of the domain scientist, in the form of tags, labels, or even an initial visualization, for example. In our vision, one key component of future research in illustrative visualization will be the effective representation of this information. Clearly, a system used by experts of different fields (such as scientists and illustrators) has to provide customized interfaces in the domain language of each of these fields. Initial steps in this direction have been made. For instance, Svakhine and Ebert [22] employ a multi-layered user interface, which ranges from novice to expert controls as well as motifs for different visualization goals (see Figure 5). Rautek et al. [20,21] present a system based on fuzzy-logic, which allows users to formulate rules for data and illustration semantics (see Figure 6). A further direction may be capturing data about the exploration and analysis process itself, which offers the potential of acquiring more information in a less intrusive way. For instance, hidden knowledge (implicit concepts which are assumed by the respective domain expert, but rarely stated explicitly) may be obtained in this way.

Conclusion

Illustrative visualization has evolved in the past years as a vivid subfield in visualization. So far, it is a tool to effectively communicate known aspects of data and therefore only used in the presentation step of the scientific knowledge gain and communication pipeline. Visualization will undergo a paradigm shift, where illustrative visualization will be integrated into all stages of the science pipeline by introducing knowledge assisted methods. Domain experts will do research using illustrative visualization systems. The system stores metadata-like derived semantics and explicitly given expert knowledge. This metadata enables scientific illustrators to make effective illustrations for a broader audience.

To make this paradigm shift happen, two major research directions need to be followed. Firstly, the seamless integration of visualization software into the workflow of illustrators, and secondly the introduction of illustrative visualization methods at early stages of the knowledge gain pipeline.

In the end, we envision integrated illustrative visualization, which offers customized interfaces familiar to the domain experts and the scientific illustrators. Such a concept may serve to enhance communication between different disciplines and also provide a basis for extracting patterns and thus automating complex processes.

References

[1] S. Bruckner and M. E. Gröller. VolumeShop: An interactive system for direct volume illustration. In Proc. of IEEE Visualization '05, pp. 671–678, 2005.

[2] S. Bruckner and M. E. Gröller : Exploded views for volume data. IEEE Trans. on Visualization and Computer Graphics 12(5):1077–1084, 2006.

[3] S. Bruckner et al.: Illustrative context-preserving exploration of volume data. IEEE Trans. on Visualization and Computer Graphics 12(6):1559–1569, 2006.

[4] S. Carpendale et al.: Distortion viewing techniques for 3-dimensional data. In Proc. of IEEE Symposium on Information Visualization '96, pp. 46–53, 1996.

[5] C. Correa et al.: Feature aligned volume manipulation for illustration and visualization. IEEE Trans. on Visualization and Computer Graphics 12(5):1069–1076, 2006.

[6] B.H. McCormick, et al.: Visualization in Scientific Computing, National Science Foundation, 1987.

[7] D. DeCarlo et al.: Suggestive contours for conveying shape. In Proceedings of ACM SIGGRAPH '03, pp. 848–855, 2003.

[8] A. Gooch et al.: A non-photorealistic lighting model for automatic technical illustration. In Proc. of ACM SIGGRAPH '98, pp. 447–452, 1998.

[9] H. Hauser: Scientific visualization: The visual extraction of knowledge from data. Springer, ch. Generalizing focus+context visualization, pp. 305–327, 2005.

[10] H. Hauser and M. Mlejnek: Interactive volume visualization of complex flow semantics. In Proc. of VMV’03, pp. 191–198, 2003.

[11] T. Judd et al.: Apparent ridges for line drawing. In Proceedings of ACM SIGGRAPH '07, pp. 19–26, 2007.

[12] T. Keahey and E. Robertson. Techniques for non-linear magnification transformations. In Proc. of IEEE Symposium on Information Visualization’96, pages 38–45, 1996.

[13] G. Kindlmann et al.: Curvature-based transfer functions for direct volume rendering:

Methods and applications. In Proc. of IEEE Visualization '03, pp. 513–520, 2003.

[14] R. Kosara et al.: Semantic depth of field. In Proc. of IEEE Symposium on Information Visualization '01, pp. 97–104, 2001.

[15] J. Krüger et al.: An interactive context preserving hotspot visualization technique. IEEE Trans. on Visualization and Computer Graphics 12(5):941–948, 2006.

[16] A. Lake et al.: Stylized rendering techniques for scalable real-time 3D animation. In Proc. Of Symposium on Non-Photorealistic Animation and Rendering '00, pp. 13–20, 2000.

[17] A. Lu et al.: Illustrative interactive stipple rendering. IEEE Trans. on Visualization and Computer Graphics, 9(2):127–138, 2003.

[18] R. Maciejewski et al.: Measuring Stipple Aesthetics in Hand-Drawn and Computer-Generated Images, IEEE Computer Graphics and Applications 28(2):62-74, 2008.

[19] P. Rheingans and D. Ebert: Volume illustration: Nonphotorealistic rendering of volume models. IEEE Trans. on Visualization and Computer Graphics, 7(3):253–264, 2001.

[20] P. Rautek, et al.: Semantic Layers for Illustrative Volume Rendering, IEEE Transactions on Visualization and Computer Graphics, 13(6):1336-1343, 2007.

[21] P. Rautek, et al.: Interaction-Dependent Semantics for Illustrative Volume, to appear in Computer Graphics Forum, 2008.

[22] N. Svakhine, et al.: Illustration Motifs for Effective Medical Volume Illustration, IEEE Computer Graphics and Applications ,vol. 25, no. 3, pp. 31-39, May/June, 2005.

[23] T. Taerum et al.: Realtime super resolution contextual close-up of clinical volumetric data. In Proc. of EuroVis ’06, pp. 347–354, 2006.

[24] I. Viola et al.: Importance-driven feature enhancement in volume visualization. IEEE Trans. on Visualization and Computer Graphics 11(4):408–418, 2005.

[25] I. Viola and M. E. Gröller: Smart visibility in visualization. In Proc. of Computational Aesthetics in Graphics, Visualization and Imaging, pp. 209–216, 2005.

[26] L. Wang et al.: The magic volume lens: An interactive focus+context technique for volume rendering. In Proc. of IEEE Visualization ’05, pp. 47–54, 2005.