Illustrative Visualization – New Technology or Useless Tautology?
The computer graphics group at TU Vienna has created some of most beautiful and effective illustrative visualizations. In this article, they share with us their unique perspective on illustrative visualization. — Kwan-Liu Ma
Authors:Peter Rautek, Stefan Bruckner, and Eduard Gröller
Technische Universität Wien
University of Bergen
In the eighties when supercomputers mass-produced data, the need arose for effective tools that aid human cognition for data exploration, hypothesis building, and reasoning. In 1987, the U.S. National Science Foundation Report “Visualization in Scientific Computing”  was published, stating the new challenges and proposing large-scale funding for scientific visualization. The field of visualization quickly started to evolve. The goal was (and still is) to generate images that show what is inside the data. Early attempts tried to establish a mapping between the data and optical properties. For example, volume visualization voxels were assigned color and transparency, and the simulation of light transport generates images following photorealistic principles. Visualization also focused on realistic light transport, as photorealism was an unquestioned paradigm of computer graphics.
Later, when visualization was already a well-established field, people were questioning the sense of the simulation of light transport for the purpose of visualization. Photorealism in many cases prohibited the effective depiction of features of interest. Since then, non-photorealistic rendering (NPR) models were adopted and used in visualization. NPR techniques are commonly inspired by artistic styles and techniques that do not focus on a realistic depiction of scenes and objects. They go beyond photorealism and express features that cannot be shown using physically correct light transport.
Scientific visualization, however, does not have the same degree of freedom in the depiction as visual art does. Visualization is bound to the depiction of the underlying data and cannot make use of all types of NPR methods. Scientific visualization needs the artistic freedom to depict features in an expressive way, while simultaneously providing insight into the underlying data. These requirements are met by a discipline that is much older than visualization - scientific illustration. Traditional illustration developed a toolbox with rendering techniques that depict knowledge in an effective way. Scientific illustrators carefully measure objects to correctly convey information about, and relations between features of interest. Abstraction is the key that enables expressive rendering techniques that go beyond reality, and are of immense value for the correct interpretation of the phenomena and the according data. Information visualization never had the option to use photorealistic techniques due to the abstract nature of the data, and therefore explored techniques like focus+context earlier. Flow- and especially volume visualization, only recently started to explore the potential of illustration techniques. Illustration is a novel approach for the presentation of knowledge in scientific visualization. Subsequently, a vivid subfield (i.e., illustrative visualization) evolved, and algorithms were developed that automate the process of generating imagery-using techniques from traditional illustration. At the beginning, low-level abstractions, i.e., “how” to render features of interest, were explored. More recently, illustrative visualization deals with high-level abstractions or the question “what” to render.
All techniques commonly utilize well-known methods that were carefully developed by traditional illustrators, and have shown to be very effective in communication. Traditional illustration techniques are a valuable source for new visualization methods. Illustrative visualization is the implementation and automation of elaborate drawing skills that were developed to improve the depiction of information, and is therefore not a new technology.
Both illustration and
visualization aim to amplify cognition using visual representations of
underlying data. Therefore, the term illustrative visualization is a
tautology. We believe that the current field of illustrative visualization will
have a considerable impact on how visualization is done in the future. In this
article, we first briefly review what has
happened so far in illustrative visualization, and later give a taste of what is to come.
What has happened
so far – Some
Visual abstraction, one of the key components of illustrative visualization, follows the abstraction techniques of traditional illustration. We see similar abstraction concepts in earlier visualization approaches. Before illustrative visualization was recognized as a new direction in visualization research, focus+context techniques were used for visual abstractions. In fact, the visual abstraction techniques of traditional illustrations have used the focus+context concept. For example, the term uneven distribution of visual resources  can mean in illustration sketchy contour drawing for contextual information combined with a detailed watercolor style for the object in focus.
Focus+context techniques have been extensively researched in information visualization. Here, the levels of transparency, saturation, sharpness, or dedicated screen-space are typical examples of visual resources distribution. In semantic depth of field , the sharpness of 2D/3D objects has been used for guiding the observer’s focus to the most relevant features, whereas context objects are blurry (Figure 1 left). Another technique utilizing the focus+context concept is the magic lens metaphor, which is somewhat related to close-up views from illustrations. Magic lens have been applied to a broad set of different data, such as map exploration  (Figure 1 right), volume data exploration , or abstract graph interaction . The resource distribution can be, for example, controlled by relevant information, e.g., a degree-of-interest (DOI) function .
A controlling mechanism in the spirit of the focus+context concept is one key component of illustrative visualization. The second component is the set of visual abstraction techniques originating from illustrations. The illustrator’s toolbox contains drawing styles like pencil, brush, or watercolor styles. In illustrative visualization, we refer to algorithms that are concerned with visual styles as low-level visual abstractions.
Line drawings are one of the most effective and difficult-to-master visual abstractions used in traditional illustration. In computer graphics, several techniques have been developed to represent mesh or volume data with feature lines to mimic hand drawn lines. These techniques are often based on local data properties such as first and second order derivatives. Typical representatives are ridge-and-valley lines , or view-dependent lines such as contours [18, 19], suggestive contours  (Figure 2), or apparent ridges . In addition to line drawing, handcrafted shading techniques such as stippling , hatching, or toon shading [8,16] have been simulated with computerized techniques. Nowadays, computer-generated stylized depictions provide good results that are similar to but still distinguishable from handcrafted illustrations .
Low-level visual abstractions, such as those mentioned above, have been the primary focus of the NPR research. In addition to these low-level techniques, the illustrator works with expressive techniques that change the layout or deform features to increase the communicative intent of the illustration. Expressive techniques such as cutaways, breakaways, close-ups, or exploded views relate to the focus+context concept. We refer to illustration-inspired focus+context techniques as high-level visual abstraction techniques. They map knowledge about the features to specific depictions, such that features are emphasized or suppressed according to their importance and the intent of the illustration.
High-level visual abstractions have been outside the
main NPR research direction. These techniques usually require inherent
relevance information (e.g., importance or DOI) to be defined on the data. They have been the main scope of
illustrative visualization research as a continuation of the focus+context
techniques. An important function
has been, for example, the steering mechanism for defining view-dependent
interactive cutaways  (Figure 3 left). Other more explicit ways of focus
definition have been used for interactive cutaways [3,15], close-ups [1,23],
exploded views  (Figure 3 right), or peel-aways  for volume data. Here
the focus is defined as a geometric region in the data (e.g., sphere or cube)
or by a segmentation mask. These techniques can be seen as interactive focus+context methods, which
have their static predecessor in traditional illustration. In literature these “what” to
render approaches are also referred as to smart visibility techniques .
A Taste of What is
Today we are at a point where many illustration techniques are available for visualization, and the question arises how to effectively use them. The goal of illustrative visualization is the development of software that enables domain experts and scientists to make illustrations of their work. In the future, experts should be able to illustrate their thinking and reasoning, and should be able to effectively communicate their derived knowledge. Systems that allow experts to render high quality illustrations without specific illustration skills can substantially support the reasoning process and enable more effective communication.
A system that allows easily
derived high quality illustrations is not
even close to becoming available and it will take more
research to realize it. The future of illustrative visualization research goes
in two directions. On one hand, the integration of illustration techniques into the
reasoning process of scientists has to be researched. On the other hand, the
seamless integration of illustrative visualization methods into the workflow of
traditional illustrators should be dealt with. In the following we give a
preview on both future research directions.
Visualization Paradigm Shift
So far, illustrative visualization is mainly seen as a tool for the effective communication of
knowledge. On the top of Figure 4, we show today’s place of illustrative
visualization in the science pipeline of knowledge gain and knowledge
communication. The domain experts acquire data and explore it, using
visualization methods of their choice, or in some cases, simply the visualization methods
that are easily available. After examination of a subset of data, the experts formulate a hypothesis
and do additional analysis on it. Again, visualization methods are used. If the
hypothesis cannot be rejected, a
model of the underlying phenomenon is formed. It is again used for further
analysis and for the presentation of gained
knowledge. For presentation purposes, the visualization is made as effective as possible to communicate the
Presently, illustrative visualization primarily aims to mimic or enhance a wide variety of traditional
illustration techniques. The used techniques have one aspect in common: the
knowledge of meaningful
entities is used to derive visual abstractions. Once a feature of an object is
known it can be abstracted. Traditional illustration utilizes visual
abstractions for the communication of knowledge. Features are identified and
assigned a degree of interest for the communicative intent. For example,
specific regions of a volume or flow data, that are known to be irrelevant for
the phenomenon in question, can be drawn in a sketchy way to provide context.
In this example abstraction is used to derive semantics like “irrelevant data”
and “sketchy drawing.” The derivation of semantics is inherent to the knowledge-gain process, and can be used in visualization software to create visual abstractions. An
example is annotation and labeling. Labels are an abstract visual
representation of the derived semantics that are able to directly relate features
of interest and their meaning.
Along these lines, it is natural that illustrative visualization was mainly used in the last step of the knowledge gain and communication pipeline, where knowledge about the data is given and can be used to derive visual abstractions. The motivation of illustrative visualization research is commonly the communication of knowledge to a nonexpert, such as for patient communication. The communication among experts was far less an issue for illustrative visualization. However, expert knowledge can be introduced earlier into the knowledge gain pipeline. Many aspects of the underlying phenomena are often already known before measurements are taken. The experts often do have knowledge that can explicitly be used to aid the exploration and analysis process. Furthermore, machine-learning algorithms can be used to imitate the knowledge-gain process and can automatically derive hypothesis and models. These intermediate representations of knowledge can be used in visualization software to derive visual abstractions. As shown on the bottom of Figure 4, the experts will explore their data using automatically derived, as well as explicitly given, knowledge. The evaluation of their hypothesis and the derived models are automatically illustrated using knowledge-assisted visualization methods. The use of illustrative visualization methods will enhance their reasoning process and also illustrate the reasoning process itself. Experts can use the illustrations for more effective communication between them. We believe that the introduction of illustrative visualization to early stages of the knowledge gain pipeline is a paradigm shift and has the potential to make expert reasoning more effective and more comprehensible. The derived semantics can be stored as metadata that is used by illustrators for the presentation of knowledge to a broader audience.
Won’t Somebody Think of the Illustrators?
The profession of an
illustrator has undergone drastic changes since the beginning of the
information age. Currently,
3D modeling and rendering software is increasingly replacing pen and paper.
Non-photorealistic rendering techniques have allowed illustrators to smoothly
handle this transition: traditional rendering can be simulated while still
taking advantage of the benefits of 3D software. However, one important aspect
is missing from the workflow: data. Typically, illustrators will rely on
external sources as well as their pre-existing domain knowledge to build a
mental model of the subject to be depicted. However, visualization aims to
provide exactly this integration – the creation of expressive images
based on measured or simulated data in an interactive environment. So why is it
that illustrators don't take advantage of the many visualization tools that
have evolved from the past 20+ years of research? One problem is that visualization software is frequently designed specifically for domain
experts, such as research scientists or
engineers, and utmost care is taken to
accurately present the original data. An illustration, on the other hand,
deliberately simplifies or distorts the data according to its communicative
intent. This difference in focus often leaves too little flexibility for the
illustrator. Moreover, user interfaces employ the language of domain experts
while illustrators are more used to terms and concepts used in graphical arts.
One direction in illustrative visualization research is to provide illustrators with sufficient artistic freedom while still enabling them to naturally interact with data from various sources. A single software package that integrates state-of-the art rendering and modeling with the visualization pipeline is still not available. However, is this just an intermediate goal? Assuming that we can solve the vast amount of technical and research challenges involved in illustrative visualization, will that not just make the profession of an illustrator redundant? If domain experts can create illustrations themselves, what is the task of an illustrator?
We do not believe that
solving the open challenges in illustrative visualization will eliminate
illustrators. Instead, we envision a shift in focus (which is already in
progress) to one of the most important aspects of illustration: visual
communication. While it is plausible that we can mimic the abstraction
techniques used by illustrators to a sufficient degree, the choice and
composition of these techniques is difficult to automate. Essentially, this
task boils down to the simulation of human cognitive processes –
something that is notoriously hard. Domain experts may be able to create
illustrations to communicate aspects of their analysis to other people, which share similar mental
models, i.e., other domain experts. Illustrators, on the other hand, have the
expertise to grasp the essentials of a complex finding and transform them into
a visualization aimed at a specific target group, which might have a very
different background. Even if all semantic properties of the data could be
derived automatically, many aspects of this transformation heavily rely on the
expert knowledge of the illustrator. Thus, illustrators are domain experts
themselves, skilled in the art and science of visual communication.
This realization makes the illustrative visualization pipeline shown in Figure 4 directly applicable to the illustrator. In this case, the input data includes the findings of the domain scientist, in the form of tags, labels, or even an initial visualization, for example. In our vision, one key component of future research in illustrative visualization will be the effective representation of this information. Clearly, a system used by experts of different fields (such as scientists and illustrators) has to provide customized interfaces in the domain language of each of these fields. Initial steps in this direction have been made. For instance, Svakhine and Ebert  employ a multi-layered user interface, which ranges from novice to expert controls as well as motifs for different visualization goals (see Figure 5). Rautek et al. [20,21] present a system based on fuzzy-logic, which allows users to formulate rules for data and illustration semantics (see Figure 6). A further direction may be capturing data about the exploration and analysis process itself, which offers the potential of acquiring more information in a less intrusive way. For instance, hidden knowledge (implicit concepts which are assumed by the respective domain expert, but rarely stated explicitly) may be obtained in this way.
Illustrative visualization has evolved in the past years as a vivid subfield in visualization. So far, it is a tool to effectively communicate known aspects of data and therefore only used in the presentation step of the scientific knowledge gain and communication pipeline. Visualization will undergo a paradigm shift, where illustrative visualization will be integrated into all stages of the science pipeline by introducing knowledge assisted methods. Domain experts will do research using illustrative visualization systems. The system stores metadata-like derived semantics and explicitly given expert knowledge. This metadata enables scientific illustrators to make effective illustrations for a broader audience.
To make this paradigm shift happen, two major research directions need to be followed. Firstly, the seamless integration of visualization software into the workflow of illustrators, and secondly the introduction of illustrative visualization methods at early stages of the knowledge gain pipeline.
In the end, we envision
integrated illustrative visualization,
which offers customized interfaces familiar to the domain experts
and the scientific illustrators. Such a concept may serve to enhance communication between different
disciplines and also provide a basis for extracting patterns and thus
automating complex processes.
 S. Bruckner and M. E. Gröller. VolumeShop: An interactive system for direct volume illustration. In Proc. of IEEE Visualization '05, pp. 671–678, 2005.
 S. Bruckner and M. E. Gröller : Exploded views for volume data. IEEE Trans. on Visualization and Computer Graphics 12(5):1077–1084, 2006.
 S. Bruckner et al.: Illustrative context-preserving exploration of volume data. IEEE Trans. on Visualization and Computer Graphics 12(6):1559–1569, 2006.
 S. Carpendale et al.: Distortion viewing techniques for 3-dimensional data. In Proc. of IEEE Symposium on Information Visualization '96, pp. 46–53, 1996.
 C. Correa et al.: Feature aligned volume manipulation for illustration and visualization. IEEE Trans. on Visualization and Computer Graphics 12(5):1069–1076, 2006.
 B.H. McCormick, et al.: Visualization in Scientific Computing, National Science Foundation, 1987.
 D. DeCarlo et al.: Suggestive contours for conveying shape. In Proceedings of ACM SIGGRAPH '03, pp. 848–855, 2003.
 A. Gooch et al.: A non-photorealistic lighting model for automatic technical illustration. In Proc. of ACM SIGGRAPH '98, pp. 447–452, 1998.
 H. Hauser: Scientific visualization: The visual extraction of knowledge from data. Springer, ch. Generalizing focus+context visualization, pp. 305–327, 2005.
 H. Hauser and M. Mlejnek: Interactive volume visualization of complex flow semantics. In Proc. of VMV’03, pp. 191–198, 2003.
 T. Judd et al.: Apparent ridges for line drawing. In Proceedings of ACM SIGGRAPH '07, pp. 19–26, 2007.
 T. Keahey and E. Robertson. Techniques for non-linear magnification transformations. In Proc. of IEEE Symposium on Information Visualization’96, pages 38–45, 1996.
 G. Kindlmann et al.: Curvature-based transfer functions for direct volume rendering:
Methods and applications. In Proc. of IEEE Visualization '03, pp. 513–520, 2003.
 R. Kosara et al.: Semantic depth of field. In Proc. of IEEE Symposium on Information Visualization '01, pp. 97–104, 2001.
 J. Krüger et al.: An interactive context preserving hotspot visualization technique. IEEE Trans. on Visualization and Computer Graphics 12(5):941–948, 2006.
 A. Lake et al.: Stylized rendering techniques for scalable real-time 3D animation. In Proc. Of Symposium on Non-Photorealistic Animation and Rendering '00, pp. 13–20, 2000.
 A. Lu et al.: Illustrative interactive stipple rendering. IEEE Trans. on Visualization and Computer Graphics, 9(2):127–138, 2003.
 R. Maciejewski et al.: Measuring Stipple Aesthetics in Hand-Drawn and Computer-Generated Images, IEEE Computer Graphics and Applications 28(2):62-74, 2008.
 P. Rheingans and D. Ebert: Volume illustration: Nonphotorealistic rendering of volume models. IEEE Trans. on Visualization and Computer Graphics, 7(3):253–264, 2001.
 P. Rautek, et al.: Semantic Layers for Illustrative Volume Rendering, IEEE Transactions on Visualization and Computer Graphics, 13(6):1336-1343, 2007.
 P. Rautek, et al.: Interaction-Dependent Semantics for Illustrative Volume, to appear in Computer Graphics Forum, 2008.
 N. Svakhine, et al.: Illustration Motifs for Effective Medical Volume Illustration, IEEE Computer Graphics and Applications ,vol. 25, no. 3, pp. 31-39, May/June, 2005.
 T. Taerum et al.: Realtime super resolution contextual close-up of clinical volumetric data. In Proc. of EuroVis ’06, pp. 347–354, 2006.
 I. Viola et al.: Importance-driven feature enhancement in volume visualization. IEEE Trans. on Visualization and Computer Graphics 11(4):408–418, 2005.
 I. Viola and M. E. Gröller: Smart visibility in visualization. In Proc. of Computational Aesthetics in Graphics, Visualization and Imaging, pp. 209–216, 2005.
 L. Wang et al.: The magic volume lens: An interactive focus+context technique for volume rendering. In Proc. of IEEE Visualization ’05, pp. 47–54, 2005.