DETAILS


COLUMNS


CONTRIBUTIONS

a a a

Non-Photorealistic Rendering

Vol.32 No.1 February 1999
ACM SIGGRAPH

Putting the Artist in the Loop



Joshua Seims
Industrial Light & Magic
“I don’t care so much whether my color is exactly the same, as long as it looks beautiful on my canvas, as beautiful as it does in nature.”

— Van Gogh [7]

Until the advent of impressionism, a painting’s canvas was supposed to be a featureless window into another world, a world that appears just like nature. The art salons of the time ridiculed impressionism, both because they deemed impressionism’s themes (people in parks, a pair of shoes in the corner) trivial, and because the painting style did not reflect reality. This new style differed from photorealistic painting in that the canvas was no longer invisible. Instead, the texture of the layers of paint on the canvas was an inherent part of the art. Artists did not faithfully replicate the colors of nature, they exaggerated them (sometimes, as in Fauvism, to an extreme degree).

Artist Needs Control

Today, one of the most interesting challenges in computer graphics is to create animations in such painterly styles. Yet, with the exception of the amazing “Paint World” sequence in What Dreams May Come, painterly rendering techniques have not been used in production environments. By a production environment, I mean places like special effects and game companies where the standards for quality are below that of fine art, yet higher than home videos. I believe a big reason for this discrepancy is the techniques we have for automatically producing painterly animations do not give the artist fine enough control. In my experience with non-photorealistic rendering, I am often frustrated by my inability to stop, reach into and tweak an automatic process. To make painterly rendering techniques more useful for production-quality work, we need to develop algorithms and interfaces that get the artist in the loop.

One reason the perfect painterly rendering algorithm remains elusive is that, unlike with photorealism, there is no single correct solution, no “rendering equation” to solve. Sometimes we have examples of paintings in the styles we want to imitate, but no one knows what those styles should look like when animated. A painterly style involves tension between the 3D world being represented and the 2D canvas plane. It is unclear how this tension should work during animation. However, even though we cannot describe the “right” way to make painterly animations, our tools should allow us to explore the full range of painterly animation styles.

Analyzing the Artistic Elements

We can get a sense of this range by analyzing the artistic elements that go into a great painting. Consider, for example, an analysis of Monet’s Four Trees by Ilan Salzberg at Columbia:

Figure 1
Figure 1: Monet’s Four Trees.

Monet’s Four Trees

Four Trees asks the viewer to make a cognitive bridge between the representation of colors and the representation of an object. The major elements in the Four Trees… create a grid pattern and do not serve to move the eye around the painting… The brushstrokes in the sky appear angled to direct toward the splash of orange between the third and fourth poplar. The strokes in the bush curve downwards into the horizontal bar. The line of the horizontal bar draws the eye along its length with a base of diagonal brushstrokes, but Monet overlaid flowing semicircular and spiraling strokes which serve to break up the eye’s movement … Color heightens the piece’s unity. The predominant colors of purple and yellow are opposite on the color wheel giving them a complimentary relationship. Monet uses that relationship to create different approaches to the idea of warmth. He succeeds in creating both a fiery vibrancy and a burning-ember warmth within a harmonic palette. [5]

Monet’s painting is much more than a set of brush strokes randomly jittered over a reference image or placed on the surface of objects. Brushstrokes do not simply follow image gradients — they direct the eye. Artists choose to paint with quick angry jabs or slow languorous strokes based on the mood they wish to convey as well as the reality they are representing. The palette choice creates an overall sensation of warmth or coldness, and the use of a palette can create a theme with repetition, and tension with contrast. Furthermore, the painted image is a distillation of the character of reality — artists add small details for decoration, and omit others for clarity. They paint a higher level of detail in important regions of the image, such as faces and specular highlights. Often they draw lines around objects to bring them out even though those lines may not appear in reality.

When these artistic elements work, the style of a painting can invoke in us feelings of beauty, action, anger, depression, excitement and joy. A big question facing painterly rendering research is how can the computer assist in putting these elements together and how do we keep them consistent during animation.

Joshua Seims worked on computer-generated watercolor and non-photorealistic performance-driven facial animation during his graduate studies at the University of Washington. He is now a Production Software Engineer at Industrial Light & Magic.

Joshua Seims
Industrial Light & Magic
P.O. Box 2459
San Rafael, CA 94912

Tel: +1-415/448-3283
Fax: +1-415/448-2233


The copyright of articles and images printed remains with the author unless otherwise indicated.

Current Tools Fall Short

Our current tools do not address this question very well. Traditional paint programs operate at too low a level of interaction. The artist must draw each frame from scratch and the computer does little more than store the colors painted. On the other hand, the painterly rendering techniques described in recent research operate at too high a level of interaction. Model-based techniques [4] take, as input, simple geometry along with a camera move. Image-based techniques [3, 2] simply require a reference video stream. Additionally, the user can set parameters affecting the brush stroke appearance. However, the user does not control the artistic decisions of how to use the strokes and palette to create an artistic composition.

Researchers in pen-and-ink illustration [6] and watercolor [1] at the University of Washington demonstrate types of artistic input that can create high-quality single frames. In the pen-and-ink system, the user hand-draws the set of strokes that are placed in the image, and uses combing, filling and interpolating tools to create a vector field that orients the strokes. Creating a vector field is a large part of the work, but allows fine tuning the image and leads to a much more appealing result. A user of the automatic watercolor program can also direct the process by segmenting the reference image into a series of mattes and choosing palettes for each matte. However, neither system takes animation into account. They require a burdensome amount of user input for each frame, and no effort is made to control frame-to-frame coherency.

However, similar types of user input could be useful for painterly animation, especially with interfaces designed for animation. In traditional animation, we have many techniques for specifying motion control information over a series of frames. Some of these techniques, such as keyframing, constraints and physically-based modeling, have useful parallels in painterly animation. One could use keyframing to interpolate vector fields and brush stroke parameters. Constraints might express artistic notions of stroke direction (eyes should follow this path, strokes should align to the object). And physically-based modeling of the paint medium could animate secondary motions such as flowing and smearing.

Other types of user input have no direct analogy to the photorealistic animation world. How does the user specify the desired detail level, either in screen space or object space? Is there a semi-automatic way the artist can control the recoloration of a reference image under an exaggerated palette? What do we do with the underlying paper texture as the camera moves around the scene? How should the user express the style of brush strokes to use in different regions, and how do we match corresponding strokes across frames? Furthermore, we would need a way to express how brush strokes change as the camera and objects move. Do the strokes stick to the object? Melt as if gas? Flow as if liquid? Break apart when zoomed in and coalesce when zoomed out? These are all difficult questions and require research in both algorithms and interfaces. Since many of these questions do not entail a single “right” answer, the artist’s input should be involved.

By involving the artists so closely with the creation of each frame, even the ideal system may require significant work from a team of professional animators, match movers, rotoscopers and painters to produce high-quality animations. The challenge is to find a level of interaction that minimizes the tedium of drawing and tracking each individual brush stroke, and yet allows the artist full control over the compositional elements that give a work of art its quality.

References

  1. Curtis, Cassidy J., Sean E. Anderson, Joshua E. Seims, Kurt W. Fleischer, David H. Salesin. “Computer-Generated Watercolor,” SIGGRAPH 97 Conference Proceedings, August 1997, pp. 401-406.
  2. Hertzmann, Aaron. “Painterly Rendering with Curved Brush Strokes of Multiple Sizes,” SIGGRAPH 98 Conference Proceedings, July 1998, pp. 453-460.
  3. Litwinowicz, Peter. “Processing Images and Video for an Impressionist Effect,” SIGGRAPH 97 Conference Proceedings, August 1997, pp. 407-414.
  4. Meier, Barbara J. “Painterly Rendering for Animation,” SIGGRAPH 96 Conference Proceedings, August 1996, pp. 477-484.
  5. The Claude Monet home page, http://www.columbia.edu/~jns16/monet_html/monet.html
  6. Salisbury, Michael P., Michael T. Wong, John F. Hughes, David H. Salesin. “Orientable Textures for Image-Based Pen-and-Ink Illustration,” SIGGRAPH 97 Conference Proceedings, August 1997, pp. 401-406.
  7. Wheldon, Keith. Van Gogh, Gallery Books, New York, 1989.