Vol.32 No.1 February 1998
Solving the Problems that Count
The excitement of doing research in computer graphics has not diminished in its 35 years of life. It remains a quick paced, intellectually challenging field that necessarily bends some traditional scientific methodologies. I think, however, that a recalibration of how we do things is in order. I offer a personal view of the "hows and whys" of that recalibration.
What Do We Do Well?
I'm just old enough to have experienced the tail end of computer graphics in its infancy. Like most people learning about computer graphics in the ’70s and ‘80s, I was introduced to interactive computer graphics through Ivan Sutherland's still remarkable Sketchpad and through the later work from Xerox PARC.
In the early ‘70s, many graphics researchers became preoccupied with getting geometry to behave itself: Warnock and Watkins developed seminal visibility algorithms; computer aided geometric design (then as now sometimes called computational geometry in the United Kingdom) had come into its own through the much earlier work of people like Coons and Bézier. Burtnyk and Wein opened us to the possibilities of interactive, interpolation-based keyframe animation. The ‘70s also saw some remarkable results come out of the University of Utah: illumination models and rendering techniques by Warnock, Gouraud, Phong and Blinn, the depth-buffer algorithm, texture mapping and filtering by Catmull, the beginnings of specialized graphics hardware by Clarke (following Evans and Sutherland) and the application of signal theory to computer graphics through Crow.
For me, the passage of computer graphics from childhood to adolescence is marked by the moment of shared belief that realistic computer graphics was achievable. I believe that threshold was crossed in 1980. That year, in perhaps the most startling and influential computer graphics paper ever published, Whitted made believers of most of us. Whitted's ray traced scenes consisted of little more than a couple of spheres and a plane, but man did they look good! Fournier, Fussell and Carpenter were coincidentally responding to a need for richer models, giving us both the ability to create stochastic or procedural models of arbitrary complexity from a small amount of initial data and by pointing the way toward the modeling of natural phenomena. Williams and others at New York Institute of Technology, building on the work of the ‘70s, were pointing the way to higher quality graphics through multiresolution methods.1 Many researchers were focusing on new forms of curves and surfaces that provided greater flexibility in modeling. Greenberg was already demonstrating a long-term vision of multidisciplinary research in image synthesis, which continues to this day. These people, together with many others, changed the game: they showed that it was possible to depict visually convincing, dynamic synthetic environments.
More recently, in its extended adolescence, computer graphics has seen the increasing use of physical models and simulation in illumination and in computer animation. Perhaps as a reaction to this trend, a variety of non-physical approaches have emerged that emulate image generation techniques from other areas, such as typography or illustration. We are seeing the influx of researchers from other areas such as computer vision, image processing and the medical sciences, in addition to our more traditional converts from the physical sciences, mathematics and arts.
Despite our growth and achievements through the ‘80s and ‘90s, our approach to solving problems in computer graphics has not significantly changed: we encounter a phenomenon we find attractive, we dig into the literature about the phenomenon, often outside our area of expertise, and we cobble together a first-order model. This does not usually give an acceptable visual model, so we plug away at it, sometimes in a disciplined way and sometimes not, refining and extending it until a satisfactory realisation is created. Then we write it up, and try to get a SIGGRAPH (or other) paper out of it. If the result sticks, we add bits and pieces to it, filling our work out into an ad hoc research programme. Otherwise we go on to the next thing.
I'm being both simplistic and a bit cynical; after all, there are some completely original algorithms in graphics that stand as remarkably pure achievements. But most of those results come from an older time when graphics was a clear screen. Incrementality is also essential to taking an initial result to closure or to adding to an open-ended research avenue. There is room, of course, for both easy and hard results. I hope that we save most of our enthusiasm for the hard results that may not yet give the nicest images.
A related observation is that I think we place overly high emphasis on the quality of implementation and not enough on fundamental thinking. It's both easier and more fun to evaluate work that has a killer implementation. However, an overemphasis on the completeness of a specific result reinforces a tendency toward both incremental and applied research.
I have characterised the past 15 years of computer graphics as its adolescence. I believe we are in need of a recalibration that would achieve a better balance among basic and applied research. To answer the question that headed this section, what we do well is short-term, applied research. But along the way, we have also demonstrated a capability to do deeper things. Computer graphics has changed. In its infancy, it was a field with far more problems than researchers; in its adolescence, that balance has shifted slightly. We no longer need to be persuaded that we can do realistic image synthesis. In fact, often there are many disjoint paths to the same "reality." With each year, our field becomes more heavily populated with excellent researchers, and there is a little less room. But compared to other fields, there's still plenty of space for everyone and there exists a growing set of unsolved, open-ended problems. The problems we have left are bigger, not smaller, because most of the smaller ones are solved.
I believe the endeavour of most of us can be defined as an enquiry of creating convincing computational depictions of visual phenomena. While many separate preoccupations fall within this mission, I think we ultimately have a common destination of transforming computer graphics into the computational science of interactive visual depiction. I have no hesitation about using the word science, for I believe that in its adulthood, computer graphics will indeed be a science. That's not all it will be: naturally, it will have an associated technology, an engineering practice and an artistic culture, but in this essay I will focus on the recalibration we need to situate computer graphics squarely on a path to science.
Problems and Priorities
In no particular order, here are some issues that I think we should be thinking about as we look forward. Let me be the first to plead guilty for not thinking as much as I should about them.
Basic Research Should Come First!
You may have noticed that I wear both academic and industrial hats. One side effect of this is that it has become much clearer in my mind that universities should be preoccupied with looking at really hard, long-term problems. In a popular management book called Built to Last, Collins and Porras entitled one of their chapters, "Try a lot of stuff and keep what works."
Exactly! The computer graphics industry has no hope of being able to do this without universities leading the charge. The reality is that industry cannot afford to try out very many crazy or otherwise nonstandard ideas. It has to pick its spots carefully. Universities should be preoccupied with scientific exploration and discovery, not product development. The latter is better left to software technicians in industry. Unfortunately, we now have to contend with a variety of externally imposed policies, whether simply misguided or crassly opportunistic, that emphasise short-term returns and immediate industrial relevance. There will be increasing pressure for research labs to do more development. Computer graphics is hardly alone in feeling this pressure, but the pressure to deliver results is more deeply felt in our area due to its reputation as an applied science.
My experience (necessarily anecdotal) is that in the process of a longer-term involvement in a scientific agenda, it is inevitable that something will be suitable for technology transfer; but I contend that this something should be serendipitous. Industry should respect this in their dealings with universities, and rather than asking for specific deliverables, it should instead be nimbly looking out for serendipitous results. This is not to say that contractual applied research should be avoided; I am simply suggesting that we all clearly understand the university's true mission is that of discovery.
Our Basic Research Is Not Basic Enough
Collins and Porras suggest that corporations should have big, hairy audacious goals. How many of us doing basic research in computer graphics have such goals? It is always difficult to define basic research, but I think two requirements are scientific uncertainty, and the potential of fundamentally changing widespread perceptions that the audience for our work will have. Our desire (and sometimes requirement) for immediate recognition through the big show at SIGGRAPH can cloud our long-term vision. I understand that there are constraints to long-term thinking. In universities, we are driven by the short-term constraints of funding and the fact that we are student-driven; in industry, we are constrained by the short-term need to get visibility and to hand technology off to product development. But I feel that those of us who are able to should be responding by setting more ambitious, longer-term goals, and our research community should recognise those efforts.2
We Don't Read Widely Enough
There is more to the graphics literature than the SIGGRAPH proceedings. Our knowledge of papers in related journals such as CVGIP, TVCG, CAGD, CAD, etc. is relatively poor. Even TOG suffers from low readership. Fortunately some areas such as rendering have very successful specialised workshops that provide a focus for experts in the area. I would also argue that our lack of background in the arts, humanities and literature causes us to misperceive or disregard the connection between our creations and the creative processes of others. How can we even formulate longer-term goals that are relevant to the rest of humanity without a deeper understanding of the world around us?
We Don't Collaborate Widely Enough
Computer graphics is a broad but shallow field (and I don't mean this pejoratively). We have several overlapping ways of making progress: we can make completely original contributions; we can gain expertise through reading the literature outside our area; or we can do it by engaging relevant experts outside our area. The first occasionally happens. The second has been a common approach, but will get harder as the problems get more technically challenging, as we push closer to the unknown in that area, or if we challenge the assumptions of that field. That leaves us with the last alternative. Collaborating more with others outside our field can have two outcomes: the direct consequence is that we may get better focused results, and the indirect consequence is that we may enhance our scientific visibility outside our area.
We Don't Export Our Research Results Enough
Universities have been quite effective, whether directly through technology transfer, or through educating students well, at exporting research results into production hardware and software systems. We have done much less well at exporting our results to other research areas. Some of the earliest results within computer graphics had a profound impact on other areas. Two that come immediately to mind are the effect of Warnock's algorithm on the development of quadtrees in image processing, and Watkin's algorithm on the evolution of line-sweep and plane-sweep algorithms in computational geometry. Our more recent successes seem somewhat muted, though this may be more an issue of our results needing time to penetrate an area. To cite one example, Barsky's work on modeling components of the human eye may well have substantial long-term implications. Nevertheless, it seems to me that we could be excellent partners for any research project involving, for example, applied geometry (e.g., to biomedical research, geographic or geologic information systems, geometric/image database systems, Web applications and so on). Such endeavours may be a distraction for some or new profession for others, especially our graduates who may well find a slower job market one day. Either way our field can probably afford to do more of this.
We Lack A Methodology For Evaluating Success
How have we measured the success of an approach? We look, of course. But what is our standard of reference, how do we measure progress and how can someone else demonstrate that they have outperformed, vindicated or contradicted our work? Here are some suggestions for improvement.
We Have Inconsistent Measures Of Accomplishment
It is inevitable that because different areas within computer graphics have different missions, we would apply different standards of research accomplishment to work in those areas. However, we have to recognise that in some areas, control for physically based animation comes immediately to mind, it appears to be especially difficult to make large increments of progress. Either that area must adapt its critical barometer to this fact, or at a higher level we will have to do a better job of normalising the judgements of experts from that area.
We Make Too Much Of SIGGRAPH
This is a tired issue, so I won't belabour it. SIGGRAPH is always exciting and each year the programme committee tries very hard to be fair. While there is room for improvement in the selection process, the fact is that we try to shoe-horn too many good papers into too short slots with too few pages devoted to too few accepted papers. The venues are impossibly large, and the production values associated with a SIGGRAPH paper are often incommensurate with the importance of the contribution. Furthermore, the fact that our conference publications are terminating (unlike most other fields, which encourage fuller versions of papers going to journals) gives us an incomplete and sometimes inaccurate view of the contributions being made. The combination of a highly selective conference as the premiere forum together with its terminating nature also has implications on tenure cases for academics. Junior faculty should not attempt to build their academic careers based on anticipated success at SIGGRAPH. As it stands, computer graphics people on average publish less than their counterparts in other areas. We need other options.
We Need A Refereed Electronic Graphics Journal
Perhaps this could be a spin-off of TOG, or even of the SIGGRAPH quarterly; now that ACM has an on-line publications repository, the time is right to get serious about a refereed electronic journal that makes it easy to incorporate animations, high-quality images and demonstrations.
The Gap Between Rich And Poor Is Growing
Funding has always been difficult in university-based computer graphics. Many of us do not have the publication records of our competitors in other fields, and computer graphics still does not have the scientific credibility of other areas (perhaps this is deserved). I also have the feeling that we're not always very nice to each other in peer reviews. Graphics faculty are exceptionally busy, productive people -- sometimes in ways that don't appear on a CV -- and we don't have the time to form lobby groups as is done in other fields. In other fields, lobbying is often done by distinguished senior people who have put aside their research and have learned the politics of funding. In computer graphics, we have relatively fewer senior academics owing to the age of our area and to the effect of industry. Furthermore, our senior academics are among our most productive researchers, and we need them to spend more time in their graphics labs than in Washington (or Ottawa, Paris, Bonn, London or Tokyo). So we have a Catch-22 situation. Our inability to play the politics of funding will result in continuing underfunding for many academic labs. Those who are already ahead on the funding curve will be able to leverage their successes, principally in attracting graduate students. I see a trend toward concentration of computer graphics talent. While anyone can start a little lab, populating it with, and retaining, excellent people won't be easy. I don't have an answer for this one.
So, What Else Should We Work On?
After the extended rant above, let me focus on some issues and challenges. First, let's dwell on pieces that that can be used to make up a whole. I cannot claim great originality in suggesting these topics.
We need to reintroduce the human back into the loop. Our models have too many parameters and are too hard to control. We need to understand how to amplify simple control variables into richer behaviour. This is especially true of the models and control systems we are building in computer animation and in rendering. Some recent efforts at interactive design and sketching are promising. We also need to develop, or to partner with people who can develop, a broader array of tactile input devices. In short, we should again become friends with our colleagues in human-computer interaction.
Virtual environments of the future need to be fully self-consistent: solid objects cannot interpenetrate (unless you want them to); lights have geometry; geometric objects have mass, inertia and volume; geometric calculations need to be robust and fast; reflectance models of most materials are not lambertian, camera motion should follow specific rules. In short, we need to revisit our first order solutions of problems in computer graphics to understand what fits together and what does not. This is far more than an engineering issue. Most of our solutions are not geared to interoperate and are not robust. I believe a great deal of basic research will come out of a disciplined reappraisal of our existing solutions.3
Our need for richer models, both geometrically and optically, will always outstrip the raw performance capabilities of our hardware. How can we apply cognitive or perceptual (and some would suggest artistic) schemata to present or compute only the minimum information needed?
The variety of modeling techniques is steadily growing. While the supporters of any one technique may think otherwise, modeling techniques will augment rather than supplant one another. This means that we will have potentially many representations of the same thing, with each representation itself potentially being multiscale. One obvious example here is that we may well have a representation of an object that is simultaneously 2D image based, 3D light-field based and also a multiscale subdivision surface with a complex, space varying BRDF (say). Intelligently choosing and managing these representations will be challenging, and it will make the problem of integration all the more difficult, as will the problem of bandwidth management.
Cognitive Modeling, Behaviours and AI
Whether due to increasing complexity or to the need for modularisation and autonomy, we will require higher-level languages that allow both the encapsulation of complex masses of data, and high-level reasoning. In computer animation and games, we need suitable abstractions for autonomous, active characters. In rendering systems, we need mechanisms that help us manage the complexity of specifying and keeping consistent the properties that we wish to attach to objects. In complex interactive systems, we need active agents that can teach, document and demonstrate tasks that normally would require expert users. All of these topics are pointing to the need to introduce cognitive and behavioural modeling languages to computer graphics.
Eugene Fiume is Head of Research and Usability at Alias|Wavefront. He is also a Professor in the Department of Computer Science at the University of Toronto. He sits on a variety of scientific, institutional and corporate boards. He received his B.Math degree from the University of Waterloo, his M.Sc. and Ph.D. degrees from the University of Toronto and did his postdoctoral studies at the University of Geneva.
The computer graphics industry is poised for another takeoff: this time to the world of commodity graphics. The potential for this industry as it mingles with other digital media and as it is continually reshaped by market forces is impossible to predict. As a field it appears that we are contributing to a science and technology that will allow us to routinely participate in convincingly realistic interactive visual environments. These environments of 25 years from now will inform and entertain us. One hopes it will free us of the stupefying monotony of today's forms of education and entertainment, but this may be the loftiest goal of them all. By contributing to computer graphics, we will not solve the deep, apparently unsolvable problems of the world, but we may one day help to enhance the quality of life for our future friends, and the quality of their dreams.
I hope that we will have the discipline to explore the intellectual and scientific breadth that computer graphics offers. I fear that instead we will be too pragmatic, and too interested in the destination rather than the ride. But I don’t want the small negatives to overwhelm the positives. When it’s all said and done, most of us are in the field now for the same reasons as when we set our first pixel: we get a kick out of being able to do new things every day, we love to make pictures and we get great stimulation from working with really smart people from many different areas.
I'm sure my colleagues would be horrified at the mere thought of implicating them in this paper. I will instead simply thank all my heroes in computer graphics, past and present.
The graphics literature.