Vol.33 No.4 November 1999

Real-time Interactive Storytelling

Scott S.Fisher
Telepresence Research Inc.
Glen Fraser

November 99 Columns
Images and Reversals Professional Chapters

Real-Time Interactive Graphics
Previous Real-Time Interactive Graphics Previous Real-Time Interactive Graphics

During a recent vacation in Vancouver, I was randomly selected off the street by a TV crew, and asked (presumably as a member of the general public) a series of questions related to computing for an upcoming show. After talking about the Internet, peripherals and so on, I was posed the final question - "what do you think computers will be able to do in five years?" I was somewhat surprised that the first thing that came into my head was neither real-time ray-tracing nor one billion polygons per second but rather, "I really hope that computers will become better tools for storytelling."

By profession, I am a software developer, and have worked on various creative and visual computing tools that can relate to storytelling - tools such as virtual reality, motion capture, virtual sets, real-time viewing and high-level animation. By hobby, I enjoy a number of creative pursuits. I dabble in computer animation and music. But my favorite and most satisfying creative pastime is writing - poems and stories. I find it a challenge to come up with good stories, and the process of writing brings me a lot of pleasure.

Iím no longer a big computer game fanatic, but in my teens I would hang around arcades, pouring my hard earned paper-route or busboy money into the games. I remember my excitement when Dragonís Lair first came out, and how awe-struck I was at its hand-drawn cel animation and new style of gameplay. Compared to most games (then as well as now), it greatly limits the playerís options for interaction - in fact, the player actually had no control over the path taken through the game - but these constraints allowed it to use a much more compelling "look" to tell its story. Fortunately for my wallet, I probably enjoyed watching others play more than I enjoyed playing myself. Many of us wanted to see the whole story, how it would "end up" - and if we werenít good enough (or rich enough) to save the princess with our own hands, we could at least choose to stand on our tip-toes and watch over the bigger guysí shoulders.

In the old days, I played text-only story games such as Enchanter or The Hitchhikerís Guide to the Galaxy on my parentsí home computer. These games could totally suspend my disbelief and immerse me in their worlds just as well as a good book, but they brought something new - they made me feel that I was in the story, in a way no book had ever done. With the first home computers I bought for myself (Atari ST and later an Amiga), I enjoyed a whole new world - graphically rich cinematic story games such as Defender of the Crown had me hooked. Later, as I succumbed and bought a home PC, I enjoyed games such as Wing Commander, various of the Kingís Quest adventures, and the last few games of the Ultima series. I sampled from other styles, such as puzzle, simulation or fighting games, but the ones I found most memorable and satisfying were games where a well-crafted story was central.

As this is a column about real-time interaction, Iíll get to the point. In our lives, many of us have had the occasion to listen to stories told by excellent, gifted storytellers. This is a special treat that everyone should experience as an adult. It can transport us back through time to an earlier era when a cultureís stories were passed on by oral tradition, or back to our childhood, when stories were a regular, often daily, part of our lives.

When an experienced storyteller (for example, a creative parent) is telling a child a story, it tends to be a real-time, interactive experience for the child. First, it can be considered real time, because the feedback and reactions of the child to the story are often integrated immediately back into the story. In a direct or indirect manner, the child becomes part of the narrative process - sometimes even a character in the narrative itself. If the child is looking bored, a twist will be added into the story. If their ears perk up at some minor descriptive detail, then that detail may be expanded and given a more integral role in the story. Second, the experience is truly interactive, because the child may ask questions for clarification ("why wasnít the princess allowed to go outside?") or for more detail ("what colour was the flower she found?"). Even when a storybook is being read to a child, although the flow and outcome of the story is predetermined, the reading tends to be an interactive process. The reader will answer questions the child asks, point out details in the drawings, make comparisons to the childís own life ("who do we know that lives on a farm?"). The point is that any required detail can be injected, thanks to the storytellerís own creativity. They can customize the story and make the experience a memorable and personal one for the child.

A rich imagination is the fertile soil from which stories can grow, and is a requirement for a good storyteller. Knowledge of the audience can be a huge benefit too - for example, knowing details about their lives, relationships and interests. To continue the plant metaphor, this insider knowledge can provide some of the "seeds" that will grow into stories. Observing and using feedback from the audience are critical skills for a great storyteller. A lot can be learned by paying attention to body language, posture, vocal responses and other subtle forms of feedback. This feedback is like water, sunshine and careful pruning - if properly applied once the story has started to grow, it can direct and control its evolution. Unfortunately for would-be computer-based storytellers, computers are currently not very good at imagining things, knowing their users or subtly observing! Letís take a closer look at these three weaknesses.

Imagination - Where Do the Stories Come From?

Since they lack the ability to be imaginative and create stories from nothing, computers must have human input to provide the elements for their stories. An author will make up the foundations for a story - settings, characters, goals, decisions, obstacles. Depending on the design of the software, they may devise multiple endings, or a variety of story branches you can follow (hmm, thereís that tree metaphor again). Although you may have a significantly different experience than someone else has using the same software, the key events are usually the same. This is because when you reach the predefined branching points, there are only a limited number of real options available. Usually winning the game means arriving via some path to one of a few winning leaves on the tree.

Usually an author decides what kind of story he wants to tell - a team may typically spend one or two years developing this into an interactive story game. Some of their time may be spent designing the game engine, if one is not already available for the project. The engine is the actual storyteller, in a sense. (Note: like human storyteller "engines," if theyíre good, they may end up being reused for different stories - although if you have a forgetful relative who always ends up telling the same story over and over, you may have trouble believing this!) It presents the story to the user by displaying the appropriate graphics and sound, and controls how the user can interact with the story. Although tuning the engine may require significant time, much of the time is spent creating the story and the elements (graphical and sound environments, characters, dialogue, interaction) necessary to tell it. The interaction itself may be defined by a series of event sequences, rules, probabilities or decision trees, which the designers must provide to the engine. It may even be defined by some form of artificial intelligence (AI) - this also requires a set of rules from the design team. In most games, the user is a primary character - usually the protagonist. As puzzles are solved and challenges overcome, more of the story is revealed. Because computers are so good at calculating and presenting numbers, some games even quantify this by providing a percentage of completion. To me this can break the suspension of disbelief - but then I suppose one can make the same calculation in a novel (letís see, page 287 of 522, thatís 55 percent complete!). Less like a book, though, in some cases you may successfully get to the end of the game while only exposing a small percentage of the overall work the design team invested into the story.

The computer doesnít know your likes and interests well enough to choose or create a story to appeal to you. Instead, you (as the consumer) must make a decision about the kind of story you want, by purchasing a particular piece of software to tell that story. This can be a hit-or-miss process, and parallels the process of buying a book to read, in that you tend to follow a similar process (favorite author, friendsí recommendations, reviewersí comments, description on the back, cover art, cost). You may choose Riven over Quake, or Kingís Quest over Ultima, depending on your interests. Each piece of software is relatively limited in the number and type of stories it can tell, because humans need to author all the "story fragments" and their associated imagery and richness. Like any creative process, this is time-consuming. One approach to break through this barrier is to link people together, such that real people can collaboratively develop original stories. This kind of networked play is still increasing in popularity, but it can be limited by the creativity of the participants, and the resulting worlds and stories tend to lack the integrity and cohesion that a focused creative effort can produce. It will be interesting to see if other approaches develop over the coming years, but itís very unlikely that computers will start fabricating custom stories for us from nothing any time soon - certainly not in the next five years.

Know Your Audience

Knowledge about the audience is sorely lacking in todayís computer-based storytelling software. Technically, there is no major problem. Itís more a matter of something lacking in the implementation - after all, computers are great at storing and processing information about things. However, after more than 20 years of computer-based stories, the best that some games can do is ask you to enter your name and choose what kind of gun you want to carry. This is one area where I expect (and hope) great progress to be made in the next five years. Some say that customization and personalization will be the next big thing in computers, and for a variety of applications (of which storytelling is but one), this should prove to be a major change - hopefully for the better. Computers that learn about you, how you work, the kinds of things you are good at or need help with, will be able to serve your work and play needs much better. With the broad application of such technology, there will likely be undesirable side effects (for example, loss-of-privacy issues or targeted advertising). But, for software that attempts to relate memorable and captivating stories that talk to you "where you are," this should be a major step forward. If computer storytellers arenít able to weave new stories for you personally, at least they should be able to take a story off the rack and tailor it for you to a custom fit, rather than one size fits all.

Feedback: Subtle and Not-So-Subtle

Currently, interactive computer stories tend to be directed by fairly heavy-handed approaches. We fly spaceships and move our avatars around on-screen using joysticks. We ask questions and type in responses using the keyboard, and we may use other specialized controllers like trackballs, steering wheels or pedals. A few applications may use voice feedback of some form, but certainly none can understand free-flowing natural languages yet. As for subtler cues - wide-open eyes, scared faces, laughing or smiling mouths - no commercial package that I know of uses these as feedback. The closest Iíve seen is that some games interpret an extended lack of input as boredom and subsequently try to tease or taunt the player back into the game. This is a small first step, but Iíd hope that in the next five years, we would start to deal with more subtle forms of feedback. Motion capture or optical tracking systems could potentially detect changes in posture, and other facial or body cues. Perhaps this might someday be called "emotion capture"? (Editorís note: See "Computer Vision for Computer Interaction" and the other focus articles in this issue.) In any case, itíll be a long time before the computer is able to register the boredom on your face during its rendition of Jack and the Beanstalk and know to evolve it into Java and the Bean Stock because it knows of your interest in the history of equities!

Are We Getting There?

There are major technological hurdles to clear, both hardware and software, before the kind of storytelling I describe becomes possible, but not all the hurdles are technological. There are major interaction and storytelling issues to resolve. Filmmakers have a century of history and experience in telling stories in that non-interactive, viewpoint-controlled medium. There are major challenges to overcome once the viewer becomes an active participant and/or camera operator. On the simplest level, they may not always look where you want them to for artistic or story-advancing reasons. Weíre getting better at figuring out ways of telling stories with a non-fixed viewpoint - the computer game and educational software industries are both making progress in these areas.

Live interactive storytelling in spoken form has been around for thousands of years, but having the computer do it is a new development. We still have lots to learn about being absentee storytellers or storytellers-once-removed now that a storytelling engine is responsible for relating the story. Although game designers have learned many tricks to help make games appear more complex and flexible, computer storytelling still essentially follows in the choose-your-own-adventure style first popularized by books of that name, in that all possible outcomes must be specified by the author. Iíd love to see storytelling become more real time as it is when one makes up a story for a child. The author may have had an ending in mind before beginning to recount the story, but the storyteller is flexible enough to change it dynamically as the story unfolds and the childís responses give him new ideas. One of my favorite visions of a futuristic storytelling device of this kind is the "Young Ladyís Illustrated Primer," described in Neal Stephensonís novel The Diamond Age. (An interesting and relevant note: even that technological marvel of a book - the Primer - required significant real-time human input on the storytelling end!) Working towards such a device that can entertain, teach and enrich peopleís lives is, to me, a very noble pursuit.

At the most recent SIGGRAPH, I was fortunate to see a demonstration of Segaís new game Shenmue, which is visually beautiful and appears to take an innovative approach to interactive storytelling. Rather than sticking to one paradigm, it seems to pick-and-choose from a variety of interactive styles, as appropriate for different parts of the story and the game. Other upcoming games, such as Ultima 9: Ascension from Origin Systems, should allow the player remarkable freedom to explore, yet also manage to tell an extremely compelling tale. When itís released next year, Halo from Bungie Software may indeed set a new standard for visual and physical realism in storytelling engines. If the story and the multi-participant interaction are as well-crafted as the demonstration they were showing this summer, it will be a great achievement.

Some say that storytelling has two aspects - the art (the creation of a story) and the craft (the working out of the details of how to tell it). Unless some revolution occurs that allows computers to become intelligent and creative, I expect that humans will continue to be sole masters of the art. However, I hope and expect that computer storytellers will become significantly more proficient in their craft, by understanding their audience and by incorporating more subtle feedback and refined forms of interaction. Computers should not supplant the other forms of storytelling we know and enjoy today, but should allow more people to have greater freedom to express their creativity through stories.

Scott S.Fisher is a Media Artist, Producer and Director whose work concentrates primarily on immersive environments and technologies of presence. Currently, he is President of Telepresence Media, a company focusing on the art and design of virtual environment and remote presence experiences. He is also on the Faculty of Environmental Information at Keio University in Fufusawa, Japan.

Glen D.Fraser is a Computer Engineer with a passion for virtual reality and other forms of real-time visual computing. He currently works at SOFTIMAGE, developing interactive viewing and animation tools.

The copyright of articles and images printed remains with the author unless otherwise indicated.