The New York University Media Research Laboratory (NYU-MRL) aims to create a sense of social space and interaction, to create virtual actors that respond believably in social ways in real time, without the intervention of human puppeteers, and to effectively use a distributed network for computing the behavior of the virtual actors. These are enabling technologies for interactive television and embodied metaverses.
In 1985, NYU-MRL used pseudo-random functions to create natural surface textures of surprisingly realistic appearance without modeling the underlying physics. This work led, for example, to a popular texture "noise function," the skin on the "Jurassic Park" dinosaurs, and the atmosphere effects in "The Lion King."
Last year, the lab applied the same approach to the problem of building real-time graphic puppets that appear to be emotionally responsive. Instead of trying to model the correct physics of human movement, they applied noise functions, together with rhythmic motions. The results, which were surprisingly lifelike, can be used to convey very subtle expressions of personality and body language. They were demonstrated in the short film "Danse Interactif" in the SIGGRAPH 94 Electronic Theatre.
NYU-MRL shares an interest in real-time parallel computation with the Laboratorio de Sistemas Integraveis at the University of Sao Paolo. Together, the two labs have now successfully extended these techniques to real-time social interaction between groups of puppets.
In this project, participants enter a darkened corridor and see a virtual room through a "picture window." A few virtual people in the virtual room are interacting socially with each other. The virtual room is rendered on a Silicon Graphics workstation and rear projected via a high-resolution projector. A ceiling-mounted video tracker tracks each participant's position. There is no trained operator intervening in the activities. Naive participants interact on their own, in complex ways, with the virtual actors.
The virtual people in the virtual room interact with each other. They move about, perhaps give each other things, look out the window, or shake hands. It's a typical social gathering. To the participant, their "talking" is heard as indistinct murmuring. It is clear that they are talking (even their tone of voice can be discerned), but it is not clear what language they are speaking.
The participants are interesting to these virtual people. One or two of them might glance over to see what a participant is up to. Another might walk over to the "window" and peer at the participant. As the participant moves about, virtual people follow the movements with their eyes. One might point, or wave. If the participant knocks on the window, one of them might knock back, and the sound is audible. If the participant gets too close, the virtual person near the "window" steps back, to maintain a safe distance. Two or more people might argue about the participant's presence, pointing and waving or raising their voices. Participants try to get the attention of the virtual people in the virtual room. They try to be noticed, and to provoke a reaction.
To the participant, the virtual people seem to be from another country or even another planet. They do not speak the same language, but there is an attempt to communicate, mostly through body language.
Each actor is computed on a different computer in a local area network (LAN). They communicate with each other over the LAN in real time at interactive frame rates. The actors actually pass entire behavior procedures to each other over the network at each animation frame, giving each other complex instructions and hints on how to respond to each other and allowing them to coordinate their responses to the participant.
The behavior description language that is passed around the network is the same language that was originally developed for NYU-MRL's interactive texture work in 1984. It has special features not usually found in programming languages (including very lightweight procedure definitions and a powerful method for fast dynamic scoping) that are necessary to realize this sort of performance.
This research focuses on situations in which people communicate richly through body language, such as parties, bar scenes, and meetings. The goals of this project include:
In particular, NYU-MRL is exploring how to convey peripheral awareness, approach/avoidance, "paying attention," "listening," etc. The lab also studies immersive scenarios involving two or more walls, to determine how simulated body language will help to convey the impression of various competing social or attention-getting activities.
Ultimately, NYU-MRL's goal is to determine to what extent the mere rhythm of interpersonal interaction can be encoded, in order to convey the impression of social complexity. For example, could one structure entire narratives in this manner?
Assessment of Potential Future Impact
Consider this possible scenario: you send your "agent" to the Metaverse library to get some information. Along the way, the agent encounters your neighbor strolling the Metaverse. You've already told your agent that if she encounters this person, she must remind him of your lunch date tomorrow.
Alternatively, for networked video games, this NYU-MRL work demonstrates that it is feasible to generate fully 3D, interactive characters. This allows networked video games to break away from the current restrictive reliance on canned CD-ROM-based video footage and animation clips.
This project also shows that it is feasible, using current technology, to apply these techniques non-invasively. People can interact in their own homes with real-time personal agents who have convincing and interesting personalities and live behind wall-size screens. Such agents can be used for entertainment, information access, or simply for company.
New York University Media Research Laboratory
New York, New York 10003 USA
New York University Media Research Laboratory
Silicon Graphics IndyXZ Presenter
Two 200 MHz Silicon Graphics Indigo2 Extremes
Macintosh Quadra 840AV
Rear projection screen
Ceiling-mounted miniature video camera
Procedural animation software created at NYU Media Research Lab
To Index mTropolis & AVRe REALWare VR System