SIGGRAPH Course: Design of Interactive Multimodal Media Systems

Design of Interactive Multimodal Media Systems

by Wendy Ju

How do you hide a six-foot gorilla in the center of a video frame?

Researchers from the University of British Columbia demonstrated this and other curiosities of human perception in their course on the Design of Interactive Multimodal Media Systems.

The course, geared for those involved in designing interactive media and applications, presented breaking research in how people attend to various aspects of visual and multi-modal information, and provided techniques for harnessing these understandings to direct how people interact with emerging technologies.

For instance, Ronald Rensink, an Assistant Professor in both the Computer Science and Psychology departments at University of British Columbia, illustrated the difference between attentional and non-attentional vision processes by asking people to count the number of times a team of players bounced a ball to one another in a short video. Afterwards, a quarter of the audience admitted that they did not see the man in the gorilla suit walk right through the center of that video segment.

Michael Halle, from the Harvard Medical School, sees immediate applicability of the ability to "design attention" in his work on computer graphics for surgery. "It's interesting to have a talk about things we see all the time, but we don't know we see... If a doctor is looking at one piece of information on the computer screen and something else is important and relevant, then we want to lead the surgeon to that new location but not necessarily distract them from something more important so this is very central to that sort of interface development."

The presentations covered a broad array of topics in the field of multimodal system design. The sessions before lunch focused on understanding human perception. Brian Fischer presented findings on how our senses interact, affecting our overall perceptions. Ronald Rensink discussed studies of human vision to demonstrate that how we see has a lot to do with what we're attending to. After lunch, the sessions were geared on how to integrate the understandings of how we sense the world in actual designs. Karon Maclean reviewed methods for physical interaction design using haptics and multimodal interfaces, and Sid Fels surveyed novel human interface technologies.

The course presenters fearlessly backed controversial positions regarding multimodal system design. They proposed, for example, that within seemingly single modes there were numerous "modules"-- such as attentional and non-attentional vision-- which could be exploited to design the user's percept of the world. They also countered the oft-held belief that such design should always be user-centric. Sid Fels, from the UBC department of Electrical and Computer Engineering, argued that since users do not always know what's going on, it is sometimes necessary to go to theories of human perception to determine how best to design tools for human use.

Above all, the lecturers of the course felt that such technologies will be a growing aspect of SIGGRAPH in the coming years. "SIGGRAPH, remember, is graphics and interactive techniques-- it's often left out. But graphics is evolving. It's about the human in the loop," affirmed Fels. "What we see is influenced by what we hear and what we touch, so it's no longer just about graphics."