Photo by: Cyrill Beeler
If you were one of the millions of moviegoers who contributed to the worldwide success of Avengers: Infinity War—the highest-grossing film of 2018—then you also got to witness the Medusa Performance Capture System in action. The team of computer scientists responsible for bringing to life characters like Thanos and the Hulk on the big screen was honored this year with a Sci-Tech Academy Award for the conception, design and engineering of Medusa.
Medusa, a comprehensive hardware and software system which has been developed, tweaked and expanded upon over the last 10 years, enables the precise digital replication of human faces, including detailed expressions and super fine physical details at high resolution. The Academy presented Sci-Tech Award certificates to the Medusa team, Thabo Beeler, Derek Bradley, Bernd Bickel and Markus Gross, at its Feb. 9, 2019, ceremony held at the Beverly Wilshire Hotel.
Credited for the initial concept, Gross, who is a professor of computer science at the Swiss Federal Institute of Technology Zürich (ETH), vice president for research at The Walt Disney Studios and director of DisneyResearch|Studios, worked with Bickel, then a doctoral student in his lab at ETH, to overcome this grand challenge in computer graphics: to create digital human faces that are indistinguishable from reality. Bickel is currently an assistant professor of computer science at the Institute of Science and Technology Austria. In 2017, he received the ACM SIGGRAPH New Researcher Award.
“Medusa is the culmination of many years of research on digital human faces and digital facial animation that we’ve been working on as part of the ongoing work in my lab and in collaboration with Disney Research,” says Gross, a longtime member of ACM SIGGRAPH and a 2012 ACM Fellow. “We got connected much more deeply with the arts and technology of special effects through our partnership with the Walt Disney Company. It gave us the insights to steer the research in such a way that we could make the best progress for advancement of facial technologies for film.”
Gross notes that at the time, researchers did not have a lot of success in bridging the so-called “uncanny valley”, which is a known phenomenon in the field that refers to the digital duplication of human faces that are not quite realistic, almost disturbingly fake in appearance.
With this Oscar honor, Medusa has staked its claim as an industry standard in special effects, achieving digital characters with highly realistic human features. This year, three out of the Oscar nominated films for best visual special effects used the Medusa system, and in recent years it has been used in numerous productions, including Star Wars: The Last Jedi and Spider-Man: Homecoming.
“The methods are really tuned into highly accurate measurement of human faces. As such our technology is not generic per se,” explains Gross. “One of the insights we’ve had in exploring facial performance is that human facial expressions usually go from a neutral state into some deformed state and then go back to that neutral-rest state. There are cycles, and examining these cycles has allowed us to build methods to track the facial surface reliably over time. We further use the randomness of the pigmentation of facial skin as invisible landmarks to reference points across consecutive frames.”
Gross is also proud of the research and development that went into the analysis of facial microgeometry, such as facial pores, which eventually allows Medusa to capture the complex geometry of human faces including minute facial details. This is essential for re-creating realistic facial expressions.
Shortly following their early partnership with Disney, Gross and Bickel were joined by Beeler and Bradley, now both research scientists at Disney Research. Developing the 3D face scanner component of the capture system became Beeler’s master thesis at the time. When Bradley, then a post-doc, joined the team, he began working on the motion capture piece and helped develop the stable face-tracking technology. Together with Beeler, the duo invented the method for separating the rigid and non-rigid components of the face performance, and they have also spent a lot of time productizing the research into an artist-friendly system.
“Bringing this research into production entailed a lot of development work, but we got a lot of input regarding potential research topics in return, one of which ended up in a publication on rigid stabilization, published at SIGGRAPH,” says Beeler. “In this work we explore the relationship between the skin surface and the underlying bone structure to separate the rigid head motion from the non-rigid face deformation, an essential step when building facial rigs and an integral part of the Medusa system.”
In fact, all of the major milestones in the development of Medusa were showcased in SIGGRAPH papers. One of the team’s images had also been used on the front page of proceedings at SIGGRAPH 2011, for which Beeler was very proud.
“We’re constantly improving Medusa through new research, and simultaneously developing the next generation performance capture technology,” says Bradley. “We can’t say much, but the future is very exciting in this field.”
This Sci-Tech award marks the second Academy Award for Gross, who won a Technical Achievement Award in 2013 for the technology that more efficiently simulates smoke and explosions in films. For Gross, this current award win is very special as it both rewards years of academic research and marks the first recognition of this kind for Disney Research.
“We worked on Medusa for literally 10 years, and I’ve been working on digital human faces since I was a post-doc, which has been some 30 years,” says Gross. “It was a beautiful experience to get recognition from the Academy for all of this work.”
Still, there is more to come. “I often compare my work to a rabbit hole,” says Beeler. “We are making great progress but the deeper we go and the more we solve, the more we realize that we are far from done. My ultimate goal is to provide technology to digitize humans holistically, with minimal effort and at maximal quality—and every day we are getting a step closer to this vision.”
By Melanie A. Farmer