SIGGRAPH 2020: ACM SIGGRAPH 2020 Courses

Vulkan is better at keeping the GPU busy than OpenGL is. OpenGL drivers need to do a lot of CPU work before handing work off to the GPU. Vulkan lets you get more power from the GPU card you already have.

Jaroslav Křivánek's research aimed at finding the one robust and efficient light transport simulation algorithm that would handle any given scene with any complexity of transport. He had a clear and unique vision of how to reach this ambitious goal. On his way, he created an impressive track of significant research contributions. In this course, his collaborators will tell the story of Jaroslav's quest for that "one" algorithm and discuss his impact and legacy.

Emerging detector technologies are capable of ultrafast capture of single photons, enabling imaging at the speed of light. Not only can these detectors be used for imaging at essentially trillion frame-per-second rates, but combining them with computational algorithms has given rise to unprecedented new imaging capabilities. Computational time-resolved imaging has enabled new techniques for 3D imaging, light transport analysis, imaging around corners or behind occluders, and imaging through scattering media such as fog, murky water, or human tissue. With applications in autonomous navigation, robotic vision, human-computer interaction, and more, this is an area of rapidly growing interest. In this course, we provide an introduction to computational time-resolved imaging and single photon sensing with a focus on hardware, applications, and algorithms. We describe various types of emerging single-photon detectors, including single-photon avalanche diodes and avalanche photodiodes, which are among the most popular time-resolved detectors. Physically accurate models for these detectors are described, including modeling parameters and noise statistics used in most computational algorithms. From the application side, we discuss the use of ultrafast active illumination for 3D imaging and transient imaging, and we describe the state of the art in non-line-of-sight imaging, which requires modelling and inverting the propagation and scattering of light from a visible surface to a hidden object and back. We describe time-resolved computational algorithms used in each of these applications and offer insights on potential future directions.

Application-domain-specific cameras that combine customized optics with modern image recovery algorithms are of rapidly growing interest, with widespread applications like ultrathin cameras for internet-of-things or drones, as well as computational cameras for microscopy and scientific imaging. Existing approaches of designing imaging optics are either heuristic or use some proxy metric on the point spread function rather than considering the image quality after post-processing. Without a true end-to-end flow of joint optimization, it remains elusive to find an optimal computational camera for a given visual task. Although this joint design concept has been the core idea of computational photography for a long time, but that only nowadays the computational tools are ready to efficiently interpret a true end-to-end imaging process via machine learning advances.

We describe the use of diffractive optics to enable lenses not only showing the compact physical appearance, but also flexible and large design degree of freedom. By building a differentiable ray or wave optics simulation model that maps the true source image to the reconstructed one, one can jointly train an optical encoder and electronic decoder. The encoder can be parameterized by the PSF of physical optics, and the decoder a convolutional neural network. By running over a broad set of images and defining domain-specific loss functions, parameters of the optics and image processing algorithms are jointly learned. We describe typical photography applications for extended depth-of-field, large field-of-view, and high-dynamic-range imaging. We also describe the generalization of this joint-design to machine vision and scientific imaging scenarios. To this point, we describe an end-to-end learned, optically coded super-resolution SPAD camera, and a hybrid optical-electronic convolutional layer based optimization of optics for image classification. Additionally, we explore lensless imaging with optimized phase masks for realizing an ultra-thin camera, a high-resolution wavefront sensing, and face detection.

Creating high-level structured 3D models of real-world indoor scenes from captured data is a fundamental task which has important applications in many fields. Given the complexity and variability of interior environments and the need to cope with noisy and partial captured data, many open research problems remain, despite the substantial progress made in the past decade. In this tutorial, we provide an up-to-date integrative view of the field, bridging complementary views coming from computer graphics and computer vision. After providing a characterization of input sources, we define the structure of output models and the priors exploited to bridge the gap between imperfect sources and desired output. We then identify and discuss the main components of a structured reconstruction pipeline, and review how they are combined in scalable solutions working at the building level. We finally point out relevant research issues and analyze research trends.

Some Thoughts on the Fresnel Term

Naty Hoffman

The Fresnel term would appear to be the best-understood part of the microfacet model --- one can simply use the original equations, or an approximation (usually Schlick's) if computation cost is at a premium. However, in this talk, we will show that all is not as it seems, and that even the humble Fresnel term can hold some surprises.

This talk builds on a previous presentation at the 2019 Eurographics Workshop on Material Appearance Modeling [Hof19], extending it into a more comprehensive overview.

The problem of seeing around corners, often referred in the broader "Non-Line-of-Sight" context, is to use sensed information from directly visible surfaces of an environment to infer properties of the scene not directly visible. For example, the geometry above presents a classic "around the corner" setting, where a flat wall is used as the visible surface, and the hidden scene is occluded by another wall. While many proposed sensing modalities have been proposed, including acoustic and RF signals, most approaches utilize photonic sensors in the visible spectrum due to the availability of hardware, and better temporal and spatial resolution. Approaches range from active time-resolved measurements, time-averaged continuous wave sources, and even to passive exploitation of ambient illumination.

Physics-based rendering algorithms generate photorealistic images by simulating the flow of light through a detailed mathematical representation of a virtual scene. In contrast, physics-based differentiable rendering algorithms focus on computing derivative of images exhibiting complex light transport effects (e.g., soft shadows, interreflection, and caustics) with respect to arbitrary scene parameters such as camera pose, object geometry (e.g., vertex positions) as well as spatially varying material properties expressed as 2D textures and 3D volumes. This new level of generality has made physics-based differentiable rendering a key ingredient for solving many challenging inverse-rendering problems, that is, the search of scene configurations optimizing user-specified objective functions, using gradient-based methods (as illustrated in the figure below). Further, these techniques can be incorporated into probabilistic inference and machine learning pipelines. For instance, differentiable renderers allow "rendering losses" to be computed with complex light transport effects captured. Additionally, they can be used as generative models that synthesize photorealistic images.

In recent years, much research has been dedicated to the development of "intelligent tools" that can assist both professionals as well as novices in the process of creation. Using the computational power of the machine, and involving advanced techniques, the tools handle complex and tedious tasks that were difficult or even impossible for humans, thereby freeing the human creator of many constraints and allowing her to concentrate on the creative process, while ensuring high-quality and valid design. This course is aimed at presenting some of the key technologies used to assist interactive creative processes. The course allows researchers and practitioners to understand these techniques more deeply, and possibly inspire them to research this subject and create intelligent tools themselves. More specifically, the course will concentrate on four main enabling technologies: geometric reasoning, physical constraints, data-driven techniques and machine learning, and crowdsourcing. In each of these areas the course will survey several recent papers and works and provide examples of using these in the creation of a variety of outputs: 3D models, animations, images, videos and more.

We use our hands every day: to grasp a cup of coffee, write text on a keyboard, or signal that we are about to say something important. We use our hands to interact with our environment and to help us communicate with each other without thinking about it. Wouldn't it be great to be able to do the same in virtual reality? However, accurate hand motions are not trivial to capture. In this course, we present the current state of the art when it comes to virtual hands. Starting with current examples for controlling and depicting hands in virtual reality (VR), we dive into the latest methods and technologies to capture hand motions. As hands can currently not be captured in every situation and as constraints stopping us from intersecting with objects are typically not available in VR, we present research on how to synthesize hand motions and simulate grasping motions. Finally, we provide an overview of our knowledge of how virtual hands are being perceived, resulting in practical tips on how to represent and handle virtual hands.

Our goals are (a) to present a broad state of the art of the current usage of hands in VR, (b) to provide more in-depth knowledge about the functioning of current hand motion tracking and hand motion synthesis methods, (c) to give insights on our perception of hand motions in VR and how to use those insights when developing new applications, and finally (d) to identify gaps in knowledge that might be investigated next. While the focus of this course is on VR, many parts also apply to augmented reality, mixed reality, and character animation in general, and some content originates from these areas.

The course starts with an overview of eye-tracking applications, distinguishing eye movement analysis from synthesis in virtual reality, games, and other venues including mobile eye tracking. The focus is on four forms of applications: diagnostic (off-line measurement), active (selection, look to shoot), passive (foveated rendering, a.k.a. gaze-contingent displays), and expressive (gaze synthesis). The course covers basic eye movement analytics, e.g., fixation count and dwell time within AOIs, as well as advanced analysis using ambient/focal attention modeling. The course concludes with an overview and a demo of how to build an interactive application using Python.

The Advanced Visualization Lab (AVL) is part of the the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign. The AVL is led by Professor Donna Cox, who coined the term "Renaissance Team", with the belief that bringing together specialists of diverse backgrounds creates a team that is greater than the sum of its parts, and members of the AVL team reflect that in our interdisciplinarity. We specialize in creating high-quality cinematic scientific visualizations of supercomputer simulations for public outreach.

Simulating dynamic deformation has been an integral component of Pixar's storytelling since Boo's shirt in Monsters, Inc. (2001). Recently, several key transformations have been applied to Pixar's core simulator Fizt that improve its speed, robustness, and generality. Starting with Coco (2017), improved collision detection and response were incorporated into the cloth solver, then with Cars 3 (2017) 3D solids were introduced, and in Onward (2020) clothing is allowed to interact with a character's body with two-way coupling.

The 3D solids are based on a fast, compact, and powerful new formulation that we have published over the last few years at SIGGRAPH. Under this formulation, the construction and eigendecomposition of the force gradient, long considered the most onerous part of the implementation, becomes fast and simple. We provide a detailed, self-contained, and unified treatment here that is not available in the technical papers.

This new formulation is only a starting point for creating a simulator that is up challenges of a production environment. One challenge is performance: we discuss our current best practices for accelerating system assembly and solver performance. Another challenge that requires considerable attention is robust collision detection and response. Much has been written about collision detection approaches such as proximity-queries, continuous collisions and global intersection analysis. We discuss our strategies for using these techniques, which provides us with valuable information that is needed to handle challenging scenarios.

Story (content) is not just the domain of directors and producers... anymore. Today, it is as important for technical directors, animators, VFX creators and interactive designers whose work is essential in making "the story" come to life. This information is particularly useful when communicating with screenwriters, directors, and producers. This course answers the question, "What is Story?" (and you don't even have to take a course in screenwriting). Knowing the basics of story enables the crew to become collaborators with the producer and director. A director may talk about their story goals; and the crew will know what specific story benchmarks they are trying to meet. This information builds from a story being more than "a sequence of events (acts) but can become a dramatic story that that builds from setup through resolution.

Having an understanding of story structure allows one to understand a story's elements in context (i.e., theme, character, setting, conflict etc.) and their relationship to classic story structure (i.e., setup, inciting incident, rising action, climax, resolution, etc.). This information is for all whose work makes the story better, but their job is not creating the story.

The following course notes are adapted from Story Structure and Development: A Guide for Animators, VFX Artists, Game Designers, and Virtual Reality Creators. CRC Publishers, a division of Taylor and Francis. Available on Amazon.