Tomographic Projector: Large Scale Volumetric Display with Uniform Viewing Experiences

Abstract: Over the past century, as display evolved, people have demanded more realistic and immersive experiences in theaters. Here, we present a tomographic projector for a volumetric display system that accommodates large audiences while providing a uniform experience. The tomographic projector combines high-speed digital micromirror and three spatial light modulators to refresh projection images at 7200 Hz. With synchronization of the tomographic projector and wearable focus-tunable eyepieces, the presented system can reconstruct 60 focal planes for volumetric representation right in front of audiences. We demonstrate proof of concept of the proposed system by implementing a miniaturized theater environment. Experimentally, we show that this system has wide expressible depth range with focus cues from 25 cm to optical infinity with sufficient tolerance while preserving high resolution and contrast. We also confirm that the proposed system provides uniform experience in a wide range of viewing zone through simulation and experiment. Additionally, the tomographic projector has capability to equalize vergence state that varies in conventional stereoscopic 3D theater according to viewing position as well as interpupillary distance. This study is concluded with thorough discussion about tomographic projectors in terms of challenges and research issues.

Authors/Presenter(s): Youngjin Jo, Seoul National University, South Korea
Seungjae Lee, Seoul National University, South Korea
Dongheon Yoo, Seoul National University, South Korea
Suyeon Choi, Seoul National University, South Korea
Dongyeon Kim, Seoul National University, South Korea
Byoungho Lee, Seoul National University, South Korea

An Integrated 6DoF Video Camera and System Design

Abstract: Designing a fully integrated 360 degree video camera supporting 6DoF head motion parallax requires overcoming many technical hurdles, including camera placement, optical design, sensor resolution, system calibration, real-time video capture, depth reconstruction, and real-time novel view synthesis. While there is a large body of work describing various system components, such as multi-view depth estimation, our paper is the first to describe a complete, reproducible system that considers the challenges arising when designing, building, and deploying a full end-to-end 6DoF video camera and playback environment. Our system includes a computational imaging software pipeline supporting on-line marker-less calibration, high-quality reconstruction, and real-time streaming and rendering. Most of our exposition is based on a professional 16-camera configuration, which will be commercially available to film producers. However, our software pipeline is generic and can handle a variety of camera geometries and configurations. The entire calibration and reconstruction software pipeline along with example datasets will be open sourced to encourage follow-up research in high-quality 6DoF video reconstruction and rendering.

Authors/Presenter(s): Albert Parra Pozo, Facebook, United States of America
Michael Toksvig, Facebook, United States of America
Terry Filiba Schrager, Facebook, United States of America
Joyce Hsu, Facebook, United States of America
Uday Mathur, RED Digital Cinema, United States of America
Alexander Sorkine Hornung, Facebook, Switzerland
Rick Szeliski, Facebook, United States of America
Brian Cabral, Facebook, United States of America

The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting

Abstract: We present "The Relightables", a volumetric capture system for photorealistic and high quality relightable full-body performance capture. While significant progress has been made on volumetric capture systems, focusing on 3D geometric reconstruction with high resolution textures, much less work has been done to recover photometric properties needed for relighting. Results from such systems lack high-frequency detail and the subject's shading is prebaked into the texture. In contrast, a large body of work has addressed relightable acquisition for image-based approaches, which photograph the subject under a set of basis lighting conditions and recombine the images to show the subject as they would appear in a target lighting environment. However, to date, these approaches have not been adapted for use in the context of a high-resolution volumetric capture system. Our method combines this ability to realistically relight humans for arbitrary environments, with the benefits of free-viewpoint volumetric capture and new levels of geometric accuracy for dynamic performances. Our subjects are recorded inside a custom geodesic sphere outfitted with 331 custom color LED lights, an array of high-resolution cameras, and a set of custom high-resolution depth sensors. Our system innovates in multiple areas: First, we designed a novel active depth sensor to capture 12.4 MP depth maps, which we describe in detail. Second, we show how to design a hybrid geometric and machine learning reconstruction pipeline to process the high resolution input and output a volumetric video. Third, we generate temporally consistent reflectance maps for dynamic performers by leveraging the information contained in two alternating color gradient illumination images acquired at 60Hz. Multiple experiments, comparisons, and applications show that The Relightables significantly improves upon the level of realism in placing volumetrically captured human performances into arbitrary CG scenes.

Authors/Presenter(s): Kaiwen Guo, Google, United States of America
Peter Lincoln, Google, United States of America
Philip Davidson, Google, United States of America
Jay Busch, Google, United States of America
Xueming Yu, Google, United States of America
Matt Whalen, Google, United States of America
Geoff Harvey, Google, United States of America
Sergio Orts Escolano, Google, United States of America
Rohit Pandey, Google, United States of America
Jason Dourgarian, Google, United States of America
Danhang Tang, Google, United States of America
Anastasia Tkach, Google, United States of America
Adarsh Kowdle, Google, United States of America
Emily Cooper, Google, United States of America
Mingsong Doums, Google, United States of America
Sean Fanello, Google, United States of America
Graham Fyffe, Google, United States of America
Christoph Rhemann, Google, United States of America
Jonathan Taylor, Google, United States of America
Paul Debevec, Google, United States of America
Shahram Izadi, Google, United States of America

Modeling Endpoint Distribution of Pointing Selection Tasks in Virtual Reality Environments

Abstract: Understanding the endpoint distribution of pointing selection tasks can reveal the underlying patterns on how users tend to acquire a target, which is one of the most essential and pervasive tasks in interactive systems. It could further aid designers to create new graphical user interfaces and interaction techniques that are optimized for accuracy, efficiency, and ease of use. Previous research has explored the modeling of endpoint distribution outside of virtual reality (VR) systems that have shown to be useful in predicting selection accuracy and guide the design of new interactive techniques. This work aims at developing an endpoint distribution of selection tasks for VR systems which has resulted in EDModel, a novel model that can be used to predict endpoint distribution of pointing selection tasks in VR environments. The development of EDModel is based on two users studies that have explored how factors such as target size, movement amplitude, and target depth affect the endpoint distribution. The model is built from the collected data and its generalizability is subsequently tested in complex scenarios with more relaxed conditions. Three applications of EDModel inspired by previous research are evaluated to show the broad applicability and usefulness of the model: correcting the bias in Fitts's law, predicting selection accuracy, and enhancing pointing selection techniques. Overall, EDModel can achieve high prediction accuracy and can be adapted to different types of applications in VR.

Authors/Presenter(s): Difeng Yu, The University of Melbourne, Xi'an Jiaotong-Liverpool University, Australia
Hai-Ning Liang, Xi'an Jiaotong-Liverpool University, China
Xueshi Lu, Xi'an Jiaotong-Liverpool University, China
Kaixuan Fan, Xi'an Jiaotong-Liverpool University, China
Barrett Ens, Monash University, Australia

Learned Large Field-of-View Imaging With Thin-Plate Optics

Abstract: Typical camera optics consist of a system of individual elements that are designed to compensate for the aberrations of a single lens. Recent computational cameras shift some of this correction task from the optics to post-capture processing, reducing the imaging optics to only a few optical elements. However, these systems only achieve reasonable image quality by limiting the field of view (FOV) to a few degrees -- effectively ignoring severe off-axis aberrations with blur sizes of multiple hundred pixels. In this paper, we propose a lens design and learned reconstruction architecture that lift this limitation and provide an order of magnitude increase in field of view using only a single thin-plate lens element. Specifically, we design a lens to produce spatially shift-invariant point spread functions, over the full FOV, that are tailored to the proposed reconstruction architecture. We achieve this with a mixture PSF, consisting of a peak and and a low-pass component, which provides residual contrast instead of a small spot size as in traditional lens designs. To perform the reconstruction, we train a deep network on captured data from a display lab setup, eliminating the need for manual acquisition of training data in the field. We assess the proposed method in simulation and experimentally with a prototype camera system. We compare our system against existing single-element designs, including an aspherical lens and a pinhole, and we compare against a complex multi-element lens,} validating high-quality large field-of-view (i.e. 53◦) imaging performance using only a single thin-plate element.

Authors/Presenter(s): YIFAN PENG, Stanford University, United States of America
QILIN SUN, King Abdullah University of Science and Technology (KAUST), Saudi Arabia
XIONG DUN, King Abdullah University of Science and Technology (KAUST), Saudi Arabia
GORDON WETZSTEIN, Stanford University, United States of America
WOLFGANG HEIDRICH, King Abdullah University of Science and Technology (KAUST), Saudi Arabia
FELIX HEIDE, Princeton University, Algolux, United States of America