Reducing Simulator Sickness with Perceptual Camera Control

Abstract: Virtual-reality provides an immersive environment but can induce cyber-sickness due to the discrepancy between visual and vestibular cues. To avoid this problem, the movement of the virtual camera needs to match the motion of the user in the real world. Unfortunately, this is usually difficult due to the mismatch between the size of the virtual environments and the space available to the users in the physical domain. The resulting constraints on the camera movement significantly hamper the adoption of virtual-reality headsets in many scenarios and make the design of the virtual environments very challenging. In this work, we study how the characteristics of the virtual camera movement (e.g., translational acceleration and rotational velocity) and the composition of the virtual environment (e.g., scene depth) contribute to perceived discomfort. Based on the results from our user experiments, we devise a computational model for predicting the magnitude of the discomfort for a given scene and camera trajectory. We further apply our model to a new path planning method which optimizes the input motion trajectory to reduce perceptual sickness. We evaluate the effectiveness of our method in improving perceptual comfort in a series of user studies targeting different applications. The results indicate that our method can reduce the perceived discomfort while maintaining the fidelity of the original navigation, and perform better than simpler alternatives.

Authors/Presenter(s): Ping Hu, Stony Brook University, United States of America
Qi Sun, Adobe Research, United States of America
Piotr Didyk, Università della Svizzera italiana (USI), Switzerland
Li-Yi Wei, Adobe Research, United States of America
Arie E. Kaufman, Stony Brook University, United States of America

DiCE: Dichoptic Contrast Enhancement for VR and Stereo Displays

Abstract: In stereoscopic displays, such as those used in VR/AR headsets, our eyes are presented with two different views. The disparity between the views is typically used to convey depth cues, but it could be also used to enhance image appearance. We devise a novel technique that takes advantage of binocular fusion to boost perceived local contrast and visual quality of images. Since the technique is based on fixed tone curves, it has negligible computational cost and it is well suited for real-time applications, such as VR rendering. To control the trade-off between contrast gain and binocular rivalry, we conduct a series of experiments to explain the factors that dominate rivalry perception in a dichoptic presentation where two images of different contrasts are displayed. With this new finding, we can effectively enhance contrast and control rivalry in mono- and stereoscopic images and in VR rendering, as confirmed in validation experiments.

Authors/Presenter(s): Fangcheng Zhong, University of Cambridge, United Kingdom
George A. Koulieris, Durham University, United Kingdom
George Drettakis, Inria, France
Martin S. Banks, UC Berkeley, United States of America
Mathieu Chambe, ENS Rennes, France
Fredo Durand, MIT, United States of America
Rafał K. Mantiuk, University of Cambridge, United Kingdom

DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression using Learned Statistics of Natural Videos

Abstract: In order to provide an immersive visual experience, modern displays require head mounting, high image resolution, low latency, as well as high refresh rate. This poses a challenging computational problem. On the other hand, the human visual system can consume only a tiny fraction of this video stream due to drastic acuity loss in the peripheral vision. Foveated rendering and compression can save computations by reducing the image quality in the peripheral vision. However, this can cause noticeable artifacts in the periphery, or, if done conservatively, would provide only modest savings. In this work, we explore a novel foveated reconstruction method that employs the recent advances in generative adversarial neural networks. We reconstruct a plausible peripheral video from a small fraction of pixels provided per frame. The idea is to project a sparse input stream of pixels onto the learned manifold of natural videos. Our method is more efficient than the state-of-the-art foveated rendering, while providing the visual experience with no noticeable quality degradation. We conducted a user study to validate our reconstruction method and compare it against existing foveated rendering and video compression techniques. Our method is fast enough to drive gaze-contingent head-mounted displays in real time on modern hardware. We plan to publish the trained network to establish a new quality bar for foveated rendering and compression as well as encourage follow-up research.

Authors/Presenter(s): Anton Kaplanyan, Facebook, United States of America
Anton Sochenov, Facebook, United States of America
Thomas Leimkuehler, Facebook, United States of America
Mikhail Okunev, Facebook, United States of America
Todd Goodall, Facebook, United States of America
Gizem Rufo, Facebook, United States of America

Wirtinger Holography for Near-eye Displays

Abstract: Near-eye displays using holographic projection are emerging as an exciting display approach for virtual and augmented reality at high-resolution without complex optical setups --- shifting optical complexity to computation. While precise phase modulation hardware is becoming available, phase retrieval algorithms are still in their infancy, and holographic display approaches resort to heuristic encoding methods or iterative methods relying on various relaxations. In this work, we depart from such existing approximations, and solve the phase retrieval problem directly revisiting complex Wirtinger derivatives. Using Wirtinger derivatives allow us to pose the phase retrieval problem as a quadratic problem which can be minimized with first-order optimization methods. The proposed Wirtinger Holography is flexible and facilitates the use of different loss functions, including learned perceptual losses parametrized by deep neural networks, as well as stochastic optimization methods. We validate this framework by demonstrating holographic reconstructions with an order of magnitude lower error, in simulation and on a experimental hardware prototype.

Authors/Presenter(s): Praneeth Chakravarthula, University of North Carolina at Chapel Hill (UNC), United States of America
Yifan Peng, Stanford University, United States of America
Joel S. Kollin, Microsoft Research, United States of America
Henry Fuchs, University of North Carolina at Chapel Hill (UNC), United States of America
Felix Heide, Princeton University, United States of America

Holographic Near-Eye Displays Based on Overlap-Add Stereograms

Abstract: Holographic near-eye displays are a key enabling technology for virtual and augmented reality (VR/AR) applications. Holographic stereograms (HS) are a method of encoding a light field into a hologram, which enables them to natively support view-dependent lighting effects. However, existing HS algorithms require the choice of a hogel size, forcing a tradeoff between spatial and angular resolution. Based on the fact that the short-time Fourier transform (STFT) connects a hologram to its observable light field, we develop the overlap-add stereogram (OLAS) as the correct method of “inverting” the light field into a hologram via the STFT. The OLAS makes more efficient use of the information contained within the light field than previous HS algorithms, exhibiting better image quality at a range of distances and hogel sizes. Most remarkably, the OLAS does not degrade spatial resolution with increasing hogel size, overcoming the spatio-angular resolution tradeoff that previous HS algorithms face. Importantly, the optimal hogel size of previous methods typically varies with the depth of every object in a scene, making the OLAS not only a hogel size–invariant method, but also nearly scene independent. We demonstrate the performance of the OLAS both in simulation and on a prototype near-eye display system, showing focusing capabilities and view-dependent effects.

Authors/Presenter(s): Nitish Padmanaban, Stanford University, United States of America
Yifan Peng, Stanford University, United States of America
Gordon Wetzstein, Stanford University, United States of America