Handheld Mobile Photography in Very Low Light

Abstract: Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, use mass-produced analog electronics that cannot easily be cooled, and are commonly used to photograph subjects that move, like children and pets. In this paper we describe a system for capturing clean, sharp, colorful photographs in light as low as 0.3 lux, where human vision becomes monochromatic and indistinct. To permit handheld photography without flash illumination, we capture, align, and combine multiple frames. Our system employs "motion metering", which uses an estimate of motion magnitudes (whether due to handshake or moving objects) to identify the number of frames and the per-frame exposure times that together minimize both noise and motion blur in a captured burst. We combine these frames using robust alignment and merging techniques that are specialized for high-noise imagery. To ensure accurate colors in such low light, we employ a learning-based auto white balancing algorithm. To prevent the photographs from looking like they were shot in daylight, we use tone mapping techniques inspired by illusionistic painting: increasing contrast, crushing shadows to black, and surrounding the scene with darkness. All of these processes are performed using the limited computational resources of a mobile device. Our system can be used by novice photographers to produce shareable pictures in a few seconds based on a single shutter press, even in environments so dim that humans cannot see clearly.

Authors/Presenter(s): Samuel W. Hasinoff, Google, United States of America
Yael Pritch, Google, United States of America
Marc Levoy, Google, United States of America
Orly Liba, Google, United States of America
Kiran Murthy, Google, United States of America
Yun-Ta Tsai, Google, United States of America
Timothy Brooks, Google, United States of America
Tianfan Xue, Google, United States of America
Nikhil Karnad, Google, United States of America
Qiurui He, Google, United States of America
Jonathan T. Barron, Google, United States of America
Dillon Sharlet, Google, United States of America
Ryan Geiss, Google, United States of America

Learning Efficient Illumination Multiplexing for Joint Capture of Reflectance and Shape

Abstract: We propose a novel framework that automatically learns the lighting patterns for efficient, joint acquisition of unknown reflectance and shape. The core of our framework is a deep neural network, with a shared linear encoder that directly corresponds to the lighting patterns used in physical acquisition, as well as non-linear decoders that output per-pixel normal and diffuse / specular information from photographs. We exploit the diffuse and normal information from multiple views to reconstruct a detailed 3D shape, and then fit BRDF parameters to the diffuse / specular information, producing texture maps as reflectance results. We demonstrate the effectiveness of the framework with physical objects that vary considerably in reflectance and shape, acquired with as few as 16~32 lighting patterns that correspond to 7~15 seconds of per-view acquisition time. Our framework is useful for optimizing the efficiency in both novel and existing setups, as it can automatically adapt to various factors, including the geometry / the lighting layout of the device and the properties of appearance.

Authors/Presenter(s): Kaizhang Kang, Zhejiang University, China
Cihui Xie, Zhejiang University, China
Chengan He, Yale University, United States of America
Mingqi Yi, Zhejiang University, China
Minyi Gu, Zhejiang University, China
Zimin Chen, Zhejiang University, China
Kun Zhou, Zhejiang University, China
Hongzhi Wu, Zhejiang University, China

Blind image super resolution with spatially variant degradations

Abstract: Existing deep learning approaches to single image super-resolution have achieved impressive results but mostly assume a setting with fixed pairs of high resolution and low resolution images. However, to robustly address realistic upscaling scenarios where the relation between high resolution and low resolution images is unknown, blind image super-resolution is required. To this end, we propose a solution that relies on three components: First, we use a degradation aware SR network to synthesize the HR image given a low resolution image and the corresponding blur kernel. Second, we train a kernel discriminator to analyze the generated high resolution image in order to predict errors present due to providing an incorrect blur kernel to the generator. Finally, we present an optimization procedure that is able to recover both the degradation kernel and the high resolution image by minimizing the error predicted by our kernel discriminator. We also show how to extend our approach to spatially variant degradations that typically arise in visual effects pipelines when compositing content from different sources and how to enable both local and global user interaction in the upscaling process.

Authors/Presenter(s): Victor Cornillère, ETH Zürich, Switzerland
Abdelaziz Djelouah, DisneyResearch|Studios, Switzerland
Yifan Wang, ETH Zürich, Switzerland
Christopher Schroers, DisneyResearch|Studios, Switzerland
Olga Sorkine-Hornung, ETH Zürich, Switzerland

Hierarchical and View-invariant Light Field Segmentation by Maximizing Entropy Rate on 4D Ray Graphs

Abstract: Image segmentation is an important first step of many image processing, computer graphics, and computer vision pipelines. Unfortunately, it remains difficult to automatically and robustly segment cluttered scenes, or scenes in which multiple objects have similar color and texture. In these scenarios, light fields offer much richer cues that can be used efficiently to drastically improve the quality and robustness of segmentations. In this paper we introduce a new light field segmentation method that respects texture appearance, depth consistency, as well as occlusion, and creates well-shaped segments that are robust under view point changes. Furthermore, our segmentation is hierarchical, i.e. with a single optimization, a whole hierarchy of segmentations with different numbers of regions is available. All this is achieved with a submodular objective function that allows for efficient greedy optimization. Finally, we introduce a new tree-array type data structure, i.e. a disjoint tree, to efficiently perform submodular optimization on very large graphs. This approach is of interest beyond our specific application of light field segmentation. We demonstrate the efficacy of our method on a number of synthetic and real data sets, and show how the obtained segmentations can be used for applications in image processing and graphics.

Authors/Presenter(s): Rui Li, KAUST, Saudi Arabia
Wolfgang Heidrich, KAUST, Saudi Arabia

Document Rectification and Illumination Correction using a Patch-based CNN

Abstract: We propose a novel learning method to rectify document images with various distortion types from a single input image. As opposed to previous learning-based methods, our approach seeks to first learn the distortion flow on input image patches rather than the entire image. We then present a robust technique to stitch the patch results into the rectified document by processing in the gradient domain. Furthermore, we propose a second network to correct the uneven illumination, further improving the readability and OCR accuracy. Due to the less complex distortion present on the smaller image patches, our patch-based approach followed by stitching and illumination correction can significantly improve the overall accuracy in both the synthetic and real datasets.

Authors/Presenter(s): Xiaoyu Li, The Hong Kong University of Science and Technology, Hong Kong
Bo Zhang, The Hong Kong University of Science and Technology, Microsoft Research Asia, Hong Kong
Jing Liao, City University of Hong Kong, Hong Kong
Pedro Sander, The Hong Kong University of Science and Technology, Hong Kong