OpenSketch: A Richly-Annotated Dataset of Product Design Sketches

Abstract: Product designers extensively use sketches to create and communicate 3D shapes and thus form an ideal audience for sketch-based modeling, non-photorealistic rendering and sketch filtering. However, sketching requires significant expertise and time, making design sketches a scarce resource for the research community. We introduce OpenSketch, a dataset of product design sketches aimed at offering a rich source of information for a variety of computer-aided design tasks. OpenSketch contains more than 400 sketches representing 12 man-made objects drawn by 7 to 15 product designers of varying expertise. We provided participants with front, side and top views of these objects, and instructed them to draw from two novel perspective viewpoints. This drawing task forces designers to construct the shape from their mental vision rather than directly copy what they see. They achieve this task by employing a variety of sketching techniques and methods not observed in prior datasets. Together with industrial design teachers, we distilled a taxonomy of line types and used it to label each stroke of the 214 sketches drawn from one of the two viewpoints. While some of these lines have long been known in computer graphics, others remain to be reproduced algorithmically or exploited for shape inference. In addition, we also asked participants to produce clean presentation drawings from each of their sketches, resulting in aligned pairs of drawings of different styles. Finally, we registered each sketch to its reference 3D model by annotating sparse correspondences. Our sketches, in combination with provided annotations, form challenging benchmarks for existing algorithms as well as a great source of inspiration for future developments. We illustrate the versatility of our data by using it to test a 3D reconstruction deep network trained on synthetic drawings, as well as to train a filtering network to convert concept sketches into presentation drawings.

Authors/Presenter(s): Yulia Gryaditskaya, Universite Cote d'Azur, Inria, France
Mark Sypesteyn, Delft University of Technology, Faculty of Industrial Design Engineering, Netherlands
Jan Willem Hoftijzer, Delft University of Technology, Faculty of Industrial Design Engineering, Netherlands
Sylvia Pont, Delft University of Technology, Perceptual Intelligence lab, Netherlands
Fredo Durand, MIT CSAIL, Inria, United States of America
Adrien Bousseau, Universite Cote d'Azur, Inria, France

Language-based Colorization of Scene Sketches

Abstract: Being natural, touchless, and fun-embracing, language-based inputs have been demonstrated effective for various tasks from image generation to literacy education for children. This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on semantic comprehension. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoon-style color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific foreground object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state of the art, and generalization user studies. Given the unique characteristics of language-based inputs, we envision a combination of our interface with a traditional scribble-based interface for a practical, multimodal colorization system, benefiting various applications.

Authors/Presenter(s): Changqing Zou, Sun Yat-sen University, Huawei Noah's Ark Lab, China
Haoran Mo, Sun Yat-sen University, China
Chengying Gao, Sun Yat-sen University, China
Ruofei Du, Google, United States of America
Hongbo Fu, City University of HongKong, Hong Kong

Data-driven Interior Plan Generation for Residential Buildings

Abstract: We propose a novel data-driven technique for automatically and efficiently generating floor plans for residential buildings with given boundaries. Central to this method is a two-stage approach that imitates the human design process by locating rooms first and then walls while adapting to the input building boundary. Based on observations of the presence of the living room in almost all floor plans, our designed learning network begins with positioning a living room and continues by iteratively generating other rooms. Then, walls are first determined by an encoder-decoder network, and then they are refined to vector representations using dedicated rules. To effectively train our networks, we construct RPLAN - a manually collected large-scale densely annotated dataset of floor plans from real residential buildings. Intensive experiments, including formative user studies and comparisons, are conducted to illustrate the feasibility and efficacy of our proposed approach. By comparing the plausibility of different floor plans, we have observed that our method substantially outperforms existing methods, and in many cases our floor plans are comparable to human-created ones.

Authors/Presenter(s): Wenming Wu, University of Science and Technology of China, China
Xiao-Ming Fu, University of Science and Technology of China, China
Rui Tang, Kujiale, China
Yuhan Wang, Kujiale, China
Yu-Hao Qi, University of Science and Technology of China, China
Ligang Liu, University of Science and Technology of China, China

Dynamic Hair Modeling from Monocular Videos using Deep Neural Networks

Abstract: We introduce a deep-learning-based framework for modeling dynamic hairs from monocular videos, which could be captured by a commodity video camera or downloaded from Internet. The framework mainly consists of two network structures, i.e., \emph{HairSpatNet} for inferring 3D spatial features of hair geometry from 2D image features, and \emph{HairTempNet} for extracting temporal features of hair motions from video frames. The spatial features are represented as 3D occupancy fields depicting the hair shapes and 3D orientation fields indicating the hair strand directions. The temporal features are represented as bidirectional 3D warping fields, describing the forward and backward motions of hair strands cross adjacent frames. Both \emph{HairSpatNet} and \emph{HairTempNet} are trained with synthetic hair data. The spatial and temporal features predicted by the networks are subsequently used for growing hair stands with both spatial and temporal consistency. Experiments demonstrate that our method is capable of constructing high-quality dynamic hair models that resemble the input video as closely as those reconstructed by the state-of-the-art multi-view method, and compares favorably to previous single-view techniques.

Authors/Presenter(s): Lingchen Yang, State Key Laboratory of CAD & CG, Zhejiang University; ZHEJIANG UNIVERSITY, China
Zefeng Shi, State Key Laboratory of CAD & CG, Zhejiang University; ZHEJIANG UNIVERSITY, China
Youyi Zheng, Zhejiang University and ZJU-FaceUnity Joint Lab of Intelligent Graphics, ZJU, China
Kun Zhou, Zhejiang University and ZJU-FaceUnity Joint Lab of Intelligent Graphics, ZHEJIANG UNIVERSITY, China