Methods, systems, and media for relighting images using predicted deep reflectance fields

US10997457B2 · kind B2 · utility

Assignee

• Google LLC · US

Inventors

• Christoph Rhemann · Santa Monica, US
• Abhimitra Meka · Riegelsberg, DE
• Matthew S. Whalen · Rumson, US
• Jessica Lynn Busch · Gardena, US
• Sofien Bouaziz · Los Gatos, US
• Geoffrey Harvey · Culver City, US
• Andrea Tagliasacchi · Mountain View, US
• Jonathan James Taylor · San Francisco, US
• Paul E. Debevec · Culver City, US
• Peter Joseph Denny · Santa Monica, US
• Sean Ryan Francesco Fanello · San Francisco, US
• Graham Fyffe · Los Angeles, US
• Jason Angelo Dourgarian · Los Angeles, US
• Xueming Yu · Yokohama, JP
• Adarsh Prakash Murthy Kowdle · San Francisco, US
• Julien Pascal Christophe Valentin · Mountain View, US
• Peter Christopher Lincoln · Chapel Hill, US
• Rohit Kumar Pandey · Mountain View, US
• Christian Häne · Berkeley, US
• Shahram Izadi · Cambridge, GB

Key dates

• Filing date: Oct 16, 2019
• Grant date: May 4, 2021
• Expiry date: Oct 16, 2039

Classification

• Technology area (CPC G): Physics
• CPC primary: G06V10/82
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

Methods, systems, and media for relighting images using predicted deep reflectance fields are provided. In some embodiments, the method comprises: identifying a group of training samples, wherein each training sample includes (i) a group of one-light-at-a-time (OLAT) images that have each been captured when one light of a plurality of lights arranged on a lighting structure has been activated, (ii) a group of spherical color gradient images that have each been captured when the plurality of lights arranged on the lighting structure have been activated to each emit a particular color, and (iii) a lighting direction, wherein each image in the group of OLAT images and each of the spherical color gradient images are an image of a subject, and wherein the lighting direction indicates a relative orientation of a light to the subject; training a convolutional neural network using the group of training samples, wherein training the convolutional neural network comprises: for each training iteration in a series of training iterations and for each training sample in the group of training samples: generating an output predicted image, wherein the output predicted image is a representation of …