Harmonizing composite images utilizing a transformer neural network

US12165284B2 · kind B2 · utility

Assignee

• Adobe Inc. · US

Inventors

• He Zhang · San Jose, US
• Jianming Zhang · Campbell, US
• Jose Ignacio Echevarria Vallespi · San Jose, US
• Kalyan Sunkavalli · San Jose, US
• Meredith Payne Stotzner · San Jose, US
• Yinglan Ma · Mountain View, US
• Zhe Lin · Fremont, US
• Elya Shechtman · Seattle, US
• Frederick Mandia · San Jose, US

Key dates

• Filing date: Mar 21, 2022
• Grant date: Dec 10, 2024
• Expiry date: Feb 7, 2043

Classification

• Technology area (CPC G): Physics
• CPC primary: G06T2207/30168
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a dual-branched neural network architecture to harmonize composite images. For example, in one or more implementations, the transformer-based harmonization system uses a convolutional branch and a transformer branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. Utilizing a decoder, the transformer-based harmonization system combines the local information and the global information from the corresponding convolutional branch and transformer branch to generate a harmonized composite image.