Generating synchronized sound from videos
US11039043B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Jan 16, 2020 |
| Grant date | Jun 15, 2021 |
| Priority date | — |
| Expiry date | Jan 16, 2040 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG11B27/10
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Embodiments herein describe an audio forwarding regularizer and an information bottleneck that are used when training a machine learning (ML) system. The audio forwarding regularizer receives audio training data and identifies visually irrelevant and relevant sounds in the training data. By controlling the information bottleneck, the audio forwarding regularizer forwards data to a generator that is primarily related to the visually irrelevant sounds, while filtering out the visually relevant sounds. The generator also receives data regarding visual objects from a visual encoder derived from visual training data. Thus, when being trained, the generator receives data regarding the visual objects and data regarding the visually irrelevant sounds (but little to no data regarding the visually relevant sounds). Thus, during an execution stage, the generator can generate sounds that are relevant to the visual objects while not adding visually irrelevant sounds to the videos.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.