Matching mouth shape and movement in digital video to alternative audio
US11436780B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Nov 23, 2020 |
| Grant date | Sep 6, 2022 |
| Priority date | — |
| Expiry date | Nov 23, 2040 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG10L2021/105
- WIPO fieldAudio-visual technology
- WIPO sectorElectrical engineering
Abstract
A method for matching mouth shape and movement in digital video to alternative audio includes deriving a sequence of facial poses including mouth shapes for an actor from a source digital video. Each pose in the sequence of facial poses corresponds to a middle position of each audio sample. The method further includes generating an animated face mesh based on the sequence of facial poses and the source digital video, transferring tracked expressions from the animated face mesh or the target video to the source video, and generating a rough output video that includes transfers of the tracked expressions. The method further includes generating a finished video at least in part by refining the rough video using a parametric autoencoder trained on mouth shapes in the animated face mesh or the target video. One or more computers may perform the operations of the method.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.