Systems and methods for video paragraph captioning using hierarchical recurrent neural networks
US10395118B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 15, 2016 |
| Grant date | Aug 27, 2019 |
| Priority date | — |
| Expiry date | Apr 27, 2038 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06V10/82
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Described herein are systems and methods that exploit hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem; that is, generating one or multiple sentences to describe a realistic video. Embodiments of the hierarchical framework comprise a sentence generator and a paragraph generator. In embodiments, the sentence generator produces one simple short sentence that describes a specific short video interval. In embodiments, it exploits both temporal- and spatial-attention mechanisms to selectively focus on visual elements during generation. In embodiments, the paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.