Multi-hop transformer for spatio-temporal reasoning and localization
US11741712B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Sep 1, 2021 |
| Grant date | Aug 29, 2023 |
| Priority date | — |
| Expiry date | Feb 25, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06V2201/07
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method for using a multi-hop reasoning framework to perform multi-step compositional long-term reasoning is presented. The method includes extracting feature maps and frame-level representations from a video stream by using a convolutional neural network (CNN), performing object representation learning and detection, linking objects through time via tracking to generate object tracks and image feature tracks, feeding the object tracks and the image feature tracks to a multi-hop transformer that hops over frames in the video stream while concurrently attending to one or more of the objects in the video stream until the multi-hop transformer arrives at a correct answer, and employing video representation learning and recognition from the objects and image context to locate a target object within the video stream.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.