Discovery of semantic similarities between images and text
US9836671B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Aug 28, 2015 |
| Grant date | Dec 5, 2017 |
| Priority date | — |
| Expiry date | Sep 29, 2035 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06V30/19173
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Disclosed herein are technologies directed to discovering semantic similarities between images and text, which can include performing image search using a textual query, performing text search using an image as a query, and/or generating captions for images using a caption generator. A semantic similarity framework can include a caption generator and can be based on a deep multimodal similar model. The deep multimodal similarity model can receive sentences and determine the relevancy of the sentences based on similarity of text vectors generated for one or more sentences to an image vector generated for an image. The text vectors and the image vector can be mapped in a semantic space, and their relevance can be determined based at least in part on the mapping. The sentence associated with the text vector determined to be the most relevant can be output as a caption for the image.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.