Method of and system for explainable knowledge-based visual question answering
US11599749B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Dec 22, 2020 |
| Grant date | Mar 7, 2023 |
| Priority date | — |
| Expiry date | Apr 25, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N5/045
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method and a system for generating an augmented scene graph of an image and for training an explainable knowledge based (KB) visual question answering (VQA) machine learning (ML) model are provided. A scene graph encoding spatial and semantic features of objects and relations between objects in the image is obtained. An augmented scene graph is generated by embedding a knowledge graph to enhance the scene graph. An embedded set of questions and associated answers related to the image are obtained. The KB VQA ML model is trained to provide an answer to a given question related to the image based on the augmented scene graph and the embedded set of questions and associated answers. The KB VQA ML model is trained to retrieve a subgraph linking the question and the associated answer as a potential explanation for the answer.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.