Self-attention-based confidence estimation of language models
US12124814B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Apr 14, 2022 |
| Grant date | Oct 22, 2024 |
| Priority date | — |
| Expiry date | Nov 7, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/042
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A confidence estimation system includes: a neural network including at least one an attention module including N heads configured to: generate attention matrices based on interactions between tokens for words in an input sequence of words, the input sequence of words including a word that is obscured; and determine the word that is obscured in the input sequence; and a confidence module configured to determine a confidence value indicative of a probability of the neural network correctly determining the word that is obscured, the confidence module determining the confidence value of the word that is obscured using a convolutional neural network that projects the attention matrices generated by the attention module over a multi-dimensional space, the attention matrices recording interactions between the tokens in the input sequence of words without information regarding the tokens for the words and the word that is obscured.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.