Use of semantic confidence metrics for uncertainty estimation in large language models
US11922126B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Jul 28, 2023 |
| Grant date | Mar 5, 2024 |
| Priority date | — |
| Expiry date | Jul 28, 2043 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F40/30
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method including receiving a user input for input to a language processing machine learning model (MLM). The method also includes generating modified inputs that are based on, and semantically related to, the user input. The method also includes executing the MLM to generate model outputs of the MLM. The MLM takes as input instances of each of the modified inputs. The method also includes sampling the model outputs using a statistical sampling strategy to generate sampled model outputs. The method also includes clustering the sampled model outputs into clusters. Each cluster of the clusters represents a distinct semantic meaning of the sampled model outputs. The method also includes generating a confidence metric for the user input. The confidence metric includes a predictive entropy of the clusters. The method also includes routing the user input based on whether the confidence metric satisfies or fails to satisfy a threshold value.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.