Use of semantic confidence metrics for uncertainty estimation in large language models
US12153892B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Jan 29, 2024 |
| Grant date | Nov 26, 2024 |
| Priority date | — |
| Expiry date | Jan 29, 2044 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F40/30
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method includes receiving a user input including natural language text. The method also includes generating modified inputs from the user input. The method also includes executing a machine learning model on the modified inputs to generate model outputs. The method also includes sampling the model outputs using a statistical sampling strategy to generate sampled model outputs. The method also includes clustering the sampled model outputs into clusters. The method also includes generating a confidence metric of the clusters. The method also includes routing, automatically in a computing system, the user input based on whether the confidence metric satisfies a threshold value.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.