Workflow for evaluating quality of artificial intelligence (AI) services using held-out data
US11429816B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Sep 6, 2018 |
| Grant date | Aug 30, 2022 |
| Priority date | — |
| Expiry date | May 6, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F21/60
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
One embodiment provides for a method for evaluation of an artificial intelligence (AI) service, the method includes partitioning, by a processor, data into in-domain data and out-of-domain data. The processor defines held-out data from the in-domain data and the out-of-domain data for evaluation by domain and sub-domain based on building a taxonomy of domains and sub-domains for the AI service. The processor further determines distribution underlying performance metrics for the held-out data using statistical processing. The processor also determines performance guarantees for multiple settings conditioned on multiple characteristics of an application scenario for the held-out data of the taxonomy based on the underlying performance metrics. The processor further provides confidence intervals based on the performance guarantees.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.