Quality assurance for labeled training data
US11803883B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jan 9, 2019 |
| Grant date | Oct 31, 2023 |
| Priority date | — |
| Expiry date | Feb 16, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06Q30/0282
- WIPO fieldIT methods for management
- WIPO sectorElectrical engineering
Abstract
Methods, systems, apparatus, and tangible non-transitory carrier media encoded with one or more computer programs for classifying an item. In accordance with particular embodiments, a labeling task is issued to workers participating in a crowdsourcing system. The labeling task includes evaluating an inferred classification that includes one or more of the class labels in a hierarchical classification taxonomy based at least in part on a description of the item and the class labels in the classification. Evaluation decisions are received from the crowdsourcing system. The classification is validated based on the evaluation decisions to obtain a validation result. The validating includes applying at least one consensus criterion to an aggregation of the received evaluation decisions. Data corresponding to one or more of the class labels in the classification is routed to respective destinations based on the validation result.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.