Evaluation of task judging results
US8868471B1 · kind B1 · utility
Assignee
Inventor
Key dates
| Filing date | Sep 21, 2011 |
| Grant date | Oct 21, 2014 |
| Priority date | — |
| Expiry date | Oct 23, 2032 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06Q10/101
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Embodiments of systems and methods can determine evaluations of the quality of task results provided by judges. Certain embodiments can determine the quality evaluations with relatively little overlap of judges (e.g., two or three judges). The quality evaluations may include determining an error rate or a confidence index for a judge or for a particular judgment made by the judge. Certain embodiments may determine the quality evaluations by intercomparing judging results from multiple judges without requiring human intervention, comparison to evaluation data known (or assumed) to be accurate, or input from an external judge review panel. In some implementations, task results can be combined with a confidence score (based at least in part on the quality evaluation of the judge who provided the task result), and this information can be used as training data to improve performance of machine learning algorithms.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.