Systems and methods for the determining annotator performance in the distributed annotation of source data
US9355360B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 19, 2013 |
| Grant date | May 31, 2016 |
| Priority date | — |
| Expiry date | Dec 18, 2034 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N20/00
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods for determining annotator performance in the distributed annotation of source data in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for clustering annotators includes obtaining a set of source data, determining a training data set representative of the set of source data, obtaining sets of annotations from a set of annotators for a portion of the training data set, for each annotator determining annotator recall metadata based on the set of annotations provided by the annotator for the training data set and determining annotator precision metadata based on the set of annotations provided by the annotator for the training data set, and grouping the annotators into annotator groups based on the annotator recall metadata and the annotator precision metadata.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.