Systems and methods for the determining annotator performance in the distributed annotation of source data
US9898701B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | May 27, 2016 |
| Grant date | Feb 20, 2018 |
| Priority date | — |
| Expiry date | May 27, 2036 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N20/00
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods for determining annotator performance in the distributed annotation of source data in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for clustering annotators includes obtaining a set of source data, determining a training data set representative of the set of source data, obtaining sets of annotations from a set of annotators for a portion of the training data set, for each annotator determining annotator recall metadata based on the set of annotations provided by the annotator for the training data set and determining annotator precision metadata based on the set of annotations provided by the annotator for the training data set, and grouping the annotators into annotator groups based on the annotator recall metadata and the annotator precision metadata.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.