Generating digital annotations for evaluating and training automatic electronic document annotation models
US11232255B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 13, 2018 |
| Grant date | Jan 25, 2022 |
| Priority date | — |
| Expiry date | Aug 8, 2038 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/09
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems, methods, and non-transitory computer-readable media are disclosed that collect and analyze annotation performance data to generate digital annotations for evaluating and training automatic electronic document annotation models. In particular, in one or more embodiments, the disclosed systems provide electronic documents to annotators based on annotator topic preferences. The disclosed systems then identify digital annotations and annotation performance data such as a time period spent by an annotator in generating digital annotations and annotator responses to digital annotation questions. Furthermore, in one or more embodiments, the disclosed systems utilize the identified digital annotations and the annotation performance data to generate a final set of reliable digital annotations. Additionally, in one or more embodiments, the disclosed systems provide the final set of digital annotations for utilization in training a machine learning model to generate annotations for electronic documents.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.