System and method for classification with effective use of manual data input and crowdsourcing
US9195910B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Apr 23, 2013 |
| Grant date | Nov 24, 2015 |
| Priority date | — |
| Expiry date | Jan 20, 2034 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/9537
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods are disclosed herein for classifying records, such as product records, using a machine learning algorithm. After training a classification model according to a machine learning algorithm using an initial training set, records are classified and high confidence classifications identified. Remaining classifications are submitted to a crowdsourcing forum that validates or invalidates the classifications or marks them as to unclear to evaluate. Invalidated classifications are automatically analyzed to identify one or both of classification values and categories having a high proportion of invalidated classifications. Requests are transmitted to analysts to generate training data that is added to the training set. The process of classifying records and obtaining crowdsourced validation thereof may then repeat. High confidence classifications may be identified using an accuracy model trained to relate an accuracy percentage to a confidence score output by the classification model.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.