System and method of data cleansing for improved data classification
US10489440B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Mar 17, 2017 |
| Grant date | Nov 26, 2019 |
| Priority date | — |
| Expiry date | Dec 2, 2037 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/334
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A system and method for data cleansing are disclosed. The method includes receiving by a data classifier, a data corpus comprising one or more words. The method further includes grouping the one or more words into a plurality of pre-defined categories and determining an overlap ratio between each pair of the plurality of pre-defined categories based on a comparison between each of the plurality of pre-defined categories. A normalized overlap ratio is computed for each of the plurality of pre-defined categories based on a combination of each overlap ratio. At least one of the plurality of pre-defined categories is determined based on a comparison between each of the normalized overlap ratio and an ideal overlap ratio. Finally, one or more common words from the selected at least one of the plurality of pre-defined categories is removed based on a frequency of occurrence of the one or more common words.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.