Tokenization platform
US9195738B2 · kind B2 · utility
Assignee
Inventor
Key dates
| Filing date | Aug 13, 2012 |
| Grant date | Nov 24, 2015 |
| Priority date | — |
| Expiry date | Aug 13, 2032 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F40/284
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A tokenization platform and method is described for accurately tokenizing character strings, including but not limited to non-delimited character strings of the type commonly used in Internet domain names and computer filenames, to accurately identify words and phrases occurring therein. In one embodiment, a phased tokenization approach is used in which the final phase is a lexical analysis-based tokenization using a dictionary. The dictionary may be advantageously created and updated based upon one or more query logs associated with respective information retrieval systems, thereby ensuring that the dictionary accurately reflects currently-used terminology and captures alternative spellings and presentations of words and phrases submitted by users.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.