Transformer-based encoding incorporating metadata
US11893346B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | May 5, 2021 |
| Grant date | Feb 6, 2024 |
| Priority date | — |
| Expiry date | Sep 30, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06V30/194
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
From metadata of a corpus of natural language text documents, a relativity matrix is constructed, a row-column intersection in the relativity matrix corresponding to a relationship between two instances of a type of metadata. An encoder model is trained, generating a trained encoder model, to compute an embedding corresponding to a token of a natural language text document within the corpus and the relativity matrix, the encoder model comprising a first encoder layer, the first encoder layer comprising a token embedding portion, a relativity embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, the training comprising adjusting a set of parameters of the encoder model.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.