Efficient transformer language models with disentangled attention and multi-step decoding
US11526679B2 · kind B2 · utility
0Cited by
0References
20Claims
0Family size
Assignee
Inventors
Key dates
| Filing date | Jun 24, 2020 |
| Grant date | Dec 13, 2022 |
| Priority date | — |
| Expiry date | Dec 24, 2040 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N20/00
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.