Semi-structured content aware bi-directional transformer
US11790885B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | May 6, 2021 |
| Grant date | Oct 17, 2023 |
| Priority date | — |
| Expiry date | Oct 16, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F40/279
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method, computer system, and a computer program product for natural language processing are provided. A first text corpus that includes semi-structured content that includes hierarchical nodes may be received. Some of the hierarchical nodes may be masked. Node embeddings and level embeddings may be generated from the semi-structured content of the first text corpus and from the masked hierarchical nodes. The node embeddings and the level embeddings may be included in a bi-directional transformer model. The bi-directional transformer model may be trained on the first text corpus by reducing loss from the bi-directional transformer model predicting the masked hierarchical nodes.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.