Slim embedding layers for recurrent neural language models
US11030997B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Nov 21, 2018 |
| Grant date | Jun 8, 2021 |
| Priority date | — |
| Expiry date | Mar 13, 2039 |
Classification
- Technology area (CPC H)Electricity
- CPC primaryH03M7/42
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Described herein are systems and methods for compressing or otherwise reducing the memory requirements for storing and computing the model parameters in recurrent neural language models. Embodiments include space compression methodologies that share the structured parameters at the input embedding layer, the output embedding layers, or both of a recurrent neural language model to significantly reduce the size of model parameters, but still compactly represent the original input and output embedding layers. Embodiments of the methodology are easy to implement and tune. Experiments on several data sets show that embodiments achieved similar perplexity and BLEU score results while only using a fraction of the parameters.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.