Augmenting attentioned-based neural networks to selectively attend to past inputs
US11829884B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Sep 25, 2020 |
| Grant date | Nov 28, 2023 |
| Priority date | — |
| Expiry date | Nov 22, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/008
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.