Sequence-to-sequence convolutional architecture
US10839790B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Dec 20, 2017 |
| Grant date | Nov 17, 2020 |
| Priority date | — |
| Expiry date | Jan 30, 2038 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/044
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Exemplary embodiments relate to improvements to neural networks for translation and other sequence-to-sequence tasks. A convolutional neural network may include multiple blocks, each having a convolution layer and gated linear units; gating may determine what information passes through to the next block level. Residual connections, which add the input of a block back to its output, may be applied around each block. Further, an attention may be applied to determine which word is most relevant to translate next. By applying repeated passes of the attention to multiple layers of the decoder, the decoder is able to work on the entire structure of a sentence at once (with no temporal dependency). In addition to better accuracy, this configuration is better at capturing long-range dependencies, better models the hierarchical syntax structure of a sentence, and is highly parallelizable and thus faster to run on hardware.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.