Associative long short-term memory neural network layers

US11010663B2 · kind B2 · utility

Assignee

• DeepMind Technologies Limited · GB

Inventors

• Ivo Danihelka · London, GB
• Nal Emmerich Kalchbrenner · London, GB
• Gregory Duncan Wayne · London, GB
• Benigno Uria-Martínez · London, GB
• Alexander Benjamin Graves · London, GB

Key dates

• Filing date: Dec 30, 2016
• Grant date: May 18, 2021
• Expiry date: Jun 11, 2039

Classification

• Technology area (CPC G): Physics
• CPC primary: G06N3/044
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

Systems, methods, and apparatus, including computer programs encoded on a computer storage medium, related to associative long short-term memory (LSTM) neural network layers configured to maintain N copies of an internal state for the associative LSTM layer, N being an integer greater than one. In one aspect, a system includes a recurrent neural network including an associative LSTM layer, wherein the associative LSTM layer is configured to, for each time step, receive a layer input, update each of the N copies of the internal state using the layer input for the time step and a layer output generated by the associative LSTM layer for a preceding time step, and generate a layer output for the time step using the N updated copies of the internal state.