Augmenting neural networks with hierarchical external memory

US11010664B2 · kind B2 · utility

Assignee

• DeepMind Technologies Limited · GB

Inventors

• Karol Kurach · Zürich, CH
• Marcin Andrychowicz · London, GB

Key dates

• Filing date: Dec 30, 2016
• Grant date: May 18, 2021
• Expiry date: Feb 9, 2040

Classification

• Technology area (CPC G): Physics
• CPC primary: G06N3/063
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

Systems, methods, devices, and other techniques are disclosed for using an augmented neural network system to generate a sequence of outputs from a sequence of inputs. An augmented neural network system can include a controller neural network, a hierarchical external memory, and a memory access subsystem. The controller neural network receives a neural network input at each of a series of time steps processes the neural network input to generate a memory key for the time step. The external memory includes a set of memory nodes arranged as a binary tree. To provide an interface between the controller neural network and the external memory, the system includes a memory access subsystem that is configured to, for each of the series of time steps, perform one or more operations to generate a respective output for the time step. The capacity of the neural network system to account for long-range dependencies in input sequences may be extended. Also, memory access efficiency may be increased by structuring the external memory as a binary tree.