Training a neural network using periodic sampling over model weights
US11922316B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Aug 13, 2020 |
| Grant date | Mar 5, 2024 |
| Priority date | — |
| Expiry date | Jul 16, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N5/046
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A computer-implemented method includes: initializing model parameters for training a neural network; performing a forward pass and backpropagation for a first minibatch of training data; determining a new weight value for each of a plurality of nodes of the neural network using a gradient descent of the first minibatch; for each determined new weight value, determining whether to update a running mean corresponding to a weight of a particular node; based on a determination to update the running mean, calculating a new mean weight value for the particular node using the determined new weight value; updating the weight parameters for all nodes based on the calculated new mean weight values corresponding to each node; assigning the running mean as the weight for the particular node when training on the first minibatch is completed; and reinitializing running means for all nodes at a start of training a second minibatch.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.