Efficient parallel training of a network model on multiple graphics processing units
US10949746B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Feb 3, 2017 |
| Grant date | Mar 16, 2021 |
| Priority date | — |
| Expiry date | Sep 21, 2039 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/09
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A system and method provides efficient parallel training of a neural network model on multiple graphics processing units. A training module reduces the time and communication overhead of gradient accumulation and parameter updating of the network model in a neural network by overlapping processes in an advantageous way. In a described embodiment, a training module overlaps backpropagation, gradient transfer and accumulation in a Synchronous Stochastic Gradient Decent algorithm on a convolution neural network. The training module collects gradients of multiple layers during backpropagation of training from a plurality of graphics processing units (GPUs), accumulates the gradients on at least one processor and then delivers the gradients of the layers to the plurality of GPUs during the backpropagation of the training. The whole model parameters can then be updated on the GPUs after receipt of the gradient of the last layer.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.