Distributed synchronous training architecture using stale weights
US12430560B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Oct 5, 2021 |
| Grant date | Sep 30, 2025 |
| Priority date | — |
| Expiry date | Aug 1, 2044 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/098
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A computer-implemented method for distributed synchronous training of a neural network model includes performing, by a worker machine of a plurality of worker machines, a forward computation of a training data set using a plurality of N layers of the neural network model. The forward computation starts at Layer 1 and proceeds through Layer N of the neural network model. The method further includes performing, by the worker machine, a backward computation of the training data set, the backward computation starting at Layer N and proceeding through Layer 1 of the neural network model. The method further includes synchronizing, by the worker machine, a plurality of gradients outputted by the neural network model during the backward computation. The synchronizing of the plurality of gradients is performed with other worker machines of the plurality of worker machines and in parallel with the backward computation.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.