Speculative training using partial gradients update
US11948352B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Mar 26, 2020 |
| Grant date | Apr 2, 2024 |
| Priority date | — |
| Expiry date | Nov 4, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/044
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
The exchange of weight gradients among the processing nodes can introduce a substantial bottleneck to the training process. Instead of remaining idle during the weight gradients exchange process, a processing node can update its own set of weights for the next iteration of the training process using the processing node's local weight gradients. The next iteration of training can be started by using these speculative weights until the weight gradients exchange process completes and a global weights update is available. If the speculative weights is close enough to the weight values from the global weights update, the training process at the processing node can continue training using the results computed from the speculative weights to reduce the overall training time.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.