Stochastic gradient boosting for deep neural networks
US10990878B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Mar 5, 2019 |
| Grant date | Apr 27, 2021 |
| Priority date | — |
| Expiry date | Oct 31, 2039 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N20/00
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Aspects described herein may allow for the application of stochastic gradient boosting techniques to the training of deep neural networks by disallowing gradient back propagation from examples that are correctly classified by the neural network model while still keeping correctly classified examples in the gradient averaging. Removing the gradient contribution from correctly classified examples may regularize the deep neural network and prevent the model from overfitting. Further aspects described herein may provide for scheduled boosting during the training of the deep neural network model conditioned on a mini-batch accuracy and/or a number of training iterations. The model training process may start un-boosted, using maximum likelihood objectives or another first loss function. Once a threshold mini-batch accuracy and/or number of iterations are reached, the model training process may begin using boosting by disallowing gradient back propagation from correctly classified examples while continue to average over all mini-batch examples.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.