Training teacher machine learning models using lossless and lossy branches
US11907845B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Aug 17, 2020 |
| Grant date | Feb 20, 2024 |
| Priority date | — |
| Expiry date | Oct 27, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG10L15/16
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Some embodiments of the present invention are directed to techniques for training teacher neural networks (TNNs) and student neural networks (SNNs). A training data set is received with a lossless set of data and a corresponding lossy set of data. Two branches of a TNN are established, with one branch trained using the lossless data (a lossless branch) and one trained using the lossy data (a lossy branch). Weights for the two branches are tied together. The lossy branch, now isolated from the lossless branch, generates a set of soft targets for initializing an SNN. These generated soft targets benefit from the training of lossless branch through the weights that were tied together between each branch, despite isolating the lossless branch from the lossy branch during soft-target generation.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.