Systems and methods for performing knowledge distillation
US11790264B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 19, 2019 |
| Grant date | Oct 17, 2023 |
| Priority date | — |
| Expiry date | Jul 30, 2040 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/082
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
The present disclosure is directed to methods and systems for knowledge distillation. Implementations of the disclosure can include executing the following actions using one or more computing devices: obtaining an initial training dataset including multiple training examples; determining sets of outputs by performing inference on the training examples with a group of pre-trained machine-learned models that have been trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a performance of each pretrained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using the distillation training dataset.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.