System and method for knowledge distillation between neural networks
US11636337B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Mar 20, 2020 |
| Grant date | Apr 25, 2023 |
| Priority date | — |
| Expiry date | May 17, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/096
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods for knowledge distillation provide supervised training of a student network with a teacher network, including inputting a batch to the teacher network, inputting the batch to the student network, generating a teacher activation map at a layer of the teacher network, generating a student activation map at a layer of the student network corresponding to the layer of the teacher network, generating a pairwise teacher similarity matrix based on the teacher activation map, generating a pairwise student similarity matrix based on the student activation map, and minimizing a knowledge distillation loss defined as a difference between the pairwise teacher similarity matrix and the pairwise student similarity matrix.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.