Systems and methods for block-sparse recurrent neural networks
US11651223B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Oct 4, 2018 |
| Grant date | May 16, 2023 |
| Priority date | — |
| Expiry date | Nov 11, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N3/09
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Described herein are systems and methods to prune deep neural network models in reducing the overall memory and compute requirements of these models. It is demonstrated that using block pruning and group lasso combined with pruning during training, block-sparse recurrent neural networks (RNNs) may be built as accurate as dense baseline models. Two different approaches are disclosed to induce block sparsity in neural network models: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, it is demonstrated that block-sparse RNNs with high sparsity can be created with small loss in accuracy. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.