Patent · US Active

Exploiting sparseness in training deep neural networks

US8700552B2 · kind B2 · utility

34Cited by

5References

20Claims

0Family size

Assignee

Microsoft Corporation · US

Inventors

Dong-Sheng Yu · Ottawa, CA
Li Deng · Redmond, US
Frank Torsten Bernd Seide · Aachen, DE
Gang Li · Beijing, CN

Key dates

Filing date	Nov 28, 2011
Grant date	Apr 15, 2014
Priority date	—
Expiry date	Aug 7, 2032

Classification

Technology area (CPC G)Physics
CPC primaryG06N3/09
WIPO fieldComputer technology
WIPO sectorElectrical engineering

Abstract

Deep Neural Network (DNN) training technique embodiments are presented that train a DNN while exploiting the sparseness of non-zero hidden layer interconnection weight values. Generally, a fully connected DNN is initially trained by sweeping through a full training set a number of times. Then, for the most part, only the interconnections whose weight magnitudes exceed a minimum weight threshold are considered in further training. This minimum weight threshold can be established as a value that results in only a prescribed maximum number of interconnections being considered when setting interconnection weight values via an error back-propagation procedure during the training. It is noted that the continued DNN training tends to converge much faster than the initial training.

Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.