Gradient normalization systems and methods for adaptive loss balancing in deep multitask networks
US11537895B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Oct 24, 2018 |
| Grant date | Dec 27, 2022 |
| Priority date | — |
| Expiry date | Nov 16, 2040 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06N20/00
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods for training a multitask network is disclosed. In one aspect, training the multitask network includes determining a gradient norm of a single-task loss adjusted by a task weight for each task, with respect to network weights of the multitask network, and a relative training rate for the task based on the single-task loss for the task. Subsequently, a gradient loss function, comprising a difference between (1) the determined gradient norm for each task and (2) a corresponding target gradient norm, can be determined. An updated task weight for the task can be determined and used in the next iteration of training the multitask network, using a gradient of the gradient loss function with respect to the task weight for the task.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.