Method, system, and computer program product for dynamically assigning an inference request to a CPU or GPU
US11714681B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jan 23, 2020 |
| Grant date | Aug 1, 2023 |
| Priority date | — |
| Expiry date | Oct 2, 2041 |
Classification
- Technology area (CPC Y)Emerging Cross-Sectional Technologies
- CPC primaryY02D10/00
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method for dynamically assigning an inference request is disclosed. A method for dynamically assigning an inference request may include determining at least one model to process an inference request on a plurality of computing platforms, the plurality of computing platforms including at least one Central Processing Unit (CPU) and at least one Graphics Processing Unit (GPU), obtaining, with at least one processor, profile information of the at least one model, the profile information including measured characteristics of the at least one model, dynamically determining a selected computing platform from between the at least one CPU and the at least one GPU for responding to the inference request based on an optimized objective associated with a status of the computing platform and the profile information, and routing, with at least one processor, the inference request to the selected computing platform. A system and computer program product are also disclosed.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.