Method and apparatus to cache key-value data in low-precision numerics for efficient generative transformer execution
US12182028B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Sep 28, 2023 |
| Grant date | Dec 31, 2024 |
| Priority date | — |
| Expiry date | Sep 28, 2043 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F2212/454
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A transformer compute apparatus and method of operation therefor. The apparatus receives matrix inputs in a first format and generates projection tokens from these inputs. Among others, the apparatus includes a first cache device configured for processing first projection tokens and a second cache device configured for processing second projection tokens. The first cache device stores the first projection tokens in a first cache region and stores these tokens converted to a second format in a second cache region. The second cache device stores the second projection tokens converted to the second format in a first cache region and stores the converted second projection tokens after being transposed. Then, a compute device performs various matrix computations with the converted first projection tokens and transposed second projection tokens. Re-processing data and expensive padding and de-padding operations for transposed storage and byte alignment can be avoided using this caching process.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.