Adaptive speculative decoding
US10911063B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Oct 3, 2019 |
| Grant date | Feb 2, 2021 |
| Priority date | — |
| Expiry date | Oct 3, 2039 |
Classification
- Technology area (CPC H)Electricity
- CPC primaryH03M7/425
- WIPO fieldBasic communication processes
- WIPO sectorElectrical engineering
Abstract
Examples herein relate to decoding tokens using speculative decoding operations to decode tokens at an offset from a token decoded by a sequential decoding operation. At a checkpoint, a determination is made as to whether tokens to be decoded by the sequential and speculative decoding operations align. If there is alignment, the speculatively decoded tokens after a discard window are committed and made available for access. If there is not alignment, the speculatively decoded tokens are discarded. A miss in alignment and a fullness level of a buffer that stores speculatively decoded tokens are assessed to determine a next offset level for a start of speculative decoding. A size of a discard window can be set using a relationship based on the offset level to improve buffer utilization and to attempt to improve changes of alignments.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.