Scalable exactly-once data processing using transactional streaming writes
US11573876B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Oct 30, 2020 |
| Grant date | Feb 7, 2023 |
| Priority date | — |
| Expiry date | Jun 5, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F2201/84
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A method for processing data exactly once using transactional stream writes includes receiving, from a client, a batch of data blocks for storage on memory hardware in communication with the data processing hardware. The batch of data blocks is associated with a corresponding sequence number and represents a number of rows of a table stored on the memory hardware. The method also includes partitioning the batch of data blocks into a plurality of sub-batches of data blocks. For each sub-batch of data blocks, the method further includes assigning the sub-batch of data blocks to a buffered stream; writing, using the assigned buffered stream, the sub-batch of data blocks to the memory hardware; updating a storage log with an intent to commit the sub-batch of data blocks using the assigned buffered stream; and committing the sub-batch of data blocks to the memory hardware.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.