Atomic incremental load for map-reduce systems on append-only file systems
US10558615B2 · kind B2 · utility
Assignee
Inventor
Key dates
| Filing date | Jun 30, 2016 |
| Grant date | Feb 11, 2020 |
| Priority date | — |
| Expiry date | Apr 22, 2038 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/1873
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Augmenting data files in a repository of an append-only file system includes maintaining a companion metadata file for each corresponding data file in a map-reduce system using the append-only file system. Each companion metadata file tracks a logical end-of-file (EOF) for each data file. Global versioning of each companion metadata is maintained. A map-reduce append job is performed for a set of data files using a current global version number for the companion metadata file. The map-reduce job including multiple append tasks. For each successful append job, a logical EOF for each appended file is incremented to a new physical EOF. For each failed append task of the append job, a logical EOF is maintained for each failed append task by not incrementing the logical EOF for each failed append task.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.