Rollback recovery with data lineage capture for data pipelines
US11556431B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Apr 21, 2021 |
| Grant date | Jan 17, 2023 |
| Priority date | — |
| Expiry date | Apr 21, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F2212/702
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Computer-readable media, methods, and systems are disclosed for performing rollback recovery with data lineage capture for data pipelines. A middle operator receives ingested input events from a source operator reading data from an external input data source. The middle operator then logs information regarding middle input events to a middle operator input log, designating the logged middle input event information as incomplete. The middle operator then processes data associated with the middle input events and updates the middle input log entries setting them to a completed logging status designation for middle input events that were consumed to produce the one or more middle output events. The middle operator then transmits the middle output events to subsequent operators. Garbage collection is performed to remove completed entries from the middle operator output log. Finally, based on receiving a recovering message from a subsequent operator, corresponding middle output events are re-sent.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.