Data pipeline branching

US10922291B2 · kind B2 · utility

Assignee

• Palantir Technologies Inc. · US

Inventors

• Vipul Shekhawat · Brooklyn, US
• Eliot Ball · London, GB
• Mikhail Proniushkin · Palo Alto, US
• Meghan Nayan · New York, US
• Mihir Rege · London, GB

Key dates

• Filing date: Dec 21, 2018
• Grant date: Feb 16, 2021
• Expiry date: Dec 21, 2038

Classification

• Technology area (CPC G): Physics
• CPC primary: G06F2201/84
• WIPO field: Computer technology
• WIPO sector: Electrical engineering

Abstract

A workbook management system provides a master branch of a data pipeline comprising a pointer(s) to a snapshot(s) of an initial dataset(s), a first logic, and a pointer(s) to a snapshot(s) of a first derived dataset(s) resulting from applying the first logic to the initial dataset(s). Responsive to user input requesting a test branch corresponding to the master branch, the system creates the test branch comprising the pointer(s) to the snapshot(s) of the initial dataset(s) and a copy of the first logic. The system receives a request to modify the test branch comprising at least one change to the copy of the first logic, and modifies the test branch independently of the master branch to include second logic reflecting the at least one change to the copy of the first logic, the pointer(s) to the snapshot(s) of the initial dataset(s), and a pointer(s) to snapshot(s) of a second derived dataset(s) resulting from applying the second logic to the initial dataset(s). Responsive to user input requesting a merge of the modified test branch into the master branch, the system updates the master branch to replace the first logic with the second logic and to replace the pointer(s) to the snapsh…