Dynamic data-ingestion pipeline
US10122783B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Nov 18, 2015 |
| Grant date | Nov 6, 2018 |
| Priority date | — |
| Expiry date | Oct 12, 2036 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/11
- WIPO fieldDigital communication
- WIPO sectorElectrical engineering
Abstract
In order to ingest data from an arbitrary source in a set of sources, a computer system accesses predefined configuration instructions. Then, the computer system generates a dynamic data-ingestion pipeline that is compatible with a Hadoop file system based on the predefined configuration instructions. This dynamic data-ingestion pipeline includes a modular arrangement of operators from a set of operators that includes: an extraction operator for extracting the data of interest from the source, a converter operator for transforming the data, and a quality-checker operator for checking the transformed data. Moreover, the computer system receives the data from the source. Next, the computer system processes the data using the dynamic data-ingestion pipeline as the data is received without storing the data in memory for the purpose of subsequent ingestion processing.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.