Meta-data driven data ingestion using MapReduce framework
US8949175B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | May 8, 2012 |
| Grant date | Feb 3, 2015 |
| Priority date | — |
| Expiry date | Jun 30, 2032 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F9/46
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A generic approach for automatically ingesting data into an HDFS (Hadoop File System) based data warehouse includes a datahub server, a generic pipelined data loading framework, and a meta-data model that, together, address data loading efficiency, data source heterogeneities, and data warehouse schema evolvement. The loading efficiency is achieved via the MapReduce scale-out solution. The meta-data model is comprised of configuration files and a catalog. The configuration file is setup per ingestion task. The catalog manages the data warehouse schema. When a scheduled data loading task is executed, the configuration files and the catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.