Managing extract, transform and load systems
US11841871B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jun 29, 2021 |
| Grant date | Dec 12, 2023 |
| Priority date | — |
| Expiry date | Sep 8, 2041 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F2201/81
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
An approach to implement an extract, transform and load system, a queue is provided for holding units of data between extraction and transformation. When units of data suffer a rate limit error in the load phase, they are requeued so they can be resubmitted for transformation. The contents of the queue are monitored and, if too many requeued units of data are detected in the queue, then this is taken as an indicator of an unacceptable number of rate limit errors and active pacing management is triggered. A retry schedule is defined for the requeued units of data. Extraction is temporarily halted to allow the requeued units of data to be retransformed without more units of data queuing up. Then, after the suspension is lifted, a pacing delay is inserted between subsequent extract events to avoid the same load phase bottleneck recurring.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.