Method and system for collecting data from data sources with commit lag to maintain data consistency in a data store
US11526491B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | May 4, 2022 |
| Grant date | Dec 13, 2022 |
| Priority date | — |
| Expiry date | May 4, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F16/2477
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A system performs a first query to retrieve a commit lag timestamp, where the commit lag timestamp specifies an earliest time instance when a record of an entity is inserted or updated, but is yet to be committed, to a data source. The system determines an inline crawl interval based on the commit lag timestamp. The system performs a second query based on the inline crawl interval to retrieve a number of record identifiers and/or modification dates. The system performs a third query based on the inline crawl interval, where the third query corresponds to records that exist in a data store. The system determines at least one identifier that is missing from the third query due to commit lag based on a difference between data corresponding to the second and third queries. The system persists data corresponding to the second query and the at least one missing identifier.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.