Moving window data deduplication in distributed storage
US11762821B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jul 29, 2022 |
| Grant date | Sep 19, 2023 |
| Priority date | — |
| Expiry date | Jul 29, 2042 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F3/0608
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
The present disclosure describes a service which provides primary in-line deduplication. A streaming application program interface (API) may allow for streaming records into a storage system with high throughput and low latency. As part of this process, the API allows user to add identifiers as a field used for data deduplication. The deduplication service keeps a moving window of the identifiers in memory and does in-line deduplication by quickly determining whether data is a duplicate. Keeping only deduplication keys in memory reduces the cost of running the service. Moreover, the real-time nature of the moving window approach allows for storing deduplication information alongside the data and accessing it immediately on read. In this regard, read after write consistency is supported, and costs are reduced.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.