Duplicate in-memory shared-intermediate data detection and reuse module in spark framework
US10311025B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Jan 11, 2017 |
| Grant date | Jun 4, 2019 |
| Priority date | — |
| Expiry date | Jan 11, 2037 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F2212/62
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
A cache management system for managing a plurality of intermediate data includes a processor, and a memory having stored thereon the plurality of intermediate data and instructions that when executed by the processor, cause the processor to perform identifying a new intermediate data to be accessed, loading the intermediate data from the memory in response to identifying the new intermediate data as one of the plurality of intermediate data, and in response to not identifying the new intermediate data as one of the plurality of intermediate data identifying a reusable intermediate data having a longest duplicate generating logic chain that is at least in part the same as a generating logic chain of the new intermediate data, and generating the new intermediate data from the reusable intermediate data and a portion of the generating logic chain of the new intermediate data not in common with the reusable intermediate data.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.