Memory optimized data shuffle
US10642520B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | Apr 18, 2017 |
| Grant date | May 5, 2020 |
| Priority date | — |
| Expiry date | Jan 6, 2038 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F3/0683
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
In a distributed data processing system with a set of multiple nodes, a first data shuffle memory pool is maintained at a data shuffle writer node, and a second data shuffle memory pool is maintained at a data shuffle reader node. The data shuffle writer node and the data shuffle reader node are part of the set of multiple nodes of the distributed data processing system. In-memory compression is performed on at least a portion of a data set from the first data shuffle memory pool. At least a portion of the compressed data is transmitted from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner. Each of the first data shuffle memory pool and the second data shuffle memory pool may include a hybrid memory configuration.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.