Anticipatory warm-up of cluster resources for jobs processed on multiple cluster nodes
US9313133B2 · kind B2 · utility
Assignee
Inventors
Key dates
| Filing date | Feb 6, 2014 |
| Grant date | Apr 12, 2016 |
| Priority date | — |
| Expiry date | Feb 21, 2034 |
Classification
- Technology area (CPC G)Physics
- CPC primaryG06F2201/815
- WIPO fieldComputer technology
- WIPO sectorElectrical engineering
Abstract
Systems and methods are disclosed for reducing latency in processing data sets in a distributed fashion. A job-queue operable for queuing data-processing jobs run on multiple nodes in a cluster may be communicatively coupled to a job analyzer. The job analyzer may be operable to read the data-processing jobs and extract information characterizing those jobs in ways that facilitate identification of resources in the cluster serviceable to run the data-processing jobs and/or data to be processed during the running of those jobs. The job analyzer may also be coupled to a resource warmer operable to warm-up a portion of the cluster to be used to run a particular data-processing job prior to the running of the job. In some embodiments, mappers and/or reducers may be extracted from the jobs and converted into compute node identifiers and/or data units identifying blocks for processing, informing the warm-up operations of the resource warmer.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.