Executing parallel jobs with message passing on compute clusters
US10148736B1 · kind B1 · utility
Assignee
Inventors
Key dates
| Filing date | May 19, 2014 |
| Grant date | Dec 4, 2018 |
| Priority date | — |
| Expiry date | Apr 5, 2036 |
Classification
- Technology area (CPC H)Electricity
- CPC primaryH04L67/34
- WIPO fieldDigital communication
- WIPO sectorElectrical engineering
Abstract
A client may submit a job to a service provider that processes a large data set and that employs a message passing interface (MPI) to coordinate the collective execution of the job on multiple compute nodes. The framework may create a MapReduce cluster (e.g., within a VPC) and may generate a single key pair for the cluster, which may be downloaded by nodes in the cluster and used to establish secure node-to-node communication channels for MPI messaging. A single node may be assigned as a mapper process and may launch the MPI job, which may fork its commands to other nodes in the cluster (e.g., nodes identified in a hostfile associated with the MPI job), according to the MPI interface. A rankfile may be used to synchronize the MPI job and another MPI process used to download portions of the data set to respective nodes in the cluster.
Source: USPTO / EPO open patent data. Objective bibliographic and citation counts.