Simple Parallel Exec, version #3 ( #2 < #4 > ... #24)

Created by georg. Last edited by georg, 4 years and one day ago.. Viewed 610 times. #3

Goal

Provide a very easy way to run one program with different commandline parameters on a bunch of computers in parallel and collect the results. There should be a simple observation that determines if the machines are still available.

Why not use something like MPI?

Well, first of all the program I want to run are not witten in C.
I could write a MPI client that executes my program with the parameters, but then I can just use a shell script.
I need a more flexibel way to grasp the results. For example some output goes to a file. A shell or perl script seams to be more convinient to do a flexible conversion.

Specification

Cluster

Linux machines
NFS
SSH (public key authentication)
permanent network connection

Features & Configurability

one master computer
list of slave computers (names or IPs)
commandline pattern with space holders for variables
validation (return value and/or existance of a file)
list with all parameter settings, which should be processed
simple task assignment: if a computer is ready with one task give him the next from the list
error detection (see next section)
in case of an error while executing one task give it to the next machine and mark that machine as dead
collecting rules: a) stdout processing (plain concat ; blockwise with parameters) b) file processing (plain concat, with parameters)

Error detection

The folling errors need to be detected:

error while connection/authentication
machine dead (for whatever reason)
connection broken
programm terminated without success

Implementation Details

Configuation and Files

Cluster file: list of computers, one per line
Parameter file: csv file, cells seperated with |, parameter names in the headline and every following line contains one parameter set. All lines have to have the same amount of cells like the headline!
Commandline pattern: a perl syntax String with parameter Variables for the parameters
Validation function: gets the parameters set and the result of the programm and returns success or not
Collection function: gets the parameters set and the result of the programm and can do whatever with it (usually print it in a file)

Error detection

Remote shell command (ssh) termination code:

0 => Success: The program has been executed with success!
otherswise => Failure: Can have the following reasons: connection failed, programm not found or terminated without success.

To check the connection and authentication run:

ssh host echo

return 0 (success): Connection is OK and machine lives. The program was either not found or it didn't terminate with 0. Either case we assume that this parameter set is somehow bad and skip it.
otherwise mark this machine as dead and reschedule the parameter set.

One case that is not covered by the above procedure is if the connection breaks. Then the ssh command is just not terminating.

Solution: asyncron periodic ping