Simple Parallel Exec, version #4 ( #3 < #5 > ... #24)

Goal

Why not use something like MPI?

• Well, first of all the program I want to run are not witten in C.
• I could write a MPI client that executes my program with the parameters, but then I can just use a shell script.
• I need a more flexibel way to grasp the results. For example some output goes to a file. A shell or perl script seams to be more convinient to do a flexible conversion.

Specification

Cluster

• Linux machines
• SSH (public key authentication)
• (SCP sequrity copy, optional)
• HTTP

Features

• one master computer with server program(Is a HTTP server). SSH and SCP is needed to get the client program to the slave and start it.
• list of slave computers (names or IPs). Every slave acts as an HTTP client.
• command specification: commandline pattern with space holders for variables and input file generation
• result specification: standart output and/or files
• validation of the results
• list with task characterised through parameters.
• timeouts and multiply task assigments if necessary (timeout, free resources and so on)
• collecting rules: a) plain concat b) blockwise with parameters
• simple statistiks: which slave did what and which parameter sets failed.

Error detection

• error while connection/authentication
• machine dead (for whatever reason)
• programm terminated without success
• programm doesn't return within timeout

Server

• initialisation: for every slave: try to start client (ssh).

• on http request for executeable: reply with the binary
• on http request for new task: reply with the next command to execute and all parameters.
• on post: validate result and mark set and collect results
• no more parameter set to process: exit and display statistik.

Client

• gets via command line: Session ID, command name (path and name of the executeable), md5 checksum of the executeable, server name and port
• check for the executeable: if not there or the MD5 checksum is wrong: fetch it from the server
• fetch a task
• run program
• check return code: if failed -> Post failture otherwise take the results and post them.
• fetch next task.
• die is there is no more task or the server is not responding.
• different settings for termination: delete executeable (if fetched), delete the client program?

Implementation Details

Configuation and Files

• Cluster file: list of computers, one per line
• Parameter file: csv file, cells seperated with |, parameter names in the headline and every following line contains one parameter set. All lines have to have the same amount of cells like the headline!
• Commandline specification: Command line: a perl syntax String with parameter Variables for the parameters; Input files: Name of the file and and parameter name to write in.
• Result specification: A result consists of a list of name and value pairs. Where name specifies the name of the particular output and value decides where the output comes from. For example output="stdout", outputfile="out.txt".
• Validation function: function template that gets the parameter set and the result of the programm and returns success or not
• Collection function: gets the parameters set and the result of the programm and can do whatever with it (usually print it in a file).

Error detection

• 0 => Success: The program has been executed with success!
• otherswise => Failure: Can have the following reasons: connection failed, programm not found or terminated without success.

ssh host echo

• return 0 (success): Connection is OK and machine lives.
• otherwise error