README for Python code for distributed computing
Joshua Stough
Washington and Lee University
2011

 License: This program is free software; you can redistribute it and/or
 modify it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 3 of the License, or (at your
 option) any later version. This program is distributed in the hope that it
 will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
 of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
 Public License for more details.

Introduction: 

Much distributed computing software is in the end, under many layers, a simple 
"ssh somewhere doSomething". This code shows that fact in as few layers as 
practicable. Uses Python's multiprocessing and subprocess modules to maintain 
a job queue and manage jobs on remote hosts.  At the top level, you generate a 
list of command lines and simply request they be executed in parallel.  These 
commands are to be as they will be seen on the unix command prompt.  For 
example, if you would write "ssh computer.at.your.domain date", then the
command line is the string 'date'. Works in Python 2.6 and 3. 

For this code to work, you must set up the ability to ssh without a 
password to the hosts to which you will distribute jobs. Seen around 
the web, that process is, from the command line:

  % cd ~
  % ssh-keygen -trsa 
  (hit return to use defaults; including no passphrase)
  % cd .ssh
  % cp id_rsa.pub authorized_keys


Notes:
    Some extraneous print statements you might want to remove, but that 
    were informative for me.

    You must change the computer_list data member of the JobDistributor
    class to reflect the collection of machines you will use.

    See the Python documentation on the multiprocessing and subprocess
    modules:
    -http://docs.python.org/library/multiprocessing.html#module-multiprocessing
    -http://docs.python.org/library/subprocess.html#module-subprocess


Files Included:

testSubmitMaster.py: The top level code, literally consisting only of the 
generation of commands (as a list of strings), and the request that they
be executed in parallel. Upon execution (via 'python testSubmitMaster.py'),
a number of outputX.dat files store the output generated by the jobs, in
the execution folder.


testScript: A unix shell script used in the commands generated by 
testSubmitMaster.  The script is minimal, resolving hostname, time, and 
accepting a process description string, and echoing this information 
along with a disk usage of the your ~ (home) directory.


submitMaster.py: Contains two functions, processCommandsInParallel and 
submitMaster.  processCommandsInParallel is called by the top level
and 1.) instantiates the submitMaster process, 2.) sends all the commands
(jobs) to the submitMaster, and 3.) waits for the submitMaster to finish.
submitMaster accepts a Pipe connection from the first function, and 
instantiates the JobDistributor and a queue. Then in a repeating loop:

    -checks the pipe for a command
        if the command is the doneyet request, then return if we're done
        else enqueue the command, as it is to be executed by a host
    -distribute as many jobs as possible, as long as there are jobs on
        the queue and the JobDistributor can accept more.
    -if there is nothing on the pipe, then wait a few seconds

The submitMaster is run as its own Process so that it is ensured CPU time.
See the Python documentation on the multiprocessing module.


jobDistributor.py: A class that maintains a list of available machines, and 
distributes jobs to them. One JobDistributor object instance is allowed,
and data members are static. The class uses subprocess to do this, 
instantiating a process (Popen) to execute the ssh command.  Process 
information is then stored in a Job object, which can in turn be queried.
The JobDistributor object maintains a dictionary that associates 
hostname with a list of Job objects, representing those jobs (and 
subprocess.Popen() calls) that are currently being executed by the host.

This class should be instantiated by an independent thread, which
periodically checks for the completion of jobs and maintains a job
queue.  See submitMaster and testSubmitMaster for guidance. When
presented with a job to distribute when all hosts are busy to their
capacity, this code will hang, waiting for a spot to open up.  It's
someone else's responsibility to keep the original caller informed
about the submissions made--again, see submitMaster.


listQueue.py: Efficient list-based Queue class, as seen in CS2/Data 
Structures. Provides ammortized constant-time enqueue and dequeue.

 






