mmf.async.pp

Task(job, server, f, *args, **kwargs)
decorate_logger(logger)	Gives a write and flush method to the logger so that it can be used by pexpect.
ServerMMF([hosts, remote_timeout, async, ...])	Server object.
ManageRemoteServer(host, server, logger)	This class will spawn a remote server, and then start a ppserver process on it.
check_hosts([hosts, ps])	Log in to each host interactively to make sure that you
kill_hosts([hosts])

Inheritance diagram for mmf.async.pp:

Tools for working with Parallel Python.

Note

You must install Parallel Python separately for this to work:

wget http://www.parallelpython.com/downloads/pp/pp-1.6.0.tar.bz2 tar -jxvf pp-1.6.0.tar.bz2 cd pp-1.6.0 python setup.py install

Presently, you must be able to ssh to the hosts without a password (setup a key using ssh-keygen). The list of hosts to use can be specified manually, or though the pp_hosts variable in the ~/.mmfrc.py initialization file. If defined, this will be used to populate the _HOSTS list. It should contain a list of strings with the host names, or a list of tuples (hostname, n_cpu) where n_cpu is an integer specifying how many cores are on the host (None for autodetection) or tuples (hostname, n_cpu, working_dir) where working_dir is the directory to change to on the remote machine before launching the ppserver.

Todo

Only launch remote servers when needed.

Todo

Maybe use the pexpect module to facilitate passwords and logging in to the remote servers. Also, provide some diagnostics if the spawn fails. Also, could add a convenience method that would try to login to all hosts to make sure that it is possible first before running.

Todo

Monitor remote connections and respawn ppservers if any die.

class mmf.async.pp.Task(job, server, f, *args, **kwargs)

Bases: object

__init__(job, server, f, *args, **kwargs)

finished

result

class mmf.async.pp.decorate_logger(logger)

Bases: object

Gives a write and flush method to the logger so that it can be used by pexpect.

Methods

flush()	Flush method: does nothing
write(*args, **kwargs)	Write method for logging handler.

__init__(logger)

flush()

Flush method: does nothing

write(*args, **kwargs)

Write method for logging handler.

class mmf.async.pp.ServerMMF(hosts=(), remote_timeout=30, async=True, use_pp=True, nice=10, port=None, ssh_tunnel=True, archive_in=False, archive_out=False, working_dir='.', use_local=True)

Bases: object

Server object.

Methods

destroy()
finished_tasks()	Iterate over all tasks, blocking until they are
get_logger(host)
start_remote_servers(hosts)	Tries to start a remote server on each specified host
submit(f, *args, **kwargs)

__init__(hosts=(), remote_timeout=30, async=True, use_pp=True, nice=10, port=None, ssh_tunnel=True, archive_in=False, archive_out=False, working_dir='.', use_local=True)

destroy()

finished_tasks()

Iterate over all tasks, blocking until they are finished.

get_logger(host)

start_remote_servers(hosts)

Tries to start a remote server on each specified host using ssh.

submit(f, *args, **kwargs)

class mmf.async.pp.ManageRemoteServer(host, server, logger)

Bases: threading.Thread

This class will spawn a remote server, and then start a ppserver process on it. The output will be directed to the logger.

There are two options for starting the remote server:

1. ssh to the remote host and launch the ppserver.py and bind it to the specified port of that server. The is the standard way of setting up the server, but allows anyone to connect and run code. This can be mitigated somewhat by starting the server with a “secret” (sent through the SSH connection, so it is not compromised), but won’t work if the remote host is firewalled. Here is how this would go:

   here:math:` ssh host1 ppserver.py -s <secret> -t 30 -p 60000 &
   here` ssh host2 ppserver.py -s <secret> -t 30 -p 60000 &
   ...

   One would connect from here using the <secret> and connecting to host1:60000 and host2:60000 etc. (but so could anyone else who found out <secret>.)

2. Use an ssh tunnel to map a local port here to the local port on the server and then bind it to a local port. In this way, only local processes on the remote host can connect (and we can act as a local process by using the tunnel). Again, processes on the remote host can use the server, so we use the secret, but presumable these are fairly well trusted.

   In this case we will have to use a sequence of ports here: one for each remote host. Here is how this would go:

   here:math:` ssh -tL 60000:localhost:60000 host1 \
         ppserver.py -i localhost -s <secret> -t 30 -p 60000 &
   here` ssh -tL 60001:localhost:60000 host2 \
         ppserver.py -i localhost -s <secret> -t 30 -p 60000 &
   ...

   Passing localhost as the interface (-i) to the server binds it to port 60000 on that server and allows it to only accept local connections to that port. (I.e. if we try to connect to host1:60000 as before it will act as if there is nothing listening.) Now we can connect to localhost:60000 and localhost:60001. These connections will be tunnelled over the ssh connection and the remote computers will interpret these as local connections. Only process running on the remote machine, or those who have tunnelled in can use this.

Methods

getName()
isAlive()
isDaemon()
is_alive()
join([timeout])
run()	Run the remote server.
setDaemon(daemonic)
setName(name)
ssh_connect([max_tries])	Return (proc, local_host) after initiating an ssh connection to the specified host.
start()

__init__(host, server, logger)

run()

Run the remote server.

ssh_connect(max_tries=100)

Return (proc, local_host) after initiating an ssh connection to the specified host.

Parameters :

max_tries : int

Maximum number of local ports to try.

Returns :

proc : spawn

pexpect spawn class representing process (or None if not started.

local_host : string

Address the ppserver should use to connect to remote workers. This is either a remote address, or a local address that has be tunnelled to the remote host.

mmf.async.pp.check_hosts(hosts=None, ps=False)

Log in to each host interactively to make sure that you can.

mmf.async.pp.kill_hosts(hosts=None)