mmf.examples.pp_example

pp
logging
time
mmf
f(x, y, z)	Function to execute on the remote server.
launch([ppservers, process_jobs_in_order])
callback(x, y, z, result_archive)	Example of a callback: will be called after the job is
run(archive)	This function must be defined in an importable module so it can be pickled.
process_job(key, job)	Unpack and process the job after it is done.

Parallel Python Example

This example shows how to use Parallel Python to distribute tasks over several local processors and on remote machines.

Note

You must install Parallel Python separately for this to work:

wget http://www.parallelpython.com/downloads/pp/pp-1.6.0.tar.bz2 tar -jxvf pp-1.6.0.tar.bz2 cd pp-1.6.0 python setup.py install

We will use two modifications. First, we will make use of the mmf.archive module for serializing our objects as this provides additional flexibility, such as the ability to serialize functions and classes.

Note

If performance is a concern, you might like to try and make your code work with the standard pickling serialization as this allows you to use the server-side caching mechanism.

Second, we will launch the remote servers using an ssh tunnel and port forwarding for security.

mmf.examples.pp_example.f(x, y, z)

Function to execute on the remote server.

mmf.examples.pp_example.launch(ppservers=('intvis07.phys.washington.edu', 'intvis08.phys.washington.edu', 'intvis11.phys.washington.edu', 'intvis15.phys.washington.edu'), process_jobs_in_order=False)

mmf.examples.pp_example.callback(x, y, z, result_archive)

Example of a callback: will be called after the job is done.

mmf.examples.pp_example.run(archive)

This function must be defined in an importable module so it can be pickled. It does the packing and unpacking of the arguments and results.

Note that this function is executed in an isolated environment (the pp module stores the actual code and does not import the whole module), thus all symbols must be imported here or explicitly passed to the server. There are several options:

1. Explicitly import what is needed in the function.
2. Pass the required information explicitly to pp.Server.
3. Pass globals() to the pp.Server.

mmf.examples.pp_example.process_job(key, job)

Unpack and process the job after it is done. This will block until the job completes, so you might like to check job.finished first.