Home

Parallel Python Example @ 2008-11-14 22:05:40
Filed under: Code  Python  Tech 
Parallel Python is one cool library. Here is an example of running jobs on a 'pool' of job servers (in this case just one ... but still) print out the results as they become available.
#!/usr/bin/env python
"""
Example script that executes jobs via a pool of job servers and prints the
results when they become ready ... not in any predetermined order.
"""

import pp

from random import randint


def sleep(name, wait):
    """
    Example function to run as a job.
    """
    time.sleep(wait)
    return "%s (%s)" %(name, wait)


def main():
    """
    Main function.
    """
    # Pool of ppservers to use as a 'server'
    job_server = pp.Server(ppservers=('127.0.0.1', ))

    # Array of all submitted jobs
    jobs = []
    #                            Function  Params                 Libs/Modules 
    #                               v       v                          v 
    jobs.append(job_server.submit(sleep, ('f1', randint(3,15)), (), ('time',)))
    jobs.append(job_server.submit(sleep, ('f2', randint(3,15)), (), ('time',)))
    jobs.append(job_server.submit(sleep, ('f3', randint(3,15)), (), ('time',)))
    jobs.append(job_server.submit(sleep, ('f4', randint(3,15)), (), ('time',)))

    # Loop
    while True:
        jobs_len = len(jobs)
        # For each job ...
        for x in range(jobs_len):
            # See if it's finished
            if jobs[x].finished == True:
                # If so, print the result, delete the job and loop again
                print jobs[x]()
                del jobs[x]
                break
        # All out of jobs? Exit!
        if jobs_len == 0:
            raise SystemExit(0)


if __name__ == '__main__':
    main()

Here is output of a few runs ...
[steve@tachikoman ~]$ python ppexample.py
f4 (3)
f1 (4)
f3 (6)
f2 (11)
[steve@tachikoman ~]$ python ppexample.py
f4 (3)
f3 (3)
f1 (4)
f2 (9)
[steve@tachikoman ~]$ python ppexamples.py 
f1 (7)
f4 (8)
f2 (13)
f3 (14)

 digg it   seed it   del.icio.us   ma.gnolia
Tags:      


 
Ohloh profile for ashcrow View Steve Milner's profile on LinkedIn A Django joint.
© 2007-2008 Steve 'Ashcrow' Milner | Studio7designs | Arbutus Photography