Parallel Python Example
@ 2008-11-14 22:05:40
Filed under: Code Python Tech
Parallel Python is one cool library. Here is an example of running jobs on a 'pool' of job servers (in this case just one ... but still) print out the results as they become available.
Here is output of a few runs ...
digg it
seed it
del.icio.us
ma.gnolia
Filed under: Code Python Tech
Parallel Python is one cool library. Here is an example of running jobs on a 'pool' of job servers (in this case just one ... but still) print out the results as they become available.
#!/usr/bin/env python
"""
Example script that executes jobs via a pool of job servers and prints the
results when they become ready ... not in any predetermined order.
"""
import pp
from random import randint
def sleep(name, wait):
"""
Example function to run as a job.
"""
time.sleep(wait)
return "%s (%s)" %(name, wait)
def main():
"""
Main function.
"""
# Pool of ppservers to use as a 'server'
job_server = pp.Server(ppservers=('127.0.0.1', ))
# Array of all submitted jobs
jobs = []
# Function Params Libs/Modules
# v v v
jobs.append(job_server.submit(sleep, ('f1', randint(3,15)), (), ('time',)))
jobs.append(job_server.submit(sleep, ('f2', randint(3,15)), (), ('time',)))
jobs.append(job_server.submit(sleep, ('f3', randint(3,15)), (), ('time',)))
jobs.append(job_server.submit(sleep, ('f4', randint(3,15)), (), ('time',)))
# Loop
while True:
jobs_len = len(jobs)
# For each job ...
for x in range(jobs_len):
# See if it's finished
if jobs[x].finished == True:
# If so, print the result, delete the job and loop again
print jobs[x]()
del jobs[x]
break
# All out of jobs? Exit!
if jobs_len == 0:
raise SystemExit(0)
if __name__ == '__main__':
main()
Here is output of a few runs ...
[steve@tachikoman ~]$ python ppexample.py
f4 (3)
f1 (4)
f3 (6)
f2 (11)
[steve@tachikoman ~]$ python ppexample.py
f4 (3)
f3 (3)
f1 (4)
f2 (9)
[steve@tachikoman ~]$ python ppexamples.py
f1 (7)
f4 (8)
f2 (13)
f3 (14)
digg it
seed it
del.icio.us
ma.gnolia


