python 分布式计算框架_Parallel

软件简介

Parallel

Python是Python进行分布式计算的开源模块，能够将计算压力分布到多核CPU或集群的多台计算机上，能够非常方便的在内网中搭建一个自组织的分布式计算平台。先从多核计算开始，普通的Python应用程序只能够使用一个CPU进程，而通过Parallel

Python能够很方便的将计算扩展到多个CPU进程中

示例代码：

#!/usr/bin/python

# File: sum_primes.py

# Author: VItalii Vanovschi

# Desc: This program demonstrates parallel computations with pp module

# It calculates the sum of prime numbers below a given integer in parallel

# Parallel Python Software: http://www.parallelpython.com

import math, sys, time

import pp

def isprime(n):

"""Returns True if n is prime and False otherwise"""

if not isinstance(n, int):

raise TypeError("argument passed to is_prime is not of 'int' type")

if n < 2:

return False

if n == 2:

return True

max = int(math.ceil(math.sqrt(n)))

i = 2

while i <= max:

if n % i == 0:

return False

i += 1

return True

def sum_primes(n):

"""Calculates sum of all primes below given integer n"""

return sum([x for x in xrange(2,n) if isprime(x)])

print """Usage: python sum_primes.py [ncpus]

[ncpus] - the number of workers to run in parallel,

if omitted it will be set to the number of processors in the system

"""

# tuple of all parallel python servers to connect with

ppservers = ()

#ppservers = ("10.0.0.1",)

if len(sys.argv) > 1:

ncpus = int(sys.argv[1])

# Creates jobserver with ncpus workers

job_server = pp.Server(ncpus, ppservers=ppservers)

else:

# Creates jobserver with automatically detected number of workers

job_server = pp.Server(ppservers=ppservers)

print "Starting pp with", job_server.get_ncpus(), "workers"

# Submit a job of calulating sum_primes(100) for execution.

# sum_primes - the function

# (100,) - tuple with arguments for sum_primes

# (isprime,) - tuple with functions on which function sum_primes depends

# ("math",) - tuple with module names which must be imported before sum_primes execution

# Execution starts as soon as one of the workers will become available

job1 = job_server.submit(sum_primes, (100,), (isprime,), ("math",))

# Retrieves the result calculated by job1

# The value of job1() is the same as sum_primes(100)

# If the job has not been finished yet, execution will wait here until result is available

result = job1()

print "Sum of primes below 100 is", result

start_time = time.time()

# The following submits 8 jobs and then retrieves the results

inputs = (100000, 100100, 100200, 100300, 100400, 100500, 100600, 100700)

jobs = [(input, job_server.submit(sum_primes,(input,), (isprime,), ("math",))) for input in inputs]

for input, job in jobs:

print "Sum of primes below", input, "is", job()

print "Time elapsed: ", time.time() - start_time, "s"

job_server.print_stats()

# Parallel Python Software: http://www.parallelpython.com