Suggestions for Python MapReduce?

python at bdurham.com python at bdurham.com
Wed Jul 22 11:42:07 EDT 2009


Phillip,

We've had great success writing simple, project specific algorithms to
split content into chunks appropriate for ETL type, Python based
processing in a hosted cloud environment like Amazon EC2 or the recently
launched Rackspace Cloud Servers. Since we're purchasing our cloud
hosting time in 1 hour blocks, we divide our data into much larger
chunks than what a traditional map-reduce technique might use. For many
of our projects, the data transfer time to and from the cloud takes the
majority of clock time.

Malcolm



More information about the Python-list mailing list