Google's MapReduce

[Terry Reedy]

<bearophileHUGS at lycos.com> wrote in message 
news:1102506918.350084.92980 at c13g2000cwb.googlegroups.com...
> (This Google article suggestion comes from the CleverCS site):
>
> http://labs.google.com/papers/mapreduce-osdi04.pdf
>
> "MapReduce is a programming model and an associated implementation for
> processing and generating large data sets. Users specify a map function
> that processes a key/value pair to generate a set of intermediate
> key/value pairs, and a reduce function that merges all intermediate
> values associated with the same intermediate key."

Summarizing groups (and sometimes simultaneously by subgroups) has been a 
standard operation for decades in both statistics packages and database 
report generators.  Python has various versions of the map and reduce 
operations that they use.  Its dicts can easily be used to group items with 
the same key.  What is somewhat specific to Google is the need to generate 
*multiple* key-value pairs from each input document.  What MapReduce does 
that is somewhat innovative is automatically parallelize the computation to 
run fault-tolerantly on a cluster of up to 1000s of machines with machine 
slowdowns and failures 'common'.  And indeed, that is the reason to use the 
system with trivial map or reduce functions, as they sometimes do.

What this does show is that Google could be regarded as a cluster 
supercomputing company, with Web search as the visible development 
application.

Terry J. Reedy