Parallel/distributed generator

[Diez B. Roggisch]

George Sakkis schrieb:
> On May 23, 2:11 pm, "Diez B. Roggisch" <d... at nospam.web.de> wrote:
> 
>> George Sakkis wrote:
>>> I'm looking for any existing packages or ideas on how to implement the
>>> equivalent of a generator (in the Python sense, i.e.
>>> http://www.python.org/dev/peps/pep-0255/) in a parallel/distributed
>>> way. As a use case, imagine a function that generates a range of
>>> primes. I'd like to be able to do something along the following lines:
>>> def iterprimes(start=1, end=None):
>>>    # ...
>>>    yield prime
>>> # rpc-related initialization
>>> ...
>>> rpc_proxy = some_rpc_lib(iterprimes, start=1e6, end=1e12)
>>> for prime in proxy:
>>>     print prime
>>> Is there any module out there that does anything close to this ?
>> Have you tried using pyro to just return a generator? After all, it's just
>> an object with a next()-method, that raises a certain exception. I can't
>> see why pyro shouldn't be able to handle that.
> 
> Just tried it:
> 
> cPickle.PicklingError: Can't pickle <type 'generator'>: attribute
> lookup __builtin__.generator failed

Hm. Maybe a bit of wrapping would help? Just like this:

class GenWrapper(object):

     def __init__(self, g):
         self._g = g

     def next(self):
         return self._g.next()

I'm not sure though if this indirection really helps.

Diez