Generators.

[Jorge Cardona]

Hi,

I was trying to create a function that receive a generator and return
a list but that each elements were computed in a diferent core of my
machine. I start using islice function in order to split the job in a
way that if there is "n" cores each "i" core will compute the elements
i,i+n,i+2n,..., but islice has a weird (to me) behavior, look:

from itertools import islice

def f(x):
    print("eval: %d"%x)
    return x

X = range(10)
g = (f(x) for x in X)

print(list(x for x in islice(g,0,None,2)))

$ python2.5 test.py
eval: 0
eval: 1
eval: 2
eval: 3
eval: 4
eval: 5
eval: 6
eval: 7
eval: 8
eval: 9
[0, 2, 4, 6, 8]
$ python2.7 test.py
eval: 0
eval: 1
eval: 2
eval: 3
eval: 4
eval: 5
eval: 6
eval: 7
eval: 8
eval: 9
[0, 2, 4, 6, 8]
$ python3.0 test.py
eval: 0
eval: 1
eval: 2
eval: 3
eval: 4
eval: 5
eval: 6
eval: 7
eval: 8
eval: 9
[0, 2, 4, 6, 8]

islice execute the function at the generator and drop the elements
that aren't in the slice. I found that pretty weird, the way that i
see generators is like an association between and indexing set (an
iterator or another generator) and a computation that is made indexed
by the indexing set, and islice is like a "transformation" on the
indexing set,it doesn't matter the result of the function, the slice
should act only on the indexing set, some other "transformation" like
takewhile act on the result so, the execution it has to be made, but
in the islice, or other "transformation" that act only in the indexing
set, the function shouldn't be executed at each element, but only on
that new set that result of the application of the "transformation" on
the original set.

I search a little bit and found that gi_frame.f_locals['.0'] holds the
inner indexing element of a generator, so i decide to define my own
islice like this:

from itertools import islice
import sys


if sys.version_info[0] != 3:
    def next(it):
        return it.next()


def f(x):
    print("eval: %d"%x)
    return x


def islice(iterable, *args):
    s = slice(*args)

    # search the deepest iter (Base indexing set)
    it = iterable
    while hasattr(it, 'gi_frame'):
        it = it.gi_frame.f_locals['.0']

    # Consume the base indexing set until the first element
    for i in range(s.start):
        it.next()

    for e in iterable:
        yield e

        # Consume the base indexing set until the next element
        for i in range(s.step-1):
            next(it)


X = range(10)
g = (f(x) for x in X)

print(list(x for x in islice(g,0,None,2)))


jcardona at terminus:/tmp$ python2.5 test.py
eval: 0
eval: 2
eval: 4
eval: 6
eval: 8
[0, 2, 4, 6, 8]
jcardona at terminus:/tmp$ python2.7 test.py
eval: 0
eval: 2
eval: 4
eval: 6
eval: 8
[0, 2, 4, 6, 8]
jcardona at terminus:/tmp$ python3.0 test.py
eval: 0
eval: 2
eval: 4
eval: 6
eval: 8
[0, 2, 4, 6, 8]

Well, it works for what i need, but is not very neat, and i think that
there it should be a formal way to act on the base indexing iterator,
such way exists? Is there a better approach to get what i need?

Thanks.


-- 
Jorge Eduardo Cardona
jorgeecardona at gmail.com
jorgeecardona.blogspot.com
------------------------------------------------
Linux registered user  #391186
Registered machine    #291871
------------------------------------------------