Question: How efficient is using generators for coroutine-like problems?

[Alex Martelli]

Carlos Ribeiro <carribeiro at gmail.com> wrote:
   ...
>     result += ["Start of processing"]

I would suggest result.append('Start of processing').  As you're
concerned with performance, see...:

kallisti:~/cb alex$ python2.4 timeit.py -s'r=[]' 'r+=["goo"]'
1000000 loops, best of 3: 1.45 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'r=[]' 'r.append("goo")'
1000000 loops, best of 3: 0.953 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'r=[]; a=r.append' 'a("goo")'
1000000 loops, best of 3: 0.556 usec per loop

i.e. you can get result.append into a local once at the start and end up
almost 3 times faster than with your approach, but even without this
you're still gaining handsomely with good old result.append vs your
preferred approach (where a singleton list gets created each time).


> Now, just because I can do it does not mean it's a good idea :-) For
> particular cases, a measurement can be done. But I'm curious about the
> generic case. What is the performance penalty of using generators in
> situations as the ones shown above?

Sorry, there's no "generic case" that I can think of.  Since
implementations of generators, list appends, etc, are allowed to change
and get optimized at any transition 2.3 -> 2.4 -> 2.5 -> ... I see even
conceptually no way to compare performance except on a specific case.

"Generally" I would expect: if you're just looping on the result, a
generator should _gain_ performance wrt making a list.  Consider cr.py:

x = map(str, range(333))

def withapp(x=x):
    result = []
    a = result.append
    for item in x: a(item)
    return result

def withgen(x=x):
    for item in x: yield item

kallisti:~/cb alex$ python2.4 timeit.py -s'import cr' 'for x in
cr.withapp(): pass'
1000 loops, best of 3: 220 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'import cr' 'for x in
cr.withgen(): pass'
1000 loops, best of 3: 200 usec per loop

The difference is more pronounced in 2.3, with the generator clocking in
at 280 usec, the appends at 370 (anybody who's interested in speed has
hopefully already downloaded 2.4 and is busy trying it out -- I can't
see any reason why not... even though one obviously can't yet deliver to
customers stuff based on what's still an alpha release, of course, at
least one can TRY it and pine for its general speedups;-).

A join is different...:

kallisti:~/cb alex$ python2.4 timeit.py -s'import cr'
'"\n".join(cr.withgen())'
1000 loops, best of 3: 274 usec per loop
kallisti:~/cb alex$ python2.4 timeit.py -s'import cr'
'"\n".join(cr.withapp())'
1000 loops, best of 3: 225 usec per loop

(speed difference was less pronounced in 2.3, 360 vs 350).  Yeah, I
know, it's not easy to conceptualize -- if looping is faster why is
joining slower?  "Implementation details" of course, and just the kind
of thing that might change any time, if some nice free optimization can
be obtained hither or yon...!


Alex