generated comprehensions

[Garth T Kidd]

I'm a little worried about getting into the habit of using list
comprehensions because I'll have to re-write the comprehensions back
in "normal" Python whenever someone tries to shove a lot of data
through them.

If we're talking normal sequences, of course, it's not that much of a
problem. If it fits in memory, it fits in memory. It's when you start
using generators because you need to that suddenly comprehensions look
a little brittle.

    def printOdds(upto): 
        for odd in [num for num in xrange(upto) if num%2]:
            print odd

... works fine if upto is 5, but just sits there chewing up memory if
upto is 10**9, at which point you abandon comprehensions and do it
properly:

    def printOdds(upto):
        for num in xrange(upto):
            if num%2:
                print odd

Other comprehension party tricks suffer similarly: 

    [a for a in xrange(upto//2) if sys.stdout.write('%s\n' % (a*2+1))]

I'm sure I'll figure out a decent rule of thumb (say, "unit test with
the biggest practical number, and get rid of comprehensions if they
turn out to be a problem", or "don't use comprehensions with
generators"), but the fracture still worries me.

Has anyone tried to figure out a generator variant on list
comprehensions? Should all list comprehensions return generators?

Regards,
Garth.

PS: Eh? ::

    >>> xrange(0,5000000000)[3]
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    OverflowError: long int too large to convert to int