generated comprehensions

[John La Rooy]

Maybe we need an 'x' comprehension? x[...] anyone?

I think 1e9 is getting toward the exceptionally large. Sure if you think
someone is going to plug it into your program someday, you should work around it
*if* it is practical to do so. If the someday isn't for 5 or 10 years, that
comprehension or whatever is going to be umpteen times faster.

Best to keep things in perspective. We generally expect to need to maintain
programs and raise those builtin limits as users expect/require it.

You have a good point. There is an area where comprehensions aren't ideal and
as a programmer in a perfect world programmers would never need to worry about the
underlying implementation.

This did occur to me on reading your post, and the result did startle me a little:

 Python 2.2.1 (#2, Apr 21 2002, 22:22:55) 
[GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> list(xrange(1e9))
Segmentation fault


On 13 May 2002 23:31:10 -0700
garth at deadlybloodyserious.com (Garth T Kidd) wrote:

> I'm a little worried about getting into the habit of using list
> comprehensions because I'll have to re-write the comprehensions back
> in "normal" Python whenever someone tries to shove a lot of data
> through them.
> 
> If we're talking normal sequences, of course, it's not that much of a
> problem. If it fits in memory, it fits in memory. It's when you start
> using generators because you need to that suddenly comprehensions look
> a little brittle.
> 
>     def printOdds(upto): 
>         for odd in [num for num in xrange(upto) if num%2]:
>             print odd
> 
> ... works fine if upto is 5, but just sits there chewing up memory if
> upto is 10**9, at which point you abandon comprehensions and do it
> properly:
> 
>     def printOdds(upto):
>         for num in xrange(upto):
>             if num%2:
>                 print odd
> 
> Other comprehension party tricks suffer similarly: 
> 
>     [a for a in xrange(upto//2) if sys.stdout.write('%s\n' % (a*2+1))]
> 
> I'm sure I'll figure out a decent rule of thumb (say, "unit test with
> the biggest practical number, and get rid of comprehensions if they
> turn out to be a problem", or "don't use comprehensions with
> generators"), but the fracture still worries me.
> 
> Has anyone tried to figure out a generator variant on list
> comprehensions? Should all list comprehensions return generators?
> 
> Regards,
> Garth.
> 
> PS: Eh? ::
> 
>     >>> xrange(0,5000000000)[3]
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in ?
>     OverflowError: long int too large to convert to int