generator expressions: performance anomaly?

[Ola Natvig]

Antoon Pardon wrote:
> Op 2005-01-18, Steve Holden schreef <steve at holdenweb.com>:
> 
>>Antoon Pardon wrote:
>>
>>
>>>Op 2005-01-18, Nick Coghlan schreef <ncoghlan at iinet.net.au>:
>>>
>>>
>>>>Raymond Hettinger wrote:
>>>>
>>>>
>>>>>[Delaney, Timothy C]
>>>>>
>>>>>
>>>>>
>>>>>>Nick's other suggestion - that genexps propagate __len__ - might
>>>>>>still be interesting. Of course, it would only be applicable for
>>>>>>unconditional genexps(i.e. no if clause).
>>>>>
>>>>>Length transparency for iterators is not as general as one would expect.  I once
>>>>>spent a good deal of effort exploring where it made sense, and I was surprised
>>>>>to find that it only rarely works out.  Length transparency is an unexpectedly
>>>>>thorny subject with many dead-ends which precludes a fully general solution such
>>>>>as that proposed by Nick.
>>>>>
>>>>>For a recap of my research, see the docstring for Lib/test/test_iterlen.py .
>>>>
>>>>"""The situation slightly more involved whenever an object allows length 
>>>>mutation during iteration. """
>>>>
>>>>Ouch. Nice understatement.
>>>>
>>>>It's rather unfortunate that we can't make use of the length information even 
>>>>when the source *doesn't* mutate, though. I'll have to think some more to see if 
>>>>I can come up with any concrete ideas for you to shoot down :)
>>>
>>>
>>>Something else I was thinking about. I think it would be nice if the
>>>python compilor could figure out whether a genexp in a list or tuple
>>>expression always generates the same list or tuple and then instead
>>>of generating code would generate the list or tuple in place.
>>>
>>
>>Since it doesn't yet optimize 2+5 to a constant-folded 7 you should 
>>realize that you are suggesting a large increase in the compiler's 
>>analytical powers.
> 
> 
> Well I can dream, cant I?
> 
> 
>>I agree it would be nice under certain circumstances, but don't forget 
>>that unlike list comprehensions (for which it would be even nicer) the 
>>whole point of generator expressions is often to defer the generation of 
>>the individual items until they are required and thereby relieve stress 
>>on memory.
>>
>>As an edge case to demonstrate the point, what about a constant but 
>>infinite sequence?
> 
> 
> Maybe I was not clear enough. I meant to limit it to cases such as
> 
>    lst = list(genexp)
>    tpl = tuple(genexp)
> 
> 
> Since in such cases the object is build in memory any way, I don't
> think it would be a problem of having them prebuilt in memory, or am
> I missing something?
> 

Perhaps it had been better if that kind of functionality were 
implemented with a different syntax, I don't know enough about how the 
python interpreter works but perhaps it should be possible to execute 
some expressions with genexp syntax when compiling to bytecode. That 
should be a quite trivial extension.

examples using '<' list-comp/genexp syntax '>':

lst = <i for i in range(100)>
or perhaps
lst = list(<i for i in range(100)>)

Would result in the same bytecode as this expression

lst = [0, 1, 2, 3, ..., 98, 99]

It could be called static list comprehensions. But if it is a usefull 
extension is another dicussion.

ola