Tuple question

[Alex Martelli]

Peter Hansen <peter at engcorp.com> wrote:
   ...
> Alex shows use of a generator... fine, but how do you
> build the tuple without storing up the results of the
> generator first somewhere else?  You can't preallocate the

You, programming in Python, can't, but the interpreter, internally, can
-- because function _PyTuple_Resize (nor being directly callable from
Pytjon) _CAN_ resize a tuple more efficiently than you imply:

> space for the tuple if you don't know how long it will
> be, but you have to preallocate the space for a tuple
> (I believe, in the interpreter anyway, if not at the
> programmer level) so you must therefore be storing the
> entire output of the generator somewhere just prior
> to the tuple creation: same problem as above.

No problem at all -- if your available memory is in one chunk, the
resize can work in place.  (If your available memory is fragmented you
can of course be hosed, since Python's can't move allocated blocks and
thus can't compact things up to cure your fragmentation; but even when
memory's tight it's not unusual for it to not be fragmented).

> 
> I know Alex knows all this (or has some additional
> info that I don't have and which he'll shortly provide),

You don't have the sources of the Python interpreter?  They're freely
available for download, why would I need to provide them?!  Anyway, just
follow, with your C debugger or whatever (I think mere code inspection
will be fine), what happens when you call x = tuple(someiterator()).

Moreover, if the 'someiterator()' iterator can return a reasonable
__len__, I believe Python 2.4 should now able to use that to estimate
the needed length in advance for greater efficiency, but I haven't
looked deeply into that, yet... indeed, some simple timeit.py tests
suggest to me that 2.4 alpha 3 only implements that optimization to
preallocate lists, not tuples.  But even if so, that's clearly a matter
of mere expediency, not one of any intrinsic problem as you make it
sound.


> so I can only assume he was reacting only to my poor
> choice of wording with 'list' and/or was ignoring the
> context of the discussion (memory usage).

Claiming that you have to have all info in memory before a tuple can be
built is simply wrong -- your previous claim that the info had to be in
a list was even "wronger", sure, but that doesn't make your current
weaker claims correct in the least.


Alex