string concatentation

[Chris Gonnerman]

----- Original Message -----
From: "Tony Rentschler" <tonyr at walterr.bellatlantic.net>
Subject: Re: string concatentation


> Thanks, the list idiom is just what I was looking for. I'm new
> to Python and am still learning how best to accomplish certain tasks. In
> this case, I can't read() the whole file in one fell swoop because the
> upstream program sends it line by line. If the list method is more
efficient,
> does that then mean that Python lists allocate more space than required as
> items are added so as to grow efficiently?

I just reviewed the source for 2.0, and it appears that a list is
represented
as a dynamic-allocated C array of generic Python objects.  This means that
creating a string object (via the .readline() method) counts as one
malloc(),
and appending it to the list requires a realloc() (actually PyMem_RESIZE()).

The other way requires a malloc() for the object creation, then a copy of
the
string data (twice, once per each source string).  Note that in this case,
the
number of bytes of data malloc()ed per pass gets progressively larger, but
in
the other case the total number malloc()ed per pass remains *relatively*
constant.  I don't know but I think the real reason for the advantage of the
list paradigm is all those excessive string copies.  Also, there will be a
free() call approximately every pass (more often I think as the string gets
larger).  Also the heap will get messier and messier because each new
malloc()
gets a larger block of memory, which probably won't match previously
free()ed
blocks.

Anyway, the list method has always worked better for me.  Anyone with more
of a
clue than I about this want to comment?