[Python-Dev] Darwin's realloc(...) implementation never shrinks allocations

[bacchusrx]

On Mon, Jan 03, 2005 at 03:55:19PM -0500, Bob Ippolito wrote:
> >Note that, with respect to http://python.org/sf/1092502, the author
> >of the (original) program was using the documented interface to a
> >file object.  It's _fileobject.read() that decides to ask for huge
> >numbers of bytes from recv() (specifically, in the
> >max(self._rbufsize, left) condition). Patched to use a fixed
> >recv_size, you of course sidestep the realloc() nastiness in this
> >particular case.
> 
> While using a reasonably sized recv_size is a good idea, using a
> smaller request size simply means that it's less likely that the
> strings will be significantly resized.  It is still highly likely they
> *will* be resized and that doesn't solve the problem that
> over-allocated strings will persist until the entire request is
> fulfilled.

You're right. I should have said, "you're more likely to get away with
it." The underlying issue still exists. My point is that the problem is
not analogous to the guy who tried to read 2GB directly from a socket
(as in http://python.org/sf/756104). 

Googling for MemoryError exceptions, you can find a number of spurious
problems on Darwin that are probably due to this bug: SpamBayes for
instance, or the thread at

http://mail.python.org/pipermail/python-list/2004-November/250625.html

bacchusrx.