Buffer growth, lazy opitimization

[Stephen D Evans]

(Not knowing anywhere better to send this observation, I'll post it to the
python newsgroup)

I have had a look at the 'Interpreter Changes and Fixes' in the Python 2.3
documentation (release 0.01)

http://www.python.org/dev/doc/devel/whatsnew/node6.html

Growing string buffers was mentioned as being 'mildly exponential'. A very
quick look in the Python CVS at python/dist/src/Objects/fileobject.c showed
that two different methods of buffer reallocation were being used in this
source file - fixed increment and doubling.

Learning from other people's experience I tend to use a factor of 1.5 for
buffer size increments. For large buffers this is faster than a fixed
increment and does not overshoot and run into memory limitations (virtual,
exhaustion) as readily as a factor of 2. This may require floating point or
division arithmetic, but it is fast when compared to a malloc/free cycle or
going virtual unnecessarily.

For a decent argument I would refer anyone to the end of Item 13 on lazy
optimization in More Exceptional C++ by Herb Sutter, Addison Wesley 2002.
This in turn refers to an article by Andrew Koenig in the September 1998
issue of JOOP (Journal of Object-Oriented Programming).


Stephen D Evans