[Python-Dev] RE: Program very slow to finish

[Andrew MacIntyre]

On Mon, 5 Nov 2001, Tim Peters wrote:

> Can any Python-Dev'er make time to dig into the advisability of making
> PyMalloc the default?  I only took time for this because I'm out sick today,
> and was looking for something mindless to occupy my fevered thoughts; alas,
> it paid off <wink>.  I recall there are still thread issues wrt PyMalloc,
> and there *were* some reports that PyMalloc was slower on some platforms.
> Against that, I'm the guy who usually gets stuck trying to explain the
> inexplicable, and malloc/free performance are so critical to Python
> performance that it's always been "a problem" that we have no idea how
> system malloc/free behave across platforms (although I suppose it's "a
> feature" that I know how to crash Win9X by provoking problems with its
> malloc <wink>).  I can't make time for it in the 2.2 timeframe, though.

My experience with the OS/2+EMX port indicates that on this platform,
PyMalloc is (in its current form) about 70% slower than the EMX malloc().
At least, when I used it for _all_ interpreter memory management...

As I recall, the test suite run time didn't change much with PyMalloc
enabled for object allocation only (fractionally slower IIRC) - the
standard WITH_PYMALLOC setup.  FWIW I'm reasonably sure I had threads
enabled for this testing too, and no problems were observed with the
threads test.

Due to the benefits PyMalloc would bring to this platform, it has been on
my todo list to research the performance hit.  My quick glance at the
code suggested a couple of things that a sophisticated optimiser should
have dealt with, but I've not had the chance to verify that this
optimisation is actually happening.

The other issue with PyMalloc has to do with C extensions - NumPy in
particular had a problem due to (I think) inconsistent use of the Python
memory interfaces, long since fixed of course.  Other less widely used
extensions may still have this issues in this regard, but we also need to
provoke some corrective action on this front too.

I'm tempted to suggest releasing the Win32 binary of 2.2b2 with PyMalloc,
just to see what you might be able to shake out, but I don't have to
support it... ;-)

One idea that occurs to me, but probably has more warts than benefits, is
to offer a PYTHON22.DLL compiled with PyMalloc (complete with
"Experimental!" labelling all over it) as a drop-in replacement for the
standard DLL.  A note that points out its possible benefits wrt the sort
of problem that kicked this off might garner enough testing to make some
progress.

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
        andymac@pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia