[Python-Dev] Re: Activating pymalloc

[Andrew MacIntyre]

On Tue, 19 Mar 2002, Tim Peters wrote:

> [Andrew MacIntyre]

{...}

> [when redirecting all (I think) mallocs to go thru pymalloc]
> > test_longexp was the pathological case.

Yes, I was redirecting everything through pymalloc for those experiments

{...}

> I then further bashed obmalloc.c to call malloc/realloc/free directly.
> test_longexp sped up significantly then, taking about 3 seconds before this
> fiddling and about 2 seconds after.
>
> So best guess is that test_longexp + pymalloc-all-the-time tickles some
> pathology in your platform's malloc/realloc/free (or did).  Do you have a
> lot of RAM?  test_longexp is a memory hog regardless.

At the time, I had a K6-2/300 with 64MB of ram and 150MB of swap-file
space (40MB preallocated).  OS/2 allocates about 70MB.  On that system,
the most I could set REPS to was about 20,500 before the avalanche started
- If I'd had another 100-150MB free on the swapfile partition it probably
would have finished, as I had a report from a user with 256MB of ram that
the python process running the test grew to about 200MB.

pymalloc'ing the whole interpreter fixed test_longexp nicely, and I was
able to set REPS to 100,000 and have test_longexp run in a second or two
and not tickle swap.

The performance hit was only apparent when running the full test suite
with the exclusively pymalloc interpreter, which took about 60% longer
than running the test suite on a standard interpreter.  As I recall,
all bar one test (test_socket?) passed, so things were solid enough
(Python 2.1 time frame).

I'd be curious as to your results in timing the whole test suite with an
all-pymalloc interpreter vs the standard interpreter, if you've still
got the bits lying around.

At the very least, the parser is probably something to look at having use
pymalloc, as it seems to generate lots of small malloc requests.

Oh, I have 256MB and an Athlon 1.4 with lots of swap space now, so
test_longexp works just fine for me now...

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
        andymac@pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia