[Python-Dev] eval() slowdown in 2.2 on MacOS X?

[Andrew MacIntyre]

On Thu, 10 Jan 2002, Andrew Kuchling wrote:

> On Tue, Jan 08, 2002 at 01:41:37PM -0500, Tim Peters wrote:
> >Break it into smaller steps so we can narrow down possible causes:
>
> You should have cc'ed Barbara on that.  I forwarded your message to her
> and she wrote back (eventually):
>
> >BTW, I forgot to pass this on yesterday, but I tried the code in Tim Peters'
> >e-mail yesterday and the delay happens during the code = compile(...)
> >statement.
>
> She's going to install sshd on her machine, so maybe this weekend I'll
> be able to log in, compile Python from source, and poke around in an
> effort to figure out what's going on.

IMHO, Barbara's problem is almost certainly related to the system
malloc(), and if that is the case the only effective antidote is
pymalloc.

However, just enabling WITH_PYMALLOC isn't enough as its currently only
configured to be used for object allocation and doesn't help the parser.

I did attach a patch enabling pymalloc for all interpreter memory
management to a long post to python-dev (which AMK might
recall this from his python-dev summaries) about my research into
test_longexp problems with the OS/2+EMX port. My research revealed that
test_longexp causes the parser to go ballistic with small mallocs.  While
pymalloc solved the test_longexp problem, using it for all interpreter
memory management caused about a 60% performance hit (on OS/2 + EMX).

On OS/2 the problem appeared to be overallocation (allocating 3-4x as much
memory as actually requested), but I recall reading a thread on
python-list wherein people reported system malloc()s that attempt to
coalesce blocks which had a similar slowdown effect in another set of
circumstances (I don't recall the details - might have been related to
dictionaries?).

Although OS/X's BSD heritage goes back to FreeBSD 3.2, I wouldn't have
expected either sort of problem from that source as I've had none of
these problems with my own FreeBSD systems - in fact last night I ran the
test suite on a CVS derived build on a 486/100 FreeBSD 4.4 system with
only 32MB of RAM and 128MB of swap and test_longexp passed & in only
minutes (the whole test suite took ~25-30 mins for the first pass, ie
without .pyc files). OS/X may have acquired a malloc() of different
heritage though.

I did keep a copy of the instrumented malloc() output from my OS/2
research if you're interested, although it probably isn't as helpful as it
could be (Python 2.0 vintage)... Likewise my crude debug malloc
wrapper... and I might be able to dig up my pymalloc_for_all patch...

--
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac@bullseye.apana.org.au  | Snail: PO Box 370
        andymac@pcug.org.au            |        Belconnen  ACT  2616
Web:    http://www.andymac.org/        |        Australia