[Python-Dev] Memory woes under Windows

[Tim Peters]

[Mark Hammond]
> This is definately wierd!  As you only mentioned Win9x, I thought I would
> give it a go on Win2k.

Thanks, Mark!  I've only got W9X machines at home.

> This is from a CVS update of only a few days ago, but it is a non-debug
> build.  PII266 with 196MB ram:
>
>          1 push       0.001
>          2 push       0.000
>          4 push       0.000
>          8 push       0.000
>         16 push       0.000
>         32 push       0.000
>         64 push       0.000
>        128 push       0.001
>        256 push       0.001
>        512 push       0.003
>       1024 push       0.006
>       2048 push       0.011
>       4096 push       0.040
>       8192 push       0.043
>      16384 push       0.103
>      32768 push       0.203
>      65536 push       0.583
>
> Things are looking OK to here - the behaviour Tim expected.  But then
> things seem to start going a little wrong:
>
>     131072 push       1.456
>     262144 push       4.763
>     524288 push      16.119
>    1048576 push      60.765

So that acts like my Win95 (which I didn't show), and somewhat like my 2nd &
3rd Win98 runs.

> All of a sudden we seem to hit N*N behaviour?

*That* part really isn't too surprising.  Python "overallocates", but by a
fixed amount independent of the current size.  This leads to quadratic-time
behavior "in theory" once a vector gets large enough.  Guido's cultural myth
for why that theory shouldn't matter is that if you keep appending to the
same vector, the OS will eventually move it to the end of the address space,
whereupon further growth simply boosts the VM high-water mark without
actually moving anything.  I call that "a cultural myth" because some
flavors of Unix did used to work that way, and some may still -- I doubt
it's ever been a valid argument under Windows, though. (you, of all people,
know how much Python's internal strategies were informed by machines nobody
uses <wink>).

So I was more surprised up to this point by the supernatural linearity of my
first W98 run (which is reproducible, btw).  But my 2nd & 3rd W98 runs (also
reproducible), and unlike your W2K run, show *worse* than quadratic
behavior.

> I gave up waiting for the next one.

Under both W98 and W95, the next one does eventually hit the MemoryError for
me, but it does take a long time.  If I thought it would help, I'd measure
it.  And *this* one is surprising, because, as you say:

> Performance monitor was showing CPU at 100%, but the Python process
> was only sitting on around 15MB of RAM (and growing _very_ slowly -
> at the rate you would expect).  Machine had tons of ram showing as
> available, and the disk was not thrashing - ie, Windows definately
> had lots of mem available, and I have no reason to believe that
> a malloc() would fail here - but certainly no one would ever want to wait
> and see :-)

How long did you wait?  If less than 10 minutes, perhaps not long enough.  I
certainly didn't expect a NULL return either, even on my tiny machine, and
certainly not on the box with 20x more RAM than the list needs.

> This was all definately built with MSVC6, SP3.

Again good to know.  I'll chew on this, but don't expect a revelation soon.

> no-room-should-ever-have-more-than-one-windows-ly y'rs

Hmm.  I *did* run these in different rooms <wink>.

no-accounting-for-windows-ly y'rs  - tim