[Python-ideas] cross platform memory usage high water mark for improved gc?

[Adam Olsen]

On Tue, Mar 11, 2008 at 1:57 PM, Aaron Watters <aaron.watters at gmail.com> wrote:
>
>
> >
> >
> > You're concerned that a new feature may increase how high of a
> > threshold you need, yet it could also exceed the "maximum" of your
> > adaptive scheme.
> >
> > I'm not convinced you need that high of a threshold anyway.  I'd like
> > to see a benchmark showing how your app performs at different levels.
>
> You are absolutely right that I can set a threshold high enough.
> With the default values Python is extremely slow for certain cases.
>  I'm arguing it should automatically detect when it is being stupid
> and attempt to fix it.  In particular I would set the maximum very
> high and start at the minimum, which might be near the current
> defaults.
>
> For example I get the following in a simple test (python2.6):
>
> > python gctest.py
> gc not disabled
> elapsed 19.2473409176
> > python gctest.py disable
> gc disabled
> elapsed 4.88715791702
>
> In this case the interpreter is spending 80% of its time trying
>  to collect non-existent garbage.  Now a newbie who
> didn't know to go fiddling with the garbage collector
> might just conclude "python is ssslllooowwww" and go
> back to using Perl or Ruby or whatever in a case like this.
>  Maybe the powers that be couldn't care less about it, I don't
> know.  (I know newbies can be irritating).
>
> The problem is quadratic also: if I double the limit the
> penalty goes up by a factor of 4.
>
>  Here is the source:
>
> def test(disable=False, limit=1000000):
>     from time import time
>     import gc
>     if disable:
>         gc.disable()
>         print "gc disabled"
>     else:
>         print "gc not disabled"
>      now = time()
>     D = {}
>     for i in range(limit):
>         D[ (hex(i), oct(i)) ] = str(i)+repr(i)
>     L = [ (y,x) for (x,y) in D.iteritems() ]
>     elapsed = time()-now
>     print "elapsed", elapsed
>
> if __name__=="__main__":
>     import sys
>     disable = False
>     if "disable" in sys.argv:
>         disable = True
>     test(disable)

Interesting.  With some further testing, it's become clear that the
problem is in gen2.  gen0 and gen1 both add a constant overhead (their
size is bounded), but gen2's size grows linearly, and with a linear
number of scans that gives quadratic performance.

I'm unsure how to best fix this.  Anything we do will effectively
disable gen2 for short-running programs, unless they do the right
thing to trigger the heuristics.  Long running programs have a little
more chance of triggering them, but may do so much later than
desirable.

Something must be done though.  The costs should be linear with time,
not quadratic.  The frequency at which an object gets scanned should
be inversely proportional to the number of objects to be scanned.


-- 
Adam Olsen, aka Rhamphoryncus