[Tutor] memory management

[Doug Stanfield]

Might I respectfully submit that this discussion has exceeded the intent of
the 'Tutor List'.  I might be wrong but it seems more appropriate to c.l.py
or maybe an offline discussion.

-Doug-

> -----Original Message-----
> From: Christian Tismer [mailto:tismer@appliedbiometrics.com]
> Sent: Monday, April 19, 1999 6:25 AM
> To: David Ascher
> Cc: Arne Mueller; tutor
> Subject: Re: [Tutor] memory management
> 
> 
> 
> 
> David Ascher wrote:
> > 
> > On Sun, 18 Apr 1999, Christian Tismer wrote:
> > 
> ...
> > > and watched my swapfile frow from small to 233 MB.
> > > Then I did a del on big. This took even more time than
> > > creating it, btw. Touching all the refcounts again seems
> > > to be quite a challenge for my harddisk. I have the strong
> > > impression that releasing objects should try to do this in
> > > reverse order.
> > 
> > Do what in reverse order?
> 
> I believe that trying to release objects in reverse order
> might be faster than in-order. At least I'm quite sure that
> it will not behave worse. Releasing objects by age is for
> sure a good guess. Python cannot know the object's age,
> but releasing list elements from back to front might
> more often be related to the object's age.
> 
> > 
> > > Without explicit measure, my guess is it takes
> > > 5 times as long to release this chunk.
> > 
> > My guess is that you're seeing OS-specific behavior.
> 
> Well, but there are many who suffer from this OS.
> 
> [most of nice didactic excourse skipped]
> 
> >   - Python's memory management is somewhat 
> object-type-specific -- some
> >     objects are allocated from buffers which are indeed not 
> completely
> >     released when the objects aren't needed -- for example, 
> frame objects
> >     (used internally) and integers are taken from 
> preallocated buffers
> >     when possible, to speed up the common case of allocation and
> >     deallocation.  You might be seeing an effect of that 
> with your range()
> >     test.  Details are in the code (see e.g. intobject.c and
> >     frameobject.c).  Interning of strings is another 
> special-case behavior
> >     which can lead to apparent memory leask which aren't real memory
> >     leaks.
> 
> Integers are allocated in a table of 1k blocks. They are chained
> to each other, forming a freelist. When a new integer is requested,
> it is taken from the freelist. Dropped integers are linked into the
> freelist, but never freed but at the program end.
> This explains the size of my swap file very well.
> An integer takes 12 bytes, this makes about 120 MB in my example,
> so this is ok.
> 
> Now, assuming you have a machine with 64 MB of memory,
> try this in a fresh PyWin session:
> 
> >>> from ptools import timing
> >>> def test(amount):
> ... 	global x
> ... 	x=range(amount)
> ... 	
> >>> timing(test, 1000000)
> (0.49, None)
> >>> timing(test, 0)
> (0.27, None)
> >>> timing(test, 1000000)
> (0.28, None)
> >>> timing(test, 0)
> (0.22, None)
> >>> timing(test, 2000000)
> (0.72, None)
> >>> timing(test, 0)
> (0.5, None)
> >>> timing(test, 3000000)
> (1.32, None)
> >>> timing(test, 0)
> (0.77, None)
> >>> timing(test, 4000000)
> (10.0, None)
> >>> timing(test, 0)
> (141.65, None)
> >>> timing(test, 3000000)
> (4.17, None)
> >>> timing(test, 0)
> (0.71, None)
> >>> timing(test, 4000000)
> (43.34, None)
> >>> timing(test, 0)
> (107.6, None)
> >>> 
> 
> As we can see, everything works fine upto 3 million numbers.
> 4 million is really bad, especially for deallocation.
> A few calculations:
> In the 3 million case, the array created by range is 4 * 3 million
> = 12 MB, the data for the integers is 12 * 3 million = 36 MB,
> together this is 48 MB which fits into memory after swapping
> most of Win98 to the disk, ok.
> 
> With 4 million elements, we reach the 64 MB limit, and the system
> goes nuts. Everything is at least 10 times slower, releasing
> seems to be at least 20 times slower. So what happens?
> 
> First, the list array is created in memory and filled with NULL.
> Then, all the integer objects are created and inserted into
> the array. Since the system cannot keep everything in memory,
> and since every place in the array and the integer freelist
> blocks will be touched once now, the whole memory will be written
> out in (small?) chunks in order to make room for the next chunk.
> So far ok, this seems to be impossible to avoid.
> 
> On deallocation, the situation seems to be similar.
> The list array is read in order, and the numbers refcounts
> are decremented. This results in freeing most of the numbers,
> but instead of returning them into the global memory pool,
> they are inserted into the freelist.
> This means, the freed integer object is written with the current
> freelist head pointer. This will again pour the whole memory
> through the disk, but doesn't explain the time behavior.
> 
> I just tested it with my 16 MB Linux machine, and no, it is not
> OS specific. Instead on timing with a function, I did
> 
> >>> x=range(1000000)
> >>> x=0
> >>>
> 
> and counted cursor blinks of my telnet shell.
> Creating the range took 12 blinks, destroying took 37 blinks.
> 
> But I think I should move this topic to the main list.
> 
> ciao - chris
> 
> -- 
> Christian Tismer             :^)   
> <mailto:tismer@appliedbiometrics.com>
> Applied Biometrics GmbH 
>      :     Have a break! Take a ride on Python's
> Kaiserin-Augusta-Allee 101   :    *Starship* 
> http://starship.python.net
> 10553 Berlin                 :     
> PGP key -> http://wwwkeys.pgp.net
> PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 
> 93E2 1FAE F6DF
>      we're tired of banana software - shipped green, ripens at home
> 
> _______________________________________________
> Tutor maillist  -  Tutor@python.org
> http://www.python.org/mailman/listinfo/tutor
>