[Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)

[M.-A. Lemburg]

On 2008-12-20 23:16, Martin v. Löwis wrote:
>>> I will try next week to see if I can come up with a smaller,
>>> submittable example.  Thanks.
>> These long exit times are usually caused by the garbage collection
>> of objects. This can be a very time consuming task.
> 
> I doubt that. The long exit times are usually caused by a bad
> malloc implementation.

With "garbage collection" I meant the process of Py_DECREF'ing the
objects in large containers or deeply nested structures, not the GC
mechanism for breaking circular references in Python.

This will usually also involve free() calls, so the malloc
implementation affects this as well. However, I've seen such long
exit times on Linux and Windows, which both have rather good
malloc implementations.

I don't think there's anything much we can do about it at the
interpreter level. Deleting millions of objects takes time and that's
not really surprising at all. It takes even longer if you have
instances with .__del__() methods written in Python.

Applications can choose other mechanisms for speeding up the
exit process in various (less clean) ways, if they have a need for
this.

BTW: Rather than using a huge in-memory dict, I'd suggest to either
use an on-disk dictionary such as the ones found in mxBeeBase or
a database.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Dec 22 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2008-12-02: Released mxODBC.Connect 1.0.0      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/