[Python-Dev] AtExit Functions

[M.-A. Lemburg]

Guido van Rossum wrote:
>>While working with mxTextTools 2.1.0b2, Mike Fletcher found that
>>he gets a fatal error when the interpreter exits.
>>
>>Some tracing indicates that the cause is the at exit function
>>of mxTextTools which clears the cache of tag tables used by
>>the Tagging Engine in mxTextTools.
>>
>>If these tables include references to (callable) Python instances,
>>Python can't properly clean them up when decref'ing them at
>>AtExit time.
>>
>>Would it be safe to simply move the call_dll_exitfunc()
>>call just before the "clear threat" code in Py_Finalize() ?
> 
> 
> You mean call_ll_exitfuncs(). :-)

Yeah... today's my typo day :-)

> I think you may be making a wrong use of Py_AtExit().  The docs state
> (since 1998):
> 
>   Since Python's internal finallization will have completed before the
>   cleanup function, no Python APIs should be called by *func*.

Hmm, and that includes Py_DECREF() and PyObject_Del() ?

In that case, I have a problem since I'm using those
two to clean up caches and free lists in the mx tools.

> I don't think it's safe to move the call forward.  (I don't know which
> line you are referring to with ``"clear threat" code'' so I don't know
> how far back you want to move it, but I think the intention is very
> clear that this should be done at the very last.)

Here's the snippet:

void
Py_Finalize(void)
{
	PyInterpreterState *interp;
	PyThreadState *tstate;

	if (!initialized)
		return;

	/* The interpreter is still entirely intact at this point, and the
	 * exit funcs may be relying on that.  In particular, if some thread
	 * or exit func is still waiting to do an import, the import machinery
	 * expects Py_IsInitialized() to return true.  So don't say the
	 * interpreter is uninitialized until after the exit funcs have run.
	 * Note that Threading.py uses an exit func to do a join on all the
	 * threads created thru it, so this also protects pending imports in
	 * the threads created via Threading.
	 */
	call_sys_exitfunc();
	initialized = 0;

	/* Get current thread state and interpreter pointer */
	tstate = PyThreadState_Get();
	interp = tstate->interp;

	/* Disable signal handling */
	PyOS_FiniInterrupts();

	/* Cleanup Codec registry */
	_PyCodecRegistry_Fini();

	/* Destroy all modules */
	PyImport_Cleanup();

	/* Destroy the database used by _PyImport_{Fixup,Find}Extension */
	_PyImport_Fini();

----------------------------------
move call_ll_exitfuncs() here
----------------------------------

	/* Debugging stuff */
#ifdef COUNT_ALLOCS
	dump_counts();
#endif

#ifdef Py_REF_DEBUG
	fprintf(stderr, "[%ld refs]\n", _Py_RefTotal);
#endif

#ifdef Py_TRACE_REFS
	if (Py_GETENV("PYTHONDUMPREFS")) {
		_Py_PrintReferences(stderr);
	}
#endif /* Py_TRACE_REFS */

	/* Now we decref the exception classes.  After this point nothing
	   can raise an exception.  That's okay, because each Fini() method
	   below has been checked to make sure no exceptions are ever
	   raised.
	*/
	_PyExc_Fini();

	/* Delete current thread */
	PyInterpreterState_Clear(interp);
	PyThreadState_Swap(NULL);
	PyInterpreterState_Delete(interp);

	PyMethod_Fini();
	PyFrame_Fini();
	PyCFunction_Fini();
	PyTuple_Fini();
	PyString_Fini();
	PyInt_Fini();
	PyFloat_Fini();

#ifdef Py_USING_UNICODE
	/* Cleanup Unicode implementation */
	_PyUnicode_Fini();
#endif

	/* XXX Still allocated:
	   - various static ad-hoc pointers to interned strings
	   - int and float free list blocks
	   - whatever various modules and libraries allocate
	*/

	PyGrammar_RemoveAccelerators(&_PyParser_Grammar);

#ifdef PYMALLOC_DEBUG
	if (Py_GETENV("PYTHONMALLOCSTATS"))
		_PyObject_DebugMallocStats();
#endif

--------------------------------
	call_ll_exitfuncs();
--------------------------------

#ifdef Py_TRACE_REFS
	_Py_ResetReferences();
#endif /* Py_TRACE_REFS */
}

> You may want to use the atexit.py module instead to schedule your
> module's cleanup action; these exit functions are called much earlier.

That's difficult to get right since I have to register such a
function from C. Also, atexit.py is not present in
Python 1.5.2.

I could probably use a hack in the module dictionary which
then triggers calling a cleanup function when the dictionary
gets cleared, but there's a problem with this: clearing
the module is easily possible for a user as well and doing
so would cause seg faults if the user continues to call
API on the module (maybe unknowingly through destructors).

Looks like the only way to "solve" the problem is by simply
leaking memory :-(

-- 
Marc-Andre Lemburg
CEO eGenix.com Software GmbH
_______________________________________________________________________
eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,...
Python Consulting:                               http://www.egenix.com/
Python Software:                    http://www.egenix.com/files/python/