[Python-Dev] refcounting vs PyModule_AddObject

[Michael Hudson]

"Martin v. Löwis" <martin at v.loewis.de> writes:

> Michael Hudson wrote:
>
>>         if (ProfilerError == NULL)
>>             ProfilerError = PyErr_NewException("hotshot.ProfilerError",
>>                                                NULL, NULL);
>>         if (ProfilerError != NULL) {
>>             Py_INCREF(ProfilerError);
>>             PyModule_AddObject(module, "ProfilerError", ProfilerError);
>>         }
>> 
>> 
>> I think the Py_INCREF should just be removed, but I'm wondering if I'm
>> missing something...
>
> It may be me who is missing something, but...

Well, quite possibly not.

> On reference is added to the dictionary, this is the one the explicit
> INCREF creates. The other reference is held in the C variable
> ProfilerError; this is the one that creating the exception object
> creates.
>
> It is convention that C variables which are explicitly used also
> hold their own references, even if they are global, and even if there
> is no procedure to clear them. The reason is that they would become
> stale if the module object went away. As there is no way to protect
> against this case, they just keep a garbage reference.

This means two things, as I see it:

1) Py_Initialize()/Py_Finalize() loops are going to leak quite a lot.
   Maybe we don't care about this.

2) In the case of the init_hotshot code above and such a loop, the
   ProfilerError object from the first interpreter will be reused by
   the second, which doesn't seem like a good idea (won't it be
   inheriting from the wrong PyExc_Exception?).

Currently running Demo/embed/loop 'import gc' crashes for a similar
kind of reason -- the gc.garbage object is shared between
interpreters, but the only reference to it is in the module's
__dict__ (well, if the module exists...).

I've been looking at this area partly to try and understand this bug:

    [ 1163563 ] Sub threads execute in restricted mode

but I'm not sure the whole idea of multiple interpreters isn't
inherently doomed :-/

Cheers,
mwh

-- 
  Premature optimization is the root of all evil.
       -- Donald E. Knuth, Structured Programming with goto Statements