[C++-sig] Object destructor not always called in embedded interpreter

[Thomas Berg]

On Tue, Oct 14, 2008 at 8:26 PM, Thomas Berg <merlin66b at gmail.com> wrote:
> On Tue, Oct 14, 2008 at 3:48 PM, Tanguy Fautré <tfautre at telenet.be> wrote:
>>> Hi Tanguy,
>>>
>>> I'm no python expert, but I think it works as follows:
>>> When you import __main__, it is added to the internal list of modules
>>> in python. It's __dict__ is therefore also referenced. So even though
>>> you no longer reference it, the __dict__ will still be referenced by
>>> the interpreter itself, and it's refcount doesn't reach 0 until you
>>> finalize the interpreter...
>>>
>>> So, what can you do? I also wanted to be able to clear the interpreter
>>> and re-run scripts in clean namespaces, without re-initializing the
>>> interpreter.
>>>
>>> This can be achieved by calling clear() on the namespace (it's a dictionary).
>>>
>>> However, that also clears the members "__builtins__", "__doc__" and
>>> "__name__", which are needed (at least __builtins__) if you want to
>>> execute code again. I solved this too, simply by re-initializing those
>>> variables again after clearing the dictionary.
>>>
>>> I also discovered that a similar initialization enables you to create
>>> other modules than the "__main__" module (using PyImport_AddModule
>>> instead of import) and execute scripts in those too, which can be
>>> useful. I don't know whether this is the "correct" way of doing things
>>> though...
>>>
>>> See below for an example.
>>>
>>> Regards,
>>> Thomas
>>>
>>> [...]
>>
>>
>> Hi Thomas,
>>
>> Thanks for the advise, it seems to work. It's also good to know how to create other modules.
>>
>> Although, I'm still surprised by this. Because I'm copying the __dict__ of __main__ instead of referencing it (I've even tried directly calling dict.copy()). So I'm surprised by the fact that the interpreter would hold a reference to my copy.
>>
>> I've also tested the following code, and it seems to do the same as yours (as far as I can tell) and can be called several time. Notice how the clear() at the end seems to only affect the copy and not the __main__.__dict__.
>>
>>
>> object main_module = import("__main__");
>> object main_namespace = main_module.attr("__dict__");
>> dict new_namespace(main_namespace);
>>
>> object ignored = exec_file("main.py", new_namespace, new_namespace);
>>
>> new_namespace.clear();
>>
>>
>> Thanks for your help!
>> Cheers,
>>
>> Tanguy
>>
>>
>> PS: I apologize if there is any problem with my posts (e.g. formatting), I'm forced to use my ISP webmail for the moment.
>>
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
> Hi Tanguy,
>
> The behaviour surprises me too.
>
> In my early attempts at embedding, I tried making copies of the
> __main__ module's dictionary too. I don't remember why, but after a
> lot of experiments I ended up just using the original dictonary, and
> clearing it after use. I didn't even notice your dictionary copying
> when I wrote the previous post...
>
> If someone knows, I would like to get an explanation to this
> behaviour. There seems to be something "magical" going on. One thing I
> tried was to make a deep copy of the __main__ module's dictionary. If
> I remember correctly, that wasn't even possible. Maybe it has some
> special properties? Maybe we should try dumping all it's attributes?
> If it was just an ordinary dictionary, deep-copying it should have
> been possible.
>
> Cheers,
> Thomas
>


After one more experiment, I think it is just the __main__ module dict
(and using shallow copies of it) that is the problem. If you modify
your example to create a custom module + dictionary instead, it works
as expected:

// ...
NamedObject::exportToPython();
{
    handle<> module_handle( PyImport_AddModule( "__test__") );
    object module = object(module_handle);
    dict module_dict = extract<dict>(module.attr("__dict__"));
    module_dict["__builtins__"] = import("__builtin__");
    module_dict["__name__"]   = "__test__";
    module_dict["__doc__"]      = object();
    exec_file("main.py", module_dict, module_dict);
}
// The module_dict goes out of scope, and everything gets deleted

By the way, I think you can't call PyImport_AddModule many times with
the same name (it will crash python), so if you want to reuse it you
have to just clear it and keep a reference to it.

Cheers,
Thomas