Advanced problem (recursive calls to python)

[Kai Sterker]

Hello folks,

I am facing a weird problem with python, which I believe might come from
recursive calls to the interpreter. OTOH, I am not sure, and since I am
fairly new to python, I want to make sure what the problem is before
fixing the wrong things.

Here's a short summary of the problem. Further down I'll get into the
details.

Basically, I have used SWIG (1.3 alpha 5) to create wrappers of my C++
code. This code itself contains calls to Python, like PyObject_CallObject,
etc.

The main() function executes a python script which in turn calls functions
of my code that execute more python code. At the most, this recursion is 3
levels deep. So far it was working, but now there is a situation where the
call to PyObject_CallObject returns NULL although I pass a valid callable
object and arguments.

Question: could that be related to the recursion?
I am unsure, because there are several calls to PyObject_CallObject in
that function, but only one of them fails.
Even more weird, if I add a PyObject_Print(...) before said call, it suddenly
works!

Any insights into this problem are welcome. I am using Python 2.1 on 
Linux.


This is the core of the problem, as I see it. But I'll give you a full
overview of the stuff in case I have missed something. So excuse me if it
gets lengthy and boring ;).

The program (Adonthell, a CRPG, adonthell.linuxgames.com) needs to execute
different (small) python scripts regularly. (several times per second).
Further, the scripts need to share the same data.

So I made a PyDict object that contains all that data:
    PyObject *data::globals.

And I compile the scripts into PyCodeObjects and run them directly with
PyEval_EvalCode(...). That way I can pass the same 'global' data to each
script and each kind of script can also get it's own local data.

I further need to access objects instantiated on C++ side from Python.
They are turned into a PyObject with the function:

PyObject *pass_instance (void *instance, const char *class_name)
{
    char class_ptr[256];
    char class_addr[256] = "_";
    char buffer[256];

    // Construct the python shadow class matching the "instance" class 
    strcat (strcpy (class_ptr, class_name), "Ptr");

    // Construct SWIG's representation of the "instance" pointer
    sprintf (buffer, "%p_p_%s", instance, class_name);
    strcat (class_addr, buffer+2);

    // Now create the Python object corresponding to "instance"
    PyObject *cls = PyDict_GetItemString (data::globals, class_ptr);
    PyObject *arg = Py_BuildValue ("(s)", class_addr);
    PyObject *res = PyObject_CallObject (cls, arg);

    // Clean up
    Py_DECREF (arg);

    // Voila: "res" is 'identical' to "instance" :)
    return res;
}

This function is the one where the error occurs under certain
circumstances (see further down):
PyObject_CallObject will return NULL even though 'cls' and 'arg' are okay,
as well as the __init__ function of the 'cls' classobject. It's the usual
code that SWIG generates, nothing special:

class landmapPtr(landmap):
    def __init__(self,this):
        self.this = this
        self.thisown = 0
        self.__class__ = landmap


Anyway, as long as I did no more as the above, everything worked fine. 
But I also need to pass a python function or method as a callback to
the C++ code. Again, storing a python function in a PyObject for later
execution via PyObject_CallObject() is trivial. But from then on, 
things went wrong. 

Here are the steps that result in above error:


I execute the first python script with
    PyRun_File( file, name, Py_file_input, data::globals, NULL );

>From that script a callback is registered, some of my wrapped classes are
instantiated and finally one of their methods is called which never returns.

At some point, the python callback is executed (so we're in the script
again), which calls some other method of a wrapped class (data::load()
- back on C++ side) from where finally over a dozen calls to above 
"pass_instance' are made. As I said above, most of them succeed, but
one does not.
But if I add a "PyObject_Print (data::globals, stdout, 0);" before that
failing call, it suddenly works!
(I thought something with the data::globals was wrong, so I wanted to see
their contents ...)

It'll also work if I leave out that callback and call data::load()
directly from the first script, or one of those precompiled ones that are
executed via PyEval_EvalCode(...).


Since both the first script and those precompiled ones get data::globals
as their global namespace, but that python callback doesn't, I suspect
that the problem lies there. But what confuses me is the fact that it is
not completely failing, only half-heartedly.

Unfortunately, I don't know enough of the python interpreter internals to
know what is allowed, and what isn't. Do I have to use multiple Python
threads instead of recursive calls? Is their a way to set the global
namespace when executing a callable PyObject? Or have I simply missed
something else?


I know that was a load of stuff, but right now I'm stuck and would
appreciate any help. 

If you are interested in the problem and like to have a closer look
you can get the code from cvs
( http://adonthell.linuxgames.com/en_frames/cvs.shtml ) 
and contact me ( ksterker at geekmail.de ) for the required data file, 
which contains, among other stuff, all the python scripts.


Well, thanks for your patience and c ya,

Kai