Multithreading Embedded C/Python : Giving each thread its own sys.stdout

[Warren Postma]

I am trying to set up an embedded Python as a multithreaded environment
where each thread gets it's own namespace, and in each namespace, sys.stdout
has been separately redirected.  I am very close. The problem is, it appears
that the command "print" actually checks __main__.sys.stdout, not whatever
namespace's sys.stdout you are currently in. Am I interpreting this
behaviour correctly, or am I goofing up somewhere. A detailed explanation of
my code follows:

1. Initialization (in the main thread),
         Py_SetProgramName(argv[0]);
         Py_Initialize();
         PyEval_InitThreads();
         PyMainThreadState = PyThreadState_Get(); // store main thread state
in a global for when I need to use it directly.
         /* initialize my own C extension modules */
         init_module1();
         init_module2();
         ...
         init_moduleN();
         PySys_SetArgv(argc, argv);
         _pydefaults(NULL); // import os,sys, and string, automatically for
convenience of my users.

2. For each thread that I create, create a new threadstate:

            PyEval_AcquireLock();
            mainInterpreterState = PyMainThreadState->interp;
            MyThread->PyState  = PyThreadState_New(mainInterpreterState);
            PyEval_ReleaseLock();

3. Then create a copy of the Builtins into a new dictionary.

    // Move into the thread state of the new thread:
    PyEval_AcquireLock();
    PyThreadState_Swap((PyThreadState *)MyThread->PyState);
    dict = PyDict_New();
     PyDict_SetItemString(dict, "__builtins__", PyEval_GetBuiltins());
     if (description != NULL)
          PyDict_SetItemString(dict, "__doc__",
PyString_FromString(description)); // describe new namespace
     _pydefaults(dict); // import os,sys, and string, automatically for
convenience of my users.
    PyThreadState_Swap(NULL);
    PyEval_ReleaseLock();


4. Then instantiate an object of my own to replace sys.stdout, execute this
once in each thread with it's own output catcher.

static char *__OutputCatcher =
"class OutputCatcher:\n"
"  'OutputCatcher - redirects sys.stdout and sys.stderr (implemented inline
in pythonint.c)'\n"
"  def __init__(self,hwrite):\n"
"    self.lines = 0\n"
"    self.hwrite = hwrite\n"
"  def write(self, msg):\n"
"    self.lines = self.lines + 1\n"
"    embstdout.write(self.hwrite,msg)\n"
"\n";

I have a C extension called embstdout, which has a method called write,
which takes a Win32 HANDLE (returned from a call to the Win32 API
CreatePipe) which allows me to create a pipe to handle the output from
Python's print statements, etecetera.

Then for each thread I just set sys.stdout and sys.stderr to a unique
instance of the handle:

// redirect output for each namespace:
static char *__OutputRedirect =
"sys.stdout = OutputCatcher(%d)\n"
"sys.stderr = OutputCatcher(%d)\n";

void RedirectStdout(PyThreadState *ThreadState, PyObject *Dict, HANDLE
hWrite)
{
    char tempbuf[1024];
    sprintf( tempbuf, __OutputRedirect, hWrite, hWrite );
     PyEval_AcquireLock();
     PyThreadState_Swap(ThreadState);
    _pyrun(Namespace,"import sys\n");
    _pyrun(Namespace,tempbuf);
    PyThreadState_Swap(NULL);
    PyEval_ReleaseLock();
};


So I can't figure out why if I do all this, I still only get one GLOBAL
sys.stdout. The last one I create is the one that ALL threads use. So if I
create 5 threads, then all 5 threads write their output to the last one.
It's like there can only be one sys.stdout instance, not one sys module per
namespace, each with it's own sys.stdout object.  What can I do?

Worst case is I must ban the use of PRINT statements in scripts, and instead
specifically have the script writers write to a handle. This is hardly
elegant though.  I would even be willing to hack the PRINT command to make
it read the current sys.stdout value from the ThreadState structure, or
something like that, if it were possible.  Has anyone else faced this little
dilemma?

Warren Postma