Some questions embedding Pythong in C (callback for variable assignment in the dictionary?)

[jordi]

Hi,

I'm starting to use Python embedded in a C program.  I'm using Python
to execute several scripts using as a variables information retrieved
for several multithread "agents" written in C.  The method is:

 1- Create a thread for each agent (I use a pool thread, every agent
run in a different thread)
 2- Precompile a different script for each agent. 
 3- Make the agent retrieve its data
 4- Convert this data to Python variables
 5- Execute a Python script.

The steps 3 to 5 are executed again and again (there is a timer that
launch the execution). As it's a multithreaded program my code is
something like this:

- In each thread initialization:

  PyEval_AcquireLock();
  interp = Py_NewInterpreter();  
  PyThreadState_Swap(interp);

  globals = PyDict_New();
  PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins());
  pCode = Py_CompileString(ScriptPython, "testPython", Py_file_input
);

  (This "ScriptPython" is different depending on the type of agent)

- Then, every time the C "agent" retrieves its data:

     PyEval_AcquireThread(interp);
     /* Here I have to modify the PyDictionary, globals or locals */
     result = PyEval_EvalCode((PyCodeObject *)pCode, globals,
0/*locals*/);
     PyEval_ReleaseThread(interp);

I have some questions:

1.- Is there any way to make the embedded Python interpret make a
callback to the C program asking for variables when they are not
defined in the Dictionary ? That is, implementing an "variable
on-demand" method. I have a problem because an "agent" can have
hundreds or thousands variables and maybe a script only uses a small
number of them. Inserting all the variables every time can be very
time consuming.   I think Lua language can do that using "fallbacks"
or "tag methods" but I don't know if Python has something like that.

2.- What is the best way to use Python embedded in a C program?.
Instead of creating a new interpreter (Py_NewInterpreter();) I tried
using only a Python interpreter and creating a different state for
each thread:

   interp = PyThreadState_New(PyMainThreadState->interp);

This solution was much worse than creating different Interpreters.  Is
it normal?
 
3.- Are there any optimization tricks to use in embedded Python to
improve performance?

4.- Have you used any other script language for embedding in a
multithreaded program? I'm looking for something portable, fast and
free for commercial use.  I have tested Python and Lua and Lua seems
more suitable (and much faster) for simple embedding in a
multithreaded program.

Best regards,

Jordi