Retrieving result from embedded execution

[Chris Angelico]

On Sat, May 12, 2012 at 3:36 AM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> On Fri, May 11, 2012 at 12:45 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> However, have you tried using a pipe for them instead of a real file? That
>> would allow you to retrieve the output without needing to pass through a
>> file in the file system. You can also replace sys.stdout/err with arbitrary
>> objects in Python space, which could then forward the output in any way you
>> want.
>
> Surely a pipe would cause a deadlock if the pipe fills up if Python
> and the C program are running in the same process/thread?

Yes, it would; how much data are you looking at retrieving, and is it
all at script-end or do you need to process it concurrently? If the
latter, then your two best options are a pipe (and running the Python
script in another thread, possibly a separate process) or a callback
of some sort (in which case the script hands you a line or block of
output and says "Deal with this and get back to me") - eg by replacing
sys.stdout, or defining a function that the Python script calls
explicitly.

This is starting to sound similar to something I did at work. A C++
program uses a block of Python code as a sort of uber-config-file. It
would initialize the Python environment, import some modules, etc,
once, and keep the globals around (this allows Python globals to be
retained). Every time script-configurable code is to be run:

1) Prepare a dictionary called Input with a whole lot of
parameters/information/etc
2) Create an empty dictionary and call it Output
3) Execute a block of code (retrieved from the database)
4) Inspect the Output dictionary and use that to control subsequent behaviour.

As a concept, it worked quite well. I do not, however, advise doing
this if your Python scripts come from untrusted sources, as it is
impossible to guarantee safety. (For that reason we actually shifted
to Javascript for that project.) But if you're using this as a
powerful and simple config-file and you know you can trust everything
that will be run, then it's a pretty awesome way of doing things. It
takes deliberately malicious code to break the system.

ChrisA