Loading modules from files through C++

[Stefan Behnel]

Roland Plüss, 17.05.2014 17:28:
> On 05/17/2014 04:01 PM, Stefan Behnel wrote:
>> Roland Plüss, 17.05.2014 15:49:
>>> On 05/17/2014 03:26 PM, Stefan Behnel wrote:
>>>> Roland Plüss, 17.05.2014 15:00:
>>>>> On 05/17/2014 01:58 PM, Stefan Behnel wrote:
>>>>>> Roland Plüss, 17.05.2014 02:27:
>>>>>>> I'm using Python in an embedded situation. In particular I have to load
>>>>>>> python scripts through a memory interface so regular python module
>>>>>>> loading can not be used. I got working so far a module loader object
>>>>>>> I've added using C++ to sys.meta_path . Now I'm totally stuck at the
>>>>>>> finally loading step.
>>>>>>>
>>>>>>> I've got this a C++ loader method "load_module(fullname)" which does
>>>>>>> load the requested module script files into a null-terminated string. I
>>>>>>> know that "load_module" has to return the module PyObject*. But I can't
>>>>>>> get the python source in the c-string into a module PyObject*.
>>>>>>> [...]
>>>>>>> Can anybody help how in gods name one is supposed to create a module
>>>>>>> from an in-memory c-string when called from within load_module (or
>>>>>>> anywhere)?
>>>>>> Looks like you want to implement a SourceLoader:
>>>>>>
>>>>>> https://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader
>>>>>>
>>>>>> I recommend implementing this in Python code instead of C code, though.
>>>>>> Much easier. Cython can help with the integration between both.
>>>>> That doesn't work in 2.x, doesn't it?
>>>> Is there a reason you have to use Py2?
>>>>
>>>> Anyway, PEP 302 predates Py3 by a couple of years:
>>>>
>>>> http://legacy.python.org/dev/peps/pep-0302/
>>> I'm willing to go to Py3 but only if the solution to the problem is
>>> simpler than getting it fixed in Py2. So some questions first:
>>>
>>> - does this importlib stuff you showed there apply to C++ land (I need
>>> to fully drive it from C++ not Python code)?
>> As I said, implementing this in Python code is much simpler than doing it
>> in C/C++ code. Basically, stop where you got the C string and do the rest
>> in Python. All your C code has to do is to take a module lookup request
>> from your custom Python module Finder and return a byte string with the
>> code. Then let your Python code wrap that in a Loader and return it to the
>> import machinery.
>
> I don't get how this is supposed to work. I'm running it as fully
> embedded Python. There is no main script. The builtin modules are added
> as C++ bound classes and a user made main script is loaded but not run
> directly (I'm hooking into a create object). For this purpose I load the
> script module using C++ code using PyImport_ImportModule(moduleName). At
> this time the module loading code has to kick in already (I've added
> this one by C++ too before). The problem is now that in this call I end
> up in my C++ loader version where there is no Python script involved. I
> came to the conclusion that I can solve this only by having the C++ end
> properly load the module. I could add Python code with
> PyRun_SimpleString but then I'm down to the same problem as before: how
> to evaluate code so it is attached to a module or type-class? As I
> understand it the problem is the same as before just pushed around a bit.

No, just run some Python code (using PyRun_SimpleString() if you have to)
and let it do whatever you like. Such as, defining a Finder class and
injecting it into the import hook. Just provide it with the entry point of
your C++ loader as a (CFunction) object when you execute it, and then let
it call that function at need whenever the Finder gets executed.

Alternatively, compile your Python integration code with Cython and link it
into your main program as yet another binary extension module.

Stefan