Loading modules from files through C++

[Stefan Behnel]

Hi,

please avoid top-posting.


Roland Plüss, 17.05.2014 15:49:
> On 05/17/2014 03:26 PM, Stefan Behnel wrote:
>> Roland Plüss, 17.05.2014 15:00:
>>> On 05/17/2014 01:58 PM, Stefan Behnel wrote:
>>>> Roland Plüss, 17.05.2014 02:27:
>>>>> I'm using Python in an embedded situation. In particular I have to load
>>>>> python scripts through a memory interface so regular python module
>>>>> loading can not be used. I got working so far a module loader object
>>>>> I've added using C++ to sys.meta_path . Now I'm totally stuck at the
>>>>> finally loading step.
>>>>>
>>>>> I've got this a C++ loader method "load_module(fullname)" which does
>>>>> load the requested module script files into a null-terminated string. I
>>>>> know that "load_module" has to return the module PyObject*. But I can't
>>>>> get the python source in the c-string into a module PyObject*.
>>>>> [...]
>>>>> Can anybody help how in gods name one is supposed to create a module
>>>>> from an in-memory c-string when called from within load_module (or
>>>>> anywhere)?
>>>> Looks like you want to implement a SourceLoader:
>>>>
>>>> https://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader
>>>>
>>>> I recommend implementing this in Python code instead of C code, though.
>>>> Much easier. Cython can help with the integration between both.
>>> That doesn't work in 2.x, doesn't it?
>> Is there a reason you have to use Py2?
>>
>> Anyway, PEP 302 predates Py3 by a couple of years:
>>
>> http://legacy.python.org/dev/peps/pep-0302/
>
> I'm willing to go to Py3 but only if the solution to the problem is
> simpler than getting it fixed in Py2. So some questions first:
> 
> - does this importlib stuff you showed there apply to C++ land (I need
> to fully drive it from C++ not Python code)?

As I said, implementing this in Python code is much simpler than doing it
in C/C++ code. Basically, stop where you got the C string and do the rest
in Python. All your C code has to do is to take a module lookup request
from your custom Python module Finder and return a byte string with the
code. Then let your Python code wrap that in a Loader and return it to the
import machinery.


> - is the C++ land of Py3 similar to Py2 or totally different?

Much the same, just slight differences. As I said, Cython can help with the
integration and would mostly cover the differences for you.

Stefan