Loading modules from files through C++

[Stefan Behnel]

Roland Plüss, 17.05.2014 18:28:
> On 05/17/2014 05:49 PM, Stefan Behnel wrote:
>> Roland Plüss, 17.05.2014 17:28:
>>> On 05/17/2014 04:01 PM, Stefan Behnel wrote:
>>>> Roland Plüss, 17.05.2014 15:49:
>>>>> On 05/17/2014 03:26 PM, Stefan Behnel wrote:
>>>>>> Roland Plüss, 17.05.2014 15:00:
>>>>>>> On 05/17/2014 01:58 PM, Stefan Behnel wrote:
>>>>>>>> Roland Plüss, 17.05.2014 02:27:
>>>>>>>>> I'm using Python in an embedded situation. In particular I have to load
>>>>>>>>> python scripts through a memory interface so regular python module
>>>>>>>>> loading can not be used. I got working so far a module loader object
>>>>>>>>> I've added using C++ to sys.meta_path . Now I'm totally stuck at the
>>>>>>>>> finally loading step.
>>>>>>>>>
>>>>>>>>> I've got this a C++ loader method "load_module(fullname)" which does
>>>>>>>>> load the requested module script files into a null-terminated string. I
>>>>>>>>> know that "load_module" has to return the module PyObject*. But I can't
>>>>>>>>> get the python source in the c-string into a module PyObject*.
>>>>>>>>> [...]
>>>>>>>>> Can anybody help how in gods name one is supposed to create a module
>>>>>>>>> from an in-memory c-string when called from within load_module (or
>>>>>>>>> anywhere)?
>>>>>>>> Looks like you want to implement a SourceLoader:
>>>>>>>>
>>>>>>>> https://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader
>>>>>>>>
>>>>>>>> I recommend implementing this in Python code instead of C code, though.
>>>>>>>> Much easier. Cython can help with the integration between both.
>>>>>>> That doesn't work in 2.x, doesn't it?
>>>>>> Is there a reason you have to use Py2?
>>>>>>
>>>>>> Anyway, PEP 302 predates Py3 by a couple of years:
>>>>>>
>>>>>> http://legacy.python.org/dev/peps/pep-0302/
>>>>> I'm willing to go to Py3 but only if the solution to the problem is
>>>>> simpler than getting it fixed in Py2. So some questions first:
>>>>>
>>>>> - does this importlib stuff you showed there apply to C++ land (I need
>>>>> to fully drive it from C++ not Python code)?
>>>> As I said, implementing this in Python code is much simpler than doing it
>>>> in C/C++ code. Basically, stop where you got the C string and do the rest
>>>> in Python. All your C code has to do is to take a module lookup request
>>>> from your custom Python module Finder and return a byte string with the
>>>> code. Then let your Python code wrap that in a Loader and return it to the
>>>> import machinery.
>>> I don't get how this is supposed to work. I'm running it as fully
>>> embedded Python. There is no main script. The builtin modules are added
>>> as C++ bound classes and a user made main script is loaded but not run
>>> directly (I'm hooking into a create object). For this purpose I load the
>>> script module using C++ code using PyImport_ImportModule(moduleName). At
>>> this time the module loading code has to kick in already (I've added
>>> this one by C++ too before). The problem is now that in this call I end
>>> up in my C++ loader version where there is no Python script involved. I
>>> came to the conclusion that I can solve this only by having the C++ end
>>> properly load the module. I could add Python code with
>>> PyRun_SimpleString but then I'm down to the same problem as before: how
>>> to evaluate code so it is attached to a module or type-class? As I
>>> understand it the problem is the same as before just pushed around a bit.
>> No, just run some Python code (using PyRun_SimpleString() if you have to)
>> and let it do whatever you like. Such as, defining a Finder class and
>> injecting it into the import hook. Just provide it with the entry point of
>> your C++ loader as a (CFunction) object when you execute it, and then let
>> it call that function at need whenever the Finder gets executed.
>>
>> Alternatively, compile your Python integration code with Cython and link it
>> into your main program as yet another binary extension module.
>
> I'm not using Cython so that's out of question. Concerning the injection
> how would this work? From the PEP I assume it would have to look like this:
> 
> # CODE #
> import sys
> class VFSModuleLoader:
>    def find_module(fullname, path=None):
>       return self if VFS.exists( vfsPathFromFullname(fullname) ) else None
>    def load_module(fullname):
>       sourceCode = VFS.read( vfsPathFromFullname(fullname))
>       ???
> # CODE #
> 
> How does ??? work? If I use "eval" I end up with the code inside
> VFSModuleLoader.load_modules as context but it should go into the global
> context as an own module.

Yeah, well, the import machinery is rather badly exposed in Py2. This got
much cleaner in Py3, especially 3.3/3.4.

Here's a hacky way to do it in Py2:

    import imp
    module = imp.new_module(module_name)
    module.__importer__ = self
    exec source_code in module.__dict__
    return module

Found here:

http://hg.python.org/cpython/file/568041fd8090/Lib/imputil.py#l284

You may have to set a couple of more special attributes on the module
(__name__? __path__? __package__? others?), but the above should at least work.

Stefan