Loading modules from files through C++

Sun May 18 15:41:03 EDT 2014

On 05/17/2014 07:05 PM, Stefan Behnel wrote:
> Roland Plüss, 17.05.2014 18:28:
>> On 05/17/2014 05:49 PM, Stefan Behnel wrote:
>>> Roland Plüss, 17.05.2014 17:28:
>>>> On 05/17/2014 04:01 PM, Stefan Behnel wrote:
>>>>> Roland Plüss, 17.05.2014 15:49:
>>>>>> On 05/17/2014 03:26 PM, Stefan Behnel wrote:
>>>>>>> Roland Plüss, 17.05.2014 15:00:
>>>>>>>> On 05/17/2014 01:58 PM, Stefan Behnel wrote:
>>>>>>>>> Roland Plüss, 17.05.2014 02:27:
>>>>>>>>>> I'm using Python in an embedded situation. In particular I have to load
>>>>>>>>>> python scripts through a memory interface so regular python module
>>>>>>>>>> loading can not be used. I got working so far a module loader object
>>>>>>>>>> I've added using C++ to sys.meta_path . Now I'm totally stuck at the
>>>>>>>>>> finally loading step.
>>>>>>>>>>
>>>>>>>>>> I've got this a C++ loader method "load_module(fullname)" which does
>>>>>>>>>> load the requested module script files into a null-terminated string. I
>>>>>>>>>> know that "load_module" has to return the module PyObject*. But I can't
>>>>>>>>>> get the python source in the c-string into a module PyObject*.
>>>>>>>>>> [...]
>>>>>>>>>> Can anybody help how in gods name one is supposed to create a module
>>>>>>>>>> from an in-memory c-string when called from within load_module (or
>>>>>>>>>> anywhere)?
>>>>>>>>> Looks like you want to implement a SourceLoader:
>>>>>>>>>
>>>>>>>>> https://docs.python.org/3.4/library/importlib.html#importlib.abc.SourceLoader
>>>>>>>>>
>>>>>>>>> I recommend implementing this in Python code instead of C code, though.
>>>>>>>>> Much easier. Cython can help with the integration between both.
>>>>>>>> That doesn't work in 2.x, doesn't it?
>>>>>>> Is there a reason you have to use Py2?
>>>>>>>
>>>>>>> Anyway, PEP 302 predates Py3 by a couple of years:
>>>>>>>
>>>>>>> http://legacy.python.org/dev/peps/pep-0302/
>>>>>> I'm willing to go to Py3 but only if the solution to the problem is
>>>>>> simpler than getting it fixed in Py2. So some questions first:
>>>>>>
>>>>>> - does this importlib stuff you showed there apply to C++ land (I need
>>>>>> to fully drive it from C++ not Python code)?
>>>>> As I said, implementing this in Python code is much simpler than doing it
>>>>> in C/C++ code. Basically, stop where you got the C string and do the rest
>>>>> in Python. All your C code has to do is to take a module lookup request
>>>>> from your custom Python module Finder and return a byte string with the
>>>>> code. Then let your Python code wrap that in a Loader and return it to the
>>>>> import machinery.
>>>> I don't get how this is supposed to work. I'm running it as fully
>>>> embedded Python. There is no main script. The builtin modules are added
>>>> as C++ bound classes and a user made main script is loaded but not run
>>>> directly (I'm hooking into a create object). For this purpose I load the
>>>> script module using C++ code using PyImport_ImportModule(moduleName). At
>>>> this time the module loading code has to kick in already (I've added
>>>> this one by C++ too before). The problem is now that in this call I end
>>>> up in my C++ loader version where there is no Python script involved. I
>>>> came to the conclusion that I can solve this only by having the C++ end
>>>> properly load the module. I could add Python code with
>>>> PyRun_SimpleString but then I'm down to the same problem as before: how
>>>> to evaluate code so it is attached to a module or type-class? As I
>>>> understand it the problem is the same as before just pushed around a bit.
>>> No, just run some Python code (using PyRun_SimpleString() if you have to)
>>> and let it do whatever you like. Such as, defining a Finder class and
>>> injecting it into the import hook. Just provide it with the entry point of
>>> your C++ loader as a (CFunction) object when you execute it, and then let
>>> it call that function at need whenever the Finder gets executed.
>>>
>>> Alternatively, compile your Python integration code with Cython and link it
>>> into your main program as yet another binary extension module.
>> I'm not using Cython so that's out of question. Concerning the injection
>> how would this work? From the PEP I assume it would have to look like this:
>>
>> # CODE #
>> import sys
>> class VFSModuleLoader:
>>    def find_module(fullname, path=None):
>>       return self if VFS.exists( vfsPathFromFullname(fullname) ) else None
>>    def load_module(fullname):
>>       sourceCode = VFS.read( vfsPathFromFullname(fullname))
>>       ???
>> # CODE #
>>
>> How does ??? work? If I use "eval" I end up with the code inside
>> VFSModuleLoader.load_modules as context but it should go into the global
>> context as an own module.
> Yeah, well, the import machinery is rather badly exposed in Py2. This got
> much cleaner in Py3, especially 3.3/3.4.
>
> Here's a hacky way to do it in Py2:
>
>     import imp
>     module = imp.new_module(module_name)
>     module.__importer__ = self
>     exec source_code in module.__dict__
>     return module
>
> Found here:
>
> http://hg.python.org/cpython/file/568041fd8090/Lib/imputil.py#l284
>
> You may have to set a couple of more special attributes on the module
> (__name__? __path__? __package__? others?), but the above should at least work.
>
> Stefan
>
>
This exec source_code in module.__dict__ , should this not also be
doable with PyEval_EvalCode? I've found a snippet like this:

# CODE #|
PyCodeObject*code =(PyCodeObject*)Py_CompileString(s,"test",Py_file_input);
PyObject*main_module =PyImport_AddModule("__main__");
PyObject*global_dict =PyModule_GetDict(main_module);
PyObject*local_dict =PyDict_New();
PyObject*obj =PyEval_EvalCode(code,global_dict,local_dict);|
# CODE #

If I would use a newly created module instead of PyImport_AddModule with
the appropriate name would this not evaluate the code (in my case a
string not a code object) in the right code space? Or did I
misunderstand the snippet there?

-- 
Yours sincerely
Plüss Roland

Leader and Head Programmer
- Game: Epsylon ( http://www.indiedb.com/games/epsylon )
- Game Engine: Drag[en]gine ( http://www.indiedb.com/engines/dragengine
, http://dragengine.rptd.ch/wiki )
- Normal Map Generator: DENormGen ( http://epsylon.rptd.ch/denormgen.php )
- As well as various Blender export scripts und game tools
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20140518/7c8c248b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20140518/7c8c248b/attachment.sig>