From dhyan at nataraj.su  Fri Jul 22 03:39:09 2022
From: dhyan at nataraj.su (Nikolay Shaplov)
Date: Fri, 22 Jul 2022 10:39:09 +0300
Subject: [C++-sig] libboost-python: how to create module in C++-runtime and
 use it from python code, in python 3.x
Message-ID: <5511655.W1rWVi7al0@thinkpad-pgpro>

I have a piece of software that embeds python using libboost-python. It 
creates a python module in C++ runtime, and then imports it from python code.

For python2.7 this works well.
See test_module.py and test27.cpp attached.
On Debian Stretch it can be build using command 

g++  test27.cpp  -I /usr/include/x86_64-linux-gnu/python2.7 -I /usr/include/
python2.7 -lboost_python -lboost_system -lpython2.7

As I've said It works as expected. Python code imports _hello_provider module 
that have been created in C++ code, and uses provide_hello() for printing.

I've tried to port it to python 3.x, and did not succeed.

I managed to write test3x.cpp, it build well on Debian Buster using command

g++ test3x.cpp  -I /usr/include/x86_64-linux-gnu/python3.9 -I /usr/include/
python3.9 -lpython3.9  -lboost_python39 -lboost_system

but it fails when I try to run it.

Traceback (most recent call last):
  File "/usr/lib/python3.9/imp.py", line 169, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 613, in _exec
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "test_module.py", line 1, in <module>
    import _hello_provider
ModuleNotFoundError: No module named '_hello_provider'

for some reason "native" python module does not see _hello_provider that have 
been created in runtime.

I guess I am doing something wrong, but I am no expert in python and python 
embedding, and I did not managed to find any good example in the internet.

So I guess I need some help here.

P.S. If remove "import" related part from examples, it also works well, so the 
only task here is to make in properly visible...



-- 
Nikolay Shaplov aka Nataraj
Fuzzing Engineer at Postgres Professional
Matrix IM: @dhyan:nataraj.su
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test3x.cpp
Type: text/x-c++src
Size: 873 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/cplusplus-sig/attachments/20220722/06cc9e4e/attachment.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test27.cpp
Type: text/x-c++src
Size: 872 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/cplusplus-sig/attachments/20220722/06cc9e4e/attachment-0001.cpp>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_module.py
Type: text/x-python
Size: 206 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/cplusplus-sig/attachments/20220722/06cc9e4e/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part.
URL: <https://mail.python.org/pipermail/cplusplus-sig/attachments/20220722/06cc9e4e/attachment.sig>

From dhyan at nataraj.su  Sat Jul 23 03:06:17 2022
From: dhyan at nataraj.su (Nikolay Shaplov)
Date: Sat, 23 Jul 2022 10:06:17 +0300
Subject: [C++-sig] libboost-python: how to create module in C++-runtime
 and use it from python code, in python 3.x
In-Reply-To: <5511655.W1rWVi7al0@thinkpad-pgpro>
References: <5511655.W1rWVi7al0@thinkpad-pgpro>
Message-ID: <7899674.L5geQnCnUD@thinkpad-pgpro>

? ?????? ?? ???????, 22 ???? 2022 ?. 10:39:09 MSK ???????????? Nikolay Shaplov 
???????:

I finally managed to fix it. 
Key point here was, that in python3.x you should manually add such modules to 
inittab(whatever it is) using PyImport_AppendInittab

Finally my sample that builds with both 2.7 anx 3.x pythons looks like this

#include <boost/python.hpp>

std::string provide_hello() {
    return "hello world provided";
}

BOOST_PYTHON_MODULE(_hello_provider) {
    using namespace boost::python;
    def("provide_hello", &provide_hello);
}

int main()
{
#if PY_MAJOR_VERSION >= 3
  PyImport_AppendInittab((char*)"_hello_provider", PyInit__hello_provider);
#else
  PyImport_AppendInittab((char*)"_hello_provider", init_hello_provider);
#endif
  Py_InitializeEx(0);

  try {
    boost::python::object modImp = boost::python::import("imp");
    PyImport_AddModule("test_module");

    modImp.attr("load_source")("test_module", "test_module.py");

    boost::python::exec("print(test_var)", 
boost::python::import("test_module").attr("__dict__"));
    boost::python::exec("print(hello_static())", 
boost::python::import("test_module").attr("__dict__"));
    boost::python::exec("print(hello_provided())", 
boost::python::import("test_module").attr("__dict__"));
  }
  catch (const boost::python::error_already_set&)
  {
    PyErr_Print();
  }
  Py_Finalize();
}

Special thanks to https://github.com/TNG/boost-python-examples/tree/main/10-Embedding authors and O02eg from linux.org.ru
-- 
Nikolay Shaplov aka Nataraj
Fuzzing Engineer at Postgres Professional
Matrix IM: @dhyan:nataraj.su
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part.
URL: <https://mail.python.org/pipermail/cplusplus-sig/attachments/20220723/3b528ada/attachment.sig>