[capi-sig] Creating type object dynamically in run-time

[Stefan Behnel]

Mateusz Loskot, 10.05.2012 14:37:
> On 10 May 2012 13:00, Stefan Behnel wrote:
>> Mateusz Loskot, 10.05.2012 13:47:
>>> On 10 May 2012 12:28, Stefan Behnel wrote:
>>>> Mateusz Loskot, 10.05.2012 13:24:
>>>>>
>>>>> I'm writing fairly complex structure of Python extensions directly using
>>>>> Python C API (version 3+ only). I define some type objects with statically,
>>>>> according to the canonical Noddy example presented in the docs.
>>>>> Some type objects have to be defined/composed dynamically in run-time.
>>>>
>>>> Do you really need a type or is a Python class enough? The letter would be
>>>> much easier to create.
>>>
>>> I'm not sure I understand.
>>> Aren't terms "class" and "type" names of the same concept, since Python 2.2?
>>
>> With "type" I meant "extension type", i.e. the one you define using C structs.
> 
> So, I probably misused "object type" as a shortcut referring
> to the pair of struct + PyTypeObject instance:
> 
> typedef struct {
>     PyObject_HEAD
> } noddy_NoddyObject;
> 
> static PyTypeObject noddy_NoddyType = { ... }
> 
>> The alternative would be a simple Python class as you would get with
>>
>>   class MyClass(object): pass
> 
> This is clear to me. I simply was confused because AFAIK there is no
> class while using Python C API. As mentioned before, I follow Eli Bendersky's:
> 
> "
> Note that when we create new types using the C API of CPython,
> there’s no "class" mentioned – we create a new "type", not a new "class".
> "

I think the emphasis is on "when we create new types". The sentence above
sounds like a tautology to me (we obviously create a type and not a class
when we create a type).


>>> What is the noddy_NoddyType in the "Defining New Types" example?
>>
>> That's an extension type.
> 
> Hmm, in the Python manual
> http://docs.python.org/release/3.2.2/extending/newtypes.html
> 
> I'm reading this:
> 
> "Moving on, we come to the crunch — the type object.
> 
> static PyTypeObject noddy_NoddyType = {
> ...
> }
> "
> 
> What makes me interpret it as noddy_NoddyType is the type object.
> 
> Simply, I understand that the pair of the
> struct noddy_NoddyObject and noddy_NoddyType
> together state the Python extension type,
> 
> Am I still confusing the terminology?

I don't care all that much about terminology myself.


>>> In my system, I'm embedding Python and I add custom Python extension
>>> too (let's call it 'emb')
>>> So, users of my embedded Python have access to 'emb' module.
>>> The 'emb' module defines number of types, some are defined statically,
>>> as the noddy_NoddyType, so users can instantiate it
>>>
>>> n = emb.Noddy()
>>> n.bar() # method defined statically in methods table of noddy_NoddyType
>>>
>>> Now, I'd like to add some types which are generated in run-time, way
>>> before the 'emb' module is appended to inittab and embedded Python is
>>> initialised.
>>
>> Why would you want to do that before initialising the Python runtime?
> 
> The 'emb' must be added to inittab before Python is initialised (that's what the
> manual says, isn't it.) So, the scheme is this:
> 
> /* 1. Dynamically generate emb.GeneratedNoddy */
> ... /* Trying to figure out how */
> 
> /* 2. Register 'emb' module as built-in */
> PyImport_AppendInittab("emb", &PyInit_emb);
> 
> /* 3. Initialise Python */
> Py_Initialize();
> 
> /* 4. we're ready to use */
> ...
> 
> /* 5. Clean-up */
> Py_Finalize();
> 
> 
> Does it make sense?

No. You don't need to declare all types before hand. You can add a type (or
function, or name, or whatever) to the module dict at any time, just as you
can in Python. Whether that's a C implemented type of a normal Python class
(or something else) doesn't matter at that point.

The inittab mechanism is just there for convenience when everything *is*
statically defined at compile time (or at least module init time).


>>> And, I'd like to enable users to instantiate them in the same way as
>>> Noddy above:
>>>
>>> d = emb.GeneratedNoddy()
>>>
>>> or allow users to use and access
>>>
>>> d  = emb.foo()
>>> d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition
>>>
>>> where:
>>>
>>> type(d)
>>> <class 'emb.GeneratedNoddy'>
>>>
>>> I hope it makes my intentions clear.
>>
>> Not clear enough. The question is what your dynamically created types
>> should do and provide. Would they need to be implemented in C (not just
>> their methods but the types themselves!) or would a Python implementation
>> suffice, potentially with methods implemented in C?
> 
> OK, I think I see where is the gap in my explanation.
> 
> 0. The generated objects will define wrappers for existing C/C++ API.
> 
> 1. I have defined contained object which needs extra steps to
> initialise within C/C++
> Namely, I need to initialise members like pointer_to_some_wrapped_api_element,
> it may be some arbitrary data or PyCapsule, etc.
> 
> /* statically
> typedef struct {
>     PyObject_HEAD
>     /* members initialised
>     PyObject* pointer_to_some_wrapped_api_element;
> } GeneratedNoddyObject;
> 
> 2. The corresponding GeneratedNoddyType is generated dynamically,
> with added set of methods, etc.
> 
> 3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy.
> 
> AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to
> compose the 'emb' module.

Not at all. It's much easier to use a Python class than to do everything in
lengthy C-API declarations.


> Alternatively, I can generate Python classes (script in text form),
> but there is one inconvenience.
> The only way to expose such classes I know is to compile and import as
> *separate* module
> using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the
> resulting structure would be:
> 
> emb
> emb.Noddy
> emb._privemb
> emb._privemb.GeneratedNoddy

Wrong again. You can execute any Python code from your C code. Look for the
PyRun_*() functions.

Note that this is even easier in Cython, where you write Python code anyway
(instead of C code).


> Given that complication of loading classes from textual form through
> intermediate module,
> I thought using Python C API to generate extension types is better.
> (I don't mine dealing with C code verbosity and complexity.)

You should. The simpler the code, the easier it is to maintain it. Why else
would you want to embed Python instead of using C for everything?


>> You may get away with using normal Python classes that inherit from an
>> extension type, for example. In that case, I'd also advise you to take a
>> look at Cython, because that makes both the implementation of extension
>> types and the creation of Python classes inheriting from them as easy as Py.
> 
> Sounds interesting, I will take a look at it.
> 
> However, I'd really like to learn canonical means of generating extension types
> dynamically using plain Python C API.
> Long story short, I assume I'm looking for Python C API equivalent of
> using type() function.

Then call type().

Stefan