[capi-sig] Creating type object dynamically in run-time

[Mateusz Loskot]

On 10 May 2012 13:00, Stefan Behnel <python_capi at behnel.de> wrote:
> Mateusz Loskot, 10.05.2012 13:47:
>> On 10 May 2012 12:28, Stefan Behnel wrote:
>>> Mateusz Loskot, 10.05.2012 13:24:
>>>>
>>>> I'm writing fairly complex structure of Python extensions directly using
>>>> Python C API (version 3+ only). I define some type objects with statically,
>>>> according to the canonical Noddy example presented in the docs.
>>>> Some type objects have to be defined/composed dynamically in run-time.
>>>
>>> Do you really need a type or is a Python class enough? The letter would be
>>> much easier to create.
>>
>> I'm not sure I understand.
>> Aren't terms "class" and "type" names of the same concept, since Python 2.2?
>
> With "type" I meant "extension type", i.e. the one you define using C structs.

So, I probably misused "object type" as a shortcut referring
to the pair of struct + PyTypeObject instance:

typedef struct {
    PyObject_HEAD
} noddy_NoddyObject;

static PyTypeObject noddy_NoddyType = { ... }

> The alternative would be a simple Python class as you would get with
>
>   class MyClass(object): pass

This is clear to me. I simply was confused because AFAIK there is no
class while using Python C API. As mentioned before, I follow Eli Bendersky's:

"
Note that when we create new types using the C API of CPython,
there’s no "class" mentioned – we create a new "type", not a new "class".
"

>> What is the noddy_NoddyType in the "Defining New Types" example?
>
> That's an extension type.

Hmm, in the Python manual
http://docs.python.org/release/3.2.2/extending/newtypes.html

I'm reading this:

"Moving on, we come to the crunch — the type object.

static PyTypeObject noddy_NoddyType = {
...
}
"

What makes me interpret it as noddy_NoddyType is the type object.

Simply, I understand that the pair of the
struct noddy_NoddyObject and noddy_NoddyType
together state the Python extension type,

Am I still confusing the terminology?

>> In my system, I'm embedding Python and I add custom Python extension
>> too (let's call it 'emb')
>> So, users of my embedded Python have access to 'emb' module.
>> The 'emb' module defines number of types, some are defined statically,
>> as the noddy_NoddyType, so users can instantiate it
>>
>> n = emb.Noddy()
>> n.bar() # method defined statically in methods table of noddy_NoddyType
>>
>> Now, I'd like to add some types which are generated in run-time, way
>> before the 'emb' module is appended to inittab and embedded Python is
>> initialised.
>
> Why would you want to do that before initialising the Python runtime?

The 'emb' must be added to inittab before Python is initialised (that's what the
manual says, isn't it.) So, the scheme is this:

/* 1. Dynamically generate emb.GeneratedNoddy */
... /* Trying to figure out how */

/* 2. Register 'emb' module as built-in */
PyImport_AppendInittab("emb", &PyInit_emb);

/* 3. Initialise Python */
Py_Initialize();

/* 4. we're ready to use */
...

/* 5. Clean-up */
Py_Finalize();


Does it make sense?

>> And, I'd like to enable users to instantiate them in the same way as
>> Noddy above:
>>
>> d = emb.GeneratedNoddy()
>>
>> or allow users to use and access
>>
>> d  = emb.foo()
>> d.bar() # added dynamically in run-time during emb.GeneratedNoddy composition
>>
>> where:
>>
>> type(d)
>> <class 'emb.GeneratedNoddy'>
>>
>> I hope it makes my intentions clear.
>
> Not clear enough. The question is what your dynamically created types
> should do and provide. Would they need to be implemented in C (not just
> their methods but the types themselves!) or would a Python implementation
> suffice, potentially with methods implemented in C?

OK, I think I see where is the gap in my explanation.

0. The generated objects will define wrappers for existing C/C++ API.

1. I have defined contained object which needs extra steps to
initialise within C/C++
Namely, I need to initialise members like pointer_to_some_wrapped_api_element,
it may be some arbitrary data or PyCapsule, etc.

/* statically
typedef struct {
    PyObject_HEAD
    /* members initialised
    PyObject* pointer_to_some_wrapped_api_element;
} GeneratedNoddyObject;

2. The corresponding GeneratedNoddyType is generated dynamically,
with added set of methods, etc.

3. The GeneratedNoddyType is exposed as emb.GeneratedNoddy.

AFAIU, generating emb.GeneratedNoddy dynamically in C makes it easier to
compose the 'emb' module.

Alternatively, I can generate Python classes (script in text form),
but there is one inconvenience.
The only way to expose such classes I know is to compile and import as
*separate* module
using PyImport_ExecCodeModule (e.g. with name '_privemb'. So, the
resulting structure would be:

emb
emb.Noddy
emb._privemb
emb._privemb.GeneratedNoddy

But, I'd like to achieve flat structure:

emb
emb.Noddy
emb.GeneratedNoddy

So, presumably I would have to walk the dictionary of emb._privemb and forward
generated classes as attributes of 'emb'. Am I correct?


Given that complication of loading classes from textual form through
intermediate module,
I thought using Python C API to generate extension types is better.
(I don't mine dealing with C code verbosity and complexity.)

> You may get away with using normal Python classes that inherit from an
> extension type, for example. In that case, I'd also advise you to take a
> look at Cython, because that makes both the implementation of extension
> types and the creation of Python classes inheriting from them as easy as Py.

Sounds interesting, I will take a look at it.

However, I'd really like to learn canonical means of generating extension types
dynamically using plain Python C API.
Long story short, I assume I'm looking for Python C API equivalent of
using type() function.

Thank you Stefan!

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net