[Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded

[Campbell Barton]

On Sun, Jul 26, 2009 at 11:01 AM, Joshua Haberman<joshua at reverberate.org> wrote:
> I'm writing a C Python extension that needs to generate PyTypeObjects
> dynamically.  Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded
> in a way that makes it difficult to achieve this goal.
>
> The documentation for Pt_TPFLAGS_HEAPTYPE says:
>
> Py_TPFLAGS_HEAPTYPE
>
>  This bit is set when the type object itself is allocated
>  on the heap. In this case, the ob_type field of its
>  instances is considered a reference to the type, and the
>  type object is INCREF’ed when a new instance is created,
>  and DECREF’ed when an instance is destroyed (this does not
>  apply to instances of subtypes; only the type referenced
>  by the instance’s ob_type gets INCREF’ed or DECREF’ed).
>
> This sounds like exactly what I want.  I want my type object INCREF'd
> and DECREF'd by its instances so it doesn't leak or get deleted
> prematurely.  If this were all that Py_TPFLAGS_HEAPTYPE did, it would
> work great for me.
>
> Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean
> "user-defined type" (as opposed to a built-in type).  It controls
> numerous subtle behaviors such as:
>
> - whether the type's name is module.type or just type.
> - whether you're allowed to set __name__, __module__, or __bases__ on the type.
> - whether you're allowed to set __class__ on instances of this type.
> - whether the module name comes from the type name or the __module__ attribute.
> - whether it will use type->tp_doc as the docstring
> - whether its repr() calls it a "class" or a "type".
> - whether you can set attributes of the type.
> - whether someone is attempting the Carlo Verre hack.
>
> So I'm stuck with an unenviable choice.  I think the lesser of two evils
> is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will
> happen is that my types will leak.  This is not as bad as having someone
> set __class__ on one of my instances, or set attributes on my type, etc.
>
> Ideally the interpreter would have a separate flag like
> Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but
> still make it possible to have dynamically generated built-in types get
> garbage collected appropriately.
>
> At the very least, the documentation I cited above should make it clear
> that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets
> INCREF'd and DECREF'd.  Based on the list of behaviors I discovered
> above, it is almost certainly not correct for a C exension type to be
> declared with Py_TPFLAGS_HEAPTYPE.
>
> Josh

Hi Joshua, recently I also needed to dynamically make subtypes from C,

I tried 2 ways of doing this, one is to do the C equivalent of calling
type("name",(bases,...), dict)  and the other is to malloc()
PyTypeObject's, fill in the slots and run PyType_Ready on them to
initialize them.

It seems the first is the expected way to make your own types so I
assume thats what your doing?, Just wondering because if you do it the
second way I think youll have more control and the types will be more
limited (like internal types).

I'm not expert enough in this area to know if malloc'ing PyTypeObject
and initializing has some other problems.

- Campbell