[Cython] [Python-Dev] C-level duck typing

[Dag Sverre Seljebotn]

On 05/28/2012 01:24 PM, Nathaniel Smith wrote:
> On Mon, May 28, 2012 at 12:09 PM, mark florisson
> <markflorisson88 at gmail.com>  wrote:
>> On 28 May 2012 12:01, Nathaniel Smith<njs at pobox.com>  wrote:
>>> On Mon, May 28, 2012 at 11:55 AM, mark florisson
>>> <markflorisson88 at gmail.com>  wrote:
>>>> On 28 May 2012 11:41, Nathaniel Smith<njs at pobox.com>  wrote:
>>>>> On Mon, May 28, 2012 at 10:13 AM, mark florisson
>>>>> <markflorisson88 at gmail.com>  wrote:
>>>>>> On 28 May 2012 09:54, mark florisson<markflorisson88 at gmail.com>  wrote:
>>>>>>> On 27 May 2012 23:12, Nathaniel Smith<njs at pobox.com>  wrote:
>>>>>>>> On Sun, May 27, 2012 at 10:24 PM, Dag Sverre Seljebotn
>>>>>>>> <d.s.seljebotn at astro.uio.no>  wrote:
>>>>>>>>> On 05/18/2012 10:30 AM, Dag Sverre Seljebotn wrote:
>>>>>>>>>>
>>>>>>>>>> On 05/18/2012 12:57 AM, Nick Coghlan wrote:
>>>>>>>>>>>
>>>>>>>>>>> I think the main things we'd be looking for would be:
>>>>>>>>>>> - a clear explanation of why a new metaclass is considered too complex a
>>>>>>>>>>> solution
>>>>>>>>>>> - what the implications are for classes that have nothing to do with the
>>>>>>>>>>> SciPy/NumPy ecosystem
>>>>>>>>>>> - how subclassing would behave (both at the class and metaclass level)
>>>>>>>>>>>
>>>>>>>>>>> Yes, defining a new metaclass for fast signature exchange has its
>>>>>>>>>>> challenges - but it means that *our* concerns about maintaining
>>>>>>>>>>> consistent behaviour in the default object model and avoiding adverse
>>>>>>>>>>> effects on code that doesn't need the new behaviour are addressed
>>>>>>>>>>> automatically.
>>>>>>>>>>>
>>>>>>>>>>> Also, I'd consider a functioning reference implementation using a custom
>>>>>>>>>>> metaclass a requirement before we considered modifying type anyway, so I
>>>>>>>>>>> think that's the best thing to pursue next rather than a PEP. It also
>>>>>>>>>>> has the virtue of letting you choose which Python versions to target and
>>>>>>>>>>> iterating at a faster rate than CPython.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This seems right on target. I could make a utility code C header for
>>>>>>>>>> such a metaclass, and then the different libraries can all include it
>>>>>>>>>> and handshake on which implementation becomes the real one through
>>>>>>>>>> sys.modules during module initialization. That way an eventual PEP will
>>>>>>>>>> only be a natural incremental step to make things more polished, whether
>>>>>>>>>> that happens by making such a metaclass part of the standard library or
>>>>>>>>>> by extending PyTypeObject.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So I finally got around to implementing this:
>>>>>>>>>
>>>>>>>>> https://github.com/dagss/pyextensibletype
>>>>>>>>>
>>>>>>>>> Documentation now in a draft in the NumFOCUS SEP repo, which I believe is a
>>>>>>>>> better place to store cross-project standards like this. (The NumPy
>>>>>>>>> docstring standard will be SEP 100).
>>>>>>>>>
>>>>>>>>> https://github.com/numfocus/sep/blob/master/sep200.rst
>>>>>>>>>
>>>>>>>>> Summary:
>>>>>>>>>
>>>>>>>>>   - No common runtime dependency
>>>>>>>>>
>>>>>>>>>   - 1 ns overhead per lookup (that's for the custom slot *alone*, no
>>>>>>>>> fast-callable signature matching or similar)
>>>>>>>>>
>>>>>>>>>   - Slight annoyance: Types that want to use the metaclass must be a
>>>>>>>>> PyHeapExtensibleType, to make the binary layout work with how CPython makes
>>>>>>>>> subclasses from Python scripts
>>>>>>>>>
>>>>>>>>> My conclusion: I think the metaclass approach should work really well.
>>>>>>>>
>>>>>>>> Few quick comments on skimming the code:
>>>>>>>>
>>>>>>>> The complicated nested #ifdef for __builtin_expect could be simplified to
>>>>>>>>   #if defined(__GNUC__)&&  (__GNUC__>  2 || __GNUC_MINOR__>  95)
>>>>>>>>
>>>>>>>> PyCustomSlots_Check should be called PyCustomSlots_CheckExact, surely?
>>>>>>>> And given that, how can this code work if someone does subclass this
>>>>>>>> metaclass?
>>>>>>>
>>>>>>> I think we should provide a wrapper for PyType_Ready, which just
>>>>>>> copies the pointer to the table and the count directly into the
>>>>>>> subclass. If a user then wishes to add stuff, the user can allocate a
>>>>>>> new memory region dynamically, memcpy the base class' stuff in there,
>>>>>>> and append some entries.
>>>>>>
>>>>>> Maybe we should also allow each custom type to set a deallocator,
>>>>>> since they are then heap types which can go out of scope. The
>>>>>> metaclass can then call this deallocator to deallocate the table.
>>>>>
>>>>> Custom types are plain old Python objects, they can use tp_dealloc.
>>>>>
>>>> If I set etp_custom_slots to something allocated on the heap, then the
>>>> (shared) metaclass would have to deallocate it. The tp_dealloc of the
>>>> type itself would be called for its instances (which can be used to
>>>> deallocate dynamically allocated memory in the objects if you use a
>>>> custom slot "pointer offset").
>>>
>>> Oh, I see. Right, the natural way to handle this would be have each
>>> user define their own metaclass with the behavior they want. Another
>>> argument for supporting multiple metaclasses simultaneously I guess...
>>>
>>> - N
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> That bludgeons your constant time type check.
>
> Not if you steal a flag, like the interpreter already does with
> Py_TPFLAGS_INT_SUBCLASS, Py_TPFLAGS_STRING_SUBCLASS, etc. I was
> referring to that argument I made earlier :-)
>
>> It's easier to just
>> reserve an extra slot for a deallocator pointer :) It would probably
>> be set to NULL in the common case anyway, since you allocate your
>> slots statically.

Subclassing: Note that even if all types has to have a PyHeapTypeObject 
structure, they are still statically allocated! So for statically 
created subclasses (which should be the majority of the cases), there's 
not going to be any deallocator...

I agree that there should be a PyExtensibleType_Ready. To keep 
allocating statically I propose that the subclass should leave some room 
open for slots from the superclass:

PyCustomSlot subclass_custom_slots[10] = {
   {SLOT_C, foo}, {SLOT_D, BAR}, {0,0}, ...
}

Then, fill in etp_count=2, etp_custom_slots=subclass_custom_slots, and 
then call PyExtensibleType_Ready(&Subclass_Type, 10); i.e., the number 
of total elements in etp_custom_slots is passed in.

One should always leave more room than one thinks one needs if the 
superclass is from another library...

Then, inheritance happens according to the following rules:

  - Slots are inherited from superclass
  - Slots in subclass with same ID overwrites superclass
  - Slots from superclass are put before slots from subclass
  - Exception raised if the number of final slots is larger than the 
limit passed in to PyExtensibleType_Ready.

(Whenever this is not sufficient, you can always manually munge the 
table after PyExtensibleType_Ready.)

Question: How to deal with possible flag bits in the ID?

Three approaches:

  a) Forget about the flags-in-ID idea; if you want flags, stick them in 
the data

  b) Embed a seperate variable for flags in every PyCustomSlot

  c) Standardize on a *hard* requirement on the bottom 8 bits being 
flags while the top 24 bits indicate incompatible slots; so for the 
purposes of inheritance, 0x12345601 would overwrite 0x12345600.

To me, b) is OK, but the 32 bit ID space is already so ridiculously huge 
that c) is a "why not"? -1 on a), it'd be rather tedious if the payload 
is an offset to the PyObject*.

Subclassing in heap-allocated types (subclasses Python side): It'd 
certainly be nice to completely ignore this for now and require making a 
sub-metaclass to support this (e.g., have tp_new parse some 
__customslots__ attribute in the class dict).

Hijacking a TP_FLAG: We could make the branch for a direct hit on 
metaclass comparison likely(), so that the branch checking tp_base on 
the metaclass unlikely(), which with branch prediction I think makes it 
very likely that there's no penalty for allowing sub-metaclasses (when 
you don't use them -- when you do, there's a slight penalty).

But here's another great argument in favour of a TP_FLAG bit: Consumers 
would then not need to import the metaclass or contain its definition 
(which is really only around in case the user imports the consumer 
before the provider...). This would make the header file the consumers 
need to bundle much lighter. So I think I'm +1.

At any rate, I would like the metaclass rendezvous to keep happening 
just because it's less confusing if "extensibletype is extensibletype" 
in general.

Anyway, the metaclass checking is a nice fallback if CPython uses all 
their flag bits.

Dag