[Cython] CEP 1001 - Custom PyTypeObject extensions

[mark florisson]

On 16 May 2012 14:03, mark florisson <markflorisson88 at gmail.com> wrote:
> On 16 May 2012 12:20, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>> On 05/14/2012 08:01 PM, Robert Bradshaw wrote:
>>>
>>> On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith<njs at pobox.com>  wrote:
>>>>
>>>> On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn
>>>> <d.s.seljebotn at astro.uio.no>  wrote:
>>>>>
>>>>> On 05/14/2012 01:34 PM, Stefan Behnel wrote:
>>>>>>
>>>>>>
>>>>>> Dag Sverre Seljebotn, 13.05.2012 21:37:
>>>>>>>
>>>>>>>
>>>>>>> Anyway, thanks for the heads up, this seems to need a bit more work.
>>>>>>> Input
>>>>>>> from somebody more familiar with this corner of the CPython API very
>>>>>>> welcome.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Wouldn't you consider python-dev an appropriate place to discuss this?
>>>>>
>>>>>
>>>>>
>>>>> Propose something for a PEP that's primarily useful to Cython without
>>>>> even
>>>>> understanding the full implications myself first?
>>>>>
>>>>> I'd rather try to not annoy people; I figured the time I have the
>>>>> CPython
>>>>> patches ready and tested is the time I ping python-dev...
>>>>
>>>>
>>>> If you want to eventually propose a PEP, you really really really
>>>> should be talking to them before. Otherwise you'll get everything
>>>> worked out just the way you want and they'll be like "what is this?
>>>> re-do it all totally differently". And they might be wrong, but then
>>>> you have to reconstruct for them the whole debate and reasoning
>>>> process and implicit assumptions that you're making and not realizing
>>>> you need to articulate, so easier to just get all the interested
>>>> people at the table to begin with. And they might be right, in which
>>>> case you just wasted however much time digging yourself into a hole
>>>> and reverse-engineering bits of CPython.
>>>>
>>>> Don't propose it as a PEP, just say "hey, we have this problem and
>>>> these constraints, and we're thinking we could solve them by something
>>>> like this; but of course that has these limitations, so I dunno. What
>>>> do you think?" And expect to spend some time figuring out what your
>>>> requirements actually are (even if you think you know already, see
>>>> above about implicit assumptions).
>>>
>>>
>>> I personally think it's a great idea to bounce ideas around here first
>>> before going to python-dev, especially as a PEP wouldn't get in until
>>> 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near
>>> term. That doesn't preclude presenting the problem and proposed
>>> solution on python-dev as well, but the purpose of this thread seems
>>> to be to think about it some, including how we're going to support
>>> things in the short term, not nail down an exact PEP for Python to
>>> accept at face value. I think we're at a point we can ping python-dev
>>> now though.
>>>
>>> To be more futureproof, we'd want an offset to PyExtendedTypeObject
>>> rather than assuming it exists at the end of PyTypeObject, but I don't
>>> see a good place to store this information, so assuming it's right
>>> there based on a bit in the flag seems a reasonable way forward.
>>
>>
>> So I posted on python-dev.
>>
>> There's a lot of "You don't need to do this"; but here's an idea I got
>> that's inspired by that discussion: We could use tp_getattr (and call it
>> directly), but pass in an interned char* which Python code can never get
>> hold of, and then that could return a void* (casted through a PyObject*, but
>> it would not be).
>
> Would we want to support monkey patching these interfaces? If so, this
> mechanism would be a bit harder than reallocing a pointer, although I
> guess a closure chain of tp_getattr functions would work :). But I
> think we want GIL-less access anyway right, which means neither
> approach would work unsynchronized.
>
>> Another alternative is to somehow handshake on a metaclass implementation;
>> and different Cython modules/NumPy/SciPy etc. would inherit from it. But
>> apart from that handshaking, and having to use a metaclass and make
>> everything more complicated for C implementors of the spec, it gives you a
>> more expensive check than just checking a flag.
>
> I agree that the flag is much easier, if you have a metaclass the
> question is again in which module to store it to get a cross-module
> working typecheck. On the other hand, if the header file provides an
> easy way to import the metaclass (monkeypatched on some module or
> living in its own module), and to allocate type instances given a
> (statically allocated) type, that would be more future-proof and
> elegant. I don't think it would be much slower, it's doing
> 'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'.

Sorry, I think you mentioned something in related in the CEP, I
couldn't find it from the enhancement list (I should have followed
your originally posted link :). So that's a good point, there are
several issues:

    - subclasses of the type exposing the interface (I don't think
that can be handled through tp_flags)
    - in the case of a metaclass approach, subclassing the metaclass
itself means type instances will no longer match the MYMETA pointer
    - exposing interfaces on metaclasses (I think this one is listed in the CEP)

I suppose the last case could either be disallowed, or one could
provide a custom tp_alloc and store (a pointer to) extra information
ahead of the object, taking care to precede any GC information. This
is pretty hacky though.

> I think for the bit flag the interface won't span subclasses, whereas
> the metaclass approach would allow subclassing but not subclassing of
> the metaclass itself (unless you instantiate the metaclass through
> itself, and check the interface against the metaclass, which only
> means the metaclass of the metaclass isn't subclassable :) (this would
> also mean a more expensive check)).
>
> I think if we implement CEP 1000, we will at that time have a generic
> way to optimize and hoist none checking/bounds checking etc, which
> will also allow us to optimize signature matching, which would mean
> the matching and unpacking isn't as performance critical. JIT
> compilers could take a similar approach.
>
>> I like a flag bit much better. I still hope that somebody more understanding
>> comes along, argues our case, and gets bit 22 reserved for our purpose :-)
>>
>> Dag
>>
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel