[Cython] CEP 1001 - Custom PyTypeObject extensions

[mark florisson]

On 16 May 2012 14:25, mark florisson <markflorisson88 at gmail.com> wrote:
> On 16 May 2012 14:03, mark florisson <markflorisson88 at gmail.com> wrote:
>> On 16 May 2012 12:20, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
>>> On 05/14/2012 08:01 PM, Robert Bradshaw wrote:
>>>>
>>>> On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith<njs at pobox.com>  wrote:
>>>>>
>>>>> On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn
>>>>> <d.s.seljebotn at astro.uio.no>  wrote:
>>>>>>
>>>>>> On 05/14/2012 01:34 PM, Stefan Behnel wrote:
>>>>>>>
>>>>>>>
>>>>>>> Dag Sverre Seljebotn, 13.05.2012 21:37:
>>>>>>>>
>>>>>>>>
>>>>>>>> Anyway, thanks for the heads up, this seems to need a bit more work.
>>>>>>>> Input
>>>>>>>> from somebody more familiar with this corner of the CPython API very
>>>>>>>> welcome.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Wouldn't you consider python-dev an appropriate place to discuss this?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Propose something for a PEP that's primarily useful to Cython without
>>>>>> even
>>>>>> understanding the full implications myself first?
>>>>>>
>>>>>> I'd rather try to not annoy people; I figured the time I have the
>>>>>> CPython
>>>>>> patches ready and tested is the time I ping python-dev...
>>>>>
>>>>>
>>>>> If you want to eventually propose a PEP, you really really really
>>>>> should be talking to them before. Otherwise you'll get everything
>>>>> worked out just the way you want and they'll be like "what is this?
>>>>> re-do it all totally differently". And they might be wrong, but then
>>>>> you have to reconstruct for them the whole debate and reasoning
>>>>> process and implicit assumptions that you're making and not realizing
>>>>> you need to articulate, so easier to just get all the interested
>>>>> people at the table to begin with. And they might be right, in which
>>>>> case you just wasted however much time digging yourself into a hole
>>>>> and reverse-engineering bits of CPython.
>>>>>
>>>>> Don't propose it as a PEP, just say "hey, we have this problem and
>>>>> these constraints, and we're thinking we could solve them by something
>>>>> like this; but of course that has these limitations, so I dunno. What
>>>>> do you think?" And expect to spend some time figuring out what your
>>>>> requirements actually are (even if you think you know already, see
>>>>> above about implicit assumptions).
>>>>
>>>>
>>>> I personally think it's a great idea to bounce ideas around here first
>>>> before going to python-dev, especially as a PEP wouldn't get in until
>>>> 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near
>>>> term. That doesn't preclude presenting the problem and proposed
>>>> solution on python-dev as well, but the purpose of this thread seems
>>>> to be to think about it some, including how we're going to support
>>>> things in the short term, not nail down an exact PEP for Python to
>>>> accept at face value. I think we're at a point we can ping python-dev
>>>> now though.
>>>>
>>>> To be more futureproof, we'd want an offset to PyExtendedTypeObject
>>>> rather than assuming it exists at the end of PyTypeObject, but I don't
>>>> see a good place to store this information, so assuming it's right
>>>> there based on a bit in the flag seems a reasonable way forward.
>>>
>>>
>>> So I posted on python-dev.
>>>
>>> There's a lot of "You don't need to do this"; but here's an idea I got
>>> that's inspired by that discussion: We could use tp_getattr (and call it
>>> directly), but pass in an interned char* which Python code can never get
>>> hold of, and then that could return a void* (casted through a PyObject*, but
>>> it would not be).
>>
>> Would we want to support monkey patching these interfaces? If so, this
>> mechanism would be a bit harder than reallocing a pointer, although I
>> guess a closure chain of tp_getattr functions would work :). But I
>> think we want GIL-less access anyway right, which means neither
>> approach would work unsynchronized.
>>
>>> Another alternative is to somehow handshake on a metaclass implementation;
>>> and different Cython modules/NumPy/SciPy etc. would inherit from it. But
>>> apart from that handshaking, and having to use a metaclass and make
>>> everything more complicated for C implementors of the spec, it gives you a
>>> more expensive check than just checking a flag.
>>
>> I agree that the flag is much easier, if you have a metaclass the
>> question is again in which module to store it to get a cross-module
>> working typecheck. On the other hand, if the header file provides an
>> easy way to import the metaclass (monkeypatched on some module or
>> living in its own module), and to allocate type instances given a
>> (statically allocated) type, that would be more future-proof and
>> elegant. I don't think it would be much slower, it's doing
>> 'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'.
>
> Sorry, I think you mentioned something in related in the CEP, I
> couldn't find it from the enhancement list (I should have followed
> your originally posted link :). So that's a good point, there are
> several issues:
>
>    - subclasses of the type exposing the interface (I don't think
> that can be handled through tp_flags)
>    - in the case of a metaclass approach, subclassing the metaclass
> itself means type instances will no longer match the MYMETA pointer
>    - exposing interfaces on metaclasses (I think this one is listed in the CEP)
>
> I suppose the last case could either be disallowed, or one could
> provide a custom tp_alloc and store (a pointer to) extra information
> ahead of the object, taking care to precede any GC information. This
> is pretty hacky though.

Hm, actually if you copy the type by value anyway, you can just modify
the tp_basicsize of the copied value... Can you elaborate on the
metaclass issue in the CEP?

>> I think for the bit flag the interface won't span subclasses, whereas
>> the metaclass approach would allow subclassing but not subclassing of
>> the metaclass itself (unless you instantiate the metaclass through
>> itself, and check the interface against the metaclass, which only
>> means the metaclass of the metaclass isn't subclassable :) (this would
>> also mean a more expensive check)).
>>
>> I think if we implement CEP 1000, we will at that time have a generic
>> way to optimize and hoist none checking/bounds checking etc, which
>> will also allow us to optimize signature matching, which would mean
>> the matching and unpacking isn't as performance critical. JIT
>> compilers could take a similar approach.
>>
>>> I like a flag bit much better. I still hope that somebody more understanding
>>> comes along, argues our case, and gets bit 22 reserved for our purpose :-)
>>>
>>> Dag
>>>
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel