[Cython] CEP 1001 - Custom PyTypeObject extensions

[mark florisson]

On 16 May 2012 12:20, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 05/14/2012 08:01 PM, Robert Bradshaw wrote:
>>
>> On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith<njs at pobox.com>  wrote:
>>>
>>> On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn
>>> <d.s.seljebotn at astro.uio.no>  wrote:
>>>>
>>>> On 05/14/2012 01:34 PM, Stefan Behnel wrote:
>>>>>
>>>>>
>>>>> Dag Sverre Seljebotn, 13.05.2012 21:37:
>>>>>>
>>>>>>
>>>>>> Anyway, thanks for the heads up, this seems to need a bit more work.
>>>>>> Input
>>>>>> from somebody more familiar with this corner of the CPython API very
>>>>>> welcome.
>>>>>
>>>>>
>>>>>
>>>>> Wouldn't you consider python-dev an appropriate place to discuss this?
>>>>
>>>>
>>>>
>>>> Propose something for a PEP that's primarily useful to Cython without
>>>> even
>>>> understanding the full implications myself first?
>>>>
>>>> I'd rather try to not annoy people; I figured the time I have the
>>>> CPython
>>>> patches ready and tested is the time I ping python-dev...
>>>
>>>
>>> If you want to eventually propose a PEP, you really really really
>>> should be talking to them before. Otherwise you'll get everything
>>> worked out just the way you want and they'll be like "what is this?
>>> re-do it all totally differently". And they might be wrong, but then
>>> you have to reconstruct for them the whole debate and reasoning
>>> process and implicit assumptions that you're making and not realizing
>>> you need to articulate, so easier to just get all the interested
>>> people at the table to begin with. And they might be right, in which
>>> case you just wasted however much time digging yourself into a hole
>>> and reverse-engineering bits of CPython.
>>>
>>> Don't propose it as a PEP, just say "hey, we have this problem and
>>> these constraints, and we're thinking we could solve them by something
>>> like this; but of course that has these limitations, so I dunno. What
>>> do you think?" And expect to spend some time figuring out what your
>>> requirements actually are (even if you think you know already, see
>>> above about implicit assumptions).
>>
>>
>> I personally think it's a great idea to bounce ideas around here first
>> before going to python-dev, especially as a PEP wouldn't get in until
>> 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near
>> term. That doesn't preclude presenting the problem and proposed
>> solution on python-dev as well, but the purpose of this thread seems
>> to be to think about it some, including how we're going to support
>> things in the short term, not nail down an exact PEP for Python to
>> accept at face value. I think we're at a point we can ping python-dev
>> now though.
>>
>> To be more futureproof, we'd want an offset to PyExtendedTypeObject
>> rather than assuming it exists at the end of PyTypeObject, but I don't
>> see a good place to store this information, so assuming it's right
>> there based on a bit in the flag seems a reasonable way forward.
>
>
> So I posted on python-dev.
>
> There's a lot of "You don't need to do this"; but here's an idea I got
> that's inspired by that discussion: We could use tp_getattr (and call it
> directly), but pass in an interned char* which Python code can never get
> hold of, and then that could return a void* (casted through a PyObject*, but
> it would not be).

Would we want to support monkey patching these interfaces? If so, this
mechanism would be a bit harder than reallocing a pointer, although I
guess a closure chain of tp_getattr functions would work :). But I
think we want GIL-less access anyway right, which means neither
approach would work unsynchronized.

> Another alternative is to somehow handshake on a metaclass implementation;
> and different Cython modules/NumPy/SciPy etc. would inherit from it. But
> apart from that handshaking, and having to use a metaclass and make
> everything more complicated for C implementors of the spec, it gives you a
> more expensive check than just checking a flag.

I agree that the flag is much easier, if you have a metaclass the
question is again in which module to store it to get a cross-module
working typecheck. On the other hand, if the header file provides an
easy way to import the metaclass (monkeypatched on some module or
living in its own module), and to allocate type instances given a
(statically allocated) type, that would be more future-proof and
elegant. I don't think it would be much slower, it's doing
'o->ob_type->tp_flags & MYFLAG' vs 'o->ob_type->ob_type == MYMETA'.

I think for the bit flag the interface won't span subclasses, whereas
the metaclass approach would allow subclassing but not subclassing of
the metaclass itself (unless you instantiate the metaclass through
itself, and check the interface against the metaclass, which only
means the metaclass of the metaclass isn't subclassable :) (this would
also mean a more expensive check)).

I think if we implement CEP 1000, we will at that time have a generic
way to optimize and hoist none checking/bounds checking etc, which
will also allow us to optimize signature matching, which would mean
the matching and unpacking isn't as performance critical. JIT
compilers could take a similar approach.

> I like a flag bit much better. I still hope that somebody more understanding
> comes along, argues our case, and gets bit 22 reserved for our purpose :-)
>
> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel