[Cython] CEP 1001 - Custom PyTypeObject extensions

[Dag Sverre Seljebotn]

On 05/14/2012 08:01 PM, Robert Bradshaw wrote:
> On Mon, May 14, 2012 at 10:05 AM, Nathaniel Smith<njs at pobox.com>  wrote:
>> On Mon, May 14, 2012 at 3:23 PM, Dag Sverre Seljebotn
>> <d.s.seljebotn at astro.uio.no>  wrote:
>>> On 05/14/2012 01:34 PM, Stefan Behnel wrote:
>>>>
>>>> Dag Sverre Seljebotn, 13.05.2012 21:37:
>>>>>
>>>>> Anyway, thanks for the heads up, this seems to need a bit more work.
>>>>> Input
>>>>> from somebody more familiar with this corner of the CPython API very
>>>>> welcome.
>>>>
>>>>
>>>> Wouldn't you consider python-dev an appropriate place to discuss this?
>>>
>>>
>>> Propose something for a PEP that's primarily useful to Cython without even
>>> understanding the full implications myself first?
>>>
>>> I'd rather try to not annoy people; I figured the time I have the CPython
>>> patches ready and tested is the time I ping python-dev...
>>
>> If you want to eventually propose a PEP, you really really really
>> should be talking to them before. Otherwise you'll get everything
>> worked out just the way you want and they'll be like "what is this?
>> re-do it all totally differently". And they might be wrong, but then
>> you have to reconstruct for them the whole debate and reasoning
>> process and implicit assumptions that you're making and not realizing
>> you need to articulate, so easier to just get all the interested
>> people at the table to begin with. And they might be right, in which
>> case you just wasted however much time digging yourself into a hole
>> and reverse-engineering bits of CPython.
>>
>> Don't propose it as a PEP, just say "hey, we have this problem and
>> these constraints, and we're thinking we could solve them by something
>> like this; but of course that has these limitations, so I dunno. What
>> do you think?" And expect to spend some time figuring out what your
>> requirements actually are (even if you think you know already, see
>> above about implicit assumptions).
>
> I personally think it's a great idea to bounce ideas around here first
> before going to python-dev, especially as a PEP wouldn't get in until
> 3.3 or 3.4 at best, and we want to do something with 2.4+ in the near
> term. That doesn't preclude presenting the problem and proposed
> solution on python-dev as well, but the purpose of this thread seems
> to be to think about it some, including how we're going to support
> things in the short term, not nail down an exact PEP for Python to
> accept at face value. I think we're at a point we can ping python-dev
> now though.
>
> To be more futureproof, we'd want an offset to PyExtendedTypeObject
> rather than assuming it exists at the end of PyTypeObject, but I don't
> see a good place to store this information, so assuming it's right
> there based on a bit in the flag seems a reasonable way forward.

So I posted on python-dev.

There's a lot of "You don't need to do this"; but here's an idea I got 
that's inspired by that discussion: We could use tp_getattr (and call it 
directly), but pass in an interned char* which Python code can never get 
hold of, and then that could return a void* (casted through a PyObject*, 
but it would not be).

Another alternative is to somehow handshake on a metaclass 
implementation; and different Cython modules/NumPy/SciPy etc. would 
inherit from it. But apart from that handshaking, and having to use a 
metaclass and make everything more complicated for C implementors of the 
spec, it gives you a more expensive check than just checking a flag.

I like a flag bit much better. I still hope that somebody more 
understanding comes along, argues our case, and gets bit 22 reserved for 
our purpose :-)

Dag