[Cython] How to define C-consts in python module scope

[Robert Bradshaw]

On Wed, Jul 20, 2011 at 2:53 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 20 July 2011 23:24, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Wed, Jul 20, 2011 at 12:53 PM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 20 July 2011 21:44, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>> On 20 July 2011 16:27, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>> On 20 July 2011 21:13, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>> On 20 July 2011 15:32, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>> On 20 July 2011 20:04, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>> On 20 July 2011 13:51, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>>>> On 20 July 2011 18:06, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>>>> On 19 July 2011 20:48, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>>>>>> On 19 July 2011 02:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>>> 2011/7/18 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I
>>>>>>>>>>>>>> got around to looking at his code (I don't think he ever created a
>>>>>>>>>>>>>> formal pull request) something came up. The idea was that we could
>>>>>>>>>>>>>> support cpdef structs and extern functions as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2011/7/19 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>>>>>>>>>>>>>>> My suggestion is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  cdef exposed enum:
>>>>>>>>>>>>>>>    ...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to
>>>>>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api
>>>>>>>>>>>>>> generation, and public used for Python-level access, with cpdef being
>>>>>>>>>>>>>> the preferred form for declaring Python-accessible types.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used
>>>>>>>>>>>>> in this case:
>>>>>>>>>>>>>
>>>>>>>>>>>>> cdef extern from "ev.h":
>>>>>>>>>>>>>    exposed enum:
>>>>>>>>>>>>>        EV_READ
>>>>>>>>>>>>>        EV_WRITE
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And what about this?
>>>>>>>>>>>>
>>>>>>>>>>>> cdef extern from "ev.h":
>>>>>>>>>>>>   cpdef enum:
>>>>>>>>>>>>       EV_READ
>>>>>>>>>>>>       EV_WRITE
>>>>>>>>>>>
>>>>>>>>>>> Yep, exactly.
>>>>>>>>>>>
>>>>>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values
>>>>>>>>>>>> will be exposed just in the matching implementation .pyx file, and not
>>>>>>>>>>>> with cimport, right?
>>>>>>>>>>>
>>>>>>>>>>> It would be an error unless there's an corresponding .pyx file (which
>>>>>>>>>>> could be empty). The idea is that one could also define extern cpdef
>>>>>>>>>>> functions and structs and wrappers would be provided.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd,
>>>>>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo'
>>>>>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed
>>>>>>>>>> but the enumerations should not be exposed in the "bar" module... Am I
>>>>>>>>>> missing something?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I believe Robert confirmed what you said and added to that that it
>>>>>>>>> would be an error to have a cpdef extern enum in a .pxd without a
>>>>>>>>> corresponding .pyx file.
>>>>>>>>>
>>>>>>>>
>>>>>>>> But what an "error" means? Cython is going to fail compilation?
>>>>>>>> Suppose I develop a package, and at setup.py install time my package
>>>>>>>> installs somewhere a .pxd full of definitions in such a way that
>>>>>>>> third-party code can cimport it. Then, while writing the third party
>>>>>>>> code, you cimport the my package .pxd, and obviously there is no .pyx
>>>>>>>> for my package, as you are writing YOUR code. What's going to happen
>>>>>>>> in this case?
>>>>>>>
>>>>>>> If you want to use cpdef in your .pxd, it means you want to expose
>>>>>>> those to Python. This means they must be importable from Python, which
>>>>>>> means you need an extension module that exposes the constants, which
>>>>>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along
>>>>>>> with your .pxd. If you don't want to expose them to Python but only to
>>>>>>> Cython, don't use cpdef. This compile time error would be issued
>>>>>>> whenever a user does a cimport of a .pxd file that has a cpdef enum
>>>>>>> with no corresponding .pyx file, i.e. the pxd is unusable, you
>>>>>>> wouldn't ship it in the first place.
>>
>> Alternatively it would be a runtime error when trying to import the
>> objects. I suppose if cimport does not inject anything into the python
>> namespace, then "cimport" could just work and "import" would work iff
>> the module is available at runtime.
>>
>>>>>> So supose I want to write a small wrapper for dlopen(), then I do:
>>>>>>
>>>>>> # dl.pxd
>>>>>> cdef from extern from "dlfcn.h":
>>>>>>    cpdef enum: RTLD_GLOBAL
>>>>>>    void * dlopen()(const char *filename, int flag);
>>>>>>
>>>>>> # dl.pyx
>>>>>> def open(filename, mode):
>>>>>>    cdef void *handle = c_dlopen(fiename,handle)
>>>>>>    return <long>handle
>>>>>>
>>>>>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL"
>>>>>>
>>>>>> Now, suppose I code my setup.py to build the ext module and install
>>>>>> dl.pxd as package data.
>>>>>>
>>>>>> Now a user runs "pip install dl" and installs my package. She wants to
>>>>>> reuse my .pxd (installed somewhere) for calling dlopen() in its own
>>>>>> Cython code:
>>>>>>
>>>>>> # user.pyx
>>>>>> from dl cimport RTLD_GLOBAL, c_dlopen
>>>>>> dlopen("somelibrary.so", RTLD_GLOBAL)
>>>>>>
>>>>>>
>>>>>> So the user code is going to fail? Then I cannot take advantage of
>>>>>> cpdef enum for developing my  "dl" module, I have to go back to
>>>>>> manually exposing the enumerations
>>>>>>
>>>>>
>>>>> Why can't you ship both dl.pxd and dl.pyx as package data?
>>>>>
>>>>
>>>> This is like asking to provide a C source in order to being able to
>>>> #include "someheader.h" in other C sources. End users do not need
>>>> implementation source code, they need declarations,interface,header
>>>> files.
>>
>> Because we all love link errors :).
>>
>>> I personally don't really mind, but what about not allowing cpdef in
>>> .pxds for enums, but only in .pyx files:
>>>
>>> from mypxd cimport my_extern_enum
>>> cpdef enum my_extern_enum
>>>
>>> Of course, the usual
>>>
>>> cpdef enum my_enum:
>>>   ...
>>>
>>> would still be allowed in .pyx files.
>>
>> Hmm... that's an idea, though it means declaring things twice. Also,
>> what about big lists of enums in, e.g. numpy.pxd or the system
>> headers?
>
> You'd declare the enum twice, but the contents once. In the pxd you
> list the items, in the pyx you merely say "ok expose this entire thing
> to Python for me".

It's not as bad for enums, but would be more of a pain for
automatically wrapped functions/structs. And redundancy is redundancy.

>> I'm perhaps OK with them being like cdef classes--the .pyx file is not
>> needed at compile time it's a runtime error on import if the
>> corresponding module is not available.
>
> With cdef classes you need the import to access the data, but with
> enums you don't so I think it's a little weird.

Alternatively, we could force you to import them iff used in a Python
context, though I'd rather have it unconditional. Think cpdef
structs/functions converting ito Python objects--even enums might be
nice to convert to Python as a new type with an __int__ function
rather than raw ints so they print nice and can be accessed like
my_enum_type.[tab]).

> I'd prefer a compile
> time error or a warning perhaps. I don't think installing source files
> is a bad idea (I'm usually disappointed when I only find .pyc files
> and not the actual source, same for .pxd).

I have similar sentiments and I prefer compile time errors as well,
but that might be too restrictive.

- Robert