[Cython] How to define C-consts in python module scope

Wed Jul 20 23:53:23 CEST 2011

On 20 July 2011 23:24, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Jul 20, 2011 at 12:53 PM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 20 July 2011 21:44, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>> On 20 July 2011 16:27, mark florisson <markflorisson88 at gmail.com> wrote:
>>>> On 20 July 2011 21:13, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>> On 20 July 2011 15:32, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>> On 20 July 2011 20:04, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>> On 20 July 2011 13:51, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>>> On 20 July 2011 18:06, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>>> On 19 July 2011 20:48, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>>>>> On 19 July 2011 02:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>>> 2011/7/18 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I
>>>>>>>>>>>>> got around to looking at his code (I don't think he ever created a
>>>>>>>>>>>>> formal pull request) something came up. The idea was that we could
>>>>>>>>>>>>> support cpdef structs and extern functions as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request.
>>>>>>>>>>>>
>>>>>>>>>>>> 2011/7/19 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>>>>>>>>>>>>>> My suggestion is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  cdef exposed enum:
>>>>>>>>>>>>>>    ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to
>>>>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api
>>>>>>>>>>>>> generation, and public used for Python-level access, with cpdef being
>>>>>>>>>>>>> the preferred form for declaring Python-accessible types.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used
>>>>>>>>>>>> in this case:
>>>>>>>>>>>>
>>>>>>>>>>>> cdef extern from "ev.h":
>>>>>>>>>>>>    exposed enum:
>>>>>>>>>>>>        EV_READ
>>>>>>>>>>>>        EV_WRITE
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And what about this?
>>>>>>>>>>>
>>>>>>>>>>> cdef extern from "ev.h":
>>>>>>>>>>>   cpdef enum:
>>>>>>>>>>>       EV_READ
>>>>>>>>>>>       EV_WRITE
>>>>>>>>>>
>>>>>>>>>> Yep, exactly.
>>>>>>>>>>
>>>>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values
>>>>>>>>>>> will be exposed just in the matching implementation .pyx file, and not
>>>>>>>>>>> with cimport, right?
>>>>>>>>>>
>>>>>>>>>> It would be an error unless there's an corresponding .pyx file (which
>>>>>>>>>> could be empty). The idea is that one could also define extern cpdef
>>>>>>>>>> functions and structs and wrappers would be provided.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd,
>>>>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo'
>>>>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed
>>>>>>>>> but the enumerations should not be exposed in the "bar" module... Am I
>>>>>>>>> missing something?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I believe Robert confirmed what you said and added to that that it
>>>>>>>> would be an error to have a cpdef extern enum in a .pxd without a
>>>>>>>> corresponding .pyx file.
>>>>>>>>
>>>>>>>
>>>>>>> But what an "error" means? Cython is going to fail compilation?
>>>>>>> Suppose I develop a package, and at setup.py install time my package
>>>>>>> installs somewhere a .pxd full of definitions in such a way that
>>>>>>> third-party code can cimport it. Then, while writing the third party
>>>>>>> code, you cimport the my package .pxd, and obviously there is no .pyx
>>>>>>> for my package, as you are writing YOUR code. What's going to happen
>>>>>>> in this case?
>>>>>>
>>>>>> If you want to use cpdef in your .pxd, it means you want to expose
>>>>>> those to Python. This means they must be importable from Python, which
>>>>>> means you need an extension module that exposes the constants, which
>>>>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along
>>>>>> with your .pxd. If you don't want to expose them to Python but only to
>>>>>> Cython, don't use cpdef. This compile time error would be issued
>>>>>> whenever a user does a cimport of a .pxd file that has a cpdef enum
>>>>>> with no corresponding .pyx file, i.e. the pxd is unusable, you
>>>>>> wouldn't ship it in the first place.
>
> Alternatively it would be a runtime error when trying to import the
> objects. I suppose if cimport does not inject anything into the python
> namespace, then "cimport" could just work and "import" would work iff
> the module is available at runtime.
>
>>>>> So supose I want to write a small wrapper for dlopen(), then I do:
>>>>>
>>>>> # dl.pxd
>>>>> cdef from extern from "dlfcn.h":
>>>>>    cpdef enum: RTLD_GLOBAL
>>>>>    void * dlopen()(const char *filename, int flag);
>>>>>
>>>>> # dl.pyx
>>>>> def open(filename, mode):
>>>>>    cdef void *handle = c_dlopen(fiename,handle)
>>>>>    return <long>handle
>>>>>
>>>>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL"
>>>>>
>>>>> Now, suppose I code my setup.py to build the ext module and install
>>>>> dl.pxd as package data.
>>>>>
>>>>> Now a user runs "pip install dl" and installs my package. She wants to
>>>>> reuse my .pxd (installed somewhere) for calling dlopen() in its own
>>>>> Cython code:
>>>>>
>>>>> # user.pyx
>>>>> from dl cimport RTLD_GLOBAL, c_dlopen
>>>>> dlopen("somelibrary.so", RTLD_GLOBAL)
>>>>>
>>>>>
>>>>> So the user code is going to fail? Then I cannot take advantage of
>>>>> cpdef enum for developing my  "dl" module, I have to go back to
>>>>> manually exposing the enumerations
>>>>>
>>>>
>>>> Why can't you ship both dl.pxd and dl.pyx as package data?
>>>>
>>>
>>> This is like asking to provide a C source in order to being able to
>>> #include "someheader.h" in other C sources. End users do not need
>>> implementation source code, they need declarations,interface,header
>>> files.
>
> Because we all love link errors :).
>
>> I personally don't really mind, but what about not allowing cpdef in
>> .pxds for enums, but only in .pyx files:
>>
>> from mypxd cimport my_extern_enum
>> cpdef enum my_extern_enum
>>
>> Of course, the usual
>>
>> cpdef enum my_enum:
>>   ...
>>
>> would still be allowed in .pyx files.
>
> Hmm... that's an idea, though it means declaring things twice. Also,
> what about big lists of enums in, e.g. numpy.pxd or the system
> headers?

You'd declare the enum twice, but the contents once. In the pxd you
list the items, in the pyx you merely say "ok expose this entire thing
to Python for me".

> I'm perhaps OK with them being like cdef classes--the .pyx file is not
> needed at compile time it's a runtime error on import if the
> corresponding module is not available.

With cdef classes you need the import to access the data, but with
enums you don't so I think it's a little weird. I'd prefer a compile
time error or a warning perhaps. I don't think installing source files
is a bad idea (I'm usually disappointed when I only find .pyc files
and not the actual source, same for .pxd).

> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>