[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Stefan Behnel]

Robert Bradshaw, 18.02.2011 23:08:
> On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote:
>> On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
>>> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
>>>> Compilation is an issue.  I think that .pxd files should be able to be
>>>> cythoned directly, since then they Cython can build any wrappers they
>>>> request.  If the file has a matching .pyx file, cythoning either one
>>>> should compile both together, since they'll produce a single Python
>>>> .so module.
>>>
>>> In this case, I think it may make more sense to consider cimporting
>>> stuff from .pyx files if no .pxd file is present.
>>
>> Can you cimport .pyx files that lack matching .pxd files?
>>
>>> In any case, this is a big change. I don't like the idea of always
>>> creating such a module (it may be empty, or have name conflicts)
>>
>> It shouldn't take to long to compile an empty module ;).  And odds are
>> noone will waste time importing it either.
>>
>>> nor the idea of conditionally compiling it (does one have to look at
>>> the contents and really understand Cython to see if a Python shadow
>>> will be created?)
>>
>> I agree here.
>>
>> Under the mantra "explicit is better than implicit", we could have
>> users add something like
>>
>>     cdef module "modname"
>>
>> to any .pxd files that should be inflated into Python modules.  .pxd
>> files without such a tag would receive the current treatment, error on
>> any cpdef, etc.  The drawback of this approach is that it makes Cython
>> more complicated, but if both behaviors are reasonable, there's
>> probably no getting around that.
>
> The other drawback is that it subverts the usual filename<->  module
> name convention that one usually expects.

+1


>>>>>> A side effect of this cpdef change would be that now even bare .pxd
>>>>>> files (no matching .pyx) would have a Python presence,
>>>>>
>>>>> Where would it live? Would we just create this module (in essence,
>>>>> acting as if there was an empty .pyx file sitting there as well)? On
>>>>> this note, it may be worth pursuing the idea of a "cython helper"
>>>>> module where common code and objects could live.
>>>>
>>>> I'm not sure exactly what you mean by "cython helper", but this sounds
>>>> like my 'bare .pyx can create a Python .so module idea above.
>>>
>>> I'm thinking of a place to put, e.g. the generator and bind-able
>>> function classes, which are now re-implemented in every module that
>>> uses them. I think there will be more cases like this in the future
>>> rather than less. C-level code could be #included and linked from
>>> "global" stores as well. However, that's somewhat tangential.

If you generate more than one file from a .pyx, including files that are 
shared between compiler runs (or even readily built as .so files), you'd 
quickly end up in dependency hell. When to regenerate that file? With what 
compiler version and config? C/C++? With/out cleanup code? And where to put 
it? Are you sure that place will be writable for all compiler runs? And 
will the result be guaranteed to be reproducible, regardless of how many 
compiler runs there were in between?


>>>>>> Unions don't really have a Python parallel,
>>>>>
>>>>> They can be a cdef class wrapping the union type.
>>>>
>>>> But I would think coercion would be difficult.  Unions are usually (in
>>>> my limited experience) for "don't worry about the type, just make sure
>>>> it fits in X bytes".  How would union->Python conversion work?
>>>
>>> There would be a wrapping type, e.g.
>>>
>>> cdef class MyUnion:
>>>     cdef union_type value

Wouldn't that have to be a pointer to the real thing instead?


>>> with a bunch of setters/getters for the values, just like there are
>>> for structs. (In fact the same code would handle structs and unions).
>>>
>>> This is getting into the wrapper-generator territory, but I'm starting
>>> to think for simple things that might be worth it.
>>
>> I think that if Cython will automatically generate a wrapper for
>>
>>     cdef public int x
>>
>> it should generate a wrapper for
>>
>>     cdef struct X: cdef public int x
>
> Or
>
>      cdef public struct X:
>          int x
>          readonly int z
>          private int z
>
> I would perhaps say that non-Pythonable non-private members in public
> structs would be a compile error.

+1, keep it safe at the beginning.


>> There really aren't that metatypes in C, so it doesn't seem like a
>> slippery slope to me.  Maybe I'm just missing something...
>>
>>>> Ok, I think we're pretty much agreed ;).  I think that the next step
>>>> is to start working on implementations of:
>>>>
>>>> * Stand alone .pxd ->  Python module
>>>
>>> I'm not sure we're agreed on this one.

Same from here. To me, that doesn't make much sense for code that wraps a 
library. And if it doesn't wrap a library, there isn't much benefit in 
writing a stand-alone .pxd in the first place. A .pyx is much more explicit 
and obvious in this case. Especially having some .pxd files that generate 
.so files and others that don't will make this very ugly.

I'd prefer adding support for cimporting from .pyx files instead, 
potentially with an automated caching generation of corresponding .pxd 
files (maybe as ".pxdg" files to make them easier to handle for users). 
However, cyclic dependencies would be tricky to handle automatically then.


>>>> * Extending class cdef/cdpef/public/readonly handling to cover enums,
>>>>   stucts, and possibly unions.
>>>
>>> This seems like the best first step.

+1


>>>> * I don't know how to handle things like dummy enums (perhaps by
>>>>   requiring all cdef-ed enums to be named).
>>>
>>> All enums in C are named.
>>
>> But my Cython declaration (exposing a C `#define CONST_A 1`):
>>
>>     cdef extern from 'mylib.h':
>>         enum: CONST_A
>>
>> is not a named enum.
>
> Ah, yes. Maybe we require a name (that would only be used in Python space).

... require it for cpdef enums, you mean?

OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to 
declare external constants correctly. After all, we loose all type 
information that way. I just saw that in math.pxd things like "M_PI" are 
declared as plain "enum" instead of "const double" or something. The type 
inferencer can't do anything with that. It might even draw the completely 
wrong conclusions.

Stefan