[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Robert Bradshaw]

On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King <wking at drexel.edu> wrote:
> On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
>> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
>>> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
>>>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote:
>>>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
>>>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote:
>>>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name
>>>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib,
>>>> >> > name)` will work in expose.pyx.
>>>> >>
>>>> >> You have also hit into the thorny issue that .pxd files are used for
>>>> >> many things. They may be pure C library declarations with no Python
>>>> >> module backing, they may be declarations of (externally implemented)
>>>> >> Python modules (such as numpy.pxd), or they may be declarations for
>>>> >> Cython-implemented modules.
>>>> >>
>>>> >> Here's another idea, what if extern blocks could contain cpdef
>>>> >> declarations, which would automatically generate a Python-level
>>>> >> wrappers for the declared members (if possible, otherwise an error)?
>>>> >
>>>> > Ah, this sounds good!  Of the three .pxd roles you list above,
>>>> > external Python modules (e.g. numpy) and Cython-implemented modules
>>>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space.
>>>> > What's missing is a way to give (where possible) declarations of
>>>> > external C libraries a Python presence.  cpdef fills this hole nicely,
>>>> > since its whole purpose is to expose Python interfaces to
>>>> > C-based elements.
>>>>
>>>> In the case of external Python modules, I'm not so sure we want to
>>>> monkey-patch our stuff in
>>>
>>> I don't think any of the changes we are suggesting would require
>>> changes to existing code, so .pxd-s with external implementations
>>> wouldn't be affected unless they brough the changes upon themselves.
>>
>> Say, in numpy.pxd, I have
>>
>> cdef extern from "...":
>>    cpdef struct obscure_internal_struct:
>>        ...
>>
>> Do we add an "obscure_internal_struct" onto the (global) numpy module?
>> What if it conflicts with a (runtime) name? This is the issue I'm
>> bringing up.
>
> Defining a cpdef *and* a non-matching external implementation should
> raise a compile-time error.  I agree that there is a useful
> distinction between external-C-library and external-Python-module .pxd
> wrappers.  Perhaps your matching blank .py or .pyx file could serve as
> a marker that the .pxd file should be inflated into its own full
> fledged python module.  I'm not even sure how you would go about
> adding attributes to the numpy module.  When/how would the
> Cython-created attributes get added?

Yes, this is exactly the issue.

> In the external-C-library case, if you define something incompatible
> in the matching .pyx or .py file, Cython will be able to see it and
> die with an appropriate error, so you can resolve your programming
> mistake.

That is only if you have a matching .pyx of .py file. Of course, there
could be a module by that name in the user's runtime environment that
we don't know about, but that currently is not in conflict with a .pxd
only file.

> If you try to override anything in a .so compiled module at runtime,
> you'd get the same kind of error you currently do trying to rebind a
> compiled class' method.

That's the desired behavior for statically-bound globals, but
implementing it is not so trivial.

>>>> (and where would we do it--on the first import of a cimporting
>>>> module?)
>>>
>>> Compilation is an issue.  I think that .pxd files should be able to be
>>> cythoned directly, since then they Cython can build any wrappers they
>>> request.  If the file has a matching .pyx file, cythoning either one
>>> should compile both together, since they'll produce a single Python
>>> .so module.
>>
>> In this case, I think it may make more sense to consider cimporting
>> stuff from .pyx files if no .pxd file is present.
>
> Can you cimport .pyx files that lack matching .pxd files?
>
>> In any case, this is a big change. I don't like the idea of always
>> creating such a module (it may be empty, or have name conflicts)
>
> It shouldn't take to long to compile an empty module ;).  And odds are
> noone will waste time importing it either.
>
>> nor the idea of conditionally compiling it (does one have to look at
>> the contents and really understand Cython to see if a Python shadow
>> will be created?)
>
> I agree here.
>
> Under the mantra "explicit is better than implicit", we could have
> users add something like
>
>    cdef module "modname"
>
> to any .pxd files that should be inflated into Python modules.  .pxd
> files without such a tag would receive the current treatment, error on
> any cpdef, etc.  The drawback of this approach is that it makes Cython
> more complicated, but if both behaviors are reasonable, there's
> probably no getting around that.

The other drawback is that it subverts the usual filename <-> module
name convention that one usually expects.

>>>> > A side effect of this cpdef change would be that now even bare .pxd
>>>> > files (no matching .pyx) would have a Python presence,
>>>>
>>>> Where would it live? Would we just create this module (in essence,
>>>> acting as if there was an empty .pyx file sitting there as well)? On
>>>> this note, it may be worth pursuing the idea of a "cython helper"
>>>> module where common code and objects could live.
>>>
>>> I'm not sure exactly what you mean by "cython helper", but this sounds
>>> like my 'bare .pyx can create a Python .so module idea above.
>>
>> I'm thinking of a place to put, e.g. the generator and bind-able
>> function classes, which are now re-implemented in every module that
>> uses them. I think there will be more cases like this in the future
>> rather than less. C-level code could be #included and linked from
>> "global" stores as well. However, that's somewhat tangential.
>
> That sounds more and more like per-pxd .so modules ;).
>
>>>> > Unions don't really have a Python parallel,
>>>>
>>>> They can be a cdef class wrapping the union type.
>>>
>>> But I would think coercion would be difficult.  Unions are usually (in
>>> my limited experience) for "don't worry about the type, just make sure
>>> it fits in X bytes".  How would union->Python conversion work?
>>
>> There would be a wrapping type, e.g.
>>
>> cdef class MyUnion:
>>    cdef union_type value
>>
>> with a bunch of setters/getters for the values, just like there are
>> for structs. (In fact the same code would handle structs and unions).
>>
>> This is getting into the wrapper-generator territory, but I'm starting
>> to think for simple things that might be worth it.
>
> I think that if Cython will automatically generate a wrapper for
>
>    cdef public int x
>
> it should generate a wrapper for
>
>    cdef struct X: cdef public int x

Or

    cdef public struct X:
        int x
        readonly int z
        private int z

I would perhaps say that non-Pythonable non-private members in public
structs would be a compile error.

> There really aren't that metatypes in C, so it doesn't seem like a
> slippery slope to me.  Maybe I'm just missing something...
>
>>> Ok, I think we're pretty much agreed ;).  I think that the next step
>>> is to start working on implementations of:
>>>
>>> * Stand alone .pxd -> Python module
>>
>> I'm not sure we're agreed on this one.
>
> Ok, it can wait while we hash out the best approach.
>
>>> * Extending class cdef/cdpef/public/readonly handling to cover enums,
>>>  stucts, and possibly unions.
>>
>> This seems like the best first step.
>>
>>> Problems with me getting started now:
>>>
>>> * I don't know where I should start mucking about in the source.
>>
>> I would start by...
>
> Excellent.  Thanks.
>
>>> * I don't know how to handle things like dummy enums (perhaps by
>>>  requiring all cdef-ed enums to be named).
>>
>> All enums in C are named.
>
> But my Cython declaration (exposing a C `#define CONST_A 1`):
>
>    cdef extern from 'mylib.h':
>        enum: CONST_A
>
> is not a named enum.

Ah, yes. Maybe we require a name (that would only be used in Python space).

>> Yep. You might want to consider creating a github fork.
>
> I've started a branch 'cdef-enums-stucts-and-unions' (no activity
> yet).
>
> Repo:
>  http://www.physics.drexel.edu/~wking/code/git/cython.git
>
> Gitweb interface:
>  http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git
>
> I can put this up on GitHub if you want, but it's easier for me to
> self-host, and `git fetch` works just as well either way ;).

Code review is much easier if you fork off the public branch and do
pull-requests, but being a DVCS you of course don't have to "choose"
where your master repo lives.

- Robert