[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[W. Trevor King]

On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote:
> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote:
>> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
>>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote:
>>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
>>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote:
>>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name
>>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib,
>>> >> > name)` will work in expose.pyx.
>>> >>
>>> >> You have also hit into the thorny issue that .pxd files are used for
>>> >> many things. They may be pure C library declarations with no Python
>>> >> module backing, they may be declarations of (externally implemented)
>>> >> Python modules (such as numpy.pxd), or they may be declarations for
>>> >> Cython-implemented modules.
>>> >>
>>> >> Here's another idea, what if extern blocks could contain cpdef
>>> >> declarations, which would automatically generate a Python-level
>>> >> wrappers for the declared members (if possible, otherwise an error)?
>>> >
>>> > Ah, this sounds good!  Of the three .pxd roles you list above,
>>> > external Python modules (e.g. numpy) and Cython-implemented modules
>>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space.
>>> > What's missing is a way to give (where possible) declarations of
>>> > external C libraries a Python presence.  cpdef fills this hole nicely,
>>> > since its whole purpose is to expose Python interfaces to
>>> > C-based elements.
>>>
>>> In the case of external Python modules, I'm not so sure we want to
>>> monkey-patch our stuff in
>>
>> I don't think any of the changes we are suggesting would require
>> changes to existing code, so .pxd-s with external implementations
>> wouldn't be affected unless they brough the changes upon themselves.
>
> Say, in numpy.pxd, I have
>
> cdef extern from "...":
>    cpdef struct obscure_internal_struct:
>        ...
>
> Do we add an "obscure_internal_struct" onto the (global) numpy module?
> What if it conflicts with a (runtime) name? This is the issue I'm
> bringing up.

Defining a cpdef *and* a non-matching external implementation should
raise a compile-time error.  I agree that there is a useful
distinction between external-C-library and external-Python-module .pxd
wrappers.  Perhaps your matching blank .py or .pyx file could serve as
a marker that the .pxd file should be inflated into its own full
fledged python module.  I'm not even sure how you would go about
adding attributes to the numpy module.  When/how would the
Cython-created attributes get added?

In the external-C-library case, if you define something incompatible
in the matching .pyx or .py file, Cython will be able to see it and
die with an appropriate error, so you can resolve your programming
mistake.

If you try to override anything in a .so compiled module at runtime,
you'd get the same kind of error you currently do trying to rebind a
compiled class' method.

>>> (and where would we do it--on the first import of a cimporting
>>> module?)
>>
>> Compilation is an issue.  I think that .pxd files should be able to be
>> cythoned directly, since then they Cython can build any wrappers they
>> request.  If the file has a matching .pyx file, cythoning either one
>> should compile both together, since they'll produce a single Python
>> .so module.
>
> In this case, I think it may make more sense to consider cimporting
> stuff from .pyx files if no .pxd file is present.

Can you cimport .pyx files that lack matching .pxd files?

> In any case, this is a big change. I don't like the idea of always
> creating such a module (it may be empty, or have name conflicts)

It shouldn't take to long to compile an empty module ;).  And odds are
noone will waste time importing it either.

> nor the idea of conditionally compiling it (does one have to look at
> the contents and really understand Cython to see if a Python shadow
> will be created?)

I agree here.

Under the mantra "explicit is better than implicit", we could have
users add something like

    cdef module "modname"

to any .pxd files that should be inflated into Python modules.  .pxd
files without such a tag would receive the current treatment, error on
any cpdef, etc.  The drawback of this approach is that it makes Cython
more complicated, but if both behaviors are reasonable, there's
probably no getting around that.

>>> > A side effect of this cpdef change would be that now even bare .pxd
>>> > files (no matching .pyx) would have a Python presence,
>>>
>>> Where would it live? Would we just create this module (in essence,
>>> acting as if there was an empty .pyx file sitting there as well)? On
>>> this note, it may be worth pursuing the idea of a "cython helper"
>>> module where common code and objects could live.
>>
>> I'm not sure exactly what you mean by "cython helper", but this sounds
>> like my 'bare .pyx can create a Python .so module idea above.
>
> I'm thinking of a place to put, e.g. the generator and bind-able
> function classes, which are now re-implemented in every module that
> uses them. I think there will be more cases like this in the future
> rather than less. C-level code could be #included and linked from
> "global" stores as well. However, that's somewhat tangential.

That sounds more and more like per-pxd .so modules ;).

>>> > Unions don't really have a Python parallel,
>>>
>>> They can be a cdef class wrapping the union type.
>>
>> But I would think coercion would be difficult.  Unions are usually (in
>> my limited experience) for "don't worry about the type, just make sure
>> it fits in X bytes".  How would union->Python conversion work?
>
> There would be a wrapping type, e.g.
>
> cdef class MyUnion:
>    cdef union_type value
>
> with a bunch of setters/getters for the values, just like there are
> for structs. (In fact the same code would handle structs and unions).
>
> This is getting into the wrapper-generator territory, but I'm starting
> to think for simple things that might be worth it.

I think that if Cython will automatically generate a wrapper for

    cdef public int x

it should generate a wrapper for

    cdef struct X: cdef public int x

There really aren't that metatypes in C, so it doesn't seem like a
slippery slope to me.  Maybe I'm just missing something...

>> Ok, I think we're pretty much agreed ;).  I think that the next step
>> is to start working on implementations of:
>>
>> * Stand alone .pxd -> Python module
>
> I'm not sure we're agreed on this one.

Ok, it can wait while we hash out the best approach.

>> * Extending class cdef/cdpef/public/readonly handling to cover enums,
>>  stucts, and possibly unions.
>
> This seems like the best first step.
>
>> Problems with me getting started now:
>>
>> * I don't know where I should start mucking about in the source.
>
> I would start by...

Excellent.  Thanks.

>> * I don't know how to handle things like dummy enums (perhaps by
>>  requiring all cdef-ed enums to be named).
>
> All enums in C are named.

But my Cython declaration (exposing a C `#define CONST_A 1`):

    cdef extern from 'mylib.h':
        enum: CONST_A

is not a named enum.

> Yep. You might want to consider creating a github fork.

I've started a branch 'cdef-enums-stucts-and-unions' (no activity
yet).

Repo:
  http://www.physics.drexel.edu/~wking/code/git/cython.git

Gitweb interface:
  http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git

I can put this up on GitHub if you want, but it's easier for me to
self-host, and `git fetch` works just as well either way ;).

-- 
This email may be signed or encrypted with GPG (http://www.gnupg.org).
The GPG signature (if present) will be attached as 'signature.asc'.
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110217/c7f87a9a/attachment.pgp>