[Cython] Static checker for cython extension dependencies?

[Robert Bradshaw]

On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Hmm, so I've looked into this a bit further, and it looks like the metadata
> isn't going to be useful for us. Many of our extensions can't be
> autogenerated by cython because they have non-trivial dependencies

meaning? In my cursory glance, these all look fine, but perhaps you
could point me at a case that isn't very nice. Also, cythonize isn't
necessarily all-or-nothing--you can always manually add Extensions to
your list.

> or depend directly on C code.

cythonize handles dependencies on C code just fine

> For these extensions, cythonize must be passed an
> instantiated Extension object, and the metadata embedded in the compiled C
> file just uses whatever metadata is attached to the Extension object.

The embedded metadata is the union of what was in the Extension object
and the extra dependencies it found.

> I think my original idea - to parse pyx files for cimports and add a test -
> should allow us to manage this more safely in the future while still
> maintaining fine-grained control over dependencies of C extensions in our
> setup.py file.
>
> That said, I'd love to hear alternate ideas.

"Parse pyx files for cimports" (recursively, handing includes and cdef
extern from ...) is essentially re-implementing cythonize. What is the
advantage of maintaining them by hand? Cythonize was invented because
people are really bad at maintaining these lists manually...and they
can be entirely deduced.

> On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw <robertwb at gmail.com> wrote:
>>
>> We write metadata in the generated C files for just this reason. You
>> can fall back to something like
>> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if
>> cython is not available.
>>
>>
>> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> > The reason we haven't done that is we would like our setup.py script to
>> > be
>> > runnable without cython being installed. I think cythonize is being
>> > invoked
>> > (or something similar to it) by setuptools, using a feature added in
>> > setuptools 18.0:
>> > https://setuptools.readthedocs.io/en/latest/history.html#id60
>> >
>> > Is there a way to use cythonize for this build workflow without
>> > importing it
>> > at the top-level in our setup.py file?
>> >
>> > FWIW, our setup.py file is here:
>> >
>> > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default
>> >
>> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw <robertwb at gmail.com>
>> > wrote:
>> >>
>> >> You should be using cythonize rather than listing and maintaining the
>> >> Extension definitions themselves.
>> >>
>> >>
>> >>
>> >> http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils
>> >>
>> >> https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing
>> >>
>> >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum
>> >> <nathan12343 at gmail.com>
>> >> wrote:
>> >> > Hi all,
>> >> >
>> >> > I'm working on a pretty large python/cython codebase (yt) that has
>> >> > many
>> >> > interdependent C extensions written in cython.
>> >> >
>> >> > I've found it to be pretty hit or miss to depend on contributors to
>> >> > manually
>> >> > update cython dependency information in our setup.py file. The
>> >> > dependency
>> >> > information seems to only be used by setuptools to trigger
>> >> > recompilation,
>> >> > but critically setuptools will happily compile a C extension for the
>> >> > first
>> >> > time if the depends information is incomplete. This means that during
>> >> > development, if I update a cython routine, there's no way to ensure
>> >> > that
>> >> > other cython routines that cimport the one I modified will be
>> >> > recompiled
>> >> > unless I manually ensure the depends information is updated whenever
>> >> > cython
>> >> > code gains or loses a cimport.
>> >> >
>> >> > To make that more concrete, here's a pull request I just made to yt
>> >> > that
>> >> > adds missing dependencies for a cython header. Without this pull
>> >> > request,
>> >> > setuptools fails to recompile these routines when
>> >> > selection_routines.pxd
>> >> > changes, causing a build failure.
>> >> >
>> >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220
>> >> >
>> >> > I think it should be possible to write a test for this by defining
>> >> > the
>> >> > dependency information outside of setup.py and parsing grep and
>> >> > looking
>> >> > for
>> >> > all cython files that cimport other cython files defined inside yt.
>> >> > However,
>> >> > before I do that, I'm curious whether anyone has done something
>> >> > similar,
>> >> > or
>> >> > if there is some other way of forcing the dependency information to
>> >> > be
>> >> > complete on the first compilation, rather than just for subsequent
>> >> > incremental recompilations during development.
>> >> >
>> >> > Thanks for your help!
>> >> >
>> >> > -Nathan
>> >> >
>> >> > _______________________________________________
>> >> > cython-devel mailing list
>> >> > cython-devel at python.org
>> >> > https://mail.python.org/mailman/listinfo/cython-devel
>> >> >
>> >> _______________________________________________
>> >> cython-devel mailing list
>> >> cython-devel at python.org
>> >> https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > cython-devel mailing list
>> > cython-devel at python.org
>> > https://mail.python.org/mailman/listinfo/cython-devel
>> >
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> https://mail.python.org/mailman/listinfo/cython-devel
>
>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>