[Cython] Cannot cythonize subclasses of setuptools.extension._Extension

Tue Apr 19 08:35:03 EDT 2016

This strayed a bit from the original topic, but yes, I should bring this up
with setuptools project. Thanks for the feedback.
(My feeling was that this was a somewhat stalled problem, reported in two
issues and persisting after a couple of years:
#209
<https://bitbucket.org/pypa/setuptools/issues/209/setup_requires-and-install_requires-dont>
and #391
<https://bitbucket.org/pypa/setuptools/issues/391/dependencies-listed-in-both-setup_requires>
)

As you said, even if it is a hackish workaround here, Cython should still
allow setuptools.extension._Extension.

On Tue, Apr 19, 2016 at 11:48 AM, Erik Bray <erik.m.bray at gmail.com> wrote:

> On Mon, Apr 18, 2016 at 6:56 PM, Manuel Nuno Melo
> <manuel.nuno.melo at gmail.com> wrote:
> > Ah, sorry Erik, you're absolutely right. I mixed my results a bit and
> must
> > elaborate:
> >
> > 'setup_requires' on its own will indeed not generate the behavior I
> > described.
> >
> > However, if you define the same dependency under 'setup_requires' AND
> > 'install_requires', then you get the mess I mentioned. Essentially, when
> you
> > reach 'install_requires' setuptools decides that the egg it has under
> .eggs
> > suffices, and just links to it.
> >
> > This isn't an issue when depending on cython because it's really just a
> > setup-time dependency. However, in our package we depend on numpy at both
> > setup- and runtime, and therefore makes sense to have it under both
> > 'requires' flags.
> >
> > This is why hooking in at the setup_requires step could be a potential
> > workaround, if we could get it to do a full dependency install instead of
> > local egg. I'm currently finishing up this approach.
>
> Interesting.  This sounds to me like a regression, as I don't think it
> used to do that.  This should probably be brought up on distutils-sig
> or in the setuptools bug tracker, rather than spending a lot of time
> hacking together a workaround or requiring changes specifically in
> Cython (though I don't disagree that distutils.Extension should be
> allowed in the first place).
>
> IIRC it used to be that if the same package was specified in both
> setup_requires and install_requires it would be built/installed twice.
> This was recognized as inefficient, and that the copy installed in the
> local .eggs directory satisfies an install_requires requirement it
> should be copied from .eggs to the system site-packages (or whatever
> the install target is).  I remember there at least being *discussion*
> about that, but not sure what changes were made.
>
> It sounds like in your case it's just adding Cython from your .eggs
> into easy-install.pth which I agree is wrong.  If that's the case it
> should be fixed in setuptools over all else.  I'll see if I can
> reproduce it.  Jason Coombs who mantains setuptools is usually quick
> to release bug fix versions if he gets a patch.
>
> Erik
>
> > On Mon, Apr 18, 2016 at 11:16 AM, Erik Bray <erik.m.bray at gmail.com>
> wrote:
> >>
> >> On Sat, Apr 16, 2016 at 1:29 PM, Manuel Nuno Melo
> >> <manuel.nuno.melo at gmail.com> wrote:
> >> > Hi Erik,
> >> > Please post your solution; I'm curious to see it.
> >>
> >> Will do in a bit. I need to see if I can distill it somewhat from its
> >> original context so that it can be better understood.
> >>
> >> > Currently, we're also using setup_requires but are moving away from it
> >> > because of two main issues:
> >> >
> >> > 1- there's no flexibility to hook into it via setuptools.Distribution
> >> > subclassing (unless you rewrite the entire __init__);
> >>
> >> I don't really think that's a big issue but maybe you've encountered
> >> some reason that it is.
> >> The worse thing about it is the general chicken-egg problem that you
> >> need to call setup() to get your setup_requires, but you may need some
> >> packages pulled in from setup_requires in order to do anything useful
> >> with setup().
> >>
> >> d2to1 [1] solves this problem quite well, but unfortunately
> >> development on it is stalled for the time being.  Another alternative
> >> that can work quite well is some version of Daniel Holth's
> >> "setup-requires" hack [2].
> >>
> >> > 2- (and more serious) setuptools installs setup_requires dependencies
> in
> >> > the
> >> > local directory, not as a system-wide install.
> >>
> >> That's a feature, not a bug.  Build requirements for a package might
> >> be in conflict with the versions of those same requirements you have
> >> already installed.  It also prevents things from being installed that
> >> are *not* required at runtime.  It makes it easier for projects to pin
> >> a little more strictly to their known-working build requirements,
> >> without the constraints imposed by supporting a wider range of runtime
> >> requirements.
> >>
> >> > Even worse, it then puts a
> >> > path link to the current directory under the system-wide
> >> > easy_install.pth.
> >> > This means that if you install from the source directory via 'sudo
> >> > ./setup.py install', you get a Cython egg downloaded into that
> >> > directory,
> >> > and its path added to sys.path (!!). Needless to say this can break in
> >> > many
> >> > nasty ways...
> >>
> >> Huh??  I'm not sure we're talking about the same thing now.  The local
> >> .eggs cache directory created for setup_requires downloads most
> >> certainly does not get added to easy_install.pth.  Are you sure you're
> >> not thinking of `setup.py develop`?
> >>
> >> [1] https://pypi.python.org/pypi/d2to1
> >> [2] https://bitbucket.org/dholth/setup-requires/src
> >>
> >> > On Sat, Apr 16, 2016 at 1:10 PM, Erik Bray <erik.m.bray at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Apr 14, 2016 21:07, "Manuel Nuno Melo" <
> manuel.nuno.melo at gmail.com>
> >> >> wrote:
> >> >> >
> >> >> > Our need to control cythonization comes from the fact that we
> >> >> > implement
> >> >> > cython as a lazy and optional dependency. Lazy in the sense that we
> >> >> > delay as
> >> >> > much as possible cythonization so that setuptools or pip have time
> to
> >> >> > install cython, if needed. Optional because we distribute both .pyx
> >> >> > and
> >> >> > cythonized .c files, and decide on which to use based on user
> flags.
> >> >> >
> >> >> > We, therefore, need an Extension class that only cythonizes if we
> >> >> > decide
> >> >> > to.
> >> >> >
> >> >> > Thanks for the feedback,
> >> >> > Manel
> >> >>
> >> >> I have a solution for exactly this in astropy_helpers which makes it
> >> >> possible, for example, to pull in Cython via setup_requires, but only
> >> >> if
> >> >> it's needed.
> >> >>
> >> >> Remind me on Monday and I can point out how it works.
> >> >>
> >> >> > On Apr 14, 2016 8:17 PM, "Matthew Brett" <matthew.brett at gmail.com>
> >> >> > wrote:
> >> >> > >
> >> >> > > On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray <
> erik.m.bray at gmail.com>
> >> >> > > wrote:
> >> >> > > > On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo
> >> >> > > > <manuel.nuno.melo at gmail.com> wrote:
> >> >> > > >> Hello devs,
> >> >> > > >>
> >> >> > > >> I'm developing the setup.py for a scientific package,
> MDAnalysis
> >> >> > > >> (see PR
> >> >> > > >> #799). We depend on distutils and setuptool. Namely, we use
> >> >> > > >> setuptools.extension.Extension class for our extensions.
> >> >> > > >>
> >> >> > > >> Some older versions of setuptools (<18.0) do filename
> >> >> > > >> cythonization
> >> >> > > >> themselves upon initialization of the Extension object.
> >> >> > > >>
> >> >> > > >> Because we want to control name cythonization ourselves I try
> to
> >> >> > > >> directly
> >> >> > > >> use distutils.extension.Extension, which has none of
> setuptools'
> >> >> > > >> cythonization. However, this doesn't work because setuptools
> >> >> > > >> patches
> >> >> > > >> distutils, so that distutils.extension.Extension effectively
> >> >> > > >> becomes
> >> >> > > >> setuptools.extension.Extension.
> >> >> > > >
> >> >> > > > I'm wondering what it is specifically you need to do in your
> >> >> > > > subclass--might it still be possible to do with a subclass of
> the
> >> >> > > > setuptools Extension?  Not saying I disagree with the overall
> >> >> > > > idea,
> >> >> > > > but I also wonder if there isn't a better way.
> >> >> > >
> >> >> > > I know this is a terrible and ugly hack, but the projects I work
> in
> >> >> > > have a 'fake_pyrex' directory, that fools setuptools into
> thinking
> >> >> > > that 'pyrex' is installed, and therefore prevents it from doing
> the
> >> >> > > .pyx -> .c filename conversions in the extension:
> >> >> > >
> >> >> > > https://github.com/regreg/regreg/blob/master/setup.py#L33
> >> >> > >
> >> >> > > Cheers,
> >> >> > >
> >> >> > > Matthew
> >> >> > > _______________________________________________
> >> >> > > cython-devel mailing list
> >> >> > > cython-devel at python.org
> >> >> > > https://mail.python.org/mailman/listinfo/cython-devel
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > cython-devel mailing list
> >> >> > cython-devel at python.org
> >> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >> >
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> cython-devel mailing list
> >> >> cython-devel at python.org
> >> >> https://mail.python.org/mailman/listinfo/cython-devel
> >> >>
> >> >
> >> >
> >> > _______________________________________________
> >> > cython-devel mailing list
> >> > cython-devel at python.org
> >> > https://mail.python.org/mailman/listinfo/cython-devel
> >> >
> >> _______________________________________________
> >> cython-devel mailing list
> >> cython-devel at python.org
> >> https://mail.python.org/mailman/listinfo/cython-devel
> >
> >
> >
> > _______________________________________________
> > cython-devel mailing list
> > cython-devel at python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
> >
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20160419/d231f90a/attachment-0001.html>