[Distutils] Role of setuptools and eggs in "modern" distributing...

[Chris Barker]

Hi folks,

I'm trying to package up a complex system and would like to do it the
correct, modern way.

In particular, this involves a bunch of compiled extensions, as well as
dependencies on both the scientific "stack" and common Web app packages.

(can you tell I'm building a web service front-end for computational code?)

This is actually a bit of a trick, because the web development community is
generally doing a good job up supporting PyPi and pip, whereas the
complications of the scientific software tools have folks relying more on
tools like Enthought and Continuum.

So far, we've been doing mostly pip and struggling with build our own for
the ugly scientific stuff (whoo hoo, fun with HDF and netcdf, and GDAL,
and....). But at the end of all this we'd like to be able to distribute and
make it easy on end users to use our tools.

I figure we we'll do one (or both) of:
- providing a custom "wheel house" with our packages and the dependencies
that are hard to come by
- provide a binstar channel with conda packages of all the same stuff but a
totally different set of "other" packages.

At the moment, I'm working on making conda packages, which brings me to my
questions.

I'm a bit confused about the role of setuptools with pip. On the one hand,
pip depends of setuptools. On the other hand, pip supposed doesn't "do"
eggs, and prefers wheels. But I find that I keep getting eggs whether I
want them or not. IN fact, as far as I can tell, the way to get pip to
instal something from git repo is:

git+https://url_to_the_repo.git#egg=name_of_package

why isn't that "wheel=name_of_package"

and will it work if setuptools was not used in the packages setup.py???

Frankly I've generally found setuptools and eggs to be overly heavy weight
and annoying -- the only reason I use setuptools at all is that I REALLY
like develop mode -- which I can now get with "pip --editable" or  does
that give me setuptools develop mode anyway, i.e. do I need to have used
setuptools.setup for it to work?

So question one: do I need to use setuptools.setup rather than plain old
distutils.setup?

Question 2:

What about setuptools: "install_requires"

I generally like the pip requirements.txt approach. It's up to the
installation tool, not the packaging tool to mange requirements. But then
again, it does make sense to declare the requirements in setup.py. But the
issue at hand is that install_requires is doing some odd things with conda:

conda, of course, is designed to manage dependencies itself, and those are
declared in the conda build meta.yaml file. Note that conda dependencies
can have nothign to do with python -- the whole point of conda -- a conda
pacakge can depend on any other conda package, including C libs, etc.

But the issue at hand is that conda build doesn't re-invent setup.py -- but
rather you generally simple call "setup.py install" from your conda build
script. Hence th issue at hand:

Using setuptools.setup, and specifying "install_requires", then kicks in
setuptools trying to resolve dependencies I don't want it to deal with.

I read Donald's "setup.py vs requirements.txt", and I guess I get it, but
it still seems quite redundant -- I see the point of separating out
 “abstract dependencies” and  “concrete dependencies”. However, the nifty
solution of only putting:

--index-url https://pypi.python.org/simple/

in the requirements.txt doesn't work for complex cases. In practice, we
find we need to specify version numbers of some dependencies, and have some
go strait into a git repo, etc. So we need a messy requirements.txt file.

And, in fact, I think that's where is belongs -- the goal of the
requirements.txt file is so pip can "do the right thing" and go grab
everything you need. But, in fact, it also is quite human readable, and
serves quite well as the documentation for the "abstract dependencies" as
well.

So I think the way to go is to keep requirements in requirements.txt, and
leave them out of the setup.py.

Can I dump setuptools.setup, as well??

Sorry, this is a bit rambling and short of clear questions, but I'm trying
to get a handle on what best practices really are these days...

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20141223/ea96bc5e/attachment-0001.html>