[AstroPy] Co-ordinating Python astronomy libraries?

[Erik Tollerud]

> Is there a standard way of combining separate packages, potentially
> developed in different repositories, into a single meta-package for
> distribution to the user? Does distutils or distribute allow something like
> this by default?

This idea of a bunch of metapackages sounds great - see below for
related thoughts.  setuptools (and hence, distribute) has the option
for "extras" - you add an extras_requires argument to the setup
function with a dictionary that maps names of a extra features to a
list of the associated packages.  So you could define a mostly empty
metapackage with just a bunch of extra_requires entries, and then
users can do

easy_install astrometapackage[optical]

to install all the optical astronomy-related packages, or

easy_install astrometapackage[all]

to get all the contained packages.

Having said this, I recall a discussion a while back on the distribute
mailing list where they were talking about not including the "extras"
feature in future releases, as it's difficult to maintain the
dependencies or something like that.  I don't know if this is actually
going to happen, but apparently it's a little-used feature... I'm also
not sure if it works with pip (the easy_install replacement to go with
distribute)


> The root of the problem is that there was no centralized build and
> test suite, nobody managing unified, integrated build testing and
> resolving the problems with the code maintainers.

A well-organized centralized scheme is definitely a good idea (you're
probably right that it'd be great for SciPy), but it seems like a
large chunk of the astronomy community doesn't work that way (and
probably never will).  Better to have the *option* of a complex
organizational/testing framework without the *requirement*.  It should
be easy to let projects naturally scale to a common organizational
framework, but only as/if they need to



> For those reluctant, we are curious about what the main obstacles are.

I'm not exactly clear on what the repository you're speaking of
encompasses (and hence what the obstacles are).  Is this just a
repository storing a whole bunch of seperate packages, or actually a
common package that is distributed with a single "python setup.py
install"?

If it's the former, the main thing in my mind is that it seems highly
redundant - there around exist repositories that are excellent (google
code, launchpad, bitbucket, github, etc.), and It's not clear to me
what is gained by making an "astronomy repository" when these other
options are most likely to have anything we would need and probably
will be more consistently maintained/upgraded given that they
typically are focused on just being a hosting service.  I also am
definitely not a fan of Trac or SVN.  The issue with Trac may be
mostly just personal preference - it always just seems kind of clunky,
behind-the-times, and difficult to use, although maybe I just didn't
try hard enough.   SVN, though, would be a non-starter -- distributed
VCS like Mercurial, git, or Bazaar are much better suited to the
rather common astronomy situation of a bunch of individuals hacking
away at something when time permits, while still supporting the
standard SVN/CVS-like centralized workflow.

If it's the latter, the biggest problem to me is that all of a sudden
it becomes *much* harder to manage distribution and have control over
the "vision."  I also can't help but think installation and
dependencies would quickly become a huge hassle given the huge
diversity of hardware and software needs that various parts of the
astronomy community generally has.  This is the main thing that would
make me leery of a centralized build/testing scheme of the depth
described above - lots of astronomers want to hack something quickly
together that "just works" and will not bother if they have to jump
through many hoops to contribute.


In my mind, the most useful thing in the short-term would be a
centralized (and actually close-to-complete!) listing of astronomy
packages well-organized by topic, with the *option* of having that
site also be the repository/host/bug tracker/wiki.  There would be an
expectation that listed packages clearly specify their requirements
and post them to pypi.python.org in the standard setup.py way .  All
the mechanisms already exist within distutils and distribute - it
there's an organized listing, it's trivial to write scripts that will
use the already-existing pypi mechanisms to either generate
metapackages as described above, or install various groupings of
astronomy packages even if the extras option disappears.  It's not
quite as straitforward, but still probably only moderately difficult
to combine these into system-specific packages.  The could also be
lots of neat ways to encourage people to combine closely related
packages using this sort of scheme.   It seems to me this can greatly
simplify the distribution problems while still leaving a lot of
flexibility for a community that's famously (notoriously?)
independently-minded.


> AAS splinter meeting, anyone?
+1



-- 
Erik Tollerud