[AstroPy] Deployment and packaging

[Erik Bray]

Hello all,

I'm just getting caught up on this discussion, though Andrew's message 
already pretty much encapsulates my thoughts on this thread so far.  I 
do have some additional notes inlined below.  But first I'd just like to 
add that packaging/deployment is of particular interest to me, and it's 
something I have a bit of experience in.

I've pulled together some large-ish projects from disjoint pieces and 
made them work quite nicely on different platforms.  This includes pure 
Python projects, as well as projects that include other big system 
dependencies including libraries, web servers, etc.  (Actually, to be 
specific, one such project involved my own Ubuntu fork based on a 
specific set of packages, so I'm quite familiar with 
Ubuntu/Debian/apt/etc.)  Unfortunately Macs are a weak point for me.

I also have pretty deep knowledge of Python's packaging mechanisms, 
including the new 'packaging' module (formerly known as distutils2).  So 
I think I can contribute there.

I agree with Andrew that an AstroPy effort should focus on being based 
on one or more "pure" Python packages.  By "pure" I don't mean to 
exclude C extensions of course; and pre-built binary packages should be 
maintained for as many platforms and Python versions as possible, to 
make things easier for users who don't have development tools on their 
systems.  This is, of course, especially important for Windows.  But 
it's important for all other platforms too--just the other day I 
couldn't get matplotlib to build due to a dependency on libpng, for 
example.  Numpy can also be tricky.  But fortunately Numpy and 
matplotlib are already available as binary packages for most platforms.

As for packages that have external, non-Python dependencies (PyRAF for 
example), I do not by any means think they should be excluded 
altogether.  But they should be able to work on a multitude of system 
configurations, and should be excluded from any installation where their 
dependencies are not bet.  At any rate, where IRAF is concerned the goal 
should be to gradually replace its pieces with Python modules.  PyRAF is 
invaluable in the meantime, but in a "perfect" world it wouldn't have to 
exist at all :)

Now if there was some desire to work on a Python + numpy + IRAF + 
SExtractor + ... all encompassing distribution (as I know there is) that 
*could* also be organized under the AstroPy banner.  But I think it 
should consist of one or more separate sub-projects, loosely coupled so 
that the best practices can be followed for different target platforms 
(i.e. .deb distributions, rpm distributions).  The pure Python part may 
have certain dependencies, but it should not be targeted for any one 
specific system layout.


On 06/15/2011 12:01 AM, Andrew Williams wrote:
> 'All in one' setups like this make life easier for people setting up new
> computers for training workshops or for new postgrads, but they make
> life a lot harder for people who already have the tools they need set
> up, and don't want to break them by installing some monolithic
> collection of loosely coupled packages. It's that latter set of expert
> users who are the ones you most want to get involved using and
> contributing to the collection...

Exactly.  We use such 'all in one' setups here, but I never use them for 
development because they're installed as root!  I have my own copies of 
them to hack on, but most of the time it's inconvenient.

> For the same reason, I'm also very much against the idea of turning this
> into another complete python distribution. Apart from Windows users,
> most people already have Python installed with the operating system.
> Installing a second copy (and getting all the other libraries you want
> to use installed under _both_ Python versions) is a real pain.
>
> There's nothing to stop people re-packaging 'astropy', or whatever the
> Python-only collection gets called, along with SExtractor, IRAF, numpy,
> and everything else, if there is demand for an 'all in one' collection.

On Ubuntu/Debian this could be a meta-package that installs all 
requirements at the system level.  I definitely think this should be 
considered, but it should not be required either.

> I'm more used to catering for the other end of the user base - the
> people who, at worst, _only_ have access to software that's part of the
> default OS install, or at best, packages maintained in the ubuntu/fedora
> software systems. For example, I wrote a pyfits-like FITS file library
> in pure Python for people who couldn't get pyfits installed on the
> machine they had to use...
>
> Another point - if this 'astropy' collection is maintained as a
> ubuntu/fedora package (plus the OSX equivalent, I forget the name), it
> will expand the available user base hugely - an awful lot of people use
> computers they don't have root access for. It's really, really hard to
> convince most sysadmins to download some random piece of software off
> the internet and install it on a machine unless it's maintained in the
> software packaging system for that OS.
>
> The more things you add, like SExtractor and numpy (which are both
> already distinct ubuntu packages), the harder it will be to package it
> properly that way.

Which is why I think any software collections should be designed for 
specific platforms in mind.  If we're talking Ubuntu, most of the 
required packages already exist.  So the only work would be making 
packages for those requirements that don't already exist, and managing 
the dependencies.  On a different platform, say, Windows, it might be 
necessary to include more software in the distribution.  But I think 
it's futile to make one exactly common base of installed software for 
all possible platforms, and to me this kind of defeats the purpose of 
basing our efforts around Python (where most of the platform 
interoperability issues should be managed at the Python level).

Eri