[AstroPy] Deployment and packaging

Wed Jun 15 02:29:20 EDT 2011

I also agree that another python distribution is not the way to go, at
least for a time, and that we should be aiming for "expert" users.  In
fact, I think most people working in astronomy are in some weird
middle ground between users and developers, so they're primarily in
that category.

I also really don't think a *directly* EPD-based solution is a good
thing to do.  It should definitely be kept compatible with EPD, but if
someone wants to by hand install and compile only the packages that
are needed for the astronomy package, that should also be an option.
That's the whole point of packaging, after all, and it would be
foolish to try to do it ourselves when all the tools that have been
mentioned already are available and work just fine (well,
sometimes...).

Larger scale, I see two different problems that are being discussed here:

1) What's the best way to package up a new python library with it's
python dependencies (possibly with standalone C-extensions)?
To me the answer is clear: PyPI with standard python tools. If
desired, it's easy to plug into those tools, bootstrapping the library
to install dependencies as needed.

2) How can we provide everything anyone using a python library might
want, even if it's not in python?
This is a *much* harder problem, and while it would be great to solve
this problem, for now I think the resources should be devoted mostly
to solving #1, given that there are other package management tools
exist. It's fairly straightforward to make the library link into
whatever extensions are present, and present helpful messages as to
where to get them if they are not.

PyPI and it's associated tools pretty much work, and it's quite easy
to write tools to make use of it to download and install arbitrary
packages.  I've done exactly this as part of astropysics, and it has
worked fine on Mac and two different Linux distros (although perhaps
there are Windows complications?)...  If the astro library goes in a
direction of a "scikits" approach, having everything on PyPI will make
it much easier to make tools that auto-install whatever kits are
desired (something scipy/scikits seems to have trouble with, because
the kit concept was tacked on later). Furthermore, a lot of the
packaging people who work on the big packaging projects use PyPI as an
easy way to monitor and update for new releases, so it's in some sense
using PyPI is two birds with one stone.

PyPI also handles binaries, although admittedly not as cleanly.  But
aren't most of the "expert" users going to have compilers? In that
case tools like pip easily install anything with a C-extension from
source.  If the concern is about binary *dependencies* that aren't in
the python ecosystem at all, the dizzying array of possible platforms
and versions to support makes me think that it's a problem that we
don't have the people-power to solve.

If someone wants the astronomy python library to call SExtractor, some
random C-code with a command line interface, or even some IDL routine,
it's actually quite easy to do that using inter-process communication
(subprocess.Popen and the like), and then it's left to the user to
install the outside requirement however it wants to be installed.
I've used this approach multiple times, and it has always worked fine
(even across different platforms if you do the python code right, as
long as the 2nd package works on said platform).  If a direct link to
some C-extension is needed, that is probably something to be avoided
as much as possible (or rendered optional a la the way the GUI and
OpenGL pieces of Traits are optional).

On Wed, Jun 15, 2011 at 12:50 AM, Vicki Laidler <laidler at stsci.edu> wrote:
> I too think that the goal of this effort should neither start with yet another python distribution, nor produce a mixed general-purpose astronomy library containing IRAF, SExtractor, or whathaveyou. Both those items produce significant barriers at both ends (the developer/deployment end, and the user/installation end).
>
> Let's at least start out by shooting for a scikit-like python library (C extensions allowed(?)) so that people who really just want a coherent python astronomy library (a la the Astron library in IDL) can have one, and people who want to make a more general astronomy library (like the unified release) can include the resulting product as part of it.
>
> Meanwhile, we can work on collecting some survey data from our potential users, segmented by the various user communities of interest -- I thought Andrew's point that it's the more expert users we want to attract to the project was a very good one; and if need be, revisit the question when we have actual measured data rather than anecdata on which to base the decision.
>
> Vicki
>
> ________________________________________
> From: astropy-bounces at scipy.org [astropy-bounces at scipy.org] on behalf of Andrew Williams [andrew at physics.uwa.edu.au]
> Sent: Wednesday, June 15, 2011 12:01 AM
> To: astropy at scipy.org
> Subject: Re: [AstroPy] Deployment and packaging
>
> On 15/06/2011 10:43 AM, Tommy Grav wrote:
>>
>> On Jun 14, 2011, at 10:11 PM, Perry Greenfield wrote:
>>> IRAF is only one example. SExtractor is another. And there are
>>> others. There isn't yet enough to replace these, and won't be for
>>> some years (if ever, it is probably silly to think everything we
>>> need will be in Python, or that there won't be some very useful
>>> things out there not written in Python).
>>
>> My opinion is that those tools belong in as separate packages. To me
>> they just don't
>
> I strongly agree - people who work with tools like these already will be
> _less_ likely to adopt an 'astropy' package if it also includes an extra
> copy of SExtractor, IRAF, and every other tool they might possibly want,
> generating conflicts with anything that's already installed.
>
> 'All in one' setups like this make life easier for people setting up new
> computers for training workshops or for new postgrads, but they make
> life a lot harder for people who already have the tools they need set
> up, and don't want to break them by installing some monolithic
> collection of loosely coupled packages. It's that latter set of expert
> users who are the ones you most want to get involved using and
> contributing to the collection...
>
> For the same reason, I'm also very much against the idea of turning this
> into another complete python distribution. Apart from Windows users,
> most people already have Python installed with the operating system.
> Installing a second copy (and getting all the other libraries you want
> to use installed under _both_ Python versions) is a real pain.
>
> There's nothing to stop people re-packaging 'astropy', or whatever the
> Python-only collection gets called, along with SExtractor, IRAF, numpy,
> and everything else, if there is demand for an 'all in one' collection.
>
> I'm more used to catering for the other end of the user base - the
> people who, at worst, _only_ have access to software that's part of the
> default OS install, or at best, packages maintained in the ubuntu/fedora
> software systems. For example, I wrote a pyfits-like FITS file library
> in pure Python for people who couldn't get pyfits installed on the
> machine they had to use...
>
> Another point - if this 'astropy' collection is maintained as a
> ubuntu/fedora package (plus the OSX equivalent, I forget the name), it
> will expand the available user base hugely - an awful lot of people use
> computers they don't have root access for. It's really, really hard to
> convince most sysadmins to download some random piece of software off
> the internet and install it on a machine unless it's maintained in the
> software packaging system for that OS.
>
> The more things you add, like SExtractor and numpy (which are both
> already distinct ubuntu packages), the harder it will be to package it
> properly that way.
>
> Andrew
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>

-- 
Erik Tollerud