[AstroPy] Deployment and packaging

[Perry Greenfield]

Some points I would like to make

1) It's clear that deployment and packaging should not be directly  
tied to the effort to develop more general and consistent astronomical  
modules for Python. This  really is a different issue than the other  
effort and nothing about this should impede that.

2) I'd say that I think all (or almost all, anyway) agree that these  
modules should support standard Python installation schemes (e.g.,  
PyPI support), and that one should not have to use some monolithic  
binary install or anything like that if one doesn't want to install  
them that way. In some respects, this is an extension of point 1.

3) It's good to understand that the typical expert on this list  
doesn't represent the average user that might want to use this  
software. Typically those participating in this discussion have  
significantly more expertise in doing software installations than the  
typical user does.

4) It's also good to understand that many users out there don't have  
root access, and don't necessary have support staff to do what they  
would like as far as root installations as Tom mentions. It's true at  
CFA and STScI, for example. This means that tools that may work  
wonderfully for you, such as RPMs, are useless to these people. Some  
have railed at the stupidity of not allowing them root, it's a fact of  
life at a number of institutions, and it isn't likely to change.

5) Besides the root issue, what works well for you, doesn't  
necessarily work well for everyone else. There is a great variety of  
systems out there that confound standard installations. Unless you  
have experience distributing software that has lots of dependences,  
you may not be aware of all the problems you may run into. As some  
have mentioned, most users will give up at the first installation  
problem. The greater the number of separate installations one has to  
do, the greater the variety of the installation scheme (from make  
variants, to distutils, to...) the more likely problems are to occur  
and the greater the likelihood of the user giving up.

6) With the standard, individualized installations one can run into  
dependency conflicts where the different packages have inconsistent  
dependency requirements. Hopefully coordinated development can avoid  
this (though there may be occasions where it is unavoidable), but  
really what is needed is regular regression testing of the packages  
with each other. This is something we (STScI) are planning on as part  
of our distribution, but it is an activity that needs to be done in  
any event.

7) Speaking of dependencies, distribution systems out there (e.g.,  
RPMs) help, but they also are subject to dependency conflicts. This is  
an area that really has no easy solution. The only existing, reliable  
solution is to control what versions are packaged together explicitly,  
and test them. This leads to two conflicting desires. The packaging  
and testing together takes time, and the more one includes, the more  
the effort (one reason why things like Linux distributions run behind  
in versions by substantial amounts). It's reliability and consistency  
vs. how current it is.

8) The scisoft distribution has some issues that people have pointed  
out already. But a lots of people have used it just because it is so  
simple to install. I have been asked often to support something like  
that for our software. But it isn't easy to update items and that a  
significant problem.

9) Is a larger package a real problem for people? Suppose it includes  
X, Y, and Z for which you have no interest in having. So long as they  
don't interfere with the installation, or break software you do care  
about, why do you care if there are there? (so long as the space  
required isn't massive or the download time isn't horrible).

10) One objection is a big binary install isn't flexible. You get what  
you get, and you can't  update it. We do plan to make it possible to  
update modules within it, and add modules to it. It is true that doing  
so uses a different scheme than the standard approach and it is more  
work. But...

My final point is: So what alternate packaged binary (or similar)  
solution that exists now that we should be using that doesn't require  
root, and doesn't require N different versions for M platforms? My  
impression from all the discussion that there isn't one. It isn't like  
we are going our own way when there is already a good solution out  
there. People don't like (for good reasons): scisoft, EPD, Sage,  
etc... So why object to trying something else? If it doesn't work, it  
won't hurt the standard way of installing software, will it? If it  
does work, it gives those that don't want to manage 18 separate  
installations something that should work in the majority of cases with  
little pain.

Perry