[AstroPy] Deployment and packaging

[Luigi Paioro]

Hello,

   I agree with Matt, Robitaille and Prasanth.

I see four different points raised during this huge thread:

1. Build up one (and just one) package (or set of non duplicated 
packages) containing the most complete and interoperable Python software 
for astronomy;

2. With respect to 1, how to handle third party dependencies, in 
particular if it is not Python stuff, e.g. legacy software like 
SExtractor, Galfit, IRAF, etc.;

3. For the astronomical Python package(s), which packaging adopt;

4. Whether or not make an effort to produce a reference Python & C. 
distribution (maybe binary) like EPD etc.


With respect to point 1. I certainly think that it would be great, but I 
would prefer having several small and well focused interoperable 
packages than a single huge package all comprehensive. Provided that all 
packages are developed by the same community (but each single package by 
a smaller team), an packaged and developed following the same 
"astronomical framework", then, in my opinion, with this approach the 
target would be centred in a more flexible and effective way,


With respect to point 2, I think that for the time being we can assume 
that the third party software is installed separately as a requirement, 
and we just provide the Python bindings (by the way, I have already 
produced a Python wrapper to SExtractor as a demo within FASE project, 
but I assume that SExtractor is installed and in the PATH). In any case 
I think that, once we have this Python framework well defined and 
stable, then we can try to collaborate with the legacy software 
developers in order to find a shared strategy for the software distribution.


Point 3... no doubts: distutils, pip and PyPI (or a separate 
implementation of PyPI).


With respect to point 4. I bring to you my experience along several 
years distributing Python software within the PANDORA initiative (R.I.P.).

We started producing software with mixed Python and C code. There were a 
lot of dependencies with third party libraries and we decided to build 
up a detailed documentation of all requirements. We didn't distribute 
the binaries of our software, but just the source to be compiled.
Result: most of the end users were discouraged at the first sight of 
such documentation and gave up; those balder that decided to give a try, 
frequently had a lot of installation problems with the third party 
software, and in those rare case where they didn't give up, they asked 
us to solve their problems (sometimes leaving to us their PC!!!); The 
remaining (very few) users that arrived to install our software, usually 
had a lot of problems 'cause there wasn't a complete compatibility 
between the libraries they installed and our software.

A mess.

Then we decided to select the versions of the dependences and build up a 
package installing all the third party software we needed (the add-ons, 
which is more or less the same idea of EPD, Sage, etc.). The add-ons 
solved the problem of incompatibility between our software and the third 
party libraries, but two new problems raised:
a) since we distributed the add-ons as sources which were automatically 
locally compiled by a script we set up, there were still a lot of 
compiling problems of the add-ons, and the users still asked us a lot of 
support, up to giving us their computers (again)!
b) after some years the add-ons were no more compatible with the most 
recent platforms and then we had to bug fix the third party software in 
order to solve their incompatibility (!!!) or upgrade it, with the 
unpleasant consequence of running back all the following 
incompatibilities in our software.

Another mess.

Presently my opinion is that there cannot be a final solution, but, at 
least to simplify the end-user life, it would be nice having an open and 
free reference Python & C. binary distribution (one for platform) a la 
EPD/CASA/etc., and developing the astro-python-packages taking into 
account this reference distribution, and leaving as a recommendation to 
support other contexts. It is clear that producing and maintaining such 
a reference distribution is a job by itself, and it requires a specific 
team of developers and maintainers willing to do this dirty job. 
Moreover, it is clear that this reference distribution must be upgraded 
periodically in order to support the new OSs, and this will implicate a 
periodical upgrade of the astro-python-packages (where needed).


Sorry for this long mail.

Luigi



Il 06/16/11 10:24, Prasanth ha scritto:
> Hello,
>
> I strongly agree with Matt Davis and Thomas Robitaille here.
>
> Many of the packages like asciidata, vo.table, ATpy, APLpy, stsci_python
> etc., are easier to install, if we have a distribution that satisfies
> the basic requirements.
>
> Right now I think the easiest way for a non expert person with no root
> access to quickly get started with using Python, is by first downloading
> EPD/Python(x,y) and then the astronomy package of interest. If there is
> an EPD like free package with most of the core dependencies, not
> specifically astronomy related, then it might be easier for a wider
> range of non expert users to get started easily.
>
> As Matt points out, this could possibly help other user communities as well.
>
> Prasanth
>
> On Thu, Jun 16, 2011 at 7:00 AM, Matt Davis <mrdavis at stsci.edu
> <mailto:mrdavis at stsci.edu>> wrote:
>
>     I can see that there seems to be substantial interest in producing
>     an easy Python installation usable by astronomers. Something like
>     EPD but free, or maybe like Python(x,y) for Windows. In fact,
>     there's no reason something like that needs to be specific to
>     astronomy. All the tools you list would be great for someone in
>     geoscience, medicine, engineering, or many other fields. I think it
>     would be fantastic if we could fill this need, but I don't think
>     it's a need specific to astronomy and for that reason it should be a
>     separate project from AstroPy (with substantial staff overlap, to be
>     sure). I have worked in Earth sciences and they face the same
>     challenges to Python adoption that astronomers do. And there is much
>     to be gained by helping other fields join the Python fold,
>     especially when it comes to things like handling large data sets,
>     statistics, pipelines, parallel processing, imaging, etc.
>
>     AstroPy, I think, should be a project separate from being packaged
>     with anything else. It can have dependencies, of course, and some
>     modules may have more dependencies than others, but the fact would
>     be that someone could install AstroPy tools into a Python
>     environment they made themselves, into stsci_python, into EPD, into
>     some Python distribution we (as a community) make, or into any other
>     Python environment that supports our distribution method.
>
>     - Matt