[Matrix-SIG] re: advocacy

[Perry Greenfield]

Paul Barrett writes:

> I feel that in the astronomical community at least Python is reaching
> critical mass.  More and more people that I talk to are interested in
> using it.  (There is even discussion of tossing the IRAF CL and
> replacing it with Python.)  So we need to make a push now to get a
> basic data analysis system operational, if we are not to loose these
> people.  We need a few more people to step forward and take charge of
> a few critical issues, for example plotting and documentation, if we
> are to succeed.
> 

Before any wild rumors get started, I'd like to clarify the above reference
to IRAF since it is probably based on comments from me. The people that 
distribute IRAF (National Optical Astronomy Observatories) have no plans
to get rid of the IRAF CL any time soon. We at the Space Telescope
Science Institute have written much of our software under the IRAF system
and are looking for an alternate scripting language for use in place
of the IRAF CL. We are investigating the use of Python as an alternate
CL that could still run most IRAF (or STSDAS) tasks. We figure that it 
will take a few months to see if this approach will work. While we
are pretty confident that we can get this to work, we would like to see
it work before we make any solid commitments to using and supporting
Python.

We--like many other astronomers apparently--have noted the shortcomings
with Python as a substitute for either the IRAF CL or IDL. Our list is
pretty much the same but I'd add one or two more items (and briefly repeat
the others). These are ones that we feel must be addressed eventually for
us to completely switch over to Python:

Previously mentioned:

1) Weak plotting support (as compared to IDL)
2) Small number of numerical utilities (as compared to IDL)
3) No support (yet) for FITS (I mean full access to images, keywords,  
   tables,...the whole 9 yards) 
4) Installation hassles.
5) Documentation lacking (especially for NumPy...in the works though)

To which I would add:

6) Weak interactive environment: It is not always appreciated by 
   non-scientists that interactive mode is where most time is spent
   looking at data. To that end, the interactive mode needs some conveniences
   that are currently lacking. Such as:
   - command (and output) logging.
   - easy system escapes (e.g., !ls in IRAF, $ls in IDL)
   - as trivial as it sounds, an intuitive exit command (like 'exit' or 'quit')
   - an easy way of re-executing moderately long command sequences or blocks
     without using an editor (e.g., you invoke a loop of several statements
     repeatedly because you need to tweak some parameters each time).
   - utility functions to start up help, or examine the namespace for
     existing data elements (much like IDL does)
   To these one can raise one more issue. I sense that Python gurus consider
   IDL users to be programming adverse, but in the spectrum of astronomical
   users, those that use IDL tend to be the ones that are most interested 
   in programming. We (STScI) also must concern ourselves with the other
   half (or whatever) that find even IDL too intimidating and are much more
   comfortable with the more shell-like environment of IRAF. There are a 
   lot of users that would not want to have to quote all strings or put
   parenthesis around all argument lists. This doesn't have a simple
   solution but I'm pretty sure that ignoring this issue will preclude
   use of Python by many astronomers (I won't speak for other fields).
7) Image display convenient for astronomers (either by saoimage or ximtool
   lookalikes or connections to either of those so that images could be
   displayed on those tools.   

After such a list some might wonder why we even consider using Python at
all. Essentially the existing situation has even more serious drawbacks
that we are trying to escape.

Should our initial efforts succeed then we need solutions to the above
issues. We make our software available to a wide community for a number
of different platforms so it is simply not a matter of solving these
problems for our own organization. If what we deliver is too hard to 
install or is to hard to use or understand in the wider astronomical
community it will be a failure (I'm sure some would argue that existing
IRAF-based software falls into that category). So it is essential, for
example, that there is a standard distribution and installation mechanism.
Fortunately, there appears to be serious interest in solving this for
the whole Python community. For our initial plans, these issues do not
all have to be solved immediately, but we have to see a way to deal with
them eventually.

Hopefully the FITS issue will be solved soon--I'm eagerly looking forward to
Paul's implementation. (And make that ~5 FITS implementations; I did a 
primitive one for some test programs we were doing here). There is reason
to believe that documentation will improve, and we can augment it where 
necessary. Numerical utilities can be developed as needed. It doesn't
seem that one giant effort is needed there.

The plotting situation seems to be somewhat muddled and one of my more 
serious concerns. I'd rather not spend the effort to develop a package
from scratch. There a number of graphics packages out there but each
has a shortcoming or two that makes us reluctant to use it. Our need
is primarily for simple 2-d plotting tasks (x-y, contour, surface, etc.)
but that has enough control to produce publication quality output.
While we don't currently support Windows  platforms, we don't want to
rule that out. Nor can we distribute commercial software. We would 
prefer it to have an object oriented interface as well. This appears
to rule out most of what is out there. PyGIST is not available on 
Windows, DISTLIN is commercial, GTK appears only to provide plotting
primitives and doesn't run on Windows, VTK is volume rendering based
and has little for 2-d plotting...about the only thing available now
appears to be pgplot.) For the moment, I'm waiting to see what David
Ascher produces; it sounds promising. Complicating our decisions is
trying to decide how Java and JPython will fit into our future. We prefer
a solution that will run in JPython as well (It almost sounds like we
constraining ourselves out of a solution!)

We are working on adding some conveniences to the interpreter by making
our own wrapper. I'm not Guido's new IDLE environment will do it for us.
I haven't had a chance to take a look at it yet.

If we do go with Python it does mean that there will be some institutional
support to help supply what is missing in the above areas.

Perry Greenfield
Space Telescope Science Institute
Science Software Group