[Numpy-discussion] Meta: too many numerical libraries doing the same thing?

[Christos Siopis <siopis@umich.edu>]

I am reviving the 'too many numerical libraries doing the same thing?'
thread, with apologies for the long delay to respond to the list's
comments. Thanks to Joe Harrington, Paul Dubois, Konrad Hinsen and Chris
Barker who took the time to respond to my posting. The funny thing is
that, as i see it, even though they were mostly trying to provide reasons
why such a project has not, and probably will not, be done, their reasons
sound to me as reasons why it *should* be done!

In essence, what i am 'proposing' is for a big umbrella organization (NSF,
NASA and IEEE come to mind) to sponsor the development of this
uber-library for numerical scientific and engineering applications. This
would be 'sold' as an infrastructure project: creating the essential
functionality that is needed in order to build most kinds of scientific
and engineering applications. It would save lots of duplication effort and
improve productivity and quality at government labs, academia and the
private sector alike. The end product would have some sort of open-source
license (this can be a thorny issue, but i am sure a mutually satisfactory
solution can be found). Alternatively (or in addition), it might be better
to specify the API and leave the implementation part (open source and/or
commercial) to others.

Below I give some reasons, based mostly on the feedback from the list, why
i think it is appropriate that this effort be pursued at such a high level
(both bureaucratically and talent-wise).

- The task is too complex to be carried out by a single group of people
  (e.g., by a single government lab):

  * Not only does it involve 'too much work', but the expertise and the
    priorities of the group won't necessarily reflect those of the
    scientific and engineering community at large.

  * The resources of a single lab/group do not suffice. Even if they did,
    project managers primarily care to produce what is needed for their
    project. If it has beneficial side-effects for the community, so much
    the better, but this is not a priority. Alas, since the proposed task
    is too general to fit in the needs of any single research group, noone
    will probably ever carry it out.

- The end product should be something that the 'average' numerical
  scientist will feel compelled to use. For instance, if the majority of
  the community does not feel at home with object-oriented techniques,
  there should be a layer (if technically possible) that would allow
  access to the library via more traditional languages (C/Fortran), all
  the while helping people to make the transition to OO, when appropriate
  --i can hear Paul saying "but it's *always* appropriate" :)

- There is a number of issues for which it is not presently clear that
  there is a good answer. Paul brought up some of these problems. For
  example, is it possible to build the library in layers such that it can
  be accessed at different levels as object components, as objects, and as
  'usual', non-OO functions all at the same time? As Paul said, "Eiffel or
  C++ versions of some NAG routines typically have methods with one or two
  arguments while the C or Fortran ones have 15 or more". Does NAG use the
  same code-base for both its non-OO and OO products?

- It is hard to find people who are good scientists, good numerical
  scientists, good programmers, have good coordinator skills and can work
  in a team. This was presented as a reason why a project such as this
  proposed cannot be done. But in fact i think these are all arguments 
  why a widely-publicised and funded project such as this would have more
  chances to succeed than small-scale, individual efforts. I would also
  imagine that numerical analysts might feel more compelled to 'publicize'
  their work to the masses if there were single, respected, quality resource
  that provides the context in which they can deposit their work (or a
  draft thereof, anyway).

- Some companies and managers are hesitant to use open-source products due
  to requirements for 'fiduciary responsibility', as Paul put it. I think 
  that a product created under the conditions that i described would
  probably pass this requirement --not in the sense that there will be an
  entity to sue, but in the sense that it would be a reputable product 
  sponsored by some of the top research organizations or even the government.

***************************************************************
/  Christos Siopis              | Tel    : 734-615-1585       \
/  Postdoctoral Research Fellow |                             \
/  Department of Astronomy      | FAX    : 734-763-6317       \
/  University of Michigan       |                             \
/  Ann Arbor, MI 48109-1090     | E-mail : siopis at umich.edu   \
/  U.S.A.  _____________________|                             \
/         / http://www.astro.lsa.umich.edu/People/siopis.html \
***************************************************************