[SciPy-dev] numpy - dual.py problems

[Arnd Baecker]

Hi,

I have now been hit the fifth time by what I would
call the `dual.py dilemma/problem`.
Let my try to explain, why I think that using `dual.py` in numpy
(at least in the way it is done right now) causes problems:

A typical scenario for "end-users" is the following:
- people will have Numeric/numarray + old scipy/old new scipy
  on their machines.

  In many cases this is the system-wide installation as done by
  the root-user (eg. via some package manager)

  The "end-user" has no root rights.
- The "end-user" hears about the great progress wrt numpy/scipy combo
  and wants to test it out.

  He downloads numpy and installs it to some place in his
  homedirectory via

    python setup.py install --prefix=<~/somewhere>

  and sets his PYTHONPATH accordingly
-  Then `import numpy` will work, but a
   `numpy.test(10)` will fail because `dual.py`
   picks his old scipy (which will be visible from the
   traceback, if he looks carefully at the path names).

- Consequence, the "end-user" will either ask a question on the
  mailing list or just quit his experiment and continue
  to work with his old installation.

I have just re-read the discussion on the mailing-list on this:

Travis' remark on this was:

"""You need to remove scipy_base from your system (or at least from the
site-packages directory you are running for Python).  I'm pretty sure
the new scipy won't work with the old scipy (which is what scipy_base
comes from).

This will need to be emphasized on install of the new system....    Get
rid of the old system first...
"""

Here I disagree: the "end-user" is in general
  a) not able to remove the old install
  b) and also does not want to remove his old install
     (usually he will have a lot of code floating around
      which relies on the old behaviour).

Later on Travis' wrote:
"""The solution is that now to get at functions that are in both numpy and
scipy and you want the scipy ones first and default to the numpy ones if
scipy is not installed,  there is a numpy.dual  module that must be
loaded separately that contains all the overlapping functions."""

I think this is fine, if a user does this in his own code,
but I have found the following `from numpy.dual import`s within numpy
  core/defmatrix.py:        from numpy.dual import inv
  lib/polynomial.py:    from numpy.dual import eigvals, lstsq
  lib/mlab.py:from numpy.dual import eig, svd
  lib/function_base.py:    from numpy.dual import i0


I think Fernando's analysis is indeed very true
(FWIW, yes, originally I had a different opinion on that ;-):
"""Travis Oliphant wrote:

> But these need to be there.  We need some way to update the functions in
> numpy if scipy is installed (so that you  can always call the numpy
> functions but get the scipy functions if they are there)...

Mmh, I'm not sure I agree with this approach.  I think this may lead to
surprising behavior, hard to find bugs, etc.  I really don't think that
the
presence of scipy should change the behavior of numpy's internals, esp.
when
we're advertising the dependency chain to be

        numpy < scipy

Finding out that in reality, there's a hidden

        |-> numpy   scipy--|
        |                  |
        |------------------|

cycle kind of threw me for a loop :)

I would argue that users should know whether they want to call scipy's
libraries or numpy's, and be clearly aware of the differences.
"""

In conclusion, I think that `dual.py` should not be
used *inside* of `numpy`.

Best, Arnd


P.S.: Fernando, I am sure you are reading all this with a smile -
yes I am converted...