[Numpy-discussion] Re: [SciPy-user] Current thoughts on future directions

[Pearu Peterson]

On Sun, 13 Mar 2005, Michiel Jan Laurens de Hoon wrote:

> Pearu Peterson wrote:
>>> Travis Oliphant wrote:
>>> 
>>>> 1) There will be a scipy_core  package which will be essentially what 
>>>> Numeric has always been (plus a few easy to install extras already in 
>>>> current scipy_core).
> ...
>>>> linalg (a lite version -- no fortran or ATLAS dependency)
>> 
>> Again, what would be the underlying linear algebra library here?
>> Numeric uses f2c version of lite lapack library. Shall we do the same but 
>> wrapping the c codes with f2py rather than by hand? f2c might be useful 
>> also in other cases to reduce fortran dependency, but only when it is 
>> critical to ease the scipy_core installation.
>> 
> If I understand Travis correctly, the idea is to use Numeric as the basis for 
> scipy_core, allowing current Numerical Python users to switch to scipy_core 
> with a minimum of trouble. So why not use Numeric's lite lapack library 
> directly? What is the advantage of repeating the c code wrapping (by f2py or 
> by hand)?

First, I wouldn't repeat wrapping c codes by hand.
But using f2py wrappers has the following advantages:
(i) maintaining the wrappers is easier (as the wrappers are generated)
(ii) one can easily link linalg_lite against optimized lapack. This is 
certainly possible with current Numeric but for a smaller set of Fortran 
compilers than when using f2py generated wrappers (for example, if a 
compiler produces uppercased symbol names then Numeric wrappers won't 
work)
(iii) scipy provides wrappers to a larger set of lapack subroutines than 
Numeric, and with f2py it is easier and less errorprone to add new 
wrappers to lapack functions than wrapping them by hand, i.e. extending 
f2py generated linalg_lite is much easier than extending the current 
Numeric lapack_lite.
(iv) and finally, f2py generated wrappers tend to be more efficient than 
Numeric hand coded wrappers. Here are some benchmark results comparing 
scipy and Numeric linalg functions:

            Finding matrix determinant
       ==================================
       |    contiguous     |   non-contiguous
----------------------------------------------
  size |  scipy  | Numeric |  scipy  | Numeric
    20 |   0.16  |   0.22  |   0.17  |   0.26     (secs for 2000 calls)
   100 |   0.29  |   0.41  |   0.28  |   0.56     (secs for 300 calls)
   500 |   0.31  |   0.36  |   0.33  |   0.45     (secs for 4 calls)
            Finding matrix inverse
       ==================================
       |    contiguous     |   non-contiguous
----------------------------------------------
  size |  scipy  | Numeric |  scipy  | Numeric
    20 |   0.28  |   0.33  |   0.27  |   0.37     (secs for 2000 calls)
   100 |   0.64  |   1.06  |   0.64  |   1.24     (secs for 300 calls)
   500 |   0.83  |   1.10  |   0.84  |   1.18     (secs for 4 calls)
       Solving system of linear equations
       ==================================
       |    contiguous     |   non-contiguous
----------------------------------------------
  size |  scipy  | Numeric |  scipy  | Numeric
    20 |   0.26  |   0.18  |   0.26  |   0.21     (secs for 2000 calls)
   100 |   0.31  |   0.35  |   0.31  |   0.52     (secs for 300 calls)
   500 |   0.33  |   0.34  |   0.35  |   0.41     (secs for 4 calls)

Remark: both scipy and Numeric are linked agaist the same ATLAS/Lapack 
library.

Pearu