[SciPy-dev] scipy status

[Pearu Peterson]

On Wed, 18 Sep 2002, eric jones wrote:

> Hey group,
> 
> I have a few days coming up here to work on getting a SciPy 0.2 out the
> door.  It looks like it is buiding and passing all test on windows, and
> I think it is doing OK on Linux.  Sun is close or finished (Skip?).
> 
> What I'd like to know is if there are any other known issues that we
> need to solve before pushing 0.2 out?  
> 
> FFT
> Pearu, how are things on the fft changes?  I never heard back from djb
> on the license issue.  I think we are in the clear to use his code, but
> he also seems to get quite exserised with people who cross him: 
>    http://cr.yp.to/distributors.html
> 
> We are statically linking his code so the comments here about installing
> it in the correct places doesn't seem to apply -- they look to be aimed
> at his executables not library files.
> 
> My vote is to move ahead with using them and then remove them later if
> he does not like what we are doing.  We can back off again to fftpack
> with the same functionality and also the fftw wrappers are still around
> for those wanting to use it and not constrained by the GPL.

fftpack2 is almost complete in the sense that wrappers to C/Fortran
libraries are finished, there are rather complete tests, and high-level
functions are documented. Right now I am fine-tuning differentiation
transform that can be sensitive to numerical noise from FFT
algorithm if the signal is very long (in my application I need the
numerical noise level to be absolutely minimal and that depends on
accuracy properties of fft libraries. Btw, djbfft turns out to be
also the most accurate fft library compared to FFTPACK and FFTW).

Anyway, fftpack2 is designed to use optimal FFT library combination for
different jobs. Currently, FFTPACK, FFTW, and DJBFFT libraries are
supported. Last two are optional and used only if system_info.py detects
them. It turns out that DJBFFT is fastest for power-of-two 1-D transforms
and  FFTW is superior for N-D transforms and non-power-of-two
transforms. Some of the timings are included at the end of this message.

What fftpack2 is missing? I haven't implemented yet some of
convenience functions such as fft2, hfft, fftshift, fftfreq, cont_ft,
zeropad. They all are Python functions and can be copied from the current
fftpack. In addition, these functions need unittests.

I think I can finish fftpack2 by this weekend so that it would be ready
for review first as a stand alone package and if you are satisfied then
it can be committed to scipy.

About djbfft and scipy.
djbfft builds from its makefile. This is fine under unix but I have no
idea if this will be an issue for Windows. Could someone try it out on
Windows? If it builds, then also test whether system_info.py can detect it
and may be adjust it accordingly?

> LINALG
> How is linalg?  This is by far the biggest build headache, but it does
> seem to be working against atlas on windows and linux and against LAPACK
> and BLAS on Sun.

linalg builds fine provided that ATLAS or LAPACK/BLAS are the latest,
built/installed properly, and preferably there are no other atlas or
lapack libraries installed on the system. I think a detailed SciPy build
HOWTO would be very useful to reduce the build headache.
INSTALL.txt should be a starting point but there are still some obsolete
instructions at the scipy site that people probably try out first and get 
into problems.

> Any other specific packages that need mentioning?

I would mention packages that lack more or less complete unittests.


Btw, I just noticed that importing scipy assumes that wxPython is
installed and DISPLAY must be working. When trying to import scipy from a
different computer without X connection, I get:

>>> import scipy

Gtk-WARNING **: cannot open display: 

and python quits. Any idea how to fix this?

Pearu


                 Fast Fourier Transform
========================================================
      |       real input       |       complex input    
--------------------------------------------------------
 size |fftpack2|  FFT  | scipy |fftpack2|  FFT  | scipy
--------------------------------------------------------
  100 |   0.38 |  0.41 |  0.37 |   0.46 |  0.41 |  0.37  secs 7000 calls
 1000 |   0.30 |  0.68 |  0.54 |   0.68 |  0.68 |  0.54  secs 2000
  256 |   0.59 |  0.89 |  0.62 |   0.67 |  0.89 |  0.63  secs 10000
  512 |   0.76 |  1.59 |  0.97 |   0.96 |  1.63 |  1.01  secs 10000
 1024 |   0.12 |  0.32 |  0.17 |   0.16 |  0.33 |  0.18  secs 1000
 2048 |   0.20 |  0.71 |  0.34 |   0.30 |  0.77 |  0.37  secs 1000
 4096 |   0.27 |  0.84 |  0.46 |   0.45 |  0.94 |  0.59  secs 500
 8192 |   0.81 |  3.21 |  2.36 |   1.36 |  3.23 |  2.46  secs 500

          Inverse Fast Fourier Transform
========================================================
      |       real input       |        complex input    
--------------------------------------------------------
 size |fftpack2|  FFT  | scipy |fftpack2|  FFT  | scipy
--------------------------------------------------------
  100 |   0.38 |  0.87 |  0.81 |   0.52 |  0.87 |  0.82  secs 7000 calls
 1000 |   0.31 |  1.19 |  1.07 |   0.85 |  1.20 |  1.06  secs 2000
  256 |   0.61 |  1.71 |  1.47 |   0.71 |  1.76 |  1.50  secs 10000
  512 |   0.80 |  2.85 |  2.24 |   1.05 |  2.84 |  2.22  secs 10000
 1024 |   0.12 |  0.54 |  0.39 |   0.18 |  0.54 |  0.39  secs 1000
 2048 |   0.22 |  1.28 |  0.85 |   0.36 |  1.24 |  0.90  secs 1000
 4096 |   0.30 |  1.45 |  1.26 |   0.52 |  1.50 |  1.22  secs 500
 8192 |   0.84 |  4.51 |  3.73 |   1.71 |  4.39 |  3.85  secs 500

      Multi-dimensional Fast Fourier Transform
========================================================
          |       real input       |       complex input    
--------------------------------------------------------
   size   |fftpack2|  FFT  | scipy |fftpack2|  FFT  | scipy
--------------------------------------------------------
  100x100 |   0.37 |  0.76 |  0.61 |   0.41 |  0.74 |  0.64  secs 100calls
 1000x100 |   0.45 |  0.64 |  0.53 |   0.49 |  0.65 |  0.55  secs 7
  256x256 |   0.44 |  0.62 |  0.55 |   0.43 |  0.63 |  0.46  secs 10
  512x512 |   0.62 |  0.83 |  0.64 |   0.63 |  0.83 |  0.60  secs 3

Differentiation of periodic functions
=====================================
 size  | optimized |    naive
-------------------------------------
   100 |      0.25 |      1.56  (secs for 1500 calls)
  1000 |      0.19 |      1.69  (secs for 300 calls)
   256 |      0.34 |      2.53  (secs for 1500 calls)
   512 |      0.36 |      2.88  (secs for 1000 calls)
  1024 |      0.31 |      2.75  (secs for 500 calls)
  2048 |      0.24 |      2.34  (secs for 200 calls)
  4096 |      0.27 |      2.32  (secs for 100 calls)
  8192 |      0.31 |      2.46  (secs for 50 calls)

 Tilbert transform of periodic functions
=========================================
 size  | optimized |    naive
-----------------------------------------
   100 |      0.22 |      1.44  (secs for 1500 calls)
  1000 |      0.15 |      1.27  (secs for 300 calls)
   256 |      0.28 |      2.05  (secs for 1500 calls)
   512 |      0.27 |      2.20  (secs for 1000 calls)
  1024 |      0.21 |      2.06  (secs for 500 calls)
  2048 |      0.16 |      1.67  (secs for 200 calls)
  4096 |      0.20 |      1.64  (secs for 100 calls)
  8192 |      0.24 |      1.70  (secs for 50 calls)

 Hilbert transform of periodic functions
=========================================
 size  | optimized |    naive
-----------------------------------------
   100 |      0.22 |      1.33  (secs for 1500 calls)
  1000 |      0.13 |      1.09  (secs for 300 calls)
   256 |      0.26 |      1.84  (secs for 1500 calls)
   512 |      0.26 |      1.88  (secs for 1000 calls)
  1024 |      0.21 |      1.74  (secs for 500 calls)
  2048 |      0.17 |      1.41  (secs for 200 calls)
  4096 |      0.19 |      1.47  (secs for 100 calls)
  8192 |      0.21 |      1.44  (secs for 50 calls)