[SciPy-dev] Ideas for scipy.sparse?

[Nathan Bell]

On Fri, Apr 11, 2008 at 12:47 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>  1) I need N-dimensional sparse arrays.  Some of the storage formats in
>  scipy.sparse (dok, coo, maybe lil) could be generalized to
>  N-dimensions, but some work would have to be done.

To make this efficient, you'd probably need a lower-level
implementation of a hash-based container like DOK.

BTW, which applications use sparse N-dimensional arrays?

>  2) I need these things to be in numpy.  I hate to start another
>  "should this go into numpy or scipy" thread, but I actually do think
>  there is a decent case for moving the core sparse arrays into numpy
>  (not the solvers though).  Please hear me out:

I don't see the need for or utility of sparse arrays in numpy.  Numpy
does low-level manipulation of dense arrays/memory buffers.

>  3) I need sparse arrays that are implemented more in C.  What do I
>  mean by this.  I am using cython for the performance critical parts of
>  my package and there are certain things (indexing in tight loops for
>  example) that I need to do in c.  Because the current sparse array
>  classes are written in pure python (with a few c++ routines underneath
>  for format conversions), this is difficult.  So...

All of the costly operations on CSR/CSC/COO matrices are done in C++.
Only lil_matrix and dok_matrix are pure-Python implementations.

Indexing sparse arrays inside a Python loop *is* slow, but there's not
much that can be done about it.

>  I think it would be a very good idea to begin moving the sparse array
>  classes to cython code.  This would be a very nice approach because it
>  could be done gradually, without breaking any of the API.  The benefit
>  is that we could improve the performance of the sparse array classes
>  drammatically, while keeping things very maintainable.

I'm not aware of any performance problems with the existing backend to
scipy.sparse.  What would you implement in cython that's not already
implemented in scipy.sparse.sparsetools?

>  3) That we begin to move their implementation to using Cython (as an
>  aside, cython does play very well with templated C++ code).  This
>  could provide a much nicer way of tying into the c++ code than using
>  swig.

In the context of scipy.sparse, what's wrong with SWIG and C++?  What
would be improved by using Cython?

>  Alright, fire away :)

:)

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/