[SciPy-dev] Ideas for scipy.sparse?

[Nathan Bell]

On Fri, Apr 11, 2008 at 4:35 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>  >  All of the costly operations on CSR/CSC/COO matrices are done in C++.
>  >  Only lil_matrix and dok_matrix are pure-Python implementations.
>
>  Yes, but to use the fast c++ code, I have to go through a slow python
>  layer - the actual csr/csc/coo classes and slow+fast = slow in my
>  book.  Also, the dok/lil formats are some of the most important and
>  they should be optimized.

Can you demonstrate a real-world scenario where the overhead is
noticeable?  In all cases that I'm aware of the number of "slow"
operations is O(1).  OTOH the "fast" code always does at least O(nnz)
operations.

You are right about the lil and dok formats.  They are currently too
slow for large-scale problems.  You can have your way with them :)

>  Think of it in terms of layers (I will use the csr format as an example):
>
>  sparse/csr.py        => top level python class that wraps the lower layers.
>  sparsetools/csr.py =>  thin swig generated python wrapper that
>  introduces overhead
>  sparsetools/csr.h  =>     fast c++ code
>
>  Using swig forces me to deal with two layers of python code before I
>  can talk to the fast c++ code.  The problem with this is that I want
>  to call the top level layer from C!  This would be like NumPy not
>  having a C API!
>
>  The cython stack would look like this
>
>  sparse/csr.pyx        => top level python class that is a C extension
>  type and can be called from C or python
>  sparsetools/csr.h  =>     fast c++ code
>
>  In that case, I could write my extension code in C and talk directly
>  to the C interface without any overhead of multiple layers of Python.

I understand your point, but it's not immediately clear to me that the
SWIG-induced overhead is actually troublesome.  For frequent calls to
simple operations I see the purpose.  However, each of functions in
sparsetools does a substantial amount of work per call.

Also, what code that would want to interface directly with csr.h
wouldn't live in sparsetools itself?

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/