[SciPy-Dev] sparsetools C++ code size

[Jaime Fernández del Río]

On Tue, Mar 4, 2014 at 1:33 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:

> Hi,
>
> In preparing the beta I've run into a practical issue. My build machine is
> not my regular (linux) one but an old one running OS X 10.6 - needed for
> the Mac binaries - with 1 Gb of RAM. I just figured out that it's basically
> impossible to compile the SWIG-generated C++ sparsetools code with it. This
> is due to the source code size more than doubling (see below) and
> everything just grinding to a halt when it hits the first sparsetools
> extension (csr_wrap.cxx).
>
> Besides that practical issue, which could be solved by setting up a build
> machine with more RAM (will take me some time), my worry is that this
> doesn't scale too well. A few more features added and it won't compile on
> more modern machines than mine either. 12 Mb of generated code from ~2k LoC
> in a few header files also feels a little crazy. What should be the
> long-term plan here?
>

I haven't checked everything, but after a quick look at csr.h and bsr.h,
the only feature of C++ that seems key to the functionality is overloading
of functions for different types. There are also calls to stl::sort and
similar here and there, but those could be replaced with numpy API calls.
It wouldn't be trivial, but not too hard either, to rewrite everything in C
using numpy's templating capabilities and have some dispatching mechanism
to get a pointer to the right function, e.g. as is done in the
implementation of np.partition:
https://github.com/numpy/numpy/blob/master/numpy/core/src/private/npy_partition.h.src#L108
Perhaps
a mix of templated C wrapped with Cython would be in line with what seems
the trend elsewhere in SciPy.


> With current master:
> $ ls -l scipy/sparse/sparsetools/*.cxx
> -rw-r--r-- 1 rgommers rgommers 4292625 Mar  4 21:58
> scipy/sparse/sparsetools/bsr_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers  750980 Mar  4 21:58
> scipy/sparse/sparsetools/coo_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers 3580216 Mar  4 21:58
> scipy/sparse/sparsetools/csc_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers  127880 Mar  4 21:58
> scipy/sparse/sparsetools/csgraph_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers 4627074 Mar  4 21:58
> scipy/sparse/sparsetools/csr_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers  264236 Mar  4 21:58
> scipy/sparse/sparsetools/dia_wrap.cxx
>
> With 0.13.x:
> $ ls -l scipy/sparse/sparsetools/*.cxx
> -rw-r--r-- 1 rgommers rgommers 2099951 Mar  4 21:58
> scipy/sparse/sparsetools/bsr_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers  451942 Mar  4 21:58
> scipy/sparse/sparsetools/coo_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers 1635183 Mar  4 21:58
> scipy/sparse/sparsetools/csc_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers  126384 Mar  4 21:58
> scipy/sparse/sparsetools/csgraph_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers 2211943 Mar  4 21:58
> scipy/sparse/sparsetools/csr_wrap.cxx
> -rw-r--r-- 1 rgommers rgommers  205412 Mar  4 21:58
> scipy/sparse/sparsetools/dia_wrap.cxx
>
> Ralf
>
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-dev
>
>


-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20140304/a8747a1a/attachment.html>