[SciPy-Dev] sparsetools C++ code size

[Ralf Gommers]

Hi,

In preparing the beta I've run into a practical issue. My build machine is
not my regular (linux) one but an old one running OS X 10.6 - needed for
the Mac binaries - with 1 Gb of RAM. I just figured out that it's basically
impossible to compile the SWIG-generated C++ sparsetools code with it. This
is due to the source code size more than doubling (see below) and
everything just grinding to a halt when it hits the first sparsetools
extension (csr_wrap.cxx).

Besides that practical issue, which could be solved by setting up a build
machine with more RAM (will take me some time), my worry is that this
doesn't scale too well. A few more features added and it won't compile on
more modern machines than mine either. 12 Mb of generated code from ~2k LoC
in a few header files also feels a little crazy. What should be the
long-term plan here?

With current master:
$ ls -l scipy/sparse/sparsetools/*.cxx
-rw-r--r-- 1 rgommers rgommers 4292625 Mar  4 21:58
scipy/sparse/sparsetools/bsr_wrap.cxx
-rw-r--r-- 1 rgommers rgommers  750980 Mar  4 21:58
scipy/sparse/sparsetools/coo_wrap.cxx
-rw-r--r-- 1 rgommers rgommers 3580216 Mar  4 21:58
scipy/sparse/sparsetools/csc_wrap.cxx
-rw-r--r-- 1 rgommers rgommers  127880 Mar  4 21:58
scipy/sparse/sparsetools/csgraph_wrap.cxx
-rw-r--r-- 1 rgommers rgommers 4627074 Mar  4 21:58
scipy/sparse/sparsetools/csr_wrap.cxx
-rw-r--r-- 1 rgommers rgommers  264236 Mar  4 21:58
scipy/sparse/sparsetools/dia_wrap.cxx

With 0.13.x:
$ ls -l scipy/sparse/sparsetools/*.cxx
-rw-r--r-- 1 rgommers rgommers 2099951 Mar  4 21:58
scipy/sparse/sparsetools/bsr_wrap.cxx
-rw-r--r-- 1 rgommers rgommers  451942 Mar  4 21:58
scipy/sparse/sparsetools/coo_wrap.cxx
-rw-r--r-- 1 rgommers rgommers 1635183 Mar  4 21:58
scipy/sparse/sparsetools/csc_wrap.cxx
-rw-r--r-- 1 rgommers rgommers  126384 Mar  4 21:58
scipy/sparse/sparsetools/csgraph_wrap.cxx
-rw-r--r-- 1 rgommers rgommers 2211943 Mar  4 21:58
scipy/sparse/sparsetools/csr_wrap.cxx
-rw-r--r-- 1 rgommers rgommers  205412 Mar  4 21:58
scipy/sparse/sparsetools/dia_wrap.cxx

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20140304/03f1a2de/attachment.html>