[SciPy-dev] Splitting sparsetools_wrap source file ?

[David Cournapeau]

Nathan Bell wrote:
>
> While better than manually splitting the _wrap file, this approach is
> still cumbersome.  There are ~35 functions in sparsetools, so a 1
> function : 1 file policy is not really scalable.

Well, having one file/function is overkill. I just think that 500-600 Mb 
of Ram to build one source file is really not good for various reasons; 
several MB source files is fundementaly a bad thing. Splitting it in 
half would already be pretty good, and that can be done easily: you just 
have to be careful about what functions to put in which file to get the 
memory reduction, though.

>
> I tried lumping all the CSR functions together and found only modest savings.

With two files, I could get from 500 Mb to around 260 Mb, which is 
already pretty good I think.

>
> Disabling the templates that unroll dense loops for the BSR matrix
> (see bsr_matvec) produced a measureable improvement in memory usage so
> I've committed this version of sparsetools.h to SVN.

the *matvec functions are indeed the ones which take a lot of memory for 
compilation, which is not surprising if they use expression template. Is 
it really useful ? I certainly do not want this to cause any performance 
penalty or anything. Putting those functions in a different file is what 
gives the most advantage in my experiment.

>
> Are you saying that g++ fails to compile on the VM, or that it starts
> swapping to disk?

Yes, that's what I am saying. I need to allocate 1 Gb of memory for 
solaris when compiling sparsetools with sun compilers right now. Even by 
today standards, that's an awful lot of memory.

cheers,

David