[SciPy-user] combine two sparse matrices

[Robin]

Hi,

I was wondering what the most (memory) efficient way of combining two
sparse matrices would be.

I am constructing a very large sparse matrix, but due to the temporary
memory required to calculate the entries I am doing it in blocks, with
the computation of each block done in a forked child process. This
returns a sparse matrix of the same dimensions as the full one, but
with a smaller number of entries. I would like to add the entries from
the block result to the 'master' copy. I can be sure that there will
be no overlap in the position of entries (ie no matrix position will
be in both sides).

What is the most memory efficient way of combining these? I noticed +=
isn't implemented, but it's not clear how that would work anyway. The
best I have done so far is adding two lil_matices (the block is
created as an lil-matrix for fancy indexing) A = A + Apartial, but as
the master copy grows this means I think that I will need double the
final memory requirement for A (to add the last block). Is there a
better way of doing this?

Also, what are the memory requirements for the conversions (.tocsc,
.tocsr etc.)? Will that mean I need double the memory anyway?

Thanks,

Robin