[Tutor] np.array operation memoryerror

[Alan Gauld]

On 14/10/2020 14:45, f.hochwart at gmx.de wrote:

> for my masterthesis I have to compute e.g. np.arrays with following
> settings:
> MemoryError: Unable to allocate 7.15 TiB for an array with shape (991600,
> 991600) and data type float64

This suggests you don't have enough memory to load your data.
You either need to increase the memory or reduce the data!

>     A =  sparse.bsr_matrix(C1 * (sparse.bsr_matrix(np.eye(dA, k = dR)) -2 *
> sparse.bsr_matrix(np.eye(dA)) + sparse.bsr_matrix(np.eye(dA, k = -dR))))

Can't help with numpy, thee is a support forum though if you visit their
web site.There may be a way of paging data or similar.

> The main question is, how can I compute big np.arrays which extend my
> memory.

The traditional way to do that would be to load "pages" of data at a
time and process each page. (It may help if you can sort the data first)
Then combine the page results to get an overall result.

But without knowing more about your data (in and out) and how you plan
on processing it we can't say much more. You might have to work a bit
harder to get your masters... :-)

But I'd start by asking on the numpy fora.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos