[Numpy-discussion] Loading a > GB file into array

[Sebastian Haase]

On Dec 20, 2007 3:22 AM, Martin Spacek <numpy at mspacek.mm.st> wrote:
> Sebastian Haase wrote:
> > b) To my knowledge, any OS Linux, Windows an OSX can max. allocate
> > about 1GB of data - assuming you have a 32 bit machine.
> > The actual numbers I measured varied  from about 700MB to maybe 1.3GB.
> > In other words, you would be right at the limit.
> > (For 64bit, you would have to make sure ALL parts are 64bit, e.g. The
> > python version must be >=2.5, python must have been compiled using a
> > 64-bit compiler *and* the windows version (XP-64))
> > This holds true the same for physical ram allocation and for memmap
> > allocation.
> > My solution to this was to "wait" for the 64bit .... not tested yet ;-)
>
> By the way, I installed 64-bit linux (ubuntu 7.10) on the same machine,
> and now numpy.memmap works like a charm. Slicing around a 15 GB file is fun!
>
Thanks for the feedback !
Did you get the kind of speed you need and/or the speed you were hoping for ?

-Sebastian