large file support

[David Ascher]

On 21 Oct 2000, Fred Clare wrote:

> How does one read large files into Python?  I have a 4 gigabyte
> file on an SGI (that has over 30 GB main memory), but when I use
> 
>   f = open("file","rb")
>   s = f.read()
> 
> on this file, I get "MemoryError".  Trying the I/O from the os module 
> produces the same result.  In fact I get the MemoryError even when 
> trying to read 2GB files.  When I run configure before building python, it
> indicates that large file support *is* enabled.

I'm not sure what the status of large file support is on SGI's.  One
thought, however, is that you may be better off using mmap to deal with
the file, as the OS can be much more efficient about what parts of the
data to load from disk.

Some docs on mmap are available: 
 
http://www.activestate.com/Products/ActivePython/Docs/lib/module-mmap.html

(I would point to the pythonlabs page, but it appears to be temporarily
down).

--david ascher