large file support

[Mats Wichmann]

On Fri, 20 Oct 2000 21:53:51 -0700 (Pacific Daylight Time), David
Ascher <DavidA at ActiveState.com> wrote:

>On 21 Oct 2000, Fred Clare wrote:
>
>> How does one read large files into Python?  I have a 4 gigabyte
>> file on an SGI (that has over 30 GB main memory), but when I use
>> 
>>   f = open("file","rb")
>>   s = f.read()
>> 
>> on this file, I get "MemoryError".  Trying the I/O from the os module 
>> produces the same result.  In fact I get the MemoryError even when 
>> trying to read 2GB files.  When I run configure before building python, it
>> indicates that large file support *is* enabled.
>
>I'm not sure what the status of large file support is on SGI's. 

Quite good, and has been for many years.

> One
>thought, however, is that you may be better off using mmap to deal with
>the file, as the OS can be much more efficient about what parts of the
>data to load from disk.

This is a good suggestion.

SGI's use the LP64 code model (as do most folks, I believe): longs and
pointers (in C) are 64-bit, integers remain 32-bit.  Just a reminder
to make sure there aren't any C int's trying to handle the large data.

Mats

Mats Wichmann

(Anti-spam stuff: to reply remove the "xyz" from the
address xyzmats at laplaza.org. Not that it helps much...)