Implementing file reading in C/Python

[Sion Arrowsmith]

Grant Edwards  <invalid at invalid> wrote:
>On 2009-01-09, Sion Arrowsmith <siona at chiark.greenend.org.uk> wrote:
>> Grant Edwards  <invalid at invalid> wrote:
>>>If I were you, I'd try mmap()ing the file instead of reading it
>>>into string objects one chunk at a time.
>> You've snipped the bit further on in that sentence where the
>> OP says that the file of interest is 2GB. Do you still want to
>> try mmap'ing it?
>Sure.  The larger the file, the more you gain from mmap'ing it.
>2GB should easily fit within the process's virtual memory
>space.

Assuming you're in a 64bit world. Me, I've only got 2GB of address
space available to play in -- mmap'ing all of it out of the question.

But I supposed that mmap'ing it chunk at a time instead of reading
chunk at a time might be worth considering.

-- 
\S -- siona at chiark.greenend.org.uk -- http://www.chaos.org.uk/~sion/
   "Frankly I have no feelings towards penguins one way or the other"
        -- Arthur C. Clarke
   her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump