Implementing file reading in C/Python

[Grant Edwards]

On 2009-01-12, Sion Arrowsmith <siona at chiark.greenend.org.uk> wrote:
> Grant Edwards  <invalid at invalid> wrote:
>>On 2009-01-09, Sion Arrowsmith <siona at chiark.greenend.org.uk> wrote:
>>> Grant Edwards  <invalid at invalid> wrote:
>>>>If I were you, I'd try mmap()ing the file instead of reading it
>>>>into string objects one chunk at a time.
>>> You've snipped the bit further on in that sentence where the
>>> OP says that the file of interest is 2GB. Do you still want to
>>> try mmap'ing it?
>>Sure.  The larger the file, the more you gain from mmap'ing it.
>>2GB should easily fit within the process's virtual memory
>>space.
>
> Assuming you're in a 64bit world. Me, I've only got 2GB of address
> space available to play in -- mmap'ing all of it out of the question.

Oh.  I assumed that decent 32-bit OSes would provide at least 3-4GB
of address space to user processes.  What OS are you using?

> But I supposed that mmap'ing it chunk at a time instead of
> reading chunk at a time might be worth considering.

I'd try mmap'ing it in large chunks (512MB maybe).

-- 
Grant Edwards                   grante             Yow! I feel like a wet
                                  at               parking meter on Darvon!
                               visi.com