Implementing file reading in C/Python

[Grant Edwards]

On 2009-01-09, Sion Arrowsmith <siona at chiark.greenend.org.uk> wrote:
> Grant Edwards  <invalid at invalid> wrote:
>>On 2009-01-09, Johannes Bauer <dfnsonfsduifb at gmx.de> wrote:
>>> I've come from C/C++ and am now trying to code some Python because I
>>> absolutely love the language. However I still have trouble getting
>>> Python code to run efficiently. Right now I have a easy task: Get a
>>> file,
>>If I were you, I'd try mmap()ing the file instead of reading it
>>into string objects one chunk at a time.
>
> You've snipped the bit further on in that sentence where the
> OP says that the file of interest is 2GB. Do you still want to
> try mmap'ing it?

Sure.  The larger the file, the more you gain from mmap'ing it.
2GB should easily fit within the process's virtual memory
space.  When you mmap a file, it doesn't take up any physical
memory. As you access different parts of it, pages are swapped
in/out by the OS's VM system. If you're using a decent OS, the
demand-paged VM system will handle things far more efficiently
than creating millions of strings and letting the Python
garbage collector clean them up.

Or does "mmap" in Python mean something completely different
than "mmap" in the C library?

-- 
Grant Edwards                   grante             Yow! Everybody gets free
                                  at               BORSCHT!
                               visi.com