python: ascii read

[Alex Martelli]

Sebastian Krause <canopus at gmx.net> wrote:

> Hello,
> 
> I tried to read in some large ascii files (200MB-2GB) in Python using
> scipy.io.read_array, but it did not work as I expected. The whole idea
> was to find a fast Python routine to read in arbitrary ascii files, to
> replace Yorick (which I use right now and which is really fast, but not
> as general as Python). The problem with scipy.io.read_array was, that it
> is really slow, returns errors when trying to process large files and it
> also changes (cuts) the files (after scipy.io.read_array processed a 2GB
> file its size was only 64MB).
> 
> Can someone give me hint how to use Python to do this job correctly and
> fast? (Maybe with another read-in routine.)

If all you need is what you say -- read a huge amount of ASCII data into
memory -- it's hard to beat
    data = open('thefile.txt').read()

mmap may in fact be preferable for many uses, but it doesn't actually
read (it _maps_ the file into memory instead).


Alex