python: ascii read

[Sebastian Krause]

I did not explictly mention that the ascii file should be read in as an 
array of numbers (either integer or float).
To use open() and read() is very fast, but does only read in the data as 
string and it also does not work with large files.

Sebastian

Alex Martelli wrote:
> Sebastian Krause <canopus at gmx.net> wrote:
> 
> 
>>Hello,
>>
>>I tried to read in some large ascii files (200MB-2GB) in Python using
>>scipy.io.read_array, but it did not work as I expected. The whole idea
>>was to find a fast Python routine to read in arbitrary ascii files, to
>>replace Yorick (which I use right now and which is really fast, but not
>>as general as Python). The problem with scipy.io.read_array was, that it
>>is really slow, returns errors when trying to process large files and it
>>also changes (cuts) the files (after scipy.io.read_array processed a 2GB
>>file its size was only 64MB).
>>
>>Can someone give me hint how to use Python to do this job correctly and
>>fast? (Maybe with another read-in routine.)
> 
> 
> If all you need is what you say -- read a huge amount of ASCII data into
> memory -- it's hard to beat
>     data = open('thefile.txt').read()
> 
> mmap may in fact be preferable for many uses, but it doesn't actually
> read (it _maps_ the file into memory instead).
> 
> 
> Alex