python: ascii read

[Sebastian Krause]

The input data is is large ascii file of astrophysical parameters 
(integer and float) of gaydynamics calculations. They should be read in 
as an array of integer and float numbers not as string (as open() and 
read() does). Then the array is used to make different plots from the 
data and do some (simple) operations: subtraction and divison of 
columns. I am using Scipy with Python 2.3.x under Linux (SuSE 9.1).

Sebastian

Robert Kern wrote:
> Sebastian Krause wrote:
> 
>> Hello,
>>
>> I tried to read in some large ascii files (200MB-2GB) in Python using 
>> scipy.io.read_array, but it did not work as I expected. The whole idea 
>> was to find a fast Python routine to read in arbitrary ascii files, to 
>> replace Yorick (which I use right now and which is really fast, but 
>> not as general as Python). The problem with scipy.io.read_array was, 
>> that it is really slow, returns errors when trying to process large 
>> files and it also changes (cuts) the files (after scipy.io.read_array 
>> processed a 2GB file its size was only 64MB).
>>
>> Can someone give me hint how to use Python to do this job correctly 
>> and fast? (Maybe with another read-in routine.)
> 
> 
> What kind of data is it? What operations do you want to perform on the 
> data? What platform are you on?
> 
> Some of the scipy.io.read_array behavior that you see look like bugs. We 
> would greatly appreciate it if you were to send a complete bug report to 
> the scipy-dev mailing list. Thank you.
>