[SciPy-dev] [SciPy-user] efficiently importing ascii data

[Travis Oliphant]

Darren Dale wrote:

>On Sunday 13 November 2005 10:46 pm, you wrote:
>  
>
>>Darren Dale wrote:
>>    
>>
>>>I am considering using scipy's fromfile function, which gives a big speed
>>>boost over io.read_array, but I don't understand what this docstring is
>>>trying to tell me:
>>>
>>>   WARNING: This function should be used sparingly, as it is not
>>>   a robust method of persistence.  But it can be useful to
>>>   read in simply-formatted or binary data quickly.
>>>      
>>>
>>It's simply trying to advertise that fromfile and tofile are very raw
>>functions.   They should work fine as far as they go, but there may be
>>easier solutions.  I don't expect the capability of these to increase.
>>But, for example, something like a TableIO could take advantage of them.
>>    
>>
>
>I was wondering if the fromstring function could be expanded to include ascii 
>strings. 
>
Hmm.  Interesting concept.  Do you mean in the same way that fromfile 
works, a single type and a simple separator?   That would be consistent 
wouldn't it...

The other thing that needs to be done is that string (and unicode) 
arrays need to convert to the numeric types, easily.  This would let you 
read in a string array and convert to numeric types quite cleanly.    
Right now, this doesn't work simply because the wrong generic functions 
are getting called in the conversion routines.  This can and should be 
changed, however.  The required code to change is in arraytypes.inc.src. 

I could see using PyInt_FromString, PyLong_FromString, 
PyFloat_FromString, and calling the complex python function for 
constructing complex numbers from a string.   Code would need to be 
written to fully support Long double and complex long double conversion, 
but initially they could just punt and use the double conversions.

Alternatively, sscanf could be called when available for the type, and 
the other approaches used when it isn't.

Anybody want a nice simple project to get themselves up to speed with 
the new code base :-)

-Travis