python: ascii read

[Des Small]

aleaxit at yahoo.com (Alex Martelli) writes:

> If your needs are very specific (you know a lot about the format of
> those huge files -- e.g. they're column-oriented, or only use
> whitespace separators and \n line termination, or other such
> specifics) you might well be able to do better -- likely even in
> Python, worst case in C.  I assume you need Numeric arrays, 2-d,
> specifically, as the result of reading your files?  Would you know
> in advance whether you're reading int or float (it might be faster
> to have two separate functions)?  Could you pre-dimension the
> Numeric array and pass it in, or do you need it to dimension itself
> dynamically based on file contents?  The less flexibility you need,
> the simpler and faster the reading can be...

The last time I wanted to be able to read large lumps of numerical
data from an ASCII file, I ended up using (f)lex, for performance
reasons.  (Pure C _might_ have been faster still, of course, but it
would _quite certainly_ also have been pure C.)

This has caused minor irritation - the code has been in use through
several upgrades of Python, and it is considered polite to recompile
to match the current C API - but I'd probably do it the same way again
in the same situation.

Des
-- 
"[T]he structural trend in linguistics which took root with the
International Congresses of the twenties and early thirties [...] had
close and effective connections with phenomenology in its Husserlian
and Hegelian versions." -- Roman Jakobson