Does Python really follow its philosophy of "Readability counts"?

[Terry Reedy]

Paul Rubin wrote:
> Terry Reedy <tjreedy at udel.edu> writes:
>> I do not see any connection, really, between what you describe above
>> and your desire for static type-checking expressed elsewhere.  When I
>> was regularly doing analysis of empirical data files, I learned
>> (sometimes the hard way, as you describe above) to **ALWAYS** run
>> preliminary checks of all fields through the entire file 
> 
> Right.  And if the file is large enough that even parsing all the
> records to check the fields takes hours, well, that's where I'm at.

So what is the problem?  Let it run overnight.  If you want more speed, 
try numpy or Cython or C functions (swigged or ctyped) or...

To guarantee no crashes, make your top level something like

for line in open('humongous.dat', 'r'):
   try:
     <parse line>
     <check fields and update summary>
   except Exception as e:
     print(line, e)

(How trivial in Python!)

If your total system (including electricity supply and busy fingers of 
other people) is unstable, or you think you might want to interrupt it, 
keep track of lines or bytes read and write a report every gigabyte or 
so with enough info to restart from the last checkpoint -- just like 
reloading your last save in a video game when your avatar dies.

Terry Jan Reedy