how to parse numeric data files

george young gry at ll.mit.edu
Tue Apr 29 13:25:19 EDT 2003 

[python 2.3a1]
We have several electronic testing machines(of various ages and
manufacturers) that spew out testing data files in various ascii formats.
Currently we have a nasty mess of awk/shell/C/fortran programs that
extract and process some data from these files.  I have a dream of
a suite of simple, clear, maintainable python programs to do these tasks.  

Ideally, for each file format(there may be a dozen or so), a clear,
concise descriptor file (yacc-like language definition? whatever)
would drive the parsing process for reading that kind of data file,
providing a nice clear API to the data.  

Is there a PyYacc or some such parser module?  Or should I use
the regular-expression module or some such lower level facility
to hard code each format?  

The trick is I hope to come up with something that our hardware
engineers can understand and maintain easily without studying
things like BNF, LALR etc. (they won't).

I hope then to use Numeric and ScientificPython to do the crunching.

I know I'm trying to do something complex very simply, but
hey! this is python right?

[Below is a sample of one of the worst formats, shortened from a 40MB file!]
-- George


PH_lot_id:			YES_2_1.25V_1.0V
Device_id:			4Mb_dc1
Operator_id:			EEA
PFD:			YES_2_1.25V_1.0V 4Mb_dc1 4M_digit_capture1 VDD=1.25V VLEAKLOW=0V V_TCSETP=1.0V
Date:			01/30/03
Test_type:			Functional_Test
Prober_id:			EG.3001
Probe_Card:			ye_sram_4Mb_1
Test_description:			4Mb SRAM1

Binlist: wafer_log
Bin: 	Opn	reprobe	 00	FAIL	Open Pins Test
Bin: 	DC	no_reprobe 	01	PASS	DC
Bin: 	db	no_reprobe	-01	FAIL	

Data_format: X,Y  SW_Bin  HW_Bin  P_F  Date  Time  Temp  FF/TF/TT  Bin_desc
Begin_data:
                         
3,2 	DC 01 	PASS 	20030411 133414 -9999.0 3/3/38 DC
  Data_format_test: suite  function  result [ res[0] unit[0] res[1] unit[1] res[2] unit[2] res[3] unit[3] ]
  Begin_data_tf:
  speccalues  spec_values PASS
     Test_number: 0
    Begin_user_data:
LEVEL SPEC VARIABLES
Eqn = 1 Sps = 1 Description = ye_sram_64Kb_vdd_1500mV_w_0mV
VDD =  1.250 V WAFER =  0.000 V V_TCSEPT =  1.000 V VLEAKLOW =  0.000 V 
Eqn = 1 Sps = 2 Description = ye_sram_64Kb_vdd_100mV_w_0mV
VDD =  0.100 V WAFER =  0.000 V V_TCSEPT =  0.000 V VLEAKLOW =  0.000 V 
TIMING SPEC VARIABLES
Eqn = 1 Sps = 1 Description = ye_sram_4Mb
t_setup =   333.3 ns t_pw_enable =   333.3 ns t_wait =   333.3 ns Period =  1000.0 ns

    End_user_data
  Connectivity_T1  leakage PASS   -10.000000 	uA     -8.999900 	uA      0.000000 	uA      0.000000 	uA 
     Test_number: 1
     Begin_limit_data:
SPRM 101,1,102001,102001
SQSL "ye_sram_4Mb"
leakage;addr_in00,addr_in01,addr_in02,addr_in03,data_out0,data_out1,data_out2,data_out3,data_out4,data_out5,data_out6,data_out7,enable,set0bar,set1bar,writebar; V;    -1.0;     -1.0;uA;   -10.0;   -10.0;uA;   -15.0;   -15.0;uA;    -1.0;     -1.0;ms;     200;     200;3;Opens ($P);10; V;    -1.0;    -1.0;;;;0;
     End_limit_data
    Data_format_pins: pin_name  PFNU0 [ value0 unit0 ] PFNU1 [ value1 unit1 ]
    Begin_data_pins:
    addr_in00 P   -10.000000 	uA  N     0.000000 	uA 
    addr_in01 P   -10.000000 	uA  N     0.000000 	uA 
    addr_in02 P   -10.000000 	uA  N     0.000000 	uA 
    addr_in03 P   -10.000000 	uA  N     0.000000 	uA 
    data_out0 P   -10.000000 	uA  N     0.000000 	uA 
    data_out1 P   -10.000000 	uA  N     0.000000 	uA 
    data_out2 P   -10.000000 	uA  N     0.000000 	uA 
    data_out3 P   -10.000000 	uA  N     0.000000 	uA 
    data_out4 P   -10.000000 	uA  N     0.000000 	uA 
    data_out5 P   -10.000000 	uA  N     0.000000 	uA 
    data_out6 P   -10.000000 	uA  N     0.000000 	uA 
    data_out7 P   -10.000000 	uA  N     0.000000 	uA 
    enable P   -10.000000 	uA  N     0.000000 	uA 
    set0bar P   -10.000000 	uA  N     0.000000 	uA 
    set1bar P   -10.000000 	uA  N     0.000000 	uA 
    writebar P   -10.000000 	uA  N     0.000000 	uA 
    End_data_pins
  Idd_at_100mV_T2  standby_current FAIL     0.004769 	 A      0.004769 	 A      0.000000 	 A      0.000000 	 A 
     Test_number: 94
     Begin_limit_data:
SPRM 101,1,102001,102001
SQSL "ye_sram_4Mb"
standby_current;vdd;uA;     0.01;uA;    1.00;Idd_at_100mV ($P);0;
     End_limit_data
    Data_format_pins: pin_name  PFNU0 [ value0 unit0 ] PFNU1 [ value1 unit1 ]
    Begin_data_pins:
    vdd F     0.004769 	 A  N     0.000000 	-
    End_data_pins
  Idd_at_Plus_1250mV_T3  standby_current PASS     0.071565 	 A      0.071565 	 A      0.000000 	 A      0.000000 	 A 
     Test_number: 100
     Begin_limit_data:
SPRM 101,1,104001,104001
SQSL "ye_sram_4Mb"
standby_current;wafer;uA;     -1.0;uA;       0;Wafer_current_at_2000mV ($P);0;
     End_limit_data
    Data_format_pins: pin_name  PFNU0 [ value0 unit0 ] PFNU1 [ value1 unit1 ]
    Begin_data_pins:
    wafer F     0.000002 	 A  N     0.000000 	-
    End_data_pins
  M02831EO  M02831EO PASS
     Test_number: 102
    Begin_user_data:
bit31_col_00-031_01 254 258 254 258 254 258 254 258 254
bit30_col_31-063_01 258 254 258 254 258 254 258 254 258

digit_cap_M0_b28-b31_even
ffeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
...
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeee00eeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

bit31_col_00-031_10 256 256 256 256 256 256 256 256 256
bit30_col_31-063_10 256 256 256 256 256 256 256 256 256

M02831 NFE = 32768 NFO = 32768 NF = 65408

 digit_cap_M0_b28-b31_odd
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee
...

More information about the Python-list mailing list