[Tutor] Python structure advice ?

[Dave S]

Alan Gauld wrote:

>>>1) batch oriented - each step of the process produces its own
>>>output file or data structure and this gets picked up by the
>>>next stage. Tis usually involved processing data in chunks
>>>- writing the first dump after every 10th set of input say.
>>>
>>>      
>>>
>>I see your point, like a static chain, one calling the next &
>>    
>>
>passing
>  
>
>>data, the problem being that the links of the chain will need to
>>remember their previous state when called again, so their output is
>>    
>>
>a
>  
>
>>function of previous data + fresh data. I guess their state could be
>>written to a file, then re-read.
>>    
>>
>
>Yes. Just to expand: the typical processing involves three files:
>1) the input which is the output of the preceding stage
>2) the output which will form input to the next stage
>3) the job log. This will contain references to any input data
>items that failed to process - typically these will be manually
>inspected, corrected and a new file created and submitted at the
>end of the batch run.
>
>BUT 3) will also contain the sequence number of the last file and/or
>last data item processed so that when the next cycle runs it knows
>where to start. It is this belt and braces approach to data
>processing and error recovery that makes mainframes so reliable,
>not just the hardware, but the whole culture there is geared to
>handling failure and being able to *recover* not just report on it.
>After all its the mainframes where the really mission critical
>software of any large enterprise runs!
>
>As an ex Unix head I learned an awful lot about reliable computing
>from the 18 months I spent working on a mainframe project. These
>guys mostly live in a highly specialised microcosm of their own
>but they have learned a lot of powerful tricks over the last 40
>years that the rest of us ignore at our peril. I strongly
>recommend that anyone who gets the chance of *a short* contract
>in mainframe land, with training, to grab the opportunity with
>both hands!
>
>< Steps off soapbox now :-) >
>
>Alan G
>Author of the Learn to Program web tutor
>http://www.freenetpages.co.uk/hp/alan.gauld
>
>
>
>  
>
You get on that soapbox whenever you want :-) , its good to hear a range 
of views !

Dave