[Tutor] Python structure advice ?

[Alan Gauld]

> >1) batch oriented - each step of the process produces its own
> >output file or data structure and this gets picked up by the
> >next stage. Tis usually involved processing data in chunks
> >- writing the first dump after every 10th set of input say.
> >
> I see your point, like a static chain, one calling the next &
passing
> data, the problem being that the links of the chain will need to
> remember their previous state when called again, so their output is
a
> function of previous data + fresh data. I guess their state could be
> written to a file, then re-read.

Yes. Just to expand: the typical processing involves three files:
1) the input which is the output of the preceding stage
2) the output which will form input to the next stage
3) the job log. This will contain references to any input data
items that failed to process - typically these will be manually
inspected, corrected and a new file created and submitted at the
end of the batch run.

BUT 3) will also contain the sequence number of the last file and/or
last data item processed so that when the next cycle runs it knows
where to start. It is this belt and braces approach to data
processing and error recovery that makes mainframes so reliable,
not just the hardware, but the whole culture there is geared to
handling failure and being able to *recover* not just report on it.
After all its the mainframes where the really mission critical
software of any large enterprise runs!

As an ex Unix head I learned an awful lot about reliable computing
from the 18 months I spent working on a mainframe project. These
guys mostly live in a highly specialised microcosm of their own
but they have learned a lot of powerful tricks over the last 40
years that the rest of us ignore at our peril. I strongly
recommend that anyone who gets the chance of *a short* contract
in mainframe land, with training, to grab the opportunity with
both hands!

< Steps off soapbox now :-) >

Alan G
Author of the Learn to Program web tutor
http://www.freenetpages.co.uk/hp/alan.gauld