[Tutor] Logfile multiplexing

[ALAN GAULD]

> Why would that be admitting defeat?
> 
> Well, it mean admitting defeat on solving the problem in python.  Yes

You could still use Python to do the sorting. But it breaks the problem 
into two separate and simpler processes. One simply sorts a file given 
a particular data layout and sort algorithm optimised for sorting where
a) the volumes dictate doing it out of memory, and
b) the data is almost sorted already.

and  the other merges pre-sorted files into one, filtering out unwanted 
lines as you go.

cron can be used to launch the sorting jobs - one per file if that 
is a constant, or via a shell script that checks the files and launches 
the python sort per file found as a background process. (You could 
replace the shell with Python too but shell is designed for this kind 
of role).

cron can also be used to then launch the merge job which can check 
that the sorting has completed, with no errors and merge the results.

> programming exercise, learning how to sort a number of files into one
> is something I'd like to crack.

The problem hasn't changed, nor has the solution technology, its how 
you architect the solution that is different. Just because its Python
doesn't mean it all has to be in a single program.

> Maybe the real lesson here is knowing which battles to fight, and a
> good developer uses the right tools for the job.

Absolutely.


Alan G.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20091111/320dba1c/attachment.htm>