[Tutor] My first real python project: LOLLERSKATES

[Chris Hengge]

All I can say to this... LOLLERSKATES.... =D

On 10/20/06, Tracy R Reed <treed at ultraviolet.org> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have been a sysadmin for quite a while now and used to do a lot of
> perl. I haven't done much programming at all in the last couple of years
> but have been meaning to dump perl for python anyhow (for the usual
> reasons) and have finally gotten around to doing it.
>
> The first itch that I wanted to scratch was something to help me find
> anomalies in our system logs. We have a lot of servers and need some
> automation for watching over them including logfiles. I have used
> logwatch and logcheck but they are both too complicated for what I
> really need not to mention practically unmaintained these days. So I
> decided to write my own.
>
> Features I wanted were:
>
> 1. A basic grep -v sort of functionality on the logfile where I have a
> file full of regexes/lines to be ignored that are applied to the
> logfiles to filter everything uninteresting out and whatever is left
> gets emailed to the admin.
>
> 2. A sort of macro system where I can substitute nasty long commonly
> used regexes in my filter/ignore file with something more easily read
> and typed.
>
> 3. Simplicity. Logcheck had several different levels of logfile events
> which it could report back on and if you wanted something ignored you
> had to put it in the right config file for whatever level it was popping
> up in. This drove me nuts because I often got the wrong one. Logwatch
> tries to analyze your logs and provide various summaries and statistics
> and other things I don't care about. I just want to see the interesting
> lines from the logfile.
>
> My code seems to have all of the above and a silly name as well:
>
> http://ultraviolet.org/Members/treed/lollerskates
>
> It has a list of logfiles to look at, a list of regexes to compare to
> each line to know what is uninteresting and to filter out/ignore, and
> emails anything left to the admin, and a simple macro facility.
>
> I would appreciate it if anyone interested would download and critique
> my code. Am I doing anything terribly un-pythonic?
>
> Questions/problems/TODO's:
>
> This is a fairly simple structured programming implementation. No OO.
> Should I be using some classes somewhere?
>
> The config file is just a module which I import which causes all of my
> configs to become globals. Globals are bad. Is there a better way or
> would just about anything else be overkill? A singleton config class or
> something? Overkill?
>
> I have several loops in this code for processing the logfiles. I once
> tried to convert these for loops to list comprehensions and totally
> confused myself and backed out the change (yeay svn!). Is there any
> benefit to list comprehensions in this case?
>
> I would kinda like to play with unit tests. Not sure how I would
> construct unit tests for this. And again, perhaps overkill. But some
> people tell me you should write the unit tests before you even begin
> coding and code until the tests are satisfied. So even on a small
> project you would have tests. I run into a couple nasty bugs I created
> which caused the script to never return anything from the logfile (so
> you don't immediately realize something is broken) where I thought "It
> sure would be nice to have a test to catch that if it ever happens again."
>
> Stuff tends to build up in the ignore file. If a line is typo'd and
> never gets used/matched you will never know. If a service is no longer
> running that we had lines in the ignore file for they will be there
> forever unused. I don't like this sort of clutter. I am really tempted
> to somehow make it keep track of when a rule is matched and if it
> doesn't match in a whole month email the admin because it doesn't need
> to be there. I am wondering if this is needless complexity or something
> useful. I think it could be useful. If so, how to save the info on when
> each line was last matched? I am thinking maybe of just having a
> dictionary keyed on the compiled regex object and the value being a
> tuple of the uncompiled regex string from the ignore file and a date
> string containing when it was last matched and pickling this data
> structure to disk to be read in each start of the program. Comments?
>
> I am currently pushing this code to around 20 systems with cfengine
> using just one ignore file for them all. If I use the scheme mentioned
> in the previous paragraph to alert me of unused rules in the ignore file
> I will get tons of such messages from machines not running a service
> that other machines of mine are. I guess that means perhaps I should
> split my ignore file out into ignore.hostname.conf or something like
> that so there is a separate one for each host.
>
> Anyhow, I appreciate any feedback on this potentially very handy
> sysadmin tool. Thanks in advance!
>
> - --
> Tracy R Reed
> http://ultraviolet.org
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFFObPK9PIYKZYVAq0RAiflAJ9jprJgGnNRXkB+nKsljFUsAUGnFwCgiJg/
> M3U24dNDtH+hAgSh7kZ40hQ=
> =nxQS
> -----END PGP SIGNATURE-----
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/tutor/attachments/20061020/83f632d6/attachment-0001.html