Log Manager application: are there components already built ?

[Barak, Ron]

Hi Guys,

I need to create a Log Manager application that is able to accept log files with various line (e.g., error messages) formatting, and create log objects that could be put into a data-base.

Some of the requirements of this Log Manager, from the perspective of a client program would be:

1) Open a log file (in line/error-message format A) in the log_files_collection (where a log is actually a set of several files, some of which might be compressed).
2) Open a log file (in line/error-message format B) in the log_files_collection (where a log is actually a set of several files, some of which might be compressed). etc...
3) Given a log and a set of error pattern objects, generate a set of error objects that correspond to error messages in the log.  Each error object represents a normalized error type, with fields from the error message text processed into attributes that allow easy cross-referencing into a database.
4) Given an error object, retrieve the 'n' log entries on either side (above or below) of the error message.
5) Given an error object, retrieve the log entries for 'n' seconds on either side of the error message.
6) Retrieve the start and end dates for a log.
7) Generate a density histogram for all messages within a specified time period for a log.
8) Generate a density histogram for messages matching a specified pattern within a specified time period for a log.
9) Retrieve all log entries matching a specified pattern within a time period.

Envisioning how the Log Manager might do each of the above, produces requirements which include:

1)  The log manager must be able to order a set of related log files so that the client sees a single, time-ordered collection of log messages.
2)  The log manager must be sequentially accessible, both forward and backward.
3)  The log manager must be randomly accessible by an event index.
4)  The log manager must be randomly accessible by event time.
5)  The log manager must allow the client to define error patterns that support regular expression matching, and the ability to convert values embedded in the log message text into keys that can be used to look up items in a ConnectivityGraph database.

(These are not all the requirements, but you get the idea)

Googling around, I found pyparsing which may help with some of the above requirements.

Anyone can suggest other "wheels" already built that I could use and save me "reinventing" them?
Namely, are there Python modules that could perform the above actions ?

Thanks,
Ron.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20081116/c45896b8/attachment.html>