Searching through two logfiles in parallel?

[Oscar Benjamin]

On 7 January 2013 22:10, Victor Hooi <victorhooi at gmail.com> wrote:
> Hi,
>
> I'm trying to compare two logfiles in Python.
>
> One logfile will have lines recording the message being sent:
>
>     05:00:06 Message sent - Value A: 5.6, Value B: 6.2, Value C: 9.9
>
> the other logfile has line recording the message being received
>
>     05:00:09 Message received - Value A: 5.6, Value B: 6.2, Value C: 9.9
>
> The goal is to compare the time stamp between the two - we can safely assume the timestamp on the message being received is later than the timestamp on transmission.
>
> If it was a direct line-by-line, I could probably use itertools.izip(), right?
>
> However, it's not a direct line-by-line comparison of the two files - the lines I'm looking for are interspersed among other loglines, and the time difference between sending/receiving is quite variable.
>
> So the idea is to iterate through the sending logfile - then iterate through the receiving logfile from that timestamp forwards, looking for the matching pair. Obviously I want to minimise the amount of back-forth through the file.
>
> Also, there is a chance that certain messages could get lost - so I assume there's a threshold after which I want to give up searching for the matching received message, and then just try to resync to the next sent message.
>
> Is there a Pythonic way, or some kind of idiom that I can use to approach this problem?

Assuming that you can impose a maximum time between the send and
recieve timestamps, something like the following might work
(untested):

def find_matching(logfile1, logfile2, maxdelta):
    buf = {}
    logfile2 = iter(logfile2)
    for msg1 in logfile1:
        if msg1.key in buf:
            yield msg1, buf.pop(msg1.key)
            continue
        maxtime = msg1.time + maxdelta
        for msg2 in logfile2:
            if msg2.key == msg1.key:
                yield msg1, msg2
                break
            buf[msg2.key] = msg2
            if msg2.time > maxtime:
                break
        else:
            yield msg1, 'No match'


Oscar