[Tutor] saveLine decked

[Avi Gross]

Peter,

Appreciated. I wrote something like this in another message before reading
yours. Indeed one of the things I found was the deque class in the
collections module. 

But I was not immediately clear on whether that would be directly
applicable. Their maximum sounded like if you exceeded it, it might either
reject the addition or throw an error. The behavior I wanted was sort of a
sliding window protocol where the oldest entry scrolled off the screen or
was simply removed. Sort of like what you might do with a moving average
that takes the average of just the last 20 days of a stock price.

But I searched some more and stand corrected. 

"New in version 2.4.

If maxlen is not specified or is None, deques may grow to an arbitrary
length. Otherwise, the deque is bounded to the specified maximum length.
Once a bounded length deque is full, when new items are added, a
corresponding number of items are discarded from the opposite end. Bounded
length deques provide functionality similar to the tail filter in Unix. They
are also useful for tracking transactions and other pools of data where only
the most recent activity is of interest."

That sounds exactly like what is needed. As long as you keep adding at the
end (using the append method) it should eventually remove from the beginning
automatically.  No need to use count and selectively remove or pop manually.

And despite all the additional functionality, I suspect it is tuned and
perhaps has parts written in C++ for added speed. I do note that any larger
log file used in the application discussed may throw things on the deque
many times but only ask it to display rarely so the former should be
optimized.

But one question before I go, Columbo style. The manual page
(https://docs.python.org/2/library/collections.html ) suggest you call deque
with an iterator. That would not necessarily meet our need as giving it the
entire file as an iterator would just grind away without any logic and keep
just the last N lines. We could arrange the logic in our own iterator, such
as a function that reads a line at a time using its own open iterator and
yields the line over but that too is problematic as to how and when you stop
and print the results. But on second look, the iterator is optional and I
tried creating a deque using just a maxlen=3 argument for illustration.

>>> from collections import deque

>>> a=deque(maxlen=3)
>>> a
deque([], maxlen=3)
>>> a.append('line 1\n')
>>> a
deque(['line 1\n'], maxlen=3)
>>> a.append('line 2\n')
>>> a.append('line 3\n')
>>> a
deque(['line 1\n', 'line 2\n', 'line 3\n'], maxlen=3)
>>> a.append('line N\n')
>>> a
deque(['line 2\n', 'line 3\n', 'line N\n'], maxlen=3)

OK, that looks right so all you need to figure out is how to print it in a
format you want.

As it happens, deque has an str and a repr that seem the same when I try to
print:

>>> a.__str__()
"deque(['line 2\\n', 'line 3\\n', 'line N\\n'], maxlen=3)"
>>> a.__repr__()
"deque(['line 2\\n', 'line 3\\n', 'line N\\n'], maxlen=3)"

So you either need to subclass deque to get your own printable version (or
use an amazing number of other Python tricks since you can, or do something
manually. 

>>> for line in a: print(line)

line 2

line 3

line N

OK, that works but my \n characters at the end of some items might suggest
using end='' in the 3.X version of print for a smaller display.

Summary: the method Peter mentions is a decent solution with no programming
or debugging overhead. It is even flexible enough, if you choose, to store
or display the lines backwards as in showing the last line that showed the
error, followed by successively earlier lines. 

Why use a limited solution when you can play with a full deck?





-----Original Message-----
From: Tutor <tutor-bounces+avigross=verizon.net at python.org> On Behalf Of
Peter Otten
Sent: Sunday, November 11, 2018 2:43 PM
To: tutor at python.org
Subject: Re: [Tutor] saveLine

Avi Gross wrote:

> Alan and others have answered the questions posed and what I am asking 
> now is to look at the function he proposed to keep track of the last 
> five lines.
> 
> There is nothing wrong with it but I wonder what alternatives people 
> would prefer. His code is made for exactly 5 lines to be buffered and 
> is quite efficient. But what if you wanted N lines buffered, perhaps 
> showing a smaller number of lines on some warnings or errors and the 
> full N in other cases?

The standard library features collections.deque. With that:

buffer = collections.deque(maxlen=N)
save_line = buffer.append

This will start with an empty buffer. To preload the buffer:

buffer = collections.deque(itertools.repeat("", N), maxlen=N)

To print the buffer:

print_buffer = sys.stdout.writelines

or, more general:

def print_buffer(items, end=""):
    for item in items:
        print(item, end=end)

Also, for smallish N:

def print_buffer(items, end=""):
    print(*items, sep=end)

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor