[Tutor] Bufferin

[Avi Gross]

After my earlier message (snipped out for space savings) focused on choosing among various methods to retain a buffer of recent lines from a file, I realized that for many, the best method is simply to import a solution others have created, often in the form of an object. Many of the methods I discussed (and undoubtedly many other ways) are likely to be incorporated using object-oriented aspects in a class. I mean if you created a class called Bufferin and set methods like:

__init__ takes an argument "n" with say a default of 5 and initializes whatever internal data structure will hold the info.

__add__ might overlay the + sign operator to take the string (or any object) passed to it and add it to the internal representation for your ordered collection of n objects and perhaps delete something to make room.

__iadd__ and __radd__ might be set to allow += to work as an overlay or to accept adding from the other side.

__str__ or perhaps __repr__ could be overlaid to print the results in whatever way you want.

And, you could just make normal named methods to allow you to say Bufferin.insert() and Bufferin.display() or whatever interface you want to use.

Using an instance of a class this way would let you have multiple active buffers active containing the last N lines (or any objects) from multiple sources.

Subclassing such an object might even let you be more selective. For example, a log may contain entries you do not want buffered such as just warnings or entries that say what time something  started. You want the last N lines for context to be mostly things that are useful in debugging. Any of the above can include specific logic that intercepts the call to add an item and only calls the class method if it is deemed to be worth keeping.

In any case, buffering is an extremely common phenomenon, and a search for something like "python circular buffer class" can provide code you might just include and use for free and that may be well tested or even more efficient. Some are probably parts of packages normally available in built-in modules. 

So, again, the solution Alan suggested as a small part of the program he was discussing, is fine. I like to get more abstract and see if there is a reasonable solution with more power for other needs. My earlier message focused on ways to either do it simpler from a human perspective or maybe even faster and then on making the size of the buffer variable. Using a class object may help not only by having a way to maintain the data without passing it repeatedly in functional calls but by allowing multiple buffers running independently. Heck, I can even imagine a log where multiple processes or threads write to the same log file in an interspersed way. Your goal in the program that began this discussion might then be to scan for a problem in ANY process or session and as soon as found, display the last N lines for just that entity. 

So you might read each line, find something like a process ID, and throw the line into an appropriate buffer. One way would be to have a master dictionary whose keys are that ID and the values are instances of your buffer object. When a failure is encountered, get the object for that key and print the contents as discussed above. You might then continue processing the log after optionally removing that object or setting it back to empty and process any other such errors till the log is completed.

Just some thoughts but also a comment I must add. To those that don't know me (meaning almost everyone) I have been known to play games that range from humor to sarcasm to word games. I chose the name Bufferin because of the similarity in wordplay but also because there is an analgesic product called Bufferin that is named that for a somewhat different reason having more to do with Chemistry. It is basically Aspirin but with added ingredients that buffer it (Magnesium and Calcium carbonates and oxides)  that are antacids so that some people experience less gastric discomfort when taking plain aspirin. Alan used "buff" as a variable, which has other meanings like someone in good physical shape or phrases like being in the buff meaning without clothes. Similar ideas in the word "buffed." I generally do not stop and explain my puns or wordplay but if something seems phrased oddly, I may be bending things a bit. I did exert some self-control by not using the name of my favorite Vampire Slayer 😊

Last comment. If the purpose of the limited Buffer is to save space by not reading the entire file at once and retaining just enough context when needed, you may want to remember to release this space too when done using it. Python garbage collection may kick in for say closing the file opened in a "for line in open(...):" or a context manager context like "with open(...) as line:" but you may want to delete things like the buffers manually when done unless they too are set up to be temporary. For a buffer size of 5, no big deal. But if you feel the need to retain the last 100K lines, perhaps in multiple buffers, ...

Avi