Best way to report progress at fixed intervals

[rdmurray at bitdance.com]

On Tue, 9 Dec 2008 at 13:27, Slaunger wrote:
> On 9 Dec., 19:35, rdmur... at bitdance.com wrote:
>>
>> I felt like a little lunchtime challenge, so I wrote something that
>> I think matches your spec, based on your sample code.  This is not
>> necessarily the best implementation, but I think it is simpler and
>> clearer than yours.  The biggest change is that the work is being
>> done in the subthread, while the main thread does the monitoring.
>>
> Well, thank you for spending your lunch time break on my little
> problem.
>
>> It would be fairly simple to enhance this so that you could pass
>> arbitrary arguments in to the worker function, in addition to
>> or instead of the loop counter.
>>
> Yes, I agree
>
>> -----------------------------------------------------------------------
>> """
>> Test module for testing generic ways of displaying progress
>> information at regular intervals.
>> """
>> import random
>> import threading
>> import time
>>
>> def work(i):
>>      """
>>      Dummy process function, which takes a random time in the interval
>>      0.0-0.5 secs to execute
>>      """
>>      print "Work step %d" % i
>>      time.sleep(0.5 * random.random())
>>
>> class Monitor(object):
>>      """
>>      This class creates an object that will execute a worker function
>>      in a loop and at regular intervals emit a progress report on
>>      how many times the function has been called.
>>      """
>>
>>      def dowork(self):
>>          """
>>          Call the worker function in a loop, keeping track of how
>>          many times it was called in self.no
>>          """
>>          for self.no in xrange(self.max_iter):
>>              self.func(self.no)
>>
>>      def __call__(self, func, verbose=True, max_iter=20, progress_interval=1.0):
> I had to look up the meaning of __call__, to grasp this, but I get
> your methology
>>          """
>>          Repeatedly call 'func', passing it the loop count, for max_iter
>>          iterations, and every progress_interval seconds report how
>>          many times the function has been called.
>>          """
>>          # Not all of these need to be instance variables, but they might
>>          # as well be in case we want to reference them in an enhanced
>>          # dowork function.
>>          self.func = func
>>          self.verbose = verbose
>>          self.max_iter=max_iter
>>          self.progress_interval=progress_interval
>>
>>          if self.verbose:
>>              print ("Work through all %d steps reporting progress every "
>>                  "%3.1f secs...") % (self.max_iter, self.progress_interval)
>>
>>          # Create a thread to run the loop, and start it going.
>>          worker = threading.Thread(target=self.dowork)
>>          worker.start()
>>
>>          # Monitoring loop.
>>          loops = 0
>>          # We're going to loop ten times per second using an integer count,
>>          # so multiply the seconds parameter by 10 to give it the same
>>          # magnitude.
>>          intint = int(self.progress_interval*10)
> Is this not an unnecessary complication?
>>          # isAlive will be false after dowork returns
>>          while worker.isAlive():
>>              loops += 1
>>              # Wait 0.1 seconds between checks so that we aren't chewing
>>              # CPU in a spin loop.
>>              time.sleep(0.1)
> Why not just call this with progress_interval directly?

Because then the program make take up to progress_interval seconds to
complete even after all the work is done.  For a long running program
and a short progress_interval that might not matter, so yes, that would
be a reasonable simplification depending on your requirements.

>>              # when the modulus (second element of divmod tuple) is zero,
>>              # then we have hit a new progress_interval, so emit the report.
> And then avoid this if expression?
>>              if not divmod(loops, intint)[1]:
>>                  print "Processed %d of %d" % (self.no, self.max_iter)
>>
>>          if verbose:
>>              print "Finished working through %d steps" % max_iter
>>
>> if __name__ == "__main__":
>>      #Create the monitor.
>>      monitor = Monitor()
>>      #Run the work function under monitoring.
>>      monitor(work)
> I was unfamiliar with this notation, but now I understand it simply
> invokes __call__. Thank you for showing me that!

Yes, it is a very nice feature of python :)

> OK. I agree this is a more elegant implementation, although I my mind,
> I find it more natural if the reporting goes on in a subthread, but

You could pretty easily rewrite it to put the reporter in the subthread,
it was just more natural to _me_ to put the worker in the subthread,
so that's how I coded it.  Note, however, that if you were to write a
GUI front end it might be important to put the worker in the background
because on some OSes it is hard to update GUI windows from anything
other than the main thread.  (I ran into this in a Windows GUI ap I
wrote using wxPython).

> that is a matter of taste, I guess. Anyway: Thank you again for
> spending your lunch break on this!

No problem, it was fun.

--RDM