Implementation of popen() with timeout

[David Bolen]

Tom Harris <tomh at optiscan.com> writes:

> I want to use one of the popen() functions to call an external program and
> capture the output, but I would like to have the facility to kill the
> external program if it does not complete within a specific time.
> Unfortunately I am using NT. Options appear to be to use the Win32 API to
> write my own, as the popen source looks fierce code. Any ideas?

What I did for something like this (some legacy Perl scripts that
would get hung up in database accesses sometimes) was to multi-thread.
The popen() was done in one thread, and a separate thread acted as the
monitor.  As long as output was being processed (indicated via a
shared flag) all was well, but if nothing was seen for a timeout, the
monitor thread killed off the process at the Win32 level.  I actually
used a separate utility (the sysinternals 'pskill' utility) although
if win32all wraps the kill process call that could also be used.

To add insult to injury, the legacy scripts could decide to exit
successfully (exit code wise) even if there was a problem, so in my
case I added some code to monitor the output and look for a final
message - if I didn't see it I'd declare a failure even if the
underlying scripts didn't.

Here's a slightly edited version of the function (sans monitoring for
that message), and the top level "execute" function that used it (note
that the FindPerl function located Perl processes that had been
running at least as long as the timeout, since the machine might be
running other Perl processes - replace that with whatever you need to
determine the appropriate child PIDs):

#
# --------------------------------------------------
#
class PerlThread (Thread):
    """This class executes a separate thread to run a perl command,
    monitoring its output and timing how long a delay there is since the
    last output.  A separate method (intended to be called from a separate
    thread) will check this time and if necessary try to kill off any
    perl processes to free up a hung system."""

    def __init__(self, command):
        Thread.__init__(self)
        self.command   = command
        self.lasttime  = 0
        self.finished  = 0
        self.result    = 0

    def monitor(self, timeout):
        """monitor(self, timeout)

        This function should be called from a separate thread, and will
        monitor the delay since last sub-process output, trying to kill
        off any perl processes if it exceeds the threshold.  When this
        function returns, the process has either completed or timed out.

        This should only be called after the start() method has already
        returned for the separate thread of execution"""
        while 1:
            if self.finished or not self.isAlive():
                break

            if (self.lasttime and
                (time.time() - self.lasttime) > timeout):
                sys.stdout.write('ERROR: Monitor Timeout waiting for command '
                                 'to complete (%ds)\n' % timeout)
                pids = FindPerl(timeout)
                sys.stdout.write('Trying to kill Perl processes %s\n' % pids)
                for curpid in pids:
                    os.system('pskill %s' % curpid)
                self.result = 255
                break

            time.sleep(5)

        return

    def run(self):
        """run(self)

        This is the independent thread that executes the command."""

        sys.stdout.write('Executing: %s\n' % self.command)
        pipe = os.popen(self.command + " 2>&1")

        while 1:
            self.lasttime = time.time()
            line = pipe.readline()
            if not line:
                break
            print "  ", line,

        self.result = pipe.close()

    def rc(self):
        return self.result


#
# --------------------------------------------------
#
def ExecuteCmd(command, timeout):
    """ExecuteCmd(command)

    Executes the command using the PerlThread class with an execution and
    monitoring thread.  Returns the result code from the execution."""

    # Create thread for execution and execute
    perl = PerlThread(command)
    perl.start()
    perl.monitor(timeout)
    perl.join()

    return perl.rc()



--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/