Python deadlock using subprocess.popen and communicate

[Atherun]

On Sep 21, 8:33 pm, Chris Rebert <c... at rebertia.com> wrote:
> On Wed, Sep 21, 2011 at 8:09 PM, Atherun <athe... at gmail.com> wrote:
> > This is on windows with python 2.6.
> > I can't seem to remove a possibility of a  deadlock in one of my
> > scripts at the moment.  Its not a constant deadlock but it appears
> > from time to time.  The code is below:
>
> > try:
>
> >        process =
> > subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
>
> >        NotDone = True
> >        data = []
> >        while NotDone:
> >            NotDone = False
>
> >            result = process.poll()
> >            if result == None:
> >                NotDone = True
> >            if NotDone:
> >                out, err = process.communicate()
> >                Log(out)
>
> > I was able to get the program to write out the stack trace for my
> > threads and it is usually deadlocked on sys.stdout.read or
> > _internal_poll of subproceess.
>
> > I've even tried the above, with using "with
> > tempfile.NamedTemporaryFiles() as buff:" and writing to the file, but
> > it still deadlocks.  In the past I work around this by running fewer
> > processes asynchronously but I would really like to get this solved so
> > I don't have to wait to see if it'll be caused with any changes I
> > make.
>
> > Any tips would be appreciated.
>
> Your polling loop is completely pointless. The code can be simplified to:
>
> process = subprocess.Popen(cmd,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
> out, err = process.communicate()
> log(out)
>
> Depending on the particular program you're running in a subprocess,
> you may need to take heed of the Note in the communicate() docs:
> "Note: The data read is buffered in memory, so *do not use this
> method* if the data size is large or unlimited." [Emphasis added]
>
> Cheers,
> Chris
> --http://rebertia.com

I'm pretty sure thats the problem, this is a generic catch all
function for running subprocesses.  It can be anything to a simple
command to a complex command with a ton of output.  I'm looking for a
better solution to handle the case of running subprocesses that have
an undetermined amount of output. (i.e. p4 edit's or tools that have
an excessive amount of logging)

I'll definitely simplify the loop, thanks for the tip on that part.