Success with subprocess communicate on Windows?

[Terry Reedy]

On 7/2/2014 7:22 PM, Wolfgang Maier wrote:

> Finally found out what the problem is:
> When I'm running your command using the cmd console, I get this output:
>
> c:\python34\lib\turtle.py:571: local variable 'rgb' is assigned to but never
> used
> c:\python34\lib\turtle.py:2936: local variable 'a21' is assigned to but
> never used
> c:\python34\lib\turtle.py:3590: local variable 'dummy' is assigned to but
> never used
> c:\python34\lib\turtle.py:3786: undefined name 'mainloop'
> c:\python34\lib\turtle.py:3969: undefined name 'mainloop'
> c:\python34\lib\turtle.py:3973: undefined name 'isdown'
> c:\python34\lib\turtle.py:3974: undefined name 'pu'
> c:\python34\lib\turtle.py:3976: undefined name 'pd'
> ..
>
> now look at the exit code:
> echo %errorlevel%
> 1

I did not know about this.

> ah, pyflakes does exit with a non-zero exit code when it finds errors in
> your file!!

I guessed that and confirmed it in pyflakes.api.main().

The same is true with >pyflakes c:/programs/python34/lib (path for my 
machine), except that then there is 20 or more times as much output.

> Now, using subprocess.check_output in IDLE:
>
>>>> msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
> Traceback (most recent call last):
>    File "<pyshell#45>", line 1, in <module>
>      msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
>    File "C:\Python34\lib\subprocess.py", line 618, in check_output
>      raise CalledProcessError(retcode, process.args, output=output)
> subprocess.CalledProcessError: Command 'pyflakes c:\python34\lib\turtle.py'
> returned non-zero exit status 1

Yes, but what puzzled me is that running
subprocess.check_output(r'pyflakes c:\programs\python34\lib')
in the regular interpreter *does* produce output instead of the error 
message. My guess is that it fills up the pipe, so that check_output 
starts reading the pipe long before pyflakes exits with status 1.

Hmm. I tried it again, and I see some but not all of the output I got at 
the command line *and* I see the exit status message. So the subprocess 
must get some of the output but then stop when it sees the exit status. 
  Thanks.

> as you said, but logical since (taken from the docs):
>
> "subprocess.check_output(args, *, input=None, stdin=None, stderr=None,
> shell=False, universal_newlines=False, timeout=None)
> Run command with arguments and return its output.
>
> If the return code was non-zero it raises a CalledProcessError. The
> CalledProcessError object will have the return code in the returncode
> attribute and any output in the output attribute."
>
> and in fact:
>>>> try:
> 	msg=subprocess.check_output(r'pyflakes c:\python34\lib\turtle.py')
> except subprocess.CalledProcessError as e:
> 	print (e.output[:81])
>
> 	
> b"c:\\python34\\lib\\turtle.py:571: local variable 'rgb' is assigned to but
> never used"
>
> So, everything's just fine except that it may be more convenient to use
> Popen().communicate() to avoid raising the error in the first place :)

As I believe I said in my first post, that is where we started -- but in 
Idle. I did not realize then that at least some of the problem is 
specific to Idle on Windows and not Python on Windows. It may have 
something to do with running in pythonw.exe

> Hope it helps this time,

Yes, it did, to focus my attention. I will see if I can get all of the 
output with communicate in the normal interpreter. Having it work in 
Idle is a different matter, as before it hung indefinitely without 
showing anything with
subprocess.check_output(r'pyflakes c:\programs\python34\lib')
or with the original Popen.communicate.
But I should start experiments in Idle with something that works right 
in the normal interpreter.
---
Doing that, this works on 2.7.7, 3.4.1+, and 3.5.0a
(and would work in posix also).

import subprocess as s
p = s.Popen([r'pyflakes', r'c:\programs\python34\lib\turtle.py'],
             stdout=s.PIPE, stderr=s.PIPE)
o,e = p.communicate()
for line in o.decode().splitlines(): print(line)

Next I will reapply the patch that did not work, add a debug print, and 
see how what did not work Windows (but worked on linux) is different 
from the above.

-- 
Terry Jan Reedy