New line conversion with Popen attached to a pty

[Jonathan Harden]

Hi,

We have a class which executes external processes in a controlled environment and does "things" specified by the client program with each line of output. To do this we have been attaching stdout from the subprocess.Popen to a pseudo terminal (pty) made with pty.openempty and opened with os.fdopen. I noticed that we kept getting a bunch of extra new line characters. 

This is all using python 2.6.4 in a centos6 environment.

After some investigation I realised we needed to use universal_newline support so I enabled it for the Popen and specified the mode in the fdopen to be rU. Things still seemed to be coming out wrong so I wrote up a test program boiling it down to the simplest cases (which is at the end of this message). The output I was testing was this:

Fake\r\nData\r\n
as seen through hexdump -C:

> hexdump -C output.txt 
00000000  46 61 6b 65 0d 0a 44 61  74 61 0d 0a              |Fake..Data..|
0000000c

Now if I do a simple subprocess.Popen and set the stdout to subprocess.PIPE, then do p.stdout.read() I get the correct output of

Fake\nData\n

When do the Popen attached to a pty I end up with

Fake\n\nData\n\n

Does anyone know why the newline conversion would be incorrect, and what I could do to fix it? In fact if anyone even has any pointers to where this might be going wrong I'd be very helpful, I've done a lot of hours of fiddling with this and googling to no avail.

Thanks,

Jonathan

#!/usr/bin/env python2.6.4
import os
import pty
import subprocess
import select
import fcntl

class TestRead(object):

    def __init__(self):
        super(TestRead, self).__init__()
        self.outputPipe()
        self.outputPty()

    def outputPipe(self):
        p1 = subprocess.Popen(
            ("/bin/cat", "output.txt"),
            stdout=subprocess.PIPE,
            universal_newlines=True
        )
        print "1: %r" % p1.stdout.read()

    def outputPty(self):
        outMaster, outSlave = pty.openpty()
        fcntl.fcntl(outMaster, fcntl.F_SETFL, os.O_NONBLOCK)

        p2 = subprocess.Popen(
            ("/bin/cat", "output.txt"),
            stdout=outSlave,
            universal_newlines=True
        )

        with os.fdopen(outMaster, 'rU') as pty_stdout:
            while True:
                try:
                    rfds, _, _ = select.select([pty_stdout], [], [], 0.1)
                    break
                except select.error:
                    continue

            for fd in rfds:
                buf = pty_stdout.read()
                print "2: %r" % buf

if __name__ == "__main__":
    t = TestRead()