Help me with a bytes decoding problem in python 3

[J]

I have a bit of a problem I hope you could help me sort out with some
code I'm porting from 2.x to 3.

I have a program with a wrapper for Popen that is called to run
certain linux shell commands and gather system info.  Essentially the
code looks something like this:


process = subprocess.Popen(command_str,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
result = process.communicate()

stdout,stderr = result

message = ['Output:\n'
                      '- returncode:\n{0}'.format(self.process.returncode)]
if stdout:
   if type(stdout) is bytes:
       stdout = stdout.decode()
   message.append('- stdout:\n{0}'.format(stdout))
if stderr:
   if type(stderr) is bytes:
       stderr = stderr.decode()
   message.append('- stderr:\n{0}'.format(stderr))
logging.debug('\n'.join(message))

So what essentially happens is that command_str is passed to this
function and is a string like this:

'lspci | grep Network'
or
'xinput list'

using the first command string, I get output like this:

In [12]: command = 'lspci |grep Network'
In [13]: process =
subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In [14]: result = process.communicate()
In [15]: stdout,stderr = result
In [16]: result
Out[16]:
(b'00:19.0 Ethernet controller: Intel Corporation 82577LC Gigabit
Network Connection (rev 06)\n07:00.0 Network controller: Intel
Corporation Ultimate N WiFi Link 5300\n',
 b'')

Which is a bytes object that is very easily converted to text by the
decode() call prior to sending "message" to the logger.

However, the program seems to now be hanging up when running the
second command_str because it's actually returning bytecode inside the
bytes object:

In [18]: command = 'xinput list'
In [19]: process =
subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
In [20]: result = process.communicate()
In [21]: stdout,stderr = result
In [22]: result
Out[22]:
(b'\xe2\x8e\xa1 Virtual core pointer
\tid=2\t[master pointer  (3)]\n\xe2\x8e\x9c   \xe2\x86\xb3 Virtual
core XTEST pointer              \tid=4\t[slave  pointer
(2)]\n\xe2\x8e\x9c   \xe2\x86\xb3 Logitech Unifying Device. Wireless
PID:1017\tid=9\t[slave  pointer  (2)]\n\xe2\x8e\x9c   \xe2\x86\xb3
SynPS/2 Synaptics TouchPad              \tid=12\t[slave  pointer
(2)]\n\xe2\x8e\x9c   \xe2\x86\xb3 MCE IR Keyboard/Mouse (ite-cir)
  \tid=14\t[slave  pointer  (2)]\n\xe2\x8e\xa3 Virtual core keyboard
                \tid=3\t[master keyboard (2)]\n    \xe2\x86\xb3
Virtual core XTEST keyboard             \tid=5\t[slave  keyboard
(3)]\n    \xe2\x86\xb3 Power Button
\tid=6\t[slave  keyboard (3)]\n    \xe2\x86\xb3 Video Bus
                \tid=7\t[slave  keyboard (3)]\n    \xe2\x86\xb3
Laptop_Integrated_Webcam_2M             \tid=8\t[slave  keyboard
(3)]\n    \xe2\x86\xb3 HID 413c:8157
\tid=10\t[slave  keyboard (3)]\n    \xe2\x86\xb3 AT Translated Set 2
keyboard            \tid=11\t[slave  keyboard (3)]\n    \xe2\x86\xb3
Dell WMI hotkeys                        \tid=13\t[slave  keyboard
(3)]\n    \xe2\x86\xb3 ITE8708 CIR transceiver
\tid=15\t[slave  keyboard (3)]\n',
 b'')

Unfortunately, what's happening here is all that extra formatting
stuff that xinput injects is also converted when doing bytes.decode()
on result[0].


In [24]: result[0].decode()
Out[24]: '⎡ Virtual core pointer                    \tid=2\t[master
pointer  (3)]\n⎜   ↳ Virtual core XTEST pointer
\tid=4\t[slave  pointer  (2)]\n⎜   ↳ Logitech Unifying Device.
Wireless PID:1017\tid=9\t[slave  pointer  (2)]\n⎜   ↳ SynPS/2
Synaptics TouchPad              \tid=12\t[slave  pointer  (2)]\n⎜   ↳
MCE IR Keyboard/Mouse (ite-cir)         \tid=14\t[slave  pointer
(2)]\n⎣ Virtual core keyboard                   \tid=3\t[master
keyboard (2)]\n    ↳ Virtual core XTEST keyboard
\tid=5\t[slave  keyboard (3)]\n    ↳ Power Button
     \tid=6\t[slave  keyboard (3)]\n    ↳ Video Bus
           \tid=7\t[slave  keyboard (3)]\n    ↳
Laptop_Integrated_Webcam_2M             \tid=8\t[slave  keyboard
(3)]\n    ↳ HID 413c:8157                           \tid=10\t[slave
keyboard (3)]\n    ↳ AT Translated Set 2 keyboard
\tid=11\t[slave  keyboard (3)]\n    ↳ Dell WMI hotkeys
      \tid=13\t[slave  keyboard (3)]\n    ↳ ITE8708 CIR transceiver
             \tid=15\t[slave  keyboard (3)]\n'

And so I believe that when that decoded string is being pushed to the
logger, the logger is choking, causing the program to just hang
waiting on something... not sure which.

Though when doing this manually via ipython3, logger seems to handle
it just fine:

In [37]: message = ['Output:\n'
'- returncode:\n{0}'.format(process.returncode)]

In [38]: message.append('- stdout:\n{0}'.format(stdout.decode()))

In [39]: logging.debug('\n'.join(message))
DEBUG:root:Output:
- returncode:
0
- stdout:
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
⎜   ↳ Logitech Unifying Device. Wireless PID:1017       id=9    [slave
 pointer  (2)]
⎜   ↳ SynPS/2 Synaptics TouchPad                id=12   [slave  pointer  (2)]
⎜   ↳ MCE IR Keyboard/Mouse (ite-cir)           id=14   [slave  pointer  (2)]
⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
   ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
   ↳ Power Button                              id=6    [slave  keyboard (3)]
   ↳ Video Bus                                 id=7    [slave  keyboard (3)]
   ↳ Laptop_Integrated_Webcam_2M               id=8    [slave  keyboard (3)]
   ↳ HID 413c:8157                             id=10   [slave  keyboard (3)]
   ↳ AT Translated Set 2 keyboard              id=11   [slave  keyboard (3)]
   ↳ Dell WMI hotkeys                          id=13   [slave  keyboard (3)]
   ↳ ITE8708 CIR transceiver                   id=15   [slave  keyboard (3)]

outside of ipython3, I'm running this and looking at the log file the
actual program generates and it always hangs up on the output from
'xinput list'.

Interestingly though, it gathers this info both before and after the
reboot action this script performs... :(

I tried attaching the actual program, but apparently it's too big :(