[issue37790] subprocess.Popen() is extremely slow

[Josh Rosenberg]

Josh Rosenberg <shadowranger+python at gmail.com> added the comment:

Have you tried switching to using Popen itself (or run, which keeps it to one layer of convenience wrapping)?

subprocess.getstatusoutput is three layers of wrapping (it calls check_output, which in turn calls run, which in turn calls Popen), and (unlike Python 2) has to decode all the output. run would avoid two layers of wrapping, can optionally receive the raw bytes instead of decoding to str, and avoids needing to wrap the whole thing in a shell (which system, older popen, and getstatusoutput all do).

Beyond that, it looks like when 3.8 releases, Popen should get *much* faster if the call meets certain conditions, see https://docs.python.org/3.8/whatsnew/3.8.html#optimizations for details. If you can make your use cases conform to those requirements (e.g. by using shutil.which to determine the absolute paths to your utilities instead of relying on PATH lookup), the speed up should eliminate (or more than eliminate) the performance regression you're seeing (per #35537, on macOS, which got the least benefit, it was still a 1.7x improvement; on other OSes, the multiplier hits 61x or 106x).

----------
nosy: +josh.r

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37790>
_______________________________________