[issue19929] subprocess: increase read buffer size

[Charles-François Natali]

New submission from Charles-François Natali:

This is a spinoff of issue #19506: currently, subprocess.communicate() uses a 4K buffer when reading data from pipes.
This was probably optimal a couple years ago, but nowadays most operating systems have larger pipes (e.g. Linux has 64K), so we might be able to gain some performance by increasing this buffer size.

For example, here's a benchmark reading from a subprocess spawning "dd if=/dev/zero bs=1M count=100":

# before, 4K buffer
$ ./python ~/test_sub_read.py 
2.72450800300021

# after, 64K buffer
$ ./python ~/test_sub_read.py 
1.2509000449999803

The difference is impressive.

I'm attaching the benchmark script so that others can experiment a bit (on multi-core machines and also different OSes).

----------
components: Library (Lib)
files: test_sub_read.py
messages: 205534
nosy: gregory.p.smith, haypo, neologix, pitrou, sbt, serhiy.storchaka
priority: normal
severity: normal
stage: needs patch
status: open
title: subprocess: increase read buffer size
type: performance
versions: Python 3.4
Added file: http://bugs.python.org/file33041/test_sub_read.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19929>
_______________________________________