[issue2523] binary buffered reading is quadratic

[Antoine Pitrou]

Antoine Pitrou <pitrou at free.fr> added the comment:

Hi Alexandre,

I first tried to use a (non-preallocated) bytearray object and, after
trying several optimization schemes, I found out that the best one
worked as well with an immutable bytes object :) I also found out that
the bytes <-> bytearray conversion costs can be noticeable in some
benchmarks.

The internal buffer is rarely reallocated because the current offset
inside it is remembered instead; also, when reading bytes from the
underlying unbuffered stream, a list of bytes objects is accumulated and
then joined at the end.

I think a preallocated bytearray would not make a lot of sense since we
can't readinto() an arbitrary position, so we still have a memory copy
from the bytes object returned by raw.read() to the bytearray buffer,
and then when returning the result to the user as a bytes object we have
another memory copy. In other words each read byte is copied twice more.

Of course, if this code was rewritten in C, different compromises would
be possible.

cheers

Antoine.

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2523>
__________________________________