[ python-Bugs-844561 ] codecs.open().readlines(sizehint) bug

[SourceForge.net]

Bugs item #844561, was opened at 2003-11-18 18:22
Message generated for change (Comment added) made by lemburg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=844561&group_id=5470

Category: Unicode
Group: Python 2.2
>Status: Closed
>Resolution: Fixed
Priority: 3
Submitted By: Jeff Epler (jepler)
Assigned to: M.-A. Lemburg (lemburg)
Summary: codecs.open().readlines(sizehint) bug

Initial Comment:
codecs.open().readlines(sizehint) can return truncated
lines.  The attached script, which uses
readlines(sizehint) to count the number of lines in a
file, demonstrates the problem.  Correct output would
be 1000 in both cases, but different values are
returned depending on sizehint because of the truncated
lines.

----------------------------------------------------------------------

>Comment By: M.-A. Lemburg (lemburg)
Date: 2004-02-26 16:26

Message:
Logged In: YES 
user_id=38388

Fixed in CVS.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2004-02-26 16:20

Message:
Logged In: YES 
user_id=38388

Ok, I'll fix codecs.py to ignore the sizehint argument then
(should not break any code; at worst it might cause problems
with MemoryOverflows).

To answer your question: whether a byte string is incomplete
or in error depends on the encoding and only the codec can
decide what to do. While the codecs do differentiate and the
error callback logic could be used to work out a correct
solution, this would require a lot of work.

----------------------------------------------------------------------

Comment By: Jeff Epler (jepler)
Date: 2004-02-26 15:50

Message:
Logged In: YES 
user_id=2772

Ignoring sizehint and reading the whole file is probably
better than truncating lines.  This change would also fix
another bug I realized exists in codecs readlines(sizehint)
currently: if it reads only part of a multi-byte character,
you get a decoding error...

A slightly more complicated approach would be to read
sizehint bytes and then while the result doesn't end in a
newline, read one more byte and decode again.  When sizehint
is large enough, doing byte-at-a-time reading of the last
half-line shouldn't be that bad for performance.  No, I
don't have a patch.

Is there a way to differentiate between "the byte string
ends with an incomplete multi-byte character" and "the byte
string contains an invalid sequence of bytes"?

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2004-02-26 10:51

Message:
Logged In: YES 
user_id=38388

Good catch. I must have overread the "whole lines" bit :-)

In that case, it's probably best to have .readlines() ignore
the sizehint argument altogether. An efficient implementation
is hard to do since the line breaking is not done at C level,
but after the data has been read.

----------------------------------------------------------------------

Comment By: Jeff Epler (jepler)
Date: 2004-02-26 02:14

Message:
Logged In: YES 
user_id=2772

To me, the phrase "*whole lines* totalling approximately
sizehint" means that no item from readlines(sizehint) will
be an incomplete line.  I don't understand why this
requirement isn't clearly indicated to you by the text you
included in your comments.

----------------------------------------------------------------------

Comment By: M.-A. Lemburg (lemburg)
Date: 2004-02-26 00:04

Message:
Logged In: YES 
user_id=38388

It's hard to say whether this is a bug or not. The sizehint
argument is not well documented and the way you use it
does not look a proper way to use it.

>From the docs:
""""
f the optional sizehint argument is present, instead of
reading up to EOF, whole lines totalling approximately
sizehint bytes (possibly after rounding up to an internal
buffer size) are read. 
""""

In your example the underlying open() implementation 
seems to round up the sizehint value to include the whole
line, while the codec.open() version will only read sizehint
bytes without any rounding (see the codecs.py 
implementation).


----------------------------------------------------------------------

Comment By: Jeff Epler (jepler)
Date: 2003-11-18 18:28

Message:
Logged In: YES 
user_id=2772

The script triggers the assertion error using at least
python 2.3.2 (locally compiled) and python 2.2.2 (redhat 9 RPM)

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=844561&group_id=5470