[Python-checkins] r81759 - in python/trunk: Lib/test/test_unicode.py Misc/NEWS

[ezio.melotti]

Author: ezio.melotti
Date: Sat Jun  5 21:21:32 2010
New Revision: 81759

Log:
Add a NEWS entry for r81758 and clarify a comment.

Modified:
   python/trunk/Lib/test/test_unicode.py
   python/trunk/Misc/NEWS

Modified: python/trunk/Lib/test/test_unicode.py
==============================================================================
--- python/trunk/Lib/test/test_unicode.py	(original)
+++ python/trunk/Lib/test/test_unicode.py	Sat Jun  5 21:21:32 2010
@@ -670,9 +670,9 @@
                               ('\xF4'+cb+'\xBF\xBF').decode, 'utf-8')
 
     def test_issue8271(self):
-        # Issue #8271: when a byte sequence is invalid, only the start byte
-        # and all the valid continuation bytes should be replaced by U+FFFD,
-        # not the number of bytes specified by the start byte.
+        # Issue #8271: during the decoding of an invalid UTF-8 byte sequence,
+        # only the start byte and the continuation byte(s) are now considered
+        # invalid, instead of the number of bytes specified by the start byte.
         # See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf (page 95,
         # table 3-8, Row 2) for more information about the algorithm used.
         FFFD = u'\ufffd'

Modified: python/trunk/Misc/NEWS
==============================================================================
--- python/trunk/Misc/NEWS	(original)
+++ python/trunk/Misc/NEWS	Sat Jun  5 21:21:32 2010
@@ -12,6 +12,14 @@
 Core and Builtins
 -----------------
 
+- Issue #8271: during the decoding of an invalid UTF-8 byte sequence, only the
+  start byte and the continuation byte(s) are now considered invalid, instead
+  of the number of bytes specified by the start byte.
+  E.g.: '\xf1\x80AB'.decode('utf-8', 'replace') now returns u'\ufffdAB' and
+  replaces with U+FFFD only the start byte ('\xf1') and the continuation byte
+  ('\x80') even if '\xf1' is the start byte of a 4-bytes sequence.
+  Previous versions returned a single u'\ufffd'.
+
 - Issue #8627: Remove bogus "Overriding __cmp__ blocks inheritance of
   __hash__ in 3.x" warning.  Also fix "XXX undetected error" that
   arises from the "Overriding __eq__ blocks inheritance ..." warning