[issue21401] python2 -3 does not warn about str/unicode to bytes conversions and comparisons

[STINNER Victor]

STINNER Victor added the comment:

Attached py2_warn_cmp_bytes_text.patch adds BytesWarning for bytes == unicode, bytes != unicode, unicode == bytes, unicode != bytes and similar comparisons with bytearray. The new warnings are added when -b or -bb command line options are used.

As a consequence, a lot of tests are failing with the patch applied and -bb command line option.

Some tests are obviously wrong (unicode expected, but the tests use bytes), but it's much more complex to fix tricky modules like urllib, os.path, json and re (sre_parse) to handle "correctly" bytes and unicode. In some other cases, the warning should be made quiet because a same test compares bytes and then text.

It also means that programs currently working fine with Python 2.7.6 with "-3 -b" options will start to see new BytesWarning warnings. Is it acceptable?

What is the purpose of -b in Python 2? Help developers to notice earlier future Unicode issues in their program, or help them to port their code to Python 3?

Maybe the new warnings should only by emited if -3 and -b options are used at the same time?

Tell me if you would like to see my work-in-progress patch to fix the whole test suite. Just the stats:

$ hg diff --stat
 Lib/_pyio.py                         |  16 ++++++++--------
 Lib/ctypes/test/test_arrays.py       |  12 ++++++------
 Lib/ctypes/test/test_buffers.py      |  20 ++++++++++----------
 Lib/ctypes/test/test_cast.py         |   2 +-
 Lib/ctypes/test/test_memfunctions.py |  10 +++++-----
 Lib/ctypes/test/test_prototypes.py   |  10 +++++-----
 Lib/ctypes/test/test_structures.py   |   2 +-
 Lib/fractions.py                     |   1 +
 Lib/sqlite3/dump.py                  |   8 ++++----
 Lib/sqlite3/test/dump.py             |   4 ++--
 Lib/sre_parse.py                     |   3 ++-
 Lib/test/string_tests.py             |   6 +++---
 Lib/test/test_builtin.py             |  14 ++++++++++----
 Lib/test/test_bytes.py               |  27 +++++++++++++++++++++++++--
 Lib/test/test_format.py              |  10 +++++++---
 Lib/test/test_future4.py             |   2 +-
 Lib/test/test_pyexpat.py             |  28 ++++++++++++++--------------
 Lib/test/test_sax.py                 |  20 ++++++++++----------
 Lib/test/test_tempfile.py            |   4 ++--
 Objects/bytearrayobject.c            |   2 +-
 Objects/stringobject.c               |   9 +++++++++
 Objects/unicodeobject.c              |   8 ++++++++
 22 files changed, 135 insertions(+), 83 deletions(-)

A funny one:

diff -r 670fb496f1f6 Lib/test/test_future4.py
--- a/Lib/test/test_future4.py  Sun May 11 23:37:26 2014 -0400
+++ b/Lib/test/test_future4.py  Tue May 13 03:28:12 2014 +0200
@@ -43,5 +43,5 @@ class TestFuture(unittest.TestCase):
 def test_main():
     test_support.run_unittest(TestFuture)
 
-if __name__ == "__main__":
+if __name__ == b"__main__":
     test_main()

----------
keywords: +patch
Added file: http://bugs.python.org/file35235/py2_warn_cmp_bytes_text.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21401>
_______________________________________