[ python-Bugs-1772788 ] chr(128) in u'only ascii' -> TypeError with misleading msg

[SourceForge.net]

Bugs item #1772788, was opened at 2007-08-13 00:54
Message generated for change (Comment added) made by effbot
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1772788&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Pekka Laukkanen (laukpe)
Assigned to: Nobody/Anonymous (nobody)
Summary: chr(128) in u'only ascii' -> TypeError with misleading msg

Initial Comment:
A test using in format "chr(x) in <string>" raises a TypeError if "x" is in range 128-255 (i.e. non-ascii) and string is unicode. This happens even if the unicode string contains only ascii data as the example below demonstrates.

Python 2.5.1 (r251:54863, May  2 2007, 16:56:35) 
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> chr(127) in 'hello'
False
>>> chr(128) in 'hello'
False
>>> chr(127) in u'hi'
False
>>> chr(128) in u'hi'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'in <string>' requires string as left operand

This can cause pretty nasty and hard-to-debug bugs in code using "in <string>" format if e.g. user provided data is converted to unicode internally. Most other string operations work nicely between normal and unicode strings and I'd say simply returning False in this situation would be ok too. Issuing a warning similarly as below might be a good idea also.  

>>> chr(128) == u''
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

Finally, the error message is somewhat misleading since the left operand is definitely a string.

>>> type(chr(128))
<type 'str'>

A real life example of code where this problem exist is telnetlib. I'll submit a separate bug about it as that problem can obviously be fixed in the library itself.


----------------------------------------------------------------------

>Comment By: Fredrik Lundh (effbot)
Date: 2007-08-21 10:48

Message:
Logged In: YES 
user_id=38376
Originator: NO

"Most other string operations work nicely between normal and unicode
strings"

Nope.  You *always* get errors if you mix Unicode with NON-ASCII data
(unless you've messed up the system's default encoding, which is a bad
thing to do if you care about portability).  Some examples:

>>> chr(128) + u"foo"
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
ordinal not in range(128)
>>> u"foo".find(chr(128))
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
ordinal not in range(128)

etc.  If there's a bug here, it's that you get a TypeError instead of a
ValueError subclass.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1772788&group_id=5470