Contradictory error messages in Python 3.4 - standard library issue!

[Peter Otten]

Harrison Chudleigh wrote:

> While working on a program, I ran into an error with the usage of the
> module tokenize. The following message was displayed.
>  File
> 
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/tokenize.py",
> line 467, in tokenize
>     encoding, consumed = detect_encoding(readline)
>   File
> 
"/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/tokenize.py",
> line 409, in detect_encoding
>     if first.startswith(BOM_UTF8):
> TypeError: startswith first arg must be str or a tuple of str, not bytes
> Undaunted, I changed the error on line 409. The line then read:
> 
> if first.startswith(BOM_UTF8):

As Steven says -- don't change the standard library.

Your problem is likely that you are opening the file containing the code you 
want to tokenize in text mode. Compare:

$ cat 42.py
42
$ python3
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize

First with the file opened in text mode:

>>> with open("42.py", "r") as f:
...     for t in tokenize.tokenize(f.readline): print(t)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.4/tokenize.py", line 468, in tokenize
    encoding, consumed = detect_encoding(readline)
  File "/usr/lib/python3.4/tokenize.py", line 408, in detect_encoding
    if first.startswith(BOM_UTF8):
TypeError: startswith first arg must be str or a tuple of str, not bytes

Now let's switch to binary mode:

>>> with open("42.py", "rb") as f:
...     for t in tokenize.tokenize(f.readline): print(t)
... 
TokenInfo(type=56 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), 
line='')
TokenInfo(type=2 (NUMBER), string='42', start=(1, 0), end=(1, 2), 
line='42\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 2), end=(1, 3), 
line='42\n')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')