Contradictory error messages in Python 3.4 - standard library issue!

[Steven D'Aprano]

Hi Harrison, and welcome!


On Fri, 17 Jun 2016 08:25 am, Harrison Chudleigh wrote:

> While working on a program, I ran into an error with the usage of the
> module tokenize.

So you have an error with the use of tokenize. Fair enough.

But why do you imagine that the errors lies in the module itself, rather
that your use of it? There are probably tens of thousands, maybe hundreds
of thousands, of people using that module (directly or indirectly). The
chances that you have identified a bug in the module that nobody else has
seen before is pretty slim.

And that certainly shouldn't be your first assumption.


> The following message was displayed. 
>  File
> "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/tokenize.py",
> line 467, in tokenize
>     encoding, consumed = detect_encoding(readline)
>   File
> "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/tokenize.py",
> line 409, in detect_encoding
>     if first.startswith(BOM_UTF8):
> TypeError: startswith first arg must be str or a tuple of str, not bytes
> Undaunted, I changed the error on line 409. The line then read:
> 
> if first.startswith(BOM_UTF8):

I'm not seeing any actual difference between the before and after:

    if first.startswith(BOM_UTF8):
    if first.startswith(BOM_UTF8):

but in another email, you state that you changed it to:

    if first.startswith('BOM_UTF8'):


Changing the source code of the standard library is almost never what you
want to be do. I've been programming in Python for 20+ years, and the
number of times I've edited the source code of the standard library to fix
a program is exactly zero.

But making random changes to the source code of the standard library without
understanding what it is doing or why is NEVER what you want to do.

All you have accomplished by editing the code is changing the situation
from "one in a 100,000 chance of a bug in the standard library"
to "certainly a bug in a non-standard module shared by absolutely nobody
else in the world".

Before we can help you, you MUST revert the module back to the way it was.
You have introduced a bug to the module, but worse, you've now made it so
that it is different from everyone else's tokenize module. Any errors you
have received after making that change, or changes, are irrelevant.

Then you need to show us how you are calling tokenize. Show us the ENTIRE
traceback, starting with the line beginning "Traceback", not just the last
line with the error message. Once we've seen that, we may be able to help
you, or we may have more questions, but that's the bare minimum we need.



-- 
Steven