[issue3594] PyTokenizer_FindEncoding() never succeeds

[Brett Cannon]

Brett Cannon <brett at python.org> added the comment:

Turns out that the NULL return value can signal an error that manifests
itself as SyntaxError("encoding problem: with BOM") thanks to the lack
of tok->filename being set in Parser/tokenizer.c:fp_setreadl() which is
called by check_coding_spec() and assumes that since tok->encoding was
never set (because fp_setreadl() returned an error value) that it had
something to do with the BOM.

The only reason this was found is because my bootstrapping of importlib
into Py3K, at some point, triggers a PyErr_Occurred() which finally
notices the error.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3594>
_______________________________________