[issue3594] PyTokenizer_FindEncoding() never succeeds

[Brett Cannon]

New submission from Brett Cannon <brett at python.org>:

Turns out that PyTokenizer_FindEncoding() never properly succeeds
because the tok_state used by it does not have tok->filename set, which
is an error condition in the tokenizer. This error has been masked by
the one place the function is used, imp.find_module() because a NULL
return is never checked for an error, but instead just assumes the
default source encoding suffices.

----------
components: Extension Modules
messages: 71397
nosy: brett.cannon, christian.heimes
priority: critical
severity: normal
status: open
title: PyTokenizer_FindEncoding() never succeeds
versions: Python 3.0

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3594>
_______________________________________