[issue18961] Non-UTF8 encoding line

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Here is a file which is accepted by Python interpreter but is rejected by the tokenize module.

$ ./python nonutf8_coding_line.py 
$ ./python -m tokenize nonutf8_coding_line.py 
nonutf8_coding_line.py: error: invalid or missing encoding declaration for 'nonutf8_coding_line.py'

Python itself checks that a line is UTF-8 encoded only if not found a magic comment. The tokenize module checks it before searching (issue14629).

----------
components: Interpreter Core, Library (Lib)
files: nonutf8_coding_line.py
messages: 197169
nosy: Arfrever, benjamin.peterson, brett.cannon, eric.snow, loewis, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Non-UTF8 encoding line
type: behavior
versions: Python 3.3, Python 3.4
Added file: http://bugs.python.org/file31648/nonutf8_coding_line.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue18961>
_______________________________________