[issue719888] tokenize module w/ coding cookie

[Mark Dickinson]

Mark Dickinson <dickinsm at gmail.com> added the comment:

With the patch, 

./python.exe Lib/test/regrtest.py test_tokenize

fails for me with the following output:

Macintosh-2:py3k dickinsm$ ./python.exe Lib/test/regrtest.py test_tokenize
test_tokenize
test test_tokenize produced unexpected output:
**********************************************************************
*** lines 2-5 of actual output doesn't appear in expected output after line 1:
+ testing: /Users/dickinsm/python_source/py3k/Lib/test/tokenize_tests-latin1-coding-cookie-and-utf8-bom-sig.txt
+ testing: /Users/dickinsm/python_source/py3k/Lib/test/tokenize_tests-no-coding-cookie-and-utf8-bom-sig-only.txt
+ testing: /Users/dickinsm/python_source/py3k/Lib/test/tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt
+ testing: /Users/dickinsm/python_source/py3k/Lib/test/tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt
**********************************************************************
1 test failed:
    test_tokenize
[65880 refs]

I get something similar on Linux.

____________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue719888>
____________________________________