[issue1420] Unicode literals in tokenize.py and tests.

[Ron Adam]

New submission from Ron Adam:

Replaced Unicode literals in tokenize.py and it's tests files with byte
literals.

Added a compile step to the test to make sure the text file used in the
test are valid python code.  This will catch changes that need to be
done in to the text (gold file) for future python versions.

----------
components: Library (Lib)
files: tokenize_patch.diff
messages: 57366
nosy: ron_adam
severity: normal
status: open
title: Unicode literals in tokenize.py and tests.
versions: Python 3.0
Added file: http://bugs.python.org/file8732/tokenize_patch.diff

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue1420>
__________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tokenize_patch.diff
Type: text/x-patch
Size: 7723 bytes
Desc: not available
Url : http://mail.python.org/pipermail/python-bugs-list/attachments/20071111/5c07bd3f/attachment.bin