[New-bugs-announce] [issue2134] function generate_tokens at tokenize.py yields wrong token for colon

[Guilherme Polo]

New submission from Guilherme Polo:

function generate_tokes at tokenize.py yields token OP (51) for colon,
while it should be token COLON (11). It probably affects other python
versions as well.

I'm attaching a minor sample that demonstrates this, running it returns
the following output:

1 'if' (1, 0) (1, 2) if a == 2:
1 'a' (1, 3) (1, 4) if a == 2:
51 '==' (1, 5) (1, 7) if a == 2:
2 '2' (1, 8) (1, 9) if a == 2:
51 ':' (1, 9) (1, 10) if a == 2:
1 'print' (2, 0) (2, 5) print 'hey'
3 "'hey'" (2, 6) (2, 11) print 'hey'
0 '' (3, 0) (3, 0)

I didn't check if there are problems with other tokens, I noticed this
with colon because I was trying to make some improvements on tabnanny.

----------
components: Library (Lib)
files: tokenize_sample.py
messages: 62509
nosy: gpolo
severity: normal
status: open
title: function generate_tokens at tokenize.py yields wrong token for colon
type: behavior
versions: Python 2.4, Python 2.5, Python 2.6
Added file: http://bugs.python.org/file9452/tokenize_sample.py

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2134>
__________________________________