[New-bugs-announce] [issue2182] tokenize: does not allow CR for a newline

[Jared Grubb]

New submission from Jared Grubb:

tokenize recognizes '\n' and '\r\n' as newlines, but does not tolerate '\r':

>>> s = "print 1\nprint 2\r\nprint 3\r"
>>> open('temp.py','w').write(s)
>>> exec(open('temp.py','r'))
1
2
3
>>> tokenize.tokenize(open('temp.py','r').readline)
1,0-1,5:	NAME	'print'
1,6-1,7:	NUMBER	'1'
1,7-1,8:	NEWLINE	'\n'
2,0-2,5:	NAME	'print'
2,6-2,7:	NUMBER	'2'
2,7-2,9:	NEWLINE	'\r\n'
3,0-3,5:	NAME	'print'
3,6-3,7:	NUMBER	'3'
3,7-3,8:	ERRORTOKEN	'\r'
4,0-4,0:	ENDMARKER	''

----------
components: Extension Modules
messages: 62959
nosy: jaredgrubb
severity: minor
status: open
title: tokenize: does not allow CR for a newline
type: behavior
versions: Python 2.5

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2182>
__________________________________