Tokenizer inconsistency wrt to new lines in comments

[George Sakkis]

The tokenize.generate_tokens function seems to handle in a context-
sensitive manner the new line after a comment:

>>> from StringIO import StringIO
>>> from tokenize import generate_tokens
>>>
>>> text = '''
... # hello world
... x = (
... # hello world
... )
... '''
>>>
>>> for t in generate_tokens(StringIO(text).readline):
...     print repr(t[1])
...
'\n'
'# hello world\n'
'x'
'='
'('
'\n'
'# hello world'
'\n'
')'
'\n'
''

Is there a reason that the newline is included in the first comment
but not in the second, or is it a bug ?

George