Tokenizer inconsistency wrt to new lines in comments

[Fredrik Lundh]

George Sakkis wrote:

>> If it was a bug it has to violate a functional requirement. I can't
>> see which one.
> 
> Perhaps it's not a functional requirement but it came up as a real
> problem on a source colorizer I use. I count on newlines generating
> token.NEWLINE or tokenize.NL tokens in order to produce <br> tags. It
> took me some time and head scratching to find out why some comments
> were joined together with the following line. Now I have to check
> whether a comment ends in new line and if it does output an extra <br>
> tag.. it works but it's a kludge.

well, the real kludge here is of course that you're writing your own 
colorizer, when you can just go and grab Pygments:

    http://pygments.org/

or, if you prefer something tiny and self-contained, something like the 
colorizer module in this directory:

    http://svn.effbot.org/public/stuff/sandbox/pythondoc/

(the element_colorizer module in the same directory gives you XHTML in 
an ElementTree instead of raw HTML, if you want to postprocess things)

</F>