[pypy-issue] [issue340] rewrite PyPy's tokenizer

[Carl Friedrich Bolz]

Carl Friedrich Bolz <cfbolz at gmx.de> added the comment:

I am not really sure that using rply's tokenizer would improve things (I should
really raise that with Alex, too). rply's tokenizer uses rlib.rsre to tokenize
stuff, which is overkill, given that the regular expressions needed for Python's
tokens are really regular, not Perl-style regular expressions.

----------
nosy: +cfbolz

________________________________________
PyPy bug tracker <tracker at bugs.pypy.org>
<https://bugs.pypy.org/issue340>
________________________________________