[Python-Dev] A standard lexer?

[Daniel Berlin]

"M.-A. Lemburg" <mal@lemburg.com> writes:


> Daniel Berlin wrote:
> > 
> > Actually, if it helps, i'm working on porting re2c to python.
> 
> Pointers ?

http://www.tildeslash.org/re2c/index.html

> 
> > Because it was written properly, it's rather simple (in fact, i've only
> > needed to modify one file, add some if's to ee if we want python
> > generation, and output the python code instead of c code)
> > The lexers it generates for c/C++ are much faster than flex lexers,
> > because they are directly coded.
> > I haven't benchmarked it against SPARK yet, but i would imagine it would
> > blow it away, for the same reason it blows away flex.
> 
> Perhaps you should also look at the tagging engine in
> mxTextTools (you know where...) ?! It's very low-level, but
> it makes nice target for optimizing parser generators since
> it provides a Python interface to raw C speed.
> 

re2c is simply a scanner generator, not a parser.
--Dan