[issue3353] make built-in tokenizer available via Python C API

[Andy]

Andy <flossyfriend at hotmail.com> added the comment:

Did that and it builds fine.

So my test procedure was:

 - checkout clean source
 - apply patch as per guidelines
 - remove the file Psrser/tokenizer.h (*)
 - ./configure
 - make
 - ./python setup.py install

Build platform: Ubuntu , gcc 4.2.3

All works fine.

thanks for the extra test files.

* - one question though. I removed the file using 'svn remove' but the
diff makes it an empty file not removed why is that? (and is it correct?)

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3353>
_______________________________________