ANN: pytoken 1.0 - native 86 machine code scanner generator

[ramb at sonic.net]

I am pleased to announce the 1.0 version of pytoken.

It is available here:

   http://code.google.com/p/pytoken/downloads/list

What is pytoken

Pytoken is a scanner generator. Given an input specification - a
bunch of regular expressions - pytoken will generate x86 machine
code that recognizes those regular expressions. Pytoken will be
most useful for programmers that want to parse complex text files.

Pytoken has separate objects for scanners and buffers.
Here is a simple example:

  import pytoken

  lexer_obj = pytoken.lexer()
  lexer_obj.add_pattern("a", 1)
  lexer_obj.add_pattern("b", 2)
  lexer_obj.compile_to_machine_code()

  buf = pytoken.lexer_state()
  buf.set_input("ab")

  tok = lexer_obj.get_token(buf)
  assert tok == 1
  tok = lexer_obj.get_token(buf)
  assert tok == 2

Pytoken has been written in a portable fashion - it is designed to support
multiple CPU types, even though only the x86 (32 bit) is supported now.

-Ram