[Compiler-sig] Initial release of parse tools for Python source

[John Ehresman]

[I tried to send this yesterday, but it appeared to bounce; apologies to
anyone who's seeing this twice] 

I have uploaded a tarball of the source for tools we are using to analyze
Python source code to ftp://archaeopteryx.com/pub/parsetools/.  This is an
preliminary release; it's working well for our purposes, but parts of it
are incomplete.  All source in this distribution is being released under
an MIT/X Consortium style license.  We will be actively developing the
tools and intend to release future versions under the same license.

The distribution contains a set of extension modules that provides the
means to tokenize Python source code, access the "pgen" parser built into
the Python runtime without going through the builtin tokenizer, and
transform parse trees created by the builtin parser to AST nodes that are
(almost) a subset of those used by the Python2C compiler (I don't use the
AssName, AssTuple, AssList nodes and intend to add a 'Delete' node). All
of the core functionality is written in C because these libraries are
being used to analyze Python source code as it is being entered or
modified. 

Unfortunately, the documentation leaves much to be desired ;).  There are
some incomplete notes in the README file and embedded in javadoc style
comments within the source itself.  There is also a logical_test.py
script which will tokenize any files given to it on the command line and
print out the tokens used on each logical line.  I intend to release
updates of this every few weeks and will set up a CVS repository at
sourceforge or on archaeopteryx.com if there is interest.

Please let me know of any questions or problems,

John

------------------------------------------------------------------------
Archaeopteryx Software, Inc.                        Wing IDE for Python 
www.archaeopteryx.com                               Take Flight!