Python AST preserving whitespace and comments

[Terry Reedy]

Michal Kwiatkowski wrote:
> Hi,
> 
> I'm working on Pythoscope[1], a unit test generator for Python and
> stumbled into the following problem. I need a way to analyze and
> modify Python AST tree, but without loosing source code formatting and
> comments. Standard library ast module discards those, so I started
> looking for other solutions. I found a library used by the 2to3
> script[2], which does different kinds of source code refactorings and
> preserves the formatting of the original file. Sadly AST generated by
> this library is not compatible with the standard one, so I can't use a
> nice interface of compiler.visitor[3]. It isn't also as well
> documented. I kind of liked the clean and descriptive list of standard
> AST nodes[4].
> 
> So here are my questions to the list. Is there any library built on
> top of lib2to3 which makes traversing it easier? The pattern matcher
> is nice, but not always feasible for the higher level view of the
> source code.
> 
> Are there any tutorials/documentation about usage of lib2to3 which I'm
> not aware of? I basically read through HACKING, README and a lot of
> source code, but would appreciate some more examples and insights.
> 
> Is the lib2to3 library meant to be used outside of the 2to3 script or
> rather not? Are there any projects that will incorporate its features
> (enriched AST in particular) in a more "official" library-like
> package? Will lib2to3 be put on PyPI anytime?
> 
> [1] http://pythoscope.org/
> [2] http://svn.python.org/view/sandbox/trunk/2to3/
> [3] http://docs.python.org/lib/module-compiler.visitor.html
> [4] http://docs.python.org/lib/module-compiler.ast.html

You found the appropriate library, but I am unaware of anything that you 
missed.  I think others have had the idea that 2to3 might grow into 
something more, but the core developers are currently focues on getting 
2.6/3.0 out by the end of September.