[Python-Dev] AST mining (was Re: Direction of PyChecker)

[Tim Peters]

[Jeremy Hylton]
> ...
> I wouldn't wait for people to argue about the right AST either.  Use
> the one Greg and Bill came up with for p2c.  It's in Tools/compiler in
> the Python distribution.  It's much simpler than the parse tree
> produced by the parser module.  And, as far as I know, no one is
> advocating a different AST.
>
> It doesn't have a query language, but neither does tokenize.py <wink>.

The likely problem I see is documentation:  exactly what is this AST, and
how does one use the compiler package?  compiler's demo.py is exactly
on-target for what people want to do here, and is very simple code, but how
would someone coming into this cold come up with this code on their own?

The import tricks don't help.  I just tried to reverse-engineer demo.py and
got stuck on the

    ast = compiler.parse(buf)

line.  Granted I've running a fever, but I'm not sure I would have thought
to study __init__.py even if not.

The relative feebleness of tokenize.py is an advantage in this respect:  it
doesn't do much, has some docs, and is all in one function, so it's
straightforward to guess what it's doing.