PEP 269

[Martin von Loewis]

Hi Jonathan,

With interest I noticed your proposal to include Pgen into the
standard library. I'm not sure about the scope of the proposed change:
Do you view pgen as a candidate for a general-purpose parser toolkit,
or do you "just" contemplate using that for variations of the Python
grammar? If the former, I think there should be a strategy already how
to expose pgen to the application; the proposed API seems
inappropriate. In particular:

- how would I integrate an alternative tokenizer?
- how could I integrate semantic actions into the parse process,
  instead of creating the canonical AST?

Of course, these questions are less interesting if the scope is to
parse Python: in that case, Python tokenization is fine, and everybody
is used to getting the Python AST.

On the specific API, I think you should drop the File functions
(parseGrammarFile, parseFile). Perhaps you can also drop the String
functions, and provide only functions that expect file-like objects.

On the naming of the API functions: I propose to use an underscore
style instead of the mixedCaps style, or perhaps to leave out any
structure (parsegrammar, buildparser, parse, symbol2string,
string2symbolmap). That would be more in line with the parser module.

Regards,
Martin