PEP 269

[Samuele Pedroni]

Hi

personally I have the following concerns about PEP 269:
- if it's purpose is to offer a framework for small languages
support, there are already modules around that support that
(SPARK, PLY ...), the only advantage of PEP 269 being speed
wrt to the pure python solutions, because of the use of the internal
CPython parser, OTOH the other solutions are more flexible...
- or if's purpose is to help experimenting with the grammar
unless support for adding keywords is added is a quite unfinished tool.

Further the PEP propose to use the actual AST format of parser module
as output format. To be honest that format is quite awful, especially
for general purpose use.

It should be considered that Jython does not contain a parser
similar to CPython one. Because of this jython does not offer parser 
module support. So implementing the PEP for Jython would require 
writing a Java or pure python equivalent of the CPython parser.

My plans for resolving the lack of parser module support were to 
to implement an higher compatibility layer based on the AST format
of tools/compiler, a more nicer format.

PEP 269 adds issues to this open problem, which I would like to
see addressed by future revisions and by further discussions.

I can live with PEP 269 implemented only for CPython, for a lack
of resources on Jython side, if is to be used for rare experimenting
with the grammar.

But it seems, as it is, a rather half-cooked solution to offer a module
for mini language support in the standard library.

regards, Samuele Pedroni.

> From: Jonathan Riehl <jriehl at spaceship.com>
> To: Martin von Loewis <loewis at informatik.hu-berlin.de>
> cc: <python-list at python.org>, <types-sig at python.org>
> MIME-Version: 1.0
> Subject: [Types-sig] Re: PEP 269
> X-BeenThere: types-sig at python.org
> X-Mailman-Version: 2.0.6 (101270)
> List-Help: <mailto:types-sig-request at python.org?subject=help>
> List-Post: <mailto:types-sig at python.org>
> List-Subscribe: <http://mail.python.org/mailman/listinfo/types-sig>, 
<mailto:types-sig-request at python.org?subject=subscribe>
> List-Id: Special Interest Group on the Python type system 
<types-sig.python.org>
> List-Unsubscribe: <http://mail.python.org/mailman/listinfo/types-sig>, 
<mailto:types-sig-request at python.org?subject=unsubscribe>
> List-Archive: <http://mail.python.org/pipermail/types-sig/>
> Date: Thu, 13 Sep 2001 14:49:32 -0500 (CDT)
> 
> Howdy all,
> 	I'm afraid Martin's attention to the PEP list has outted me
> before I was able to post about this myself.  Anyway, for those
> interested, I wrote a PEP for the exposure of pgen to the Python
> interpreter.  You may view it at:
> 
> http://python.sourceforge.net/peps/pep-0269.html
> 
> 	I am looking for comments on this PEP, and below, I address some
> interesting issues raised by Martin.  Furthermore, I already have a
> parially functioning reference implementation, and should be pestered to
> make it available shortly.
> 
> Thanks,
> -Jon
> 
> On Tue, 11 Sep 2001, Martin von Loewis wrote:
> 
> > Hi Jonathan,
> >
> > With interest I noticed your proposal to include Pgen into the
> > standard library. I'm not sure about the scope of the proposed change:
> > Do you view pgen as a candidate for a general-purpose parser toolkit,
> > or do you "just" contemplate using that for variations of the Python
> > grammar?
> 
> I am thinking of going for the low hanging fruit first (a Python centric
> pgen module), and then adding more functionality for later releases of
> Python (see below.)
> 
> > If the former, I think there should be a strategy already how
> > to expose pgen to the application; the proposed API seems
> > inappropriate. In particular:
> >
> > - how would I integrate an alternative tokenizer?
> > - how could I integrate semantic actions into the parse process,
> >   instead of creating the canonical AST?
> 
> The current change proposed is somewhat restrained by the Python 2.2
> release schedule, and will initially only address building parsers that
> use the Python tokenizer.  If the module misses 2.2 release, I'd like to
> make it more functional and provide the ability to override the Python
> tokenizer.  I may also add methods to export all the data found in the DFA
> structure.
> 
> I am unsure what the purpose of integration of semantics into the parse
> process buys us besides lower memory overhead.  In C/C++ such coupling is
> needed because of the TYPEDEF/IDENTIFIER tokenization problem, but I
> don't see Python and future Python-like, LL(1), languages needing such
> hacks.  Finally, I am prone to enforce the separation of the backend
> actions from the AST.  This allows the AST to be used for a variety of
> purposes, rather than those intended by the initial parser developer.
> 
> > Of course, these questions are less interesting if the scope is to
> > parse Python: in that case, Python tokenization is fine, and everybody
> > is used to getting the Python AST.
> 
> An interesting note to make about this is that the since the nonterminal
> integer values are generated by pgen, pgen AST's are not currently
> compatible with the parser module AST's.  Perhaps such unification may be
> slated for future work (I know Fred left room in the parser AST datatype
> for identification of the grammar that generated the AST using an integer
> value, but using this would be questionable in a "rapid parser
> development" environment.)
> 
> > On the specific API, I think you should drop the File functions
> > (parseGrammarFile, parseFile). Perhaps you can also drop the String
> > functions, and provide only functions that expect file-like objects.
> 
> I am open to further discussion on this, but I would note that filename
> information is used (and useful) when reporting syntax errors.  I think
> that the "streaming" approach to parsing is another hold over from days
> where memory constraints ruled (much like binding semantics to the parser
> itself.)
> 
> > On the naming of the API functions: I propose to use an underscore
> > style instead of the mixedCaps style, or perhaps to leave out any
> > structure (parsegrammar, buildparser, parse, symbol2string,
> > string2symbolmap). That would be more in line with the parser module.
> 
> I would like to hear more about this from the Pythonati.  I am currently
> following the naming conventions I use at work, which of course is most
> natural for me at home. :)
> 
> >
> > Regards,
> > Martin
> >
> >
> 
> 
> _______________________________________________
> Types-SIG mailing list
> Types-SIG at python.org
> http://mail.python.org/mailman/listinfo/types-sig