[Python-ideas] Hooking between lexer and parser

[Neil Girdhar]

Maybe if every production has a link to its parent, then the spaces after a
newline followed by statement reduce to indentation followed by statement,
which reduces to indent or dedent  or nothing followed by statement based
on the parent's indentation level?  In other words the parent (a file_input
e.g.) has active control of the grammar of its children?

On Sat, Jun 6, 2015 at 9:36 AM, s.krah <stefan at bytereef.org> wrote:

>
>
> *Neil Girdhar <mistersheik at gmail.com <mistersheik at gmail.com>>* wrote:
> > Along with the grammar, you also give it code that it can execute as it
> matches each symbol in a rule.  In Python for example, as it matches each
> argument passed to a function, it would keep track of the count of *args,
> **kwargs, and  keyword arguments, and regular arguments, and then raise a
> syntax error if it encounters anything out of order.  Right now that check
> is done in validate.c, which is really annoying.
>
> Agreed.  For 3.4 it was possible to encode these particular semantics into
> the grammar
> itself, but it would no longer be LL(1).
>
> If I understood correctly, you wanted to handle lexing and parsing
> together.  How
> would the INDENT/DEDENT tokens be generated?
>
> For my private ast generator, I did the opposite: I wanted to formalize
> the token
> preprocessing step, so I have:
>
>     lexer -> parser1 (generates INDENT/DEDENT) -> parser2 (generates the
> ast directly)
>
>
> It isn't slower than what is in Python right now and you can hook into the
> token stream
> at any place.
>
>
>
> Stefan Krah
>
>
>  --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/WTFHSUbfU20/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/WTFHSUbfU20/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150606/159e41c6/attachment.html>