invalid-token syntax hook (was Re: Hack request: rational numbers)

[Alex Martelli]

"Tim Peters" <tim.one at home.com> wrote in message
news:mailman.980590234.29036.python-list at python.org...
    [snip]
> That said, I believe your only hope of getting new literal syntax is to
> embed it in a general proposal.  Some notation for rational literals is
just
> one of an unbounded number of similar desires.  Without a general scheme
for

What I was thinking, too.  So what about the "invalid-token syntax
error hook" idea -- whenever the compiler is about to raise a
SyntaxError because of an invalid token ('3r','@@x',etc), if
sys.invalidTokenHook (or something of that ilk) is set to some
callable, that callable is invoked, passed the invalid-token string,
and given a chance to either return None (confirming that, yes, it
IS a syntax error) or a tuple (possibly empty or singleton) of tokens
to be used instead.

Not sure how (if?) a module specifies what hook it wants to
be set during its own compilation -- perhaps it can't, in which
case a module whose source contains syntactically invalid
tokens can only be compiled after 'something else' has set
the (global?) hook 'appropriately'.

No, doesn't look like a necessarily great idea to me, either.

It allows a very modest amount of syntax extension (token
level only -- really nothing more than the sugariest kind)
and modules using it become not-understandable unless
what hook they use is known/documented.  At least it does
not allow _altering_ syntax in general (as only currently
invalid tokens would ever been passed to the hook), but,
it _would_ make it harder or impossibly to extent this
level of syntax in the future (a currently-invalid token could
not become valid in the future without risking a breakage
to some existing hook).

Consider it a strawman just to try and fathom if there IS
some halfway-good general idea for extensibility...


Alex