[ python-Feature Requests-1634034 ] Show "expected" token on syntax error

[SourceForge.net]

Feature Requests item #1634034, was opened at 2007-01-12 13:03
Message generated for change (Comment added) made by oliver_gramberg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1634034&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Parser/Compiler
Group: Python 2.6
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Oliver Gramberg (oliver_gramberg)
Assigned to: Nobody/Anonymous (nobody)
Summary: Show "expected" token on syntax error

Initial Comment:

I suggest that the parser, when reporting a syntax
error, should make use of its knowlegde of which token
type is expected at the position where the error
occurred. This results in more helpful error messages:

-----------------------------------------------------
>>> for a in (8,9)
  File "<stdin>", line 1
    for a in (8,9)
                 ^
SyntaxError: invalid syntax - COLON expected
-----------------------------------------------------
>>> for a in (8,9: print a,
  File "<stdin>", line 1
    for a in (8,9: print a,
                 ^
SyntaxError: invalid syntax: RPAR expected
-----------------------------------------------------

I tried the following patch (for pythonrun.c). It works
well in the shell both interactively and in scripts,
as well as in IDLE. But it's not complete:
- It doesn't always print useful messages (only for
fixed-size terminal token types, I assume.)
- There sure are cases where more than one token type
is allowed in a position. I believe I have seen that
this information is available too somewhere in the
parser, but it is not forwarded to the err_input
routine.

It's even nicer to show "')'" instead of "RPAR"...

-----------------------------------------------------
/* Set the error appropriate to the given input error code (see errcode.h) */

static void
err_input(perrdetail *err)
{
	PyObject *v, *w, *errtype;
	PyObject* u = NULL;
	char *msg = NULL;
	errtype = PyExc_SyntaxError;
	switch (err->error) {
	case E_SYNTAX:
		errtype = PyExc_IndentationError;
		if (err->expected == INDENT)
			msg = "expected an indented block";
		else if (err->token == INDENT)
			msg = "unexpected indent";
		else if (err->token == DEDENT)
			msg = "unexpected unindent";
		else {
			char buf[50];
			errtype = PyExc_SyntaxError;
			if(err->expected != -1) {
				snprintf(buf, 48, "invalid syntax - %.16s expected\0",
					_PyParser_TokenNames[err->expected]);
				msg = buf;
			} else {
				msg = "invalid syntax";
			}
		}
		break;
		...
-----------------------------------------------------

I am willing to help work on this.

Regards
-Oliver


----------------------------------------------------------------------

>Comment By: Oliver Gramberg (oliver_gramberg)
Date: 2007-03-30 11:44

Message:
Logged In: YES 
user_id=206204
Originator: YES

Pfa a diff for my patch.

Regards
-Oliver

File Added: pythonrun.patch

----------------------------------------------------------------------

Comment By: Sean Gillespie (sean_gillespie)
Date: 2007-03-28 23:37

Message:
Logged In: YES 
user_id=1744567
Originator: NO

Your patch seems to work.

I agree that showing the token (as in ")") would indeed be much more
useful, and it would be pretty easy to implement.

However, I think that you should generate a diff for your patch.  Its
incredibly hard to read over SF.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1634034&group_id=5470