[IronPython] Continuations and IronPythonConsole

[Curt Hagenlocher]

I use Python largely for a) immediate gratification (ie interactively)
and b) as an embedded runtime interpreter.  I've been stuck at
version 1.52 of CPython for some time now, so my expectations may
be a bit different than someone using a more modern version.

Naturally, in starting to play with IronPython, I've used solely
the IronPythonConsole program.  In doing so, I've discovered that
continuations largely seem broken.  (By continuations, I mean the
"..." that the interpreter prints to show that it wants more input.)

Here are four separate cases:

1. >>> a = '
causes an infinite loop instead of returning an error
2. >>> a = '''
causes an infinite loop instead of giving a continuation
3. >>> a = (
causes an error instead of giving a continuation
4. >>> def a():
works correctly

The first problem can be fixed in Tokenizer.next() at line 333
by adding " || ch == EOF" to the or clause.

The second problem can be fixed by adding a new section to that
same "if/else if/else if" construction.  The new section reads
	} else if (isTriple && ch == EOF) {
		setEnd();
		return TokenKind.EofToken;
	}
This change works because of special-case code in IronPython.cs
that catches exceptions, and, if the token that caused the problem
is an EofToken, tells the DoInteractive loop to perform a
continuation.

Unfortunately, this also exposes a problem in DoInteractive().
Its logic for deciding whether or not a continuation is necessary
is primitive and broken.  At line 178, the logic says 

	"If this is the first line we've read, then show a
	continuation unless there was a (non-EOF) exception or
	the line parsed correctly in its entirety.  If we've
	already read more than one line, then show continuations
	until the user enters a blank line."

What should probably change is that the Stmt class should support
a virtual method to indicate whether or not a correct parse
indicates the end of the statement, or whether the user should be
prompted with continuations until a blank line is entered.  This
will distinguish statements of the first type from statements
like "class" and "def", without having to explicity compare the
statement type.

It's been ages since I read the CPython source, so I wasn't able
to find how this is handled there.  I do seem to remember that, at
least in 1.x versions, the parser had three different mode flags
depending on whether or not it was being invoked interactively,
on a single line or on a file.
	
I'm not sure how to solve the third problem.  The exception is
thrown from Parser.finishTupleValue() when it doesn't get the
expected right parenthesis.  But I was expecting the nextToken()
function to return a NewlineToken, and it didn't.  I hope to look
into this further in the next few days.

Jim, do you expect to keep development private for now?

--
Curt Hagenlocher
curt at hagenlocher.org