[Python-ideas] Transportable indent level markers. >>>===<<<

[Nick Coghlan]

On Thu, Dec 15, 2011 at 1:19 PM, Ron Adam <ron3200 at gmail.com> wrote:
>> You'd probably also want an explicit ";;" token to force a
>> token.NEWLINE into the token stream.
>
> That isn't needed.  Any of these in the middle of a line will add a new
> line and back up, so the next call to tok_get() will find it, and so on.

OK, take the way you're thinking (indent +1, indent 0, indent -1) and
instead think in terms of starting a suite, terminating a statement
and terminating a suite:

/// -> {:
;;; -> ;;
\\\ -> :}

Now do you see why I'm saying you're needlessly complicating things?
Suite delimiters and statement terminators (or separators) are the way
full whitespace insensitivity is normally handled when designing a
language syntax. There's no reason to get creative here when the
standard terminology and conventions would work just fine.

For example, it shouldn't be difficult to create a variant of the
tokenize module's tokeniser that adds the following rules:

{: -> emits OP(':'), NEWLINE, INDENT and increments the parenlevel
;; -> emits NEWLINE
:} -> emits NEWLINE, DEDENT and decrements the parenlevel

and a variant of the untokenizer() that looks ahead and *emits* those
character sequences when applicable.

That should be enough to let you use Python code in whitespace
insensitive environments without changing the semantics:

Encoding for transport: tokenize() normally, untokenize() with suite delimiters
Decoding from transport: tokenize() with suite delimiter support,
untokenize() normally

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia