[Python-bugs-list] [ python-Bugs-410155 ] tokenize.NL undocumented
noreply@sourceforge.net
noreply@sourceforge.net
Wed, 21 Mar 2001 16:24:16 -0800
Bugs item #410155, was updated on 2001-03-20 13:28
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=410155&group_id=5470
Category: Documentation
Group: None
>Status: Closed
Priority: 5
Submitted By: Jürgen Hermann (jhermann)
Assigned to: Guido van Rossum (gvanrossum)
Summary: tokenize.NL undocumented
Initial Comment:
In section 17.5 of the Python 2.0 Library Reference,
the constant "NL" is not documented.
----------------------------------------------------------------------
>Comment By: Guido van Rossum (gvanrossum)
Date: 2001-03-21 16:24
Message:
Logged In: YES
user_id=6380
The time machne strikes again. Skip Montanaro added docs
for NL in February. For proof, see:
http://python.sourceforge.net/devel-docs/lib/module-tokenize.html
----------------------------------------------------------------------
Comment By: Jürgen Hermann (jhermann)
Date: 2001-03-20 19:21
Message:
Logged In: YES
user_id=39128
tokenize.NL seems to be an empty line, as opposed to
token.NEWLINE which is a logical line end. Insofar NL is
comparable to COMMENT, since it has no semantic content and
is ignorable by a parser.
1 a = \
2 2
3
4 b=4
emits these tokens:
1,0-1,1: NAME 'a'
1,2-1,3: OP '='
2,0-2,1: NUMBER '2'
2,1-2,2: NEWLINE '\012'
3,0-3,1: NL '\012'
4,0-4,1: NAME 'b'
4,1-4,2: OP '='
4,2-4,3: NUMBER '4'
4,3-4,4: NEWLINE '\012'
5,0-5,0: ENDMARKER ''
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-03-20 18:22
Message:
Logged In: YES
user_id=31435
Yes, it should be, but Ping doesn't know about NL: Guido
added that one. Python's real tokenizer has no such token
type. It was added because-- unlike Python's real
tokenizer --tokenizer.py passes *everything* on to the
client, including blank lines and comments, so passes on
newlines that aren't NEWLINEs too. Reassigned to Guido in
case he remembers more about the details ...
----------------------------------------------------------------------
Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2001-03-20 14:58
Message:
Logged In: YES
user_id=3066
Ping, should NL be documented?
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=105470&aid=410155&group_id=5470