[Python-checkins] CVS: python/dist/src/Parser tokenizer.c,2.37,2.38

[Guido van Rossum]

Update of /projects/cvsroot/python/dist/src/Parser
In directory eric:/home/guido/hp/mal/py-patched/Parser

Modified Files:
	tokenizer.c 
Log Message:
Marc-Andre Lemburg: add new string token types u"..." and ur"..."
(Unicode and raw Unicode).


Index: tokenizer.c
===================================================================
RCS file: /projects/cvsroot/python/dist/src/Parser/tokenizer.c,v
retrieving revision 2.37
retrieving revision 2.38
diff -C2 -r2.37 -r2.38
*** tokenizer.c	1998/12/04 18:51:01	2.37
--- tokenizer.c	2000/03/10 22:56:54	2.38
***************
*** 592,595 ****
--- 592,596 ----
  	/* Identifier (most frequent token!) */
  	if (isalpha(c) || c == '_') {
+ 		/* Process r"", u"" and ur"" */
  		switch (c) {
  		case 'r':
***************
*** 598,601 ****
--- 599,611 ----
  			if (c == '"' || c == '\'')
  				goto letter_quote;
+ 			break;
+ 		case 'u':
+ 		case 'U':
+ 			c = tok_nextc(tok);
+ 			if (c == 'r' || c == 'R')
+ 				c = tok_nextc(tok);
+ 			if (c == '"' || c == '\'')
+ 				goto letter_quote;
+ 			break;
  		}
  		while (isalnum(c) || c == '_') {