[Python-checkins] r79724 - in python/branches/py3k: Misc/NEWS Parser/tokenizer.c
benjamin.peterson
python-checkins at python.org
Sun Apr 4 00:55:48 CEST 2010
Author: benjamin.peterson
Date: Sun Apr 4 00:55:48 2010
New Revision: 79724
Log:
Merged revisions 79723 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
........
r79723 | benjamin.peterson | 2010-04-03 17:48:51 -0500 (Sat, 03 Apr 2010) | 1 line
ensure that the locale does not affect the tokenization of identifiers
........
Modified:
python/branches/py3k/ (props changed)
python/branches/py3k/Misc/NEWS
python/branches/py3k/Parser/tokenizer.c
Modified: python/branches/py3k/Misc/NEWS
==============================================================================
--- python/branches/py3k/Misc/NEWS (original)
+++ python/branches/py3k/Misc/NEWS Sun Apr 4 00:55:48 2010
@@ -12,6 +12,8 @@
Core and Builtins
-----------------
+- Ensure that tokenization of identifiers is not affected by locale.
+
- Issue #1222585: Added LDCXXSHARED for C++ support. Patch by Arfrever.
- Raise a TypeError when trying to delete a T_STRING_INPLACE struct member.
Modified: python/branches/py3k/Parser/tokenizer.c
==============================================================================
--- python/branches/py3k/Parser/tokenizer.c (original)
+++ python/branches/py3k/Parser/tokenizer.c Sun Apr 4 00:55:48 2010
@@ -179,6 +179,16 @@
#else /* PGEN */
+/* Ensure that the locale does not interfere with tokenization. */
+
+static int
+ascii_isalnum(int c)
+{
+ return (('a' <= c && c <= 'z') ||
+ ('A' <= c && c <= 'Z') ||
+ ('0' <= c && c <= '9'));
+}
+
static char *
error_ret(struct tok_state *tok) /* XXX */
{
@@ -245,7 +255,7 @@
} while (t[0] == '\x20' || t[0] == '\t');
begin = t;
- while (isalnum(Py_CHARMASK(t[0])) ||
+ while (ascii_isalnum(Py_CHARMASK(t[0])) ||
t[0] == '-' || t[0] == '_' || t[0] == '.')
t++;
More information about the Python-checkins
mailing list