[Python-checkins] cpython: #11454: Reduce email module load time, improve surrogate check efficiency.

[r.david.murray]

http://hg.python.org/cpython/rev/520490c4c388
changeset:   84349:520490c4c388
user:        R David Murray <rdmurray at bitdance.com>
date:        Wed Jun 26 12:06:21 2013 -0400
summary:
  #11454: Reduce email module load time, improve surrogate check efficiency.

The new _has_surrogates code was suggested by Serhiy Storchaka.  See
the issue for timings, but it is far faster than any other alternative,
and also removes the load time that we previously incurred from compiling
the complex regex this replaces.

files:
  Lib/email/utils.py |  14 ++++++++++----
  1 files changed, 10 insertions(+), 4 deletions(-)


diff --git a/Lib/email/utils.py b/Lib/email/utils.py
--- a/Lib/email/utils.py
+++ b/Lib/email/utils.py
@@ -54,10 +54,16 @@
 specialsre = re.compile(r'[][\\()<>@,:;".]')
 escapesre = re.compile(r'[\\"]')
 
-# How to figure out if we are processing strings that come from a byte
-# source with undecodable characters.
-_has_surrogates = re.compile(
-    '([^\ud800-\udbff]|\A)[\udc00-\udfff]([^\udc00-\udfff]|\Z)').search
+def _has_surrogates(s):
+    """Return True if s contains surrogate-escaped binary data."""
+    # This check is based on the fact that unless there are surrogates, utf8
+    # (Python's default encoding) can encode any string.  This is the fastest
+    # way to check for surrogates, see issue 11454 for timings.
+    try:
+        s.encode()
+        return False
+    except UnicodeEncodeError:
+        return True
 
 # How to deal with a string containing bytes before handing it to the
 # application through the 'normal' interface.

-- 
Repository URL: http://hg.python.org/cpython