UNICODE mode for regular expressions - time to change the default?

[John Machin]

On Apr 6, 5:50 am, John Nagle <n... at animats.com> wrote:
>    Regular expressions are compiled in ASCII mode
> unless
> Unicode mode is specified to "rc.compile".  The difference is that regular
> expressions in ASCII mode don't recognize things like
> Unicode whitespace, even when applied to Unicode strings.

AFAICT, the default is that \s, \d, etc are interpreted according to
the current locale's properties. Specifying re.U changes that to use
the unicodedata properties instead. There is no such thing as "ASCII
mode".

> For example, Unicode character 0x00A0 is a "NO-BREAK SPACE", which is
> a form of whitespace. It's the Unicode equivalent of HTML's " ".
> This can create some strange bugs.
>
>    Is the current default good?  Or is it time to compile all regular
> expressions in Unicode mode by default?  It shouldn't hurt processing of
> ASCII strings to do that.

Believe it or not: there are folk out there who have data which is
encoded in 8-bit encodings which are not ASCII and for which
"\xA0".decode('whatever') does not produce u"\xA0" ... it could for
example be a box-drawing character or a letter:

>>> import unicodedata as ucd
>>> "\xA0".decode('koi8-r')
u'\u2550'
>>> ucd.name(_)
'BOX DRAWINGS DOUBLE HORIZONTAL'
>>> "\xA0".decode('cp850')
u'\xe1'
>>> ucd.name(_)
'LATIN SMALL LETTER A WITH ACUTE'
>>>

Problem number 2: It's probable that users in locale X wouldn't want a
match to succeed on a character that is regarded as a digit (say) in
distant locale Y, but if found in a data file in locale X is probably
more indicative of having read binary data instead of Unicode text:

>>> ucd.name(u"\u0f20")
'TIBETAN DIGIT ZERO'
>>> re.match(ur"\d", u"\u0f20")
>>> re.match(ur"\d", u"\u0f20", re.U)
<_sre.SRE_Match object at 0x00EFC9C0>


>  The current setup is really a legacy of when
> most things in Python didn't work in Unicode mode, and you didn't want to
> introduce Unicode unnecessarily.   It's another one of those obscure
> Unicode "gotchas" that really should go away.

It's the ASCII-centric mindset that creates gotchas and really should
go away :-)

HTH,
John