python regex character group matches

[Steven D'Aprano]

On Wed, 17 Sep 2008 09:27:47 -0400, christopher taylor wrote:

> hello python-list!
> 
> the other day, i was trying to match unicode character sequences that
> looked like this:
> 
> \\uAD0X...

It is not clear what this is supposed to be. Is that matching a literal 
pair of backslashes, or a single escaped backslash, or a single unicode 
character with codepoint AD0X, or what?

If I read it *literally*, then you're trying to match:

backslash backslash lowercase-u uppercase-A uppercase-D zero uppercase-X

Is that what you intended to match?


> my issue, is that the pattern i used was returning:
> 
> [ '\\uAD0X', '\\u1BF3', ... ]

Unless you are using Python 3, I see that you aren't actually dealing 
with Unicode strings, you're using byte strings. Is that deliberate?


> when i expected:
> 
> [ '\\uAD0X\\u1BF3', ]

I make that to be a string of length 12. Is that what you are expecting?

>>> len('\\uAD0X\\u1BF3')
12


> the code looks something like this:
> 
> pat = re.compile("(\\\u[0-9A-F]{4})+", re.UNICODE|re.LOCALE) 
> #print pat.findall(txt_line)
> results = pat.finditer(txt_line)

First point: I don't think the UNICODE flag does what you think it does. 
It redefines the meaning of special escape sequences \b etc. Since you 
aren't using any special escape sequences, I'm going to guess that you 
think it turns your search string into Unicode. It doesn't. (Apologies in 
advance if I guessed wrong.) I don't think you need either the UNICODE or 
LOCALE flag for this search: they don't seem to have any effect.

Secondly: you will generally save yourself a lot of trouble when writing 
regexes to use raw strings, because backslashes in the regex engine clash 
with backslashes in Python strings. But there's a gotcha: backslash 
escapes behave differently in ordinary strings and the re engine.

In an ordinary string, the sequence backslash-char is treated as a 
literal backslash-char if it isn't a special escape. So:

>>> len('\t')  # special escape
1
>>> len('\u')  # not a special escape
2

But that's not the case in the re engine! As the Fine Manual says:

    The special sequences consist of "\" and a character from 
    the list below. If the ordinary character is not on the 
    list, then the resulting RE will match the second character. 
    For example, \$ matches the character "$".

http://docs.python.org/lib/re-syntax.html

So all of these match the same thing:
    re.compile('\\u')
    re.compile(r'\u')
    re.compile('u')

To match a literal backslash-u, you need to escape the backslash before 
the engine joins it to the u: r'\\u'.

Putting it all together again:

pat = re.compile(r"(\\u[0-9A-F]{4})+") 

will probably do what you want, assuming I have guessed what you want 
correctly!



-- 
Steven