Correct handling of case in unicode and regexps

[Devin Jeanpierre]

On Sat, Feb 23, 2013 at 1:12 PM, MRAB <python at mrabarnett.plus.com> wrote:
> The basic rule is that a series of characters in the regex must match a
> series of characters in the text, with no partial matches in either.
>
> For example, 'ss' can match 'ß', but 's' can't match 'ß' because that
> would be matching part of 'ß'.
>
> In a regex like 's+', you're asking it to match one or more repetitions
> of 's', but that would mean that 's' would have to match part of 'ß' in
> the first iteration and the remainder of 'ß' in the second iteration.

That makes sense. I'll have to think about this and run some tests
through regex, as well.

Thanks!

> Although it's theoretically possible to do that, the code is already
> difficult enough. The cost outweighs the potential benefit.
>
> If you'd like to have a go at implementing it, the code _is_ open
> source. :-)

Actually, the reason it's relevant to me is that I'm reimplementing
the re module using a more automata theoretic approach (it's my second
attack at the problem). Also, I've read the _sre source code and it's
unpleasant. Is regex much better?

At least the way I'm planning on going about it, supporting this is
easier, as long as one can figure out what it means to match halfway
inside a ß. Since case folding is a homomorphism*, I can case fold the
regex** and case fold the input and then I'm done. Case folding of the
input can be done character by character, and to emulate the regex
module behavior I'd need to check at certain places whether or not I'm
in the middle of a casefolding expansion, and fail if so. On the other
hand, if I don't emulate the regex module's behavior in at least some
cases, I'd need to figure out what the value of a match of 's' against
'ß' would be.

[*]  i.e. it can be done character by character (see Unicode 3.13
Default Case Algorithms)
[**] Not as trivial as it sounds, but still easy. [ßa-z] goes to e.g.
[a-z]|ss (not [ssa-z]).

-- Devin