Diacretical incensitive search

[Peter Otten]

Olive wrote:

> One feature that seems to be missing in the re module (or any tools that I
> know for searching text) is "diacretical incensitive search". I would like
> to have a match for something like this:
> 
> re.match("franc", "français")
> 
> in about the same whay we can have a case incensitive search:
> 
> re.match("(?i)fran", "Français").
> 
> Another related and more general problem (in the sense that it could
> easily be used to solve the first problem) would be to translate a string
> removing any diacritical mark:
> 
> nodiac("Français") -> "Francais"
> 
> The algorithm to write such a function is trivial but there are a lot of
> mark we can put on a letter. It would be necessary to have the list of
> "a"'s with something on it. i.e. "à,á,ã", etc. and this for every letter.
> Trying to make such a list by hand would inevitably lead to some symbols
> forgotten (and would be tedious).

[Python3.3]

>>> unicodedata.normalize("NFKD", "Français").encode("ascii", 
"ignore").decode()
'Francais'

import sys
from collections import defaultdict
from unicodedata import name, normalize

d = defaultdict(list)
for i in range(sys.maxunicode):
    c = chr(i)
    n = normalize("NFKD", c)[0]
    if ord(n) < 128 and n.isalpha(): # optional
        d[n].append(c)

for k, v in d.items():
    if len(v) > 1:
        print(k, "".join(v))

See also <http://effbot.org/zone/unicode-convert.htm>

PS: Be warned that experiments on the console may be misleading:

>>> unicodedata.normalize("NFKD", "ç")
'c'
>>> ascii(_)
"'c\\u0327'"