aligning a set of word substrings to sentence

[Steven Bethard]

Fredrik Lundh wrote:
> Steven Bethard wrote:
> 
> 
>>>>I feel like there should be a simpler solution (maybe with the re
>>>>module?) but I can't figure one out.  Any suggestions?
>>>
>>>using the finditer pattern I just posted in another thread:
>>>
>>>tokens = ['She', "'s", 'gon', 'na', 'write', 'a', 'book', '?']
>>>text = '''\
>>>She's gonna write
>>>a book?'''
>>>
>>>import re
>>>
>>>tokens.sort() # lexical order
>>>tokens.reverse() # look for longest match first
>>>pattern = "|".join(map(re.escape, tokens))
>>>pattern = re.compile(pattern)
>>>
>>>I get
>>>
>>>print [m.span() for m in pattern.finditer(text)]
>>>[(0, 3), (3, 5), (6, 9), (9, 11), (12, 17), (18, 19), (20, 24), (24, 25)]
>>>
>>>which seems to match your version pretty well.
>>
>>That's what I was looking for.  Thanks!
> 
> 
> except that I misread your problem statement; the RE solution above allows the
> tokens to be specified in arbitrary order.  if they've always ordered, you can re-
> place the code with something like:
> 
>     # match tokens plus optional whitespace between each token
>     pattern = "\s*".join("(" + re.escape(token) + ")" for token in tokens)
>     m = re.match(pattern, text)
>     result = (m.span(i+1) for i in range(len(tokens)))
> 
> which is 6-7 times faster than the previous solution, on my machine.

Ahh yes, that's faster for me too.  Thanks again!

STeVe