using re.finditer()

[Erik Johnson]

    I am still fairly new to Python and trying to learn to put RE's to good
use. I am a little confused about the finditer() method. It is documented
like so:

      finditer( pattern, string)

Return an iterator over all non-overlapping matches for the RE pattern in
string. For each match, the iterator returns a match object. Empty matches
are included in the result unless they touch the beginning of another match.
New in version 2.2.


    I would say this documentation is not quite right (or incomplete at
best) because it doesn't document any restriction about multiline matching,
but it certainly seems to have one.  This would seem to be an ideal
application for finditer()...

#! /usr/bin/python

import re

html = """
<table>
  <tr>
    <td>Data 1-1</td>
    <td>Data 1-2</td>
    <td>Data 1-3</td>
  </tr>
  <tr>
    <td>Data 2-1</td>
    <td>Data 2-2
    </td>
    <td>Data 2-3</td>
  </tr>
</table>
"""

pat = r'<td.*?>(.*?)</td>'
for match in re.finditer(pat, html):
  print match.group(1)


    The iterator returned seems to work fine to step through items that
happen to be contained within one line. That is, you can step through flat,
one-line td's, but if you want to step through tr's, this doesn't work (run
this code and notice Data 2-2 is not there). finditer() doesn't accept a
flag like re.DOTALL, as re.match() and re.search() do. It seems a shame not
to be able to put an otherwise smart design to use.

    One work around to this is by applying a regualr RE like
'<tr.*?>(.*?)</tr>', finding the first match, and then chopping off the
found part as you go.  Another is to change the pattern to something like:
pat = r'<td.*?>([\n\w\s\d-]*?)</td>'    I guess I will get one of those
implemented and get this task done, but I am still interested in learning to
use RE's better.
Interestingly, using pat = r'<td.*?>([\r\n.]*?)</td>' does NOT work, and I
don't understand why - can someone explain that?
What exactly would be the equivalent set for dot with re.DOTALL turned on
([\w\W\r\n] works here, but does that really cover it)? Other ideas?

    I can think of split & join substituion tricks & the like to replace \n
with something else, then put it back in, but that get's kinda messy and
requires you to find some special substitution character that's not
elsewhere. I want to apply this to dynamically generated text that I don't
control and keep this as general as possible.  I'm wondering if I'm not
missing something here - is there no way to make finditer() work
(straightforwardly) on multilines using just a simple dot RE?

Thanks for taking the time to read my post! :)

-ej