Regex single quotes in scraper script?

Rock rock at py-nosp
Fri Jul 16 22:32:39 EDT 2004


Hi, I started using a python based screen scraper called newsscraper I
downloaded from sourceforge.
http://sourceforge.net/projects/newsscraper/.   I have created many python
templates that work just fine from their examples however I ran into a road
block with sites that use single quotes instead of double quotes for
specifying url in their web pages.

For example:  <a href='http://www.foo/'>

instead of the usual
                     <a href="http://www.foo/">

Being a real newbie with this I think I found the area of code that parses
the href.  It is in a file called parsefns.py
the full excerpt is listed below but here is the regex line that I believe
is not dealing with single quote.

m = re.search(r'href\s*=\s*"?([^>" ]+)["> ]', text, re.I)

I have tried many different variations but no luck and no luck getting hold
of the author.  Any ideas?  Thx.

---------------------
def get_href(text, base_url=None):
    """get_href(text[, base_url]) -> href or None

    Extract the URL out of an HREF tag.  If base_url is provided,
    will attempt to resolve relative links.

    """
    m = re.search(r'href\s*=\s*"?([^>" ]+)["> ]', text, re.I)
    if not m:
        return None
    link = m.group(1)
    if base_url and not link.lower().startswith("http"):
        import urlparse
        link = urlparse.urljoin(base_url, link)
    return link
===============





More information about the Python-list mailing list