Scrapy/XPath help

Always Learning cbrowning at ou.edu
Fri Dec 21 16:21:08 EST 2012


Hello all. I'm new to Python, but have been playing around with it for a few weeks now, following tutorials, etc. I've spun off on my own and am trying to do some basic web scraping. I've used Firebug/View XPath in Firefox for some help with the XPaths, however, I still am receiving errors when I try to run this script. If you could help, it would be greatly appreciated!

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from cbb_info.items import CbbInfoItem, Field

class GameInfoSpider(BaseSpider):
    name = "game_info"
    allowed_domains = ["www.sbrforum.com"]
    start_urls = [
        'http://www.sbrforum.com/betting-odds/ncaa-basketball/',
        ]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        toplevels = hxs.select("//div[@class='eventLine-value']")
        items = []
        for toplevels in toplevels:
            item = CbbInfoItem()
            item ["teams"] = toplevels.select("/span[@class='team-name'/text()").extract()
            item ["lines"] = toplevels.select("/div[@rel='19']").extract()
            item.append(item)
        return items



More information about the Python-list mailing list