Parsing/Crawler Questions..

[Philip Semanchuk]

On Mar 4, 2009, at 4:44 PM, bruce wrote:

> Hi...
>
> Sorry that this is a bit off track. Ok, maybe way off track!
>
> But I don't have anyone to bounce this off of..
>
> I'm working on a crawling project, crawling a college website, to  
> extract
> course/class information. I've built a quick test app in python to  
> crawl the
> site. I crawl at the top level, and work my way down to getting the  
> required
> course/class schedule. The app works. I can consistently run it and  
> extract
> the information. The required information is based upon an XPath  
> analysis of
> the DOM for the given pages that I'm parsing.
>
> My issue is now that I have a "basic" app that works, I need to  
> figure out
> how I guarantee that I'm correctly crawling the site. How do I know  
> when
> I've got an error at a given node/branch, so that the app knows that  
> it's
> not going to fetch the underlying branch/nodes of the tree..
>
> When running the app, I can get 5000 classes on one run, 4700 on  
> antoher,
> etc... So I need some method of determining when I get a "complete"  
> tree...
>
> How do I know when I have a complete "tree"!


hi Bruce,
To put this another way, you're trying to convince yourself that your  
program is correct, yes? For instance, you're worried that you might  
be doing something like discovering a URL on a site but failing to  
pursue that URL, yes?

The standard way of testing any program is to test known input and  
look for expected output. Repeat as necessary. In your case that would  
mean crawling a site where you know all of the URLs and to see if your  
program finds them all. And that, of course, isn't proof of  
correctness, it just means that that particular site didn't trigger  
any error conditions that would cause your program to misbehave.

I think every modern OS makes it easy to run a Web server on your  
local machine. You might want to set up suite of test sites on your  
machine and point your program at localhost. That way you can build a  
site to test your application in areas you fear it may be weak.

I'm unclear on what you're using to parse the pages, but (X)HTML is  
very often invalid in the strict sense of validity. If the tools  
you're using expect/insist on well-formed XML or valid HTML, they'll  
be disappointed on most sites and you'll probably be missing URLs. The  
canonical solution for parsing real-world Web pages with Python is  
BeautifulSoup.

HTH
Philip