web spider and password protected pages

[jdonnell]

"Nevertheless, perhaps you'll still post the answer here so
that others who come along later can benefit from your
experience in the same way that you benefited from reading
whatever page you found (even if you didn't benefit from
my suggestions...). "

Your funny :) Perhaps you should take your own advice. My guess is that
the google search you described will return different results in a few
months. Your first post won't benefit those who "come along later".

"The spider hangs (waits for username and
pass) when I hit a page that requires .htaccess authentication."

I was using urllib.
urllib2 doesn't have this problem. Simply switching urllib to urllib2
fixed the problem