Slurping Web Pages

[Ian Bicking]

On Sat, 2003-01-25 at 12:54, Tony Dunn wrote:
> I've started a new project where I need to slurp web pages from a site that
> use cookies to authenticate access.  I've used *urllib* in the past to grab
> *public* web pages, but I'm not sure the best way to go about dealing with
> the cookie issue.

You may want to try httpsession, which will keep track of your cookies. 
I believe it's at webunit.sf.net -- there's also another module around
for keeping track of cookies while using urllib, but I can't remember
the name of it anymore.  It's probably on Parnassus; but httpsession has
worked well for me in the past.

-- 
Ian Bicking           Colorstudy Web Development
ianb at colorstudy.com   http://www.colorstudy.com
PGP: gpg --keyserver pgp.mit.edu --recv-keys 0x9B9E28B7
4869 N Talman Ave, Chicago, IL 60625 / (773) 275-7241