How to keep cookies when making http requests (Python 2.7)
Luca Cerone
luca.cerone at gmail.com
Tue Aug 27 06:17:54 EDT 2013
>
> > Let me make an additional remark however: you should
> > not expect to get complete details in a list like this - but only
> > hints towards a solution for your problem (i.e.
> > there remains some work for you).
> > Thus, I expect you to read the "cookielib/cookiejar" documentation
> > (part of Python's standard documentation) in order to understand
> > my example code - before I would be ready to provide further details.
Ok so after reading the documentation for urllib2 and cookielib I came up with the following code:
#START
from urllib2 import urlopen , Request
from cookielib import CookieJar
import re
regex = re.compile(r'<span class=\'x\'>\{(.*)\}<span class=\'x\'>')
base_url = "http://quiz.gambitresearch.com"
job_url = base_url + "/job/"
cookies = CookieJar()
r = Request(base_url) #prepare the request object
cookies.add_cookie_header(r) #allow to have cookies
R = urlopen(r) #read the url
cookies.extract_cookies(R,r) #take the cookies from the response R and adds #them to the request object
#build the new url
t = R.read()
v = str(eval(regex.findall(t)[0]))
job_url = job_url + v
# Here I create a new request to the url containing the email address
r2 = Request(job_url)
cookies.add_cookie_header(r2) #I prepare the request for cookies adding the cookies that I extracted before.
#perform the request and print the page
R2 = urlopen(r2)
t2 = R2.read()
print job_url
print t2
#END
This still doesn't work, but I really can't understand why.
As far as I have understood first I have to instantiate a Request object
and allow it to receive and set cookies (I do this with r = Request() and cookies.add_cookie_header(r))
Next I perform the request (urlopen), save the cookies in the CookieJar (cookies.extract_cookies(R,r)).
I evaluate the new address and I create a new Request for it (r2 = Request)
I add the cookies stored in the cookiejar in my new request (cookies.add_cookie_header(r2))
Then I perform the request (R2 = urlopen(r2)) and read the page (t2 = R2.read())
What am I doing wrong? Do I misunderstand something in the process?
Thanks again in advance for the help,
Cheers,
Luca
More information about the Python-list
mailing list