How to keep cookies when making http requests (Python 2.7)

[dieter]

Luca Cerone <luca.cerone at gmail.com> writes:

> ...
> Ok so after reading the documentation for urllib2 and cookielib I came up with the following code:
>
> #START
> from urllib2 import urlopen , Request
> from cookielib import CookieJar
> import re
> regex = re.compile(r'<span class=\'x\'>\{(.*)\}<span class=\'x\'>')
>
> base_url = "http://quiz.gambitresearch.com"
> job_url  = base_url + "/job/"
>
> cookies = CookieJar()
> r = Request(base_url) #prepare the request object
> cookies.add_cookie_header(r) #allow to have cookies
> R = urlopen(r) #read the url
> cookies.extract_cookies(R,r) #take the cookies from the response R and adds #them to the request object 

"adds them to the request object" should be "adds them to the cookie jar".

> #build the new url
> t = R.read()
> v = str(eval(regex.findall(t)[0]))
> job_url = job_url + v
>
>
> # Here I create a new request to the url containing the email address
> r2 = Request(job_url)
> cookies.add_cookie_header(r2) #I prepare the request for cookies adding the cookies that I extracted before.
>
> #perform the request and print the page
> R2 = urlopen(r2)
> t2 = R2.read()
> print job_url
> print t2
> #END
>
> This still doesn't work, but I really can't understand why.
> As far as I have understood first I have to instantiate a Request object
> and allow it to receive and set cookies (I do this with r = Request() and cookies.add_cookie_header(r))
> Next I perform the request (urlopen),  save the cookies in the CookieJar (cookies.extract_cookies(R,r)).
>
> I evaluate the new address and I create a new Request for it (r2 = Request)
> I add the cookies stored in the cookiejar in my new request (cookies.add_cookie_header(r2))
> Then I perform the request (R2 = urlopen(r2)) and read the page (t2 = R2.read())
>
> What am I doing wrong?

With respect to cookie handling, you do everything right.

There may be other problems with the (wider) process.
Analysing the responses of your requests (reading the status codes,
the response headers and the response bodies) may provide hints
towards the problem.

>Do I misunderstand something in the process?

Not with respect to cookie handling.