using urllib or httplib to post with ENCTYPE=mulitpart/form-data

[John J. Lee]

Kevin Carlson <nskhcarlso at bellsouth.net> writes:

> Never mind.  I don't think the ClientCookie is going to help me
> here. As someone had eluded earlier in this thread, the cookie is not
> coming back as a header.  I was looking through the headers and
> couldn't see a Set-Cookie header -- thought I was going crazy.
> 
> As it turns out, one of the pages in the frameset contains Javascript
> that creates the cookie. I can parse that for the Cookie and send it
> back in subsequent requests.
[...]

Yes, that's a pain.  I'm contemplating writing a Javascript/Python
thing that would solve some of these problems for HTTP cookies and
HTML forms.  Vapourware.

Anyway, there is still some point in ClientCookie even for people who
have Javascript cookies, though.  ATM, it knows which cookies to
accept and return to the server, and can do that for all requests --
so you just have to set a cookie once, then forget about it and just
call urlopen (or OpenerDirector.open).  In future, I imagine it will
give you access to a proper constructor for Cookie structs.

import urllib2, urlparse

import ClientCookie  # 0.4.1a required

cs = ClientCookie.Cookies()
opener = ClientCookie.build_opener(ClientCookie.HTTPHandler(c))

request = urllib2.Request("http://www.example.com")
response = opener.open(request)
cookies = your_cookie_parsing_routine(request, response)
for cookie in cookies:
    cs.set_cookie_if_ok(cookie, request)

# go on using opener.open() -- cookies get sent back when they should be
# (and any that arrive via the traditional HTTP route will be set, if
#  appropriate)
opener.open("http://www.example.com/blah.html")
opener.open("http://www.example.com/foo.html")


def your_cookie_parsing_routine(request, response):
    data = response.read()

    # Parse name, value, domain, path from data (ie. the HTML) here.

    # If domain and path are not specified, set the domain_specified and
    # path_specified flags accordingly, and use the request values:

    req_domain, req_path = urlparse.urlparse(request,get_full_url())
    if not domain_specified:
        domain = req_domain
    if not path_specified:
        path = req_path

    # Create a Cookie struct:
    # (ClientCookie will aquire a better way of doing this eventually)
    c = Cookie(0, name, value,
               None, 0,
               domain, domain_specified, domain.startswith("."),
               path, path_specified,
               secure,  # true if must only be sent back via https
               time.time()+(3600*24*365),  # expires
               0, "", "", {})

    return [c]  # you may be parsing out more than one, of course


Hmm, probably ClientCookie's cookie parsers and constructors should be
factored out of Cookies, so you can supply your own parser and not
need to construct Cookie structs by hand... more refactoring ahead
(but no more interface changes, I hope!).


John