Can anyone suggest a good HTTP/1.1 web client?

[Jean-Paul Calderone]

On Tue, 16 Dec 2008 12:18:17 -0800 (PST), Kottiyath <n.kottiyath at gmail.com> wrote:
>Hi all,
>    I have to connect to a secure website every second to get the data
>and then post to it. I have been investigating on many web clients in
>python, but nothing fits the bill properly.
>    The ones I tried implementing are:
>    1. httplib based - I created myself. (I cannot use urllib2 since I
>have to transfer files, and urllib2 doesnt have multipart content-type
>support)
>    2. Twisted web client.
>    I also looked at mechanize etc too.
>
>    The problems I face are -
>    1. I liked twisted a lot, but when I implemented it, I found that
>client support is there only for twisted.web and not twisted.web2.
>Since I connect to the same website every time, I would like to have
>persistent connections and since twisted.web is HTTP/1.0, persistent
>connection support is not yet there. Without persistent connections, I
>would have to have TCP connection handshake everytime and it is taking
>too much time.

I'm working on a new HTTP 1.1 client for Twisted.  You can follow the
progress at <http://twistedmatrix.com/trac/ticket/886> or check out the
development branch <svn://svn.twistedmatrix.com/svn/Twisted/branches/expressive-http-client-886> and give it a try.

Persistent connections are not yet implemented, but there's not a whole
lot left to do to support them.  That means pipelining isn't supported
yet, but once persistent connections are supported pipelining will be
simple.

As with any Twisted protocol, it's just as easy to run over SSL as over
TCP, so HTTPS is essentially supported for free.  There is not yet
support for proxies, but this is planned as well and should be only
a little more work to implement.
 
> [snip]
>
>   Is there any good web client which I can use straight up? Or would
>I have to implement the whole thing myself? It looks like a big beast
>and I was wondering whether python provides it straight up.

If you'd like to help out with the new Twisted HTTP client, that would be
wonderful.  Even if you can just try it out and report any problems you run
into, that would be immensely helpful.

Jean-Paul