Can anyone suggest a good HTTP/1.1 web client?

[Saju Pillai]

On Dec 17, 1:18 am, Kottiyath <n.kottiy... at gmail.com> wrote:
> Hi all,
>     I have to connect to a secure website every second to get the data
> and then post to it. I have been investigating on many web clients in
> python, but nothing fits the bill properly.
>     The ones I tried implementing are:
>     1. httplib based - I created myself. (I cannot use urllib2 since I
> have to transfer files, and urllib2 doesnt have multipart content-type
> support)
>     2. Twisted web client.
>     I also looked at mechanize etc too.
>
>     The problems I face are -
>     1. I liked twisted a lot, but when I implemented it, I found that
> client support is there only for twisted.web and not twisted.web2.
> Since I connect to the same website every time, I would like to have
> persistent connections and since twisted.web is HTTP/1.0, persistent
> connection support is not yet there. Without persistent connections, I
> would have to have TCP connection handshake everytime and it is taking
> too much time.
>     2. Since I connect to the website every second, I have to have
> many connections running at the same time. I am worried that creating
> threads for each connection is going to be a big problem (esp if the
> server response is slow), since the processor will get swamped -
> especially since there are many other activities going on in the
> machine.
>    3. I would also like to pipe line the requests - esp if the
> response is slow.
>
>    Other requirements:
>    1. HTTPS Support
>    2. Connection through proxy.


You could take a look at pycurl - python bindings for libcurl.
http://pycurl.sourceforge.net/

srp

>
>    Is there any good web client which I can use straight up? Or would
> I have to implement the whole thing myself? It looks like a big beast
> and I was wondering whether python provides it straight up.
>
> Regards
> K