Using a proxy with urllib2

[Rob Wolfe]

"Jack" <nospam at invalid.com> writes:

> I'm trying to use a proxy server with urllib2.
> So I have managed to get it to work by setting the environment
> variable:
> export HTTP_PROXY=127.0.0.1:8081
>
> But I wanted to set it from the code. However, this does not set the proxy:
>     httpproxy = '127.0.0.1:3129'
>     proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
>     opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
>     urllib2.install_opener(opener)

Works for me.
How do you know that the proxy is not set?

> I'm using it from a web.py URL handler file, not sure if it matters.

I don't think so.

> I have another question though. It seems that using either of the
> methods above, the proxy will be global. What if I want to use
> a proxy with one site, but not with another site? Or even use a
> proxy for some URLs but not others? The proxy having to be global
> is really not convenient. Is there any way to do per-fetch proxy? 

Try this:

<code>
import urllib2

def getopener(proxy=None):
    opener = urllib2.build_opener(urllib2.HTTPHandler)
    if proxy:
        proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
        opener.add_handler(proxy_support)
    return opener

def fetchurl(url, opener):
    f = opener.open(url)
    data = f.read()
    f.close()
    return data

print fetchurl('http://www.python.org', getopener('127.0.0.1:8081'))
</code>

HTH,
Rob