URLLIB2 problems (solved)

[Anthony_Barker]

After a bit of frustration and trying a bunch of things (reading the
urllib2.py file) i found some documentation by Jeremy and Moshe Zadka.

ProxyHandler([proxies])'
     Cause requests to go through a proxy.  If PROXIES is given, it
     must be a dictionary mapping protocol names to URLs of proxies.
     The default is to read the list of proxies from the environment
     variables `PROTOCOL_proxy'.

So I added the environment variable to my .bash_profile in the format

export HTTP_PROXY="http://user:pass@proxy.com:8080/"

relogged in and checked my environment variables.

launched the python interpreter

import urllib2
f = urllib2.urlopen('http://www.python.org')
print f.read()

It works!!


Just a quick note - if you want to find out the realm of your proxy. 2
methods I found:
1) use netscape 4.7
2) telnet in on port (eg 8080) 
Type GET / HTTP/1.0

You will get something like this:
HTTP/1.0 407 Proxy Authentication Required
Date: Thu, 14 Feb 2002 16:57:27 GMT
Content-Length: 233
Content-Type: text/html
Server: NetCache (NetApp/5.1R2D15)
Connection: keep-alive
Proxy-Authenticate: basic realm="NETCACHE2"

Telling us that it uses basic authentication (not MD5) and the realm
is "NETCACHE2"

> > I am having problems with python and our company's proxy server, which
> > requires  authentication.
> > 
> > here is the code I am using:
> > 
> > import urllib2 
> > proxy = urllib2.ProxyHandler({"http":"http://myname:mypass@proxyserver.company.com:8080/"})
> > opener = urllib2.build_opener(proxy) 
> > f = urllib2.urlopen('http://www.python.org') 
> > 
> > results
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in ?
> >   File "/usr/lib/python2.2/urllib2.py", line 138, in urlopen
> >     return _opener.open(url, data)
> >   File "/usr/lib/python2.2/urllib2.py", line 322, in open
> >     '_open', req)
> >   File "/usr/lib/python2.2/urllib2.py", line 301, in _call_chain
> >     result = func(*args)
> >   Fil
> 
> e "/usr/lib/python2.2/urllib2.py", line 785, in http_open
> >     return self.do_open(httplib.HTTP, req)
> >   File "/usr/lib/python2.2/urllib2.py", line 763, in do_open
> >     raise URLError(err)
> > urllib2.URLError: <urlopen error (101, 'Network is unreachable')>
> > 
> > Under windows error is the same except for the last line (and line
> > numbers)
> > URLError: <urlopen error (10051, 'Network is unreachable')>
> > 
> > Both perl and wget work. wget only supports Basic Authentication, so I
> > assume it is the same. Thinking it might be a network type issue I
> > changed the domain names to IP addresses. (results follow)
> > 
> > Any help would be appreciated................
> > 
> > File "<stdin>", line 1, in ?
> >   File "/usr/lib/python2.2/urllib2.py", line 138, in urlopen
> >     return _opener.open(url, data)
> >   File "/usr/lib/python2.2/urllib2.py", line 322, in open
> >     '_open', req)
> >   File "/usr/lib/python2.2/urllib2.py", line 301, in _call_chain
> >     result = func(*args)
> >   File "/usr/lib/python2.2/urllib2.py", line 785, in http_open
> >     return self.do_open(httplib.HTTP, req)
> >   File "/usr/lib/python2.2/urllib2.py", line 763, in do_open
> >     raise URLError(err)
> > urllib2.URLError: <urlopen error (111, 'Connection refused')>