Client side network programming

[fishboy]

On Wed, 02 Jun 2004 06:23:20 -0400, "Roysun_rohit"
<roysun_rohit at rediffmail.com> wrote:

>I am interested in making a search engine which takes web sites
>iteratively, and downloads the web page or has to perform some search
>across the web pages.
>I am unsucessful to do so. My machine works through a proxy server and the
>internet connection is through 24 hour lease line. when ever i try the
>code it gives :
>
>Traceback (innermost last):
>  File "http-getfile-urllib2.py", line 19, in ?
>    urllib.urlretrieve(remoteaddr, localname)
>  File "/usr/lib/python1.5/urllib.py", line 66, in urlretrieve
>    return _urlopener.retrieve(url, filename, reporthook)
>  File "/usr/lib/python1.5/urllib.py", line 186, in retrieve
>    fp = self.open(url)
>  File "/usr/lib/python1.5/urllib.py", line 159, in open
>    return getattr(self, name)(url)
>  File "/usr/lib/python1.5/urllib.py", line 260, in open_http
>    h = httplib.HTTP(host)
>  File "/usr/lib/python1.5/httplib.py", line 53, in __init__
>    if host: self.connect(host, port)
>  File "/usr/lib/python1.5/httplib.py", line 81, in connect
>    self.sock.connect(host, port)
>IOError: [Errno socket error] (101, 'Network is unreachable')
>============================================================
>The code is like this:-
>#!/usr/bin/env python
>import os, sys, urllib, urlparse, socket
>showlines = 6
>try:
>    servername, filename = sys.argv[1:3]
>except:
>    servername, filename = 'www.igib.res.in', '/sarsanalysis.html'
>    
>remoteaddr = 'http://%s%s' % (servername, filename)
>if len(sys.argv) == 4:
>   localname = sys.argv[3]
>else:
>   (scheme, server, path, parms, query, frag) =
>urlparse.urlparse(remoteaddr)
>   localname = os.path.split(path)[1]
>   
>print remoteaddr, localname
>urllib.urlretrieve(remoteaddr, localname)
>remotedata = open(localname).readlines()
>for line in remotedata[:showlines]: print line,
>============================================================
>I am new to the internet programming as well as python. please guide me,
>how to solve this one.

urllib will work with proxies.  Just set your environment to point at
it before you start python.  Like this (copied from the urllib doc):

% http_proxy="http://www.someproxy.com:3128"
% export http_proxy
% python

hth,
><{{{*>