Passing environment variable to HTTP server

Tue May 8 22:18:45 EDT 2001

"Doug Fort" <dougfort at downright.com> wrote in ...
> William Park wrote:
>
> > When I send a request to CGI script, the browser sends environment
> > variables, such as HTTP_REFERER, HTTP_USER_AGENT, etc.  How can I modify
> > the environment variables that are sent to CGI script?
> >
> > Eg. if remote CGI script only responds to Netscape (from
> > HTTP_USER_AGENT), but I would still like to browse and download using
> > Python script.
> >
> > --William Park, Open Geometry Consulting, Mississauga, Ontario, Canada.
> >   8 CPU cluster, (Slackware) Linux, Python, LaTeX, vim, mutt
> >
> >
>
> To impersonate a browser we send the 'User-agent' header.  However there's
> usually more to it than that: other custom headers or specialized cookies.
> I recommend capturing a browser session with Ethereal
> http://www.ethereal.com/ and duplicating the headers exactly.
>
> Note that even if you are impersonating a browsee, you should adhere to
the
> site's robot.txt file and <meta> tags.  There is an excellent Python
> module for checking robots.txt.
>
Since you already have Python, Sam Rushing's proxy server (found on
www.nightmare.com) would be an excellent tool for capturing the dialog
between client and server. Here's a quick example:

D:\Steve\Projects\Python\medusa>C:\Python20\python proxysample.py 127.0.0.1
80
<== (0) 'GET /holdenweb/default.htm HTTP/1.0\015'
<== (0) 'Connection: Keep-Alive\015'
<== (0) 'User-Agent: Mozilla/4.7 [en] (Win98; I)\015'
<== (0) 'Host: 127.0.0.1:8080\015'
<== (0) 'Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
image/png, */*\015'
<== (0) 'Accept-Encoding: gzip\015'
<== (0) 'Accept-Language: en\015'
<== (0) 'Accept-Charset: iso-8859-1,*,utf-8\015'
<== (0) '\015'
Connected
==> (0) 'HTTP/1.0 200 Ok\015'
==> (0) 'Server: Xitami\015'
==> (0) 'Content-Type: text/html\015'
==> (0) 'Content-Length: 164\015'
==> (0) 'Last-Modified: Sat, 27 Jan 2001 20:36:56 GMT\015'
==> (0) 'Connection: Keep-Alive\015'
==> (0) 'Keep-Alive: timeout=30, max=25\015'
==> (0) '\015'
==> (0) '<html>\015'
==> (0) '<head>\015'
==> (0) '    <title>Holden Web: Operations Failure</title>\015'
==> (0) '</head>\015'
==> (0) '<body>\015'
==> (0) '<h2>Web Process Not Running</h2>\015'
==> (0) '<p>Please tell Steve</p>\015'
==> (0) '</body>\015'
==> (0) '</html>\015'
==> (0) '\015'
Connected
<== (0) 'GET / HTTP/1.0\015'
<== (0) 'Connection: Keep-Alive\015'
<== (0) 'User-Agent: Mozilla/4.7 [en] (Win98; I)\015'
<== (0) 'Host: 127.0.0.1:8080\015'
<== (0) 'Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
image/png, */*\015'
<== (0) 'Accept-Encoding: gzip\015'
<== (0) 'Accept-Language: en\015'
<== (0) 'Accept-Charset: iso-8859-1,*,utf-8\015'
<== (0) '\015'
==> (0) 'HTTP/1.0 200 Ok\015'
==> (0) 'Server: Xitami\015'
==> (0) 'Content-Type: text/html\015'
==> (0) 'Content-Length: 5589\015'
==> (0) 'Last-Modified: Tue, 20 Mar 2001 17:22:48 GMT\015'
==> (0) 'Connection: Keep-Alive\015'
==> (0) 'Keep-Alive: timeout=30, max=25\015'
==> (0) '\015'
==> (0) '<HTML><HEAD><!--Revised: 2 Jan 1999 By: iMatix Corporation-->\015'
==> (0) '<TITLE>Welcome To Xitami v2.4d7</TITLE></HEAD>\015'
==> (0) '<BODY BGCOLOR="#87CEFA">\015'
==> (0) '<TABLE WIDTH=100%><TR><TD ALIGN=LEFT>\015'
==> (0) ' | <A href="http://www.imatix.com/">iMatix home page</A><BR>\015'
==> (0) ' | <A HREF="/xitami/index.htm"> Xitami home page</A><BR>\015'
==> (0) ' | <A HREF="/lrpersonal/"> Class Web</A>\015'
==> (0) '<TD ALIGN=RIGHT>\015'
==> (0) '<IMG SRC="/images/xitami1.gif" ALT="Xitami"\015'
==> (0) ' WIDTH=100 HEIGHT=75 ALIGN=RIGHT HSPACE=10>\015'
==> (0) '<FONT SIZE=4><B>Xitami</B><BR><FONT SIZE=2>Version 2.4d7\015'
==> (0) '</TABLE>\015'
==> (0) '<HR>\015'
==> (0) '\015'
==> (0) '<TABLE>\015'
==> (0) '<TR><TD VALIGN=TOP>\015'
==> (0) '<H1>Welcome To Xitami v2.4d7</H1>\015'
==> (0) '\015'
==> (0) '<P>Welcome to Xitami, the <EM>fast</EM> web server. You should be
accessing\015'
==> (0) 'these pages through Xitami now. To configure the server, use
the\015'
==> (0) '<A HREF="/admin">web-based admin</A> function. This is
password-protected. On\015'
==> (0) 'UNIX and OS/2 you should edit the password file (xitami.aut or
equivalent)\015'
==> (0) 'after installation. Under Windows this is done at installation
time.\015'
==> (0) '\015'

... and lots, lots more.

regards
 Steve