Building browser-like GET request

[Björn Keil]

On 21 Apr., 23:28, Gilles Ganault <nos... at nospam.com> wrote:
> I'd like to download pages from a site, but it checks whether
> the requests are coming from a live user or a script; If the latter,
> the server returns a blank page.
>
> Using a proxy (Paros), I can see what information my script and
> FireFox send, and there are a lot of information that Python is
> missing:

Well, I am brand new to Python, so it takes me a lot of guessing, but
since it seems you're using urlib2:

On http://docs.python.org/lib/module-urllib2.html is written that you
may add custom headers to your http requests.
Either by calling "addheader()" or by passing a dictionary with
headers to the constructor.

I hope that helped and I wasn't telling things you already new.
As a sidenote: For the task you describe I'd rather use an actual
sniffer - such as Wireshark (http://en.wikipedia.org/wiki/Wireshark),
than logs of a Proxy... Not sure wether Wireshark works under Windows,
though.

Good luck!