view http info when browsing

[John J. Lee]

santoshamb at yahoo.com (ben) writes:

> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it.  The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script.  I think if I could see the raw http data
> that comes to my browser life would be much easier.  Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
> 
> I've looked at plugins like httpwatch and iewatch.  httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional.  iewatch has a free trial, but they don't show cookie
> info, which is important.
> 
> Does anybody know a good way of view the raw http?
[...]

from http://wwwsearch.sourceforge.net/bits/GeneralFAQ.html :

| I want to see what my web browser is doing, but standard network
| sniffers like ethereal or netcat (nc) don't work for HTTPS. How do I
| sniff HTTPS traffic?
| 
| Three good options: 
| 
| Mozilla plugin: livehttpheaders. 
| ieHTTPHeaders does the same for MSIE. 
| Use lynx -trace, and filter out the junk with a script. 
| 
| I'm told you can also use a proxy like proxomitron (never tried it
| myself). There's also a commercial MSIE plugin.

URLs:

http://www.ethereal.com/

http://livehttpheaders.mozdev.org/

http://www.blunck.info/iehttpheaders.html

http://lynx.browser.org/

http://www.proxomitron.info/

http://www.simtec.ltd.uk/


There are plugins that show what session & permanent cookies your
browser has stored, too.  Both Mozilla and Konqueror (& Safari and
Opera too, I guess?) do this as standard, but for IE:

http://www.codeproject.com/shell/cookiespy.asp?df=100&forumid=1989&exp=0&select=21591


John