How to apply the user's HTML environment in a Python programme?

[Ramchandra Apte]

On Monday, 1 October 2012 19:49:27 UTC+5:30, BobAalsma  wrote:
> Op vrijdag 21 september 2012 16:15:30 UTC+2 schreef Joel Goldstick het volgende:
> 
> > On Fri, Sep 21, 2012 at 9:58 AM, BobAalsma wrote:
> 
> > 
> 
> > > Op vrijdag 21 september 2012 15:36:11 UTC+2 schreef Jerry Hill het volgende:
> 
> > 
> 
> > >> On Fri, Sep 21, 2012 at 9:31 AM, BobAalsma wrote:
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> > Thanks, Joel, yes, but as far as I'm aware these would all require the Python programme to have the user's username and password (or "credentials"), which I wanted to avoid.
> 
> > 
> 
> > >>
> 
> > 
> 
> > >>
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> No matter what you do, your web service is going to have to
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> authenticate with the remote web site.  The details of that
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> authentication are going to vary with each remote web site you want to
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> connect to.
> 
> > 
> 
> > >>
> 
> > 
> 
> > >>
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> --
> 
> > 
> 
> > >>
> 
> > 
> 
> > >> Jerry
> 
> > 
> 
> > >
> 
> > 
> 
> > > Hmm, from the previous posts I get the impression that I could best solve this by asking the user for the specific combination of username, password and URL + promising not to keep any of that...
> 
> > 
> 
> > >
> 
> > 
> 
> > > OK, that does sound doable - thank you all
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > I recommend that you write your program to read pages that are not
> 
> > 
> 
> > protected.  Once you get that working, you can go back and figure out
> 
> > 
> 
> > how you want to get the username/password from your 'friends' and add
> 
> > 
> 
> > that in.  Also look up Beautiful Soup (version 4) for a great library
> 
> > 
> 
> > to parse the pages that you retrieve
> 
> > 
> 
> > >
> 
> > 
> 
> > > Bob
> 
> > 
> 
> > > --
> 
> > 
> 
> > > http://mail.python.org/mailman/listinfo/python-list
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > 
> 
> > -- 
> 
> > 
> 
> > Joel Goldstick
> 
> 
> 
> Joel, 
> 
> 
> 
> I've spent some time with this but don't really understand my results - some help would be appreciated.
> 
> I've built a tester that will read my LinkedIn home page, which is password protected.
> 
> When I use that method for reading other people's pages, the program is redirected to the LinkedIn login page.
> 
> When I paste the URLs for the other people's pages in any browser, the requested pages are shown.
> 
> 
> 
> Bob

Not all the authentication information is in the URL.
Some of it is in cookies in the browser.