Python: 404 Error when trying to login a webpage by using 'urllib' and 'HTTPCookieProcessor'

[Chris Angelico]

On Sun, Jan 12, 2014 at 11:17 PM, KMeans Algorithm <bilbaow at gmail.com> wrote:
> What am I doing wrong? Thank you very much.

I can't say what's actually wrong, but I have a few ideas for getting
more information out of the system...

> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())

You don't do anything with this opener - could you have a cookie problem?

> req = urllib2.Request(url, login)
>
> But I get a "404" error (Not Found). The page "https://www.mysite.com/loginpage" does exist (note please the httpS, since I'm not sure if this the key of my problem).
>
> If I try with
>
> -------
> resp = urllib2.urlopen(url)
> --------
> (with no 'login' data), it works ok but, obviously, I'm not logged in.

Note that adding a data parameter changes the request from a GET to a
POST. I'd normally expect the server to respond 404 to both or
neither, but it's theoretically possible.

It's also possible that you're getting redirected, and that (maybe
because cookies aren't being retained??) the destination is 404. I'm
not familiar with urllib2, but if you get a response object back, you
can call .geturl() on it - no idea how that goes with HTTP errors,
though.

You may want to look at the exception's .reason attribute - might be
more informative than .code.

As a last resort, try firing up Wireshark or something and watch
exactly what gets sent and received. I went looking through the docs
for a "verbose" mode or a "debug" setting but can't find one - that'd
be ideal if it exists, though.

Hope that's of at least some help!

ChrisA