Getting a 401 from requests.get, but not when logging in via the browser.

[Eli the Bearded]

In comp.lang.python,  <dcwhatthe at gmail.com> wrote, in reply to me:
> "What do you think it is doing?"
> I thought the timeout was waiting for a successful connection.

A successful *connection* and a successful *authentication* are
different things. 

$ telnet example.com 80
Trying 255.11.22.123...
Connected to example.com
Escape character is '^]'.

[...]

There's a connection. No authentication, however.

> "Are you sure the site is using HTTPBasicAuth()? Because if it's not,
> that would explain how the same credentials can fail. (It could also
> be something else, like a site returning "401 Unauthorized" because
> it doesn't like your User-Agent.)"
> 
> Yes, that's what I'm getting.
> 
> No, I don't know if it's using Basic Authentication.  If I log in
> through the browser, then it pops up for an id and password.
> 
> How do I find out what type of Authentication is applicable?  

Look at the WWW-Authenticate: header.

For an example, back to telnet again.

$ telnet example.com 80
Trying 255.11.22.123...
Connected to example.com
Escape character is '^]'.
GET /digest/ HTTP/1.1
Host: example.com

HTTP/1.1 401 Unauthorized
Date: Mon, 20 Apr 2020 20:42:25 GMT
Server: Apache/2.4.41 (Unix) OpenSSL/1.0.2k
WWW-Authenticate: Digest realm="File Resources", nonce="RyTO776jBQA=5fe3887c65536842f2ebb8ad6cf39bb6b5ec9b66", algorithm=MD5, domain="/digest/", qop="auth"
Content-Length: 381
Connection: close
Content-Type: text/html; charset=iso-8859-1
...



$ telnet example.com 80
Trying 255.11.22.123...
Connected to example.com
Escape character is '^]'.
GET /basic/ HTTP/1.1
Host: example.com

HTTP/1.1 401 Unauthorized
Date: Mon, 20 Apr 2020 20:45:22 GMT
Server: Apache/2.4.41 (Unix) OpenSSL/1.0.2k
WWW-Authenticate: Basic realm="Restricted Resources"
Content-Length: 381
Connection: close
Content-Type: text/html; charset=iso-8859-1
...


There are other ways to authenticate besides those two, but those
are the ones I've used that operate on the HTTP level and in browsers.

http://www.iana.org/assignments/http-authschemes/http-authschemes.xhtml

That list is supposedly all of the auth schemes, I don't know how many
are widely implemented. Certainly some of them, like "Bearer" I've
seen for APIs, but not using a browser password GUI. Bearer is a very
common way to authenticate for APIs.

If you don't understand what the site is asking for, it may be very
difficult for you to satisfy it.

Elijah
------
understands all of this at a low level and not well at a library level