Getting a 401 from requests.get, but not when logging in via the browser.

[dcwhatthe at gmail.com]

On Monday, April 20, 2020 at 5:02:23 PM UTC-4, Eli the Bearded wrote:
> In comp.lang.python,  dc wrote, in reply to me:
> > "What do you think it is doing?"
> > I thought the timeout was waiting for a successful connection.
> 
> A successful *connection* and a successful *authentication* are
> different things. 
> 
> $ telnet example.com 80
> Trying 255.11.22.123...
> Connected to example.com
> Escape character is '^]'.
> 
> [...]
> 
> There's a connection. No authentication, however.
> 
> > "Are you sure the site is using HTTPBasicAuth()? Because if it's not,
> > that would explain how the same credentials can fail. (It could also
> > be something else, like a site returning "401 Unauthorized" because
> > it doesn't like your User-Agent.)"
> > 
> > Yes, that's what I'm getting.
> > 
> > No, I don't know if it's using Basic Authentication.  If I log in
> > through the browser, then it pops up for an id and password.
> > 
> > How do I find out what type of Authentication is applicable?  
> 
> Look at the WWW-Authenticate: header.
> 
> For an example, back to telnet again.
> 
> $ telnet example.com 80
> Trying 255.11.22.123...
> Connected to example.com
> Escape character is '^]'.
> GET /digest/ HTTP/1.1
> Host: example.com
> 
> HTTP/1.1 401 Unauthorized
> Date: Mon, 20 Apr 2020 20:42:25 GMT
> Server: Apache/2.4.41 (Unix) OpenSSL/1.0.2k
> WWW-Authenticate: Digest realm="File Resources", nonce="RyTO776jBQA=5fe3887c65536842f2ebb8ad6cf39bb6b5ec9b66", algorithm=MD5, domain="/digest/", qop="auth"
> Content-Length: 381
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
> ...
> 
> 
> 
> $ telnet example.com 80
> Trying 255.11.22.123...
> Connected to example.com
> Escape character is '^]'.
> GET /basic/ HTTP/1.1
> Host: example.com
> 
> HTTP/1.1 401 Unauthorized
> Date: Mon, 20 Apr 2020 20:45:22 GMT
> Server: Apache/2.4.41 (Unix) OpenSSL/1.0.2k
> WWW-Authenticate: Basic realm="Restricted Resources"
> Content-Length: 381
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
> ...
> 
> 
> There are other ways to authenticate besides those two, but those
> are the ones I've used that operate on the HTTP level and in browsers.
> 
> http://www.iana.org/assignments/http-authschemes/http-authschemes.xhtml
> 
> That list is supposedly all of the auth schemes, I don't know how many
> are widely implemented. Certainly some of them, like "Bearer" I've
> seen for APIs, but not using a browser password GUI. Bearer is a very
> common way to authenticate for APIs.
> 
> If you don't understand what the site is asking for, it may be very
> difficult for you to satisfy it.
> 
> Elijah
> ------
> understands all of this at a low level and not well at a library level

Been years since I used Telnet.  I didn't even think that Windows had it anymore.

I tried telneting the landing page, i.e. without the specific node that requires the login.  So e.g.

Telnet thissite.oh.gov 80

, but it returns a 400 Bad Request.  Before that, the Telnet screen is completely blank ; I have to press a key before it returns the Bad Request.


Roger on knowing what the site is asking for.  But I don't know how to determine that.

Will continue tomorrow, thanks for your assistance.