[issue34357] situation where urllib3 works, but urllib does not work

[Martin Panter]

Martin Panter <vadmium+py at gmail.com> added the comment:

I can’t get it to hang. Does your computer or Internet provider have a proxy or firewall that may be interfering?

Perhaps it is worth comparing the HTTP header fields being sent and received. You can enable debug messages to see the request sent, and print the response fields directly. Most important things to look for are the Content-Length and Transfer-Encoding (if any) fields in the response.

>>> import urllib.request
>>> url = "https://cinnamon-spices.linuxmint.com/json/applets.json"
>>> handler = urllib.request.HTTPSHandler(debuglevel=1)
>>> opener = urllib.request.build_opener(handler)
>>> resp = opener.open(url)
send: b'GET /json/applets.json HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: cinnamon-spices.linuxmint.com\r\nUser-Agent: Python-urllib/3.6\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server header: Date header: Content-Type header: Content-Length header: Connection header: Last-Modified header: ETag header: X-Sucuri-Cache header: X-XSS-Protection header: X-Frame-Options header: X-Content-Type-Options header: X-Sucuri-ID header: Accept-Ranges $
>>> print(response.info())
Server: Sucuri/Cloudproxy
Date: Mon, 13 Aug 2018 07:18:11 GMT
Content-Type: application/json
Content-Length: 70576
Connection: close
Last-Modified: Mon, 13 Aug 2018 07:25:14 GMT
ETag: "113b0-5734bfe97145e"
X-Sucuri-Cache: HIT
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-Sucuri-ID: 11014
Accept-Ranges: bytes


>>> data = resp.read()
>>> len(data)
70576

Another experiment would be to try “http.client” directly, which I understand is used by both the built-in “urllib.request” module, and “urllib3”:

from http.client import HTTPSConnection
conn = HTTPSConnection("cinnamon-spices.linuxmint.com")
headers = {  # Same header fields sent by “urllib.request”
    "Accept-Encoding": "identity",
    "Host": "cinnamon-spices.linuxmint.com",
    "User-Agent": "Python-urllib/3.6",
    "Connection": "close",
}
conn.request("GET", "/json/applets.json", headers=headers)
resp = conn.getresponse()
print(resp.msg)
data = resp.read()

Try removing the “Connection: close” field from the request. Occasionally this triggers bad server behaviour (see Issue 12849); maybe your server or proxy is affected.

----------
nosy: +martin.panter
stage:  -> test needed
type:  -> behavior
versions: +Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue34357>
_______________________________________