urllib.request.urlopen fails with https (was: Re: Stock quote APi)

[Irv Kalb]

Thanks to Chris A and Roger C for their information.  Very helpful and I am working on both of their suggestions.  But now I've run into a new related problem.

I still am trying to get stock information using Standard Library calls for now (although I understand that the requests module makes this kind of thing very easy).

My general approach (Python 3) is to build up a URL by concatenating an parameters, and make a call to url.request.urlopen.  I then do a 'read' on what was returned. For example, in my class, I use the following code to get weather data from weatherman.org:

    URL = 'http://api.openweathermap.org/data/2.5/weather?q=' + city + '&mode=xml'+ '&APPID=' + API_KEY
    response = urllib.request.urlopen(URL)
    responseString = str(response.read())

I then have some code to parse out the pieces of the returned information.  This works great and an excellent example of the power of using API's with Python.


In looking for a stock data API, Chris suggested  alpha vantage.co (and Roger made a nice suggestion as to how to parse it).  But when I use their API, I get an error:  

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

I have tried a number of different API's and found that if the URL starts with 'https', it always fails.  Even a simple example that I found here:  https://pythonspot.com/urllib-tutorial-python-3/ <https://pythonspot.com/urllib-tutorial-python-3/>  fails the same way.

Here's what I see when I try that example:

>>> import urllib.request
>>> html = urllib.request.urlopen('https://arstechnica.com').read()
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<pyshell#6>", line 1, in <module>
    html = urllib.request.urlopen('https://arstechnica.com').read()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>
>>> 

All my tests return the same tracebacks.

Am I doing something wrong?  Is there another way (besides using the requests module which DOES work for me) to get data from an https URL?

Any help would be appreciated.

Thanks,

Irv