[New-bugs-announce] [issue24667] OrderedDict.popitem() raises KeyError

[Fabian]

New submission from Fabian:

While testing pywikibot using requests and urllib3 on Python 3.6 we got an interesting error:

======================================================================
ERROR: testQueryApiGetter (tests.wikidataquery_tests.TestApiSlowFunctions)
Test that we can actually retreive data and that caching works.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/xZise/pywikibot-core/tests/wikidataquery_tests.py", line 252, in testQueryApiGetter
    data = w.query(q)
  File "/home/travis/build/xZise/pywikibot-core/pywikibot/data/wikidataquery.py", line 601, in query
    data = self.getDataFromHost(fullQueryString)
  File "/home/travis/build/xZise/pywikibot-core/pywikibot/data/wikidataquery.py", line 563, in getDataFromHost
    resp = http.fetch(url)
  File "/home/travis/build/xZise/pywikibot-core/pywikibot/comms/http.py", line 359, in fetch
    error_handling_callback(request)
  File "/home/travis/build/xZise/pywikibot-core/pywikibot/comms/http.py", line 276, in error_handling_callback
    raise request.data
  File "/home/travis/build/xZise/pywikibot-core/pywikibot/comms/http.py", line 255, in _http_process
    auth=auth, timeout=timeout, verify=True)
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/adapters.py", line 337, in send
    conn = self.get_connection(request.url, proxies)
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/adapters.py", line 251, in get_connection
    conn = self.poolmanager.connection_from_url(url)
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py", line 139, in connection_from_url
    return self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py", line 125, in connection_from_host
    self.pools[pool_key] = pool
  File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/packages/urllib3/_collections.py", line 66, in __setitem__
    _key, evicted_value = self._container.popitem(last=False)
KeyError: ('https', 'eu.wiktionary.org', 443)

Now that doesn't make much sense, as OrderedDict.popitem() only returns a KeyError if it's empty but then not with that error message. Unfortunately I can't reproduce it without doing the complete pywikibot test suite. When I only execute the tests where it occurred I don't get any failures. So I modified the output in urllib3 and returned the key as well as the content before popitem(last=False) is called (I shortened the values):

Key: ('https', 'bn.wikipedia.org', 443)
Content: OrderedDict([(('https', 'bn.wikipedia.org', 443), <requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0x7fe483d991d0>), (('https', 'bs.wikipedia.org', 443), <…0x7fe483d99470>), (('https', 'ca.wikipedia.org', 443), <…0x7fe483da0e10>), (('https', 'cs.wikipedia.org', 443), <…0x7fe483dfc908>), (('https', 'da.wikipedia.org', 443), <…0x7fe483d98c88>), (('https', 'de.wikipedia.org', 443), <…0x7fe4911e06a0>), (('https', 'diq.wikipedia.org', 443), <…0x7fe483d6b400>), (('https', 'dsb.wikipedia.org', 443), <…0x7fe483d6bcf8>), (('https', 'en.wikipedia.org', 443), <…0x7fe483dc1da0>), (('https', 'eo.wikipedia.org', 443), <…0x7fe483d6cf28>), (('https', 'es.wikipedia.org', 443), <…0x7fe483d6b4a8>), (('https', 'fa.wikipedia.org', 443), <…0x7fe483d27278>), (('https', 'fi.wikipedia.org', 443), <…0x7fe483d6ce10>), (('https', 'fr.wikipedia.org', 443), <…0x7fe483e24cf8>), (('https', 'frr.wikipedia.org', 443), <…0x7fe483d39390>), (('https', 'ga.wikipedia.org', 443), <…0x7fe483c99668>), (('https', 'gl.wikipedia.org', 443), <…0x7fe483ca3f98>), (('https', 'als.wikipedia.org', 443), <…0x7fe483c4fcf8>), (('https', 'hu.wikipedia.org', 443), <…0x7fe483cd09e8>)])

As you can see it is not empty and the key in the KeyError is the first key in the OrderedDict. Also the key there is different from the key in the original exception I noticed so it's not a specific key that failed.

I don't think versions before Python 3.6 are affected as we had tests running on Python 3.5 (before Travis switched to 3.6 recently) and these all worked. Also not all popitem() calls in that line fail. I'm using Python 3.6.0a0 (default:d6c91b8242d2, Jul 18 2015, 16:36:01).

See also: https://github.com/shazow/urllib3/issues/680 and https://phabricator.wikimedia.org/T106212

----------
components: Library (Lib)
messages: 246937
nosy: xZise
priority: normal
severity: normal
status: open
title: OrderedDict.popitem() raises KeyError
versions: Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24667>
_______________________________________