[issue36755] [2.7] test_httplib leaked [8, 8, 8] references with OpenSSL 1.1.1

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

AMD64 Fedora Rawhide Refleaks 2.7 buildbot:
https://buildbot.python.org/all/#/builders/190/builds/18

test_httplib leaked [8, 8, 8] references, sum=24


When I run the test on my Fedora 29 ("OpenSSL 1.1.1b FIPS  26 Feb 2019"), I can reproduce leak:

$ ./python -m test -R 3:3 -m test.test_httplib.HTTPSTest.test_local_bad_hostname test_httplib
...
test_httplib leaked [8, 8, 8] references, sum=24
...


My bet is that the issue is related to OpenSSL 1.1.1 which changes how a TLS connection is terminated.


Running the test in verbose mode logs a traceback:

$ ./python -m test -v test_httplib 
...
test_local_bad_hostname (test.test_httplib.HTTPSTest) ... ----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 48554)
Traceback (most recent call last):
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 293, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 655, in __init__
    self.handle()
  File "/home/vstinner/prog/python/2.7/Lib/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/home/vstinner/prog/python/2.7/Lib/BaseHTTPServer.py", line 310, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/home/vstinner/prog/python/2.7/Lib/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/home/vstinner/prog/python/2.7/Lib/ssl.py", line 754, in recv
    return self.read(buflen)
  File "/home/vstinner/prog/python/2.7/Lib/ssl.py", line 641, in read
    v = self._sslobj.read(len)
error: [Errno 104] Connection reset by peer
----------------------------------------
 server (('127.0.0.1', 44923):44923 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [30/Apr/2019 02:40:01] code 404, message File not found
 server (('127.0.0.1', 44923):44923 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [30/Apr/2019 02:40:01] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok


Without -v, the test fails with:

vstinner at apu$ ./python -m test test_httplib
Run tests sequentially
0:00:00 load avg: 0.63 [1/1] test_httplib
Traceback (most recent call last):
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 293, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/vstinner/prog/python/2.7/Lib/SocketServer.py", line 655, in __init__
    self.handle()
  File "/home/vstinner/prog/python/2.7/Lib/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/home/vstinner/prog/python/2.7/Lib/BaseHTTPServer.py", line 310, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/home/vstinner/prog/python/2.7/Lib/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/home/vstinner/prog/python/2.7/Lib/ssl.py", line 754, in recv
    return self.read(buflen)
  File "/home/vstinner/prog/python/2.7/Lib/ssl.py", line 641, in read
    v = self._sslobj.read(len)
error: [Errno 104] Connection reset by peer
test test_httplib produced unexpected output:
**********************************************************************
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56044)
----------------------------------------

**********************************************************************

== Tests result: FAILURE ==

1 test failed:
    test_httplib

Total duration: 281 ms
Tests result: FAILURE


My attempt to fix the warning:

diff --git a/Lib/BaseHTTPServer.py b/Lib/BaseHTTPServer.py
index 3df3323a97..8fe29e9d3e 100644
--- a/Lib/BaseHTTPServer.py
+++ b/Lib/BaseHTTPServer.py
@@ -332,6 +332,12 @@ class BaseHTTPRequestHandler(SocketServer.StreamRequestHandler):
             self.log_error("Request timed out: %r", e)
             self.close_connection = 1
             return
+        except socket.error as exc:
+            # Using ssl and OpenSSL 1.1.1, sometimes readline() can fail
+            # with error(104, 'Connection reset by peer')
+            self.close_connection = 1
+            exc = None
+            return
 
     def handle(self):
         """Handle multiple requests if necessary."""

----------
components: Tests
messages: 341127
nosy: vstinner
priority: normal
severity: normal
status: open
title: [2.7] test_httplib leaked [8, 8, 8] references with OpenSSL 1.1.1
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36755>
_______________________________________