[Cryptography-dev] Minimal Test Case, bad interaction between "cryptography" and "wsgi" (Re: [modwsgi] mod_wsgi and creating new SSL connections inside the "body" iterator)

Paul Kehrer paul.l.kehrer at gmail.com
Thu Dec 3 21:58:02 EST 2015 

Can you reproduce this if you set:

WSGIApplicationGroup %{GLOBAL}

?

We're aware of a current issue with lock and library initialization related to Python subinterpreters (which mod_wsgi uses). Fixing this is...very hard, but https://github.com/pyca/cryptography/issues/2299 covers most of what we know right now.

-Paul (reaperhulk)

On December 3, 2015 at 8:54:23 PM, Jesus Cea (jcea at jcea.es) wrote:

Context for "Cryptography" people:  

Using "cryptography" and then trying to create a "ssl.SSLContext()"  
hangs the progress when running under mod_wsgi. I can not reproduce this  
issue with a standalone script, it must be run inside mod_wsgi. May be  
useful to know that, in daemon mode, mod_wsgi uses several threads to  
monitor health and do maintenance, beside the main wsgi application.  

Mail thread in the mod_wsgi mailing list in  
<https://groups.google.com/forum/#!topic/modwsgi/WFLao7kOqxE>.  


On 04/12/15 01:00, Graham Dumpleton wrote:  
> You don’t happen to be using PostgreSQL in the same web application  
> process are you?  

No. I have quite a few modules in my apache, but I am running WSGI in  
daemon mode.  

I have a minimal testcase. I can not reproduce this issue OUTSIDE of  
mod_wsgi, so it seems to be a bad interaction between mod_wsgi and  
"cryptography" library.  

The minimal test case, mod_wsgi compiled to use Python 3.5.0, is:  

"""  
from cryptography.hazmat.primitives.ciphers import Cipher  
from cryptography.hazmat.primitives.ciphers import algorithms as \  
Cipher_algorithms  
from cryptography.hazmat.primitives.ciphers import modes as \  
Cipher_modes  
from cryptography.hazmat.backends import default_backend as \  
Cipher_backend  

import ssl  

def application(environ, start_response) :  
cipher = Cipher(  
Cipher_algorithms.AES(b'e'*16),  
Cipher_modes.ECB(),  
backend = Cipher_backend())  

ssl.SSLContext(ssl.PROTOCOL_SSLv23) # <- HANGS HERE!!!  

crypto_iv = cipher.encryptor()  

start_response('200 OK', [])  
return [b'SUCCESS!\r\n']  
"""  

This code hangs the HTTP request in my machine.  

I am using Cryptography 1.1.1 and mod_wsgi 4.4.21. This server has  
OpenSSL 1.0.1. It is an Ubuntu 12.04 fully patched.  

A "thread apply all bt" in GDB shows this:  

"""  
Thread 19 (Thread 0x7fa947867700 (LWP 11497)):  
#0 0x00007fa952a3a6f3 in select () from /lib/x86_64-linux-gnu/libc.so.6  
#1 0x00007fa952f4d815 in apr_sleep (t=<optimized out>) at  
time/unix/time.c:246  
#2 0x00007fa94cb78ee0 in wsgi_monitor_thread (thd=<optimized out>,  
data=<optimized out>) at src/server/mod_wsgi.c:8658  
#3 0x00007fa952d13e9a in start_thread () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#4 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6  
#5 0x0000000000000000 in ?? ()  

Thread 18 (Thread 0x7fa9477e6700 (LWP 11498)):  
#0 0x00007fa952d180fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#1 0x00007fa94c792aff in take_gil () at Python/condvar.h:103  
#2 0x00007fa94c793099 in PyEval_RestoreThread () at Python/ceval.c:450  
#3 0x00007fa94c7ccb2b in PyGILState_Ensure () from  
/usr/local/lib/libpython3.5m.so.1.0  
#4 0x00007fa94cb7ac9a in wsgi_deadlock_thread (thd=<optimized out>,  
data=<optimized out>) at src/server/mod_wsgi.c:8468  
#5 0x00007fa952d13e9a in start_thread () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#6 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6  
#7 0x0000000000000000 in ?? ()  

Thread 17 (Thread 0x7fa947765700 (LWP 11499)):  
#0 0x00007fa952d17d84 in pthread_cond_wait@@GLIBC_2.3.2 () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#1 0x00007fa94cb8325d in wsgi_worker_acquire (id=0) at  
src/server/mod_wsgi.c:8044  
#2 wsgi_daemon_worker (thread=<optimized out>, p=0x1788588) at  
src/server/mod_wsgi.c:8166  
#3 wsgi_daemon_thread (thd=0x15cae30, data=<optimized out>) at  
src/server/mod_wsgi.c:8425  
#4 0x00007fa952d13e9a in start_thread () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#5 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6  
#6 0x0000000000000000 in ?? ()  

[QUITE A FEW THREADS WITH THE SAME STACK]  

Thread 3 (Thread 0x7fa947057700 (LWP 11513)):  
#0 0x00007fa952d180fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#1 0x00007fa94c792aff in take_gil () at Python/condvar.h:103  
#2 0x00007fa94c793099 in PyEval_RestoreThread () at Python/ceval.c:450  
#3 0x00007fa94c7ccb2b in PyGILState_Ensure () from  
/usr/local/lib/libpython3.5m.so.1.0  
#4 0x00007fa9363c51d1 in invoke_callback ()  
from  
/usr/local/lib/python3.5/site-packages/_cffi_backend.cpython-35m-x86_64-linux-gnu.so  
#5 0x00007fa9361aa819 in ffi_closure_unix64_inner () from  
/usr/lib/x86_64-linux-gnu/libffi.so.6  
#6 0x00007fa9361aab7c in ffi_closure_unix64 () from  
/usr/lib/x86_64-linux-gnu/libffi.so.6  
#7 0x00007fa94b7585e5 in ?? () from  
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0  
#8 0x00007fa94b7583c8 in ?? () from  
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0  
#9 0x00007fa94b76efbb in EC_KEY_generate_key () from  
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0  
#10 0x00007fa94bab3998 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.0.0  
#11 0x00007fa94474d6d5 in _ssl__SSLContext () from  
/usr/local/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so  
#12 0x00007fa94c7190a4 in tp_new_wrapper () from  
/usr/local/lib/libpython3.5m.so.1.0  
#13 0x00007fa94c7023e9 in PyCFunction_Call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#14 0x00007fa94c79d04d in PyEval_EvalFrameEx () at Python/ceval.c:4651  
#15 0x00007fa94c79dab6 in _PyEval_EvalCodeWithName () at Python/ceval.c:3962  
#16 0x00007fa94c79dba8 in PyEval_EvalCodeEx () at Python/ceval.c:3983  
#17 0x00007fa94c6df076 in function_call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#18 0x00007fa94c6ade0a in PyObject_Call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#19 0x00007fa94c714ab5 in slot_tp_new () from  
/usr/local/lib/libpython3.5m.so.1.0  
#20 0x00007fa94c7191e3 in type_call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#21 0x00007fa94c6ade0a in PyObject_Call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#22 0x00007fa94c79711e in PyEval_EvalFrameEx () at Python/ceval.c:4883  
#23 0x00007fa94c79dab6 in _PyEval_EvalCodeWithName () at Python/ceval.c:3962  
#24 0x00007fa94c79dba8 in PyEval_EvalCodeEx () at Python/ceval.c:3983  
#25 0x00007fa94c6df076 in function_call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#26 0x00007fa94c6ade0a in PyObject_Call () from  
/usr/local/lib/libpython3.5m.so.1.0  
#27 0x00007fa94c793787 in PyEval_CallObjectWithKeywords () at  
Python/ceval.c:4526  
#28 0x00007fa94cb81822 in Adapter_run (object=0x7fa946f16ea0,  
self=0x7fa944723210) at src/server/mod_wsgi.c:2897  
#29 wsgi_execute_script (r=0x7fa940004980) at src/server/mod_wsgi.c:3726  
#30 0x00007fa94cb83c69 in wsgi_hook_daemon_handler (c=<optimized out>)  
at src/server/mod_wsgi.c:12236  
#31 wsgi_process_socket (bucket_alloc=0x7fa940006e40, sock=<optimized  
out>, p=0x7fa940005e90, daemon=<optimized out>)  
at src/server/mod_wsgi.c:8007  
#32 wsgi_daemon_worker (thread=<optimized out>, p=0x17a7638) at  
src/server/mod_wsgi.c:8336  
#33 wsgi_daemon_thread (thd=0x1795368, data=<optimized out>) at  
src/server/mod_wsgi.c:8425  
#34 0x00007fa952d13e9a in start_thread () from  
/lib/x86_64-linux-gnu/libpthread.so.0  
#35 0x00007fa952a4138d in clone () from /lib/x86_64-linux-gnu/libc.so.6  
#36 0x0000000000000000 in ?? ()  

[MORE BORING THREADS]  

Thread 1 (Thread 0x7fa9537c2700 (LWP 11094)):  
#0 0x00007fa952a359d3 in poll () from /lib/x86_64-linux-gnu/libc.so.6  
#1 0x00007fa952f49563 in apr_poll (aprset=0x7fffca6fd500, num=1,  
nsds=0x7fffca6fd55c, timeout=<optimized out>)  
at poll/unix/poll.c:120  
#2 0x00007fa94cb859ab in wsgi_daemon_main (daemon=0x15ca3d8,  
p=0x159a138) at src/server/mod_wsgi.c:8944  
#3 wsgi_start_process (p=0x159a138, daemon=0x15ca3d8) at  
src/server/mod_wsgi.c:9665  
#4 0x00007fa94cb87494 in wsgi_start_daemons (p=0x159a138) at  
src/server/mod_wsgi.c:9877  
#5 0x00000000004370f0 in ap_run_pre_mpm ()  
#6 0x000000000048454c in worker_run ()  
#7 0x0000000000433618 in ap_run_mpm ()  
#8 0x000000000042b340 in main ()  
"""  

It is looks like some kind of callback is registered by "cryptography"  
and when calling "ssl.SSLContext()" that callback tries to get the GIL  
when it already has it, hanging. My speculation, I am not familiar with  
the inner working of OpenSSL neither "cryptography", but the traceback  
of "Thread 3" looks quite suspicious.  

In recent Python releases, the GIL is a NOP if not threads are created.  
Lets try something... Lets create another idle thread.  

"""  
from cryptography.hazmat.primitives.ciphers import Cipher  
from cryptography.hazmat.primitives.ciphers import algorithms as \  
Cipher_algorithms  
from cryptography.hazmat.primitives.ciphers import modes as \  
Cipher_modes  
from cryptography.hazmat.backends import default_backend as \  
Cipher_backend  

import ssl, threading, time  

def dummy() :  
cipher = Cipher(  
Cipher_algorithms.AES(b'e'*16),  
Cipher_modes.ECB(),  
backend = Cipher_backend())  

ssl.SSLContext(ssl.PROTOCOL_SSLv23)  

crypto_iv = cipher.encryptor()  

while True :  
pass  

t = threading.Thread(target = dummy)  
t.setDaemon(True)  
t.start()  
print("SSSS")  

time.sleep(2)  

print("OK!")  
"""  

This code works right. I can only reproduce the issue inside mod_wsgi.  

--  
Jesús Cea Avión _/_/ _/_/_/ _/_/_/  
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/  
Twitter: @jcea _/_/ _/_/ _/_/_/_/_/  
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/  
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/  
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/  
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz  

_______________________________________________  
Cryptography-dev mailing list  
Cryptography-dev at python.org  
https://mail.python.org/mailman/listinfo/cryptography-dev  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20151203/0c4c4939/attachment-0001.html>

More information about the Cryptography-dev mailing list