[Python-Dev] Import lock considered mysterious

[P.J. Eby]

At 02:48 PM 7/22/2011 +0200, Antoine Pitrou wrote:

>See http://bugs.python.org/issue9260
>
>There's a patch there but it needs additional sophistication to remove
>deadlocks when doing concurrent circular imports.

I don't think that approach can work, as PEP 302 loaders can 
currently assume the global import lock is being held when they 
run...  and in general, there are too many global data structures in 
sys that need to be protected by code that uses such things.

A simpler solution to Greg's problem would be to have a timeout on 
attempts to acquire the import lock, and have it fail with a 
RuntimeError describing the problem.  (*Not* an ImportError, mind 
you, which might get ignored and trigger a different code path.)

The timeout would need to be on the order of seconds to prevent false 
positives, and there'd need to be a way to change or remove the 
timeout in the event somebody really needs to.  But it would 
eliminate the mysteriousness.  A unique and Google-able error message 
would let someone find a clear explanation of what's going on, as well.

A second thing that *could* be helpful would be to issue a warning 
when a new thread is started (or waited on) while the import lock is 
held.  This is already known to be a bad thing to do.

The tricky part is issuing the warning for the right caller level, 
but I suppose you could walk back up the call stack until you found 
some module-level code, and then fingered that line of code as the culprit.

Yes, that might do it: the code for starting or waiting on a thread, 
could check to see if the import lock is held by the current thread, 
and if so, walk up the stack to find a module frame (one where 
f_globals is f_locals and __name__ in f_locals and 
sys.modules[__name__].__dict__ is f_locals), and if one is found, 
issue a warning about not starting or waiting on threads in module-level code.

Between that and the timeout, the mysteriousness could be completely 
done away with, without throwing a monkey wrench into the current 
import mechanisms.