[Python-Dev] Import lock considered mysterious

[Greg Ewing]

I recently encountered a very mysterious bug in which
a background thread would inexplicably hang while attempting
to make a connection using httplib.

Debugging as far as I could at the Python level led to
the surprising conclusion that it was hanging while
using the ascii codec to encode the host name.

I couldn't imagine why, until I resorted to breaking into
it with gdb and found that it was trying to import
"codecs.ascii", and blocked on acquiring the import lock.

The reason for *that* was that my main module was a stub
that imported the real main module, which did all its
work directly from the module code. So the whole program
was effectively running inside an import statement and
holding onto the import lock.

Once I realised what was happening, it was easy to fix,
but I'm a bit disturbed about how difficult it was to
track down the cause.

This whole episode seems to have resulted from a collision
between two rather obscure things: that import statements
involve locking things, and that some fairly innocuous
looking calls, such as s.encode('ascii'), can trigger an
import.

I'm wondering whether anything can be done to make problems
like this less likely to occur, or at least to make the
cause more readily apparent. One shouldn't really have to
use gdb to track down bugs in Python code.

Is it really necessary to hold the import lock for so long?
Presumably the import lock is there to protect access to
things like sys.modules. Is that all? Could it be released
after the module code is loaded and sys.modules has been
updated, but before executing the module body?

-- 
Greg