Dealing with ImportLock deadlock in Import Hooks

[Arnaud Fontaine]

Hello,

I'm currently working on implementing Import Hooks (PEP302) with Python
2.7 to be able to import modules whose code is in ZODB. However, I have
stumbled upon a widely known issue about import deadlock[0][1] (note
that this issue is not directly related to ZODB, but a more general
question about dealing with ImportLock deadlock for Import Hooks),
basically:

  Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a
  package containing dynamic modules) handled by Import Hooks I
  implemented, so ImportLock is acquired before even running the hooks
  (Python/import.c:PyImport_ImportModuleLevel()). Then, these import
  hooks try to load objects from ZODB and a request is sent and handled
  by another thread (Thread 2) which itself tries to import another
  module. Of course, this causes a deadlock because the first thread
  still holds ImportLock.

I have thought about the following solutions:

* Backport the patch applied in python 3.3 from issue 9260[0]. This
  would be the best option because it would mean that even when trying
  to import any module from package 'foo', other modules and packages
  can be imported, which would solve my issue. However, I'm not sure it
  could be released into python 2.7?

* Within the hooks, protect the Import Hooks with a separate lock for
  the loader method. This would prevent any other thread to import any
  modules from 'foo' package but still allows to call the finder method
  (ignoring module fullname not starting with 'foo') along with other
  finder methods, so that other ZODB modules can be imported.

  Then, in the loader method, until the module is actually inserted into
  sys.modules and then other load_module() PEP302 responsabilities being
  taken care of (such as exec the code), release the ImportLock so that
  Thread 2 can process requests and send objects back to Thread 1.

  However, even after trying to understand import.c, I'm not sure at all
  that this is enough and that releasing ImportLock would not have nasty
  side-effects, any thoughts about that?

* Fix the code in ZODB to not avoid import but to me this seems like a
  dirty hack because it could happen again and I would prefer to fix
  this issue once and for all.

Any thoughts or suggestion welcome, thanks!

Regards,
-- 
Arnaud Fontaine

[0] http://bugs.python.org/issue9260
[1] http://mail.python.org/pipermail/python-dev/2003-February/033436.html