[Python-Dev] Design Philosophy: Performance vs Robustness/Maintainability

[Antoine Pitrou]

On Tue, 18 Jul 2017 09:08:08 -0700
Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> Nick Coughlan:
> -------------

It is "Nick Coghlan" not "Coughlan".

> > As another example of this: while trading the global import lock for
> > per-module locks eliminated most of the old import deadlocks, it turns
> > out that it *also* left us with some fairly messy race conditions and
> > more fragile code (I still count that particular case as a win
> > overall, but it definitely raises the barrier to entry for maintaining
> > that code).
> >
> > Unfortunately, these are frequently cases where the benefits are
> > immediately visible (e.g. faster benchmark results, removing
> > longstanding limitations on user code), but the downsides can
> > literally take years to make themselves felt (e.g. higher defect rates
> > in the interpreter, subtle bugs in previously correct user code that
> > are eventually traced back to interpreter changes).  

I'll reply here again: the original motivation for the per-module
import lock was not performance but correctness.

The import deadlocks were really in the category of "subtle bugs" that
only occur in certain timing conditions (especially when combined with
PyImport_ImportModuleNoBlock and/or stdlib modules which can try to
import stuff silently, such as the codecs module). So we traded a
category of "subtle bugs" due to a core design deficiency for another
category of "subtle bugs" due to an imperfect implementation, the
latter being actually fixable incrementally :-)

Disclaimer: I wrote the initial per-module lock implementation.

Regards

Antoine.