The tragic tale of the deadlocking Python queue

[Chris Angelico]

On Thu, Aug 17, 2017 at 6:31 PM,  <breamoreboy at gmail.com> wrote:
> I found it interesting, possibly some of you may feel the same way so here it is https://codewithoutrules.com/2017/08/16/concurrency-python/

Starts out with a false premise.

"""
Writing programs with concurrency, programs with multiple threads, is
hard. Without threads code is linear: line 2 is executed after line 1,
with nothing happening in between. Add in threads, and now changes can
happen behind your back.
"""

With nothing happening in between. Except for (asynchronous) signal
handlers, cyclic garbage collection, and a whole lot of other things
that can already happen in between otherwise-adjacent pieces of code.
And that's without even considering that other processes in the system
are doing things.

Edit: Oh, turns out the author knows this. Okay then. Criticism withdrawn.

The fact remains, though, that most of the problems listed come from
improper use of mutable global state. The article would be a lot more
factual (but a lot shorter and less emotive) if it just stuck to its
main point, which is about how __del__ is asynchronous.

Okay, so we get down to one thing: locks acquired in __del__ are
dangerous. And I'd say that yes, they really should be. The hard
problem is not "how can we make a re-entrant queue", but "how can we
avoid the use of locks inside __del__". The solution, in my opinion,
is not to use queues with ResourceWarning (which might mean handling
ResourceWarning specially, or handling all warnings differently).

Concurrent programming is not as hard as a lot of people make out.
It's just an extension of otherwise well-known good programming habits
like being careful of mutable global state. The only difference is
that, with single-threaded code, you have all these implicit locks
around your functions, so you can pretend that things are atomic when
they're not.

ChrisA