Mutable global state and threads

[Chris Angelico]

On Wed, Jan 4, 2017 at 5:41 PM, Kev Dwyer <kevin.p.dwyer at gmail.com> wrote:
> Hello List,
>
> I came across some threading code in Some Other place recently and wanted to
> sanity-check my assumptions.
>
> The code (below) creates a number of threads; each thread takes the last
> (index -1) value from a global list of integers, increments it by one and
> appends the new value to the list.
>
> The originator of the code expected that when all the threads completed, the
> list would be an ascending sequence of integers, for example if the original
> list was [0] and two threads mutated it twice each, the final state would be
> [0, 1, 2, 3, 4].
>
> Here is a version of the code (slightly simplified and modified to allow
> changing the number of threads and mutations).
>
>
> import sys
> import threading
>
>
> class myThread(threading.Thread):
>
>     def __init__(self, nmutations):
>         threading.Thread.__init__(self)
>         self.nmutations = nmutations
>
>     def run(self):
>         mutate(self.nmutations)
>         # print (L)
>         return
>
> def mutate(nmutations):
>     n = nmutations
>     while n:
>         L.append(L[-1 ]+ 1)
>         n -= 1
>     return
>
>
> def main(nthreads=2, nmutations=2):
>     global L
>     L = [0]
>     threads = [myThread(nmutations) for i in range(nthreads)]

You can drop the myThread class and instead instantiate
threading.Thread directly:

threads = [threading.Thread(target=mutate, args=(nmutations,))
    for i in range(nthreads)]
> Firstly, is it true that the statement
>
> L.append(L[-1 ]+ 1)
>
> is not atomic, that is the thread might evaluate L[-1] and then yield,
> allowing another thread to mutate L, before incrementing and appending?

That is indeed true. If the code were all run sequentially (setting
nthreads to 1), the last element in the final list would be equal to
nthreads*nmutations, and you can mess around with the numbers to find
out exactly how far short it is.

Python guarantees that certain primitive operations (such as the list
append itself) won't be broken, but anything that involves application
code can be split up. So you can be confident that you'll end up with
a list of integers and not a segfault, but they might not even be
consecutive.

> Secondly, the original code printed the list at the end of a thread's run
> method to examine the state of the list.  I don't think this would work
> quite as expected, because the thread might yield after mutating the list
> but before printing, so the list could have been mutated before the print
> was executed.  Is there a way to display the state of the list before any
> further mutations take place?

At the end of one thread's run, there are other threads still running.
So you're correct again; another thread could change the list. What
you could possibly do mess with locking, but before I advise that, I'd
have to see what you're trying to accomplish - toy examples are hard
to mess with. Bear in mind that simple code like this can't actually
benefit from threads, as Python (or at least, CPython) won't run two
of them at once.

> (Disclaimer: I understand that sanity, mutable global state and threads are
> unlikely bedfellows and so promise never to try anything like this in
> production code).

Well, I lost my sanity shortly after becoming a programmer, so I've
happily used mutable globals in threaded programs. :) You can use them
quite happily as long as you know what you're doing. But if you find
yourself tossing in heaps of locks to try to reclaim your sanity, you
may want to consider asyncio instead. Instead of locking and unlocking
to say "don't yield here", you explicitly say "yield here". The
downside is that you have to have *everything* cope with that - and
not everything can. (For a long time, it was easy to establish a
socket connection asynchronously, but hard to look up "www.python.org"
without potentially stalling.) They're two models that can both be
used to solve a lot of the same problems.

By and large, your analysis is correct. Have fun threading! :)

ChrisA