Mutable global state and threads

[Chris Angelico]

On Wed, Jan 4, 2017 at 5:41 PM, Kev Dwyer <kevin.p.dwyer at gmail.com> wrote:
> Hello List,
>
> I came across some threading code in Some Other place recently and wanted to
> sanity-check my assumptions.
>
> The code (below) creates a number of threads; each thread takes the last
> (index -1) value from a global list of integers, increments it by one and
> appends the new value to the list.
>
> The originator of the code expected that when all the threads completed, the
> list would be an ascending sequence of integers, for example if the original
> list was [0] and two threads mutated it twice each, the final state would be
> [0, 1, 2, 3, 4].
>
> Here is a version of the code (slightly simplified and modified to allow
> changing the number of threads and mutations).
>
>
> import sys
> import threading
>
>
> class myThread(threading.Thread):
>
>     def __init__(self, nmutations):
>         threading.Thread.__init__(self)
>         self.nmutations = nmutations
>
>     def run(self):
>         mutate(self.nmutations)
>         # print (L)
>         return
>
> def mutate(nmutations):
>     n = nmutations
>     while n:
>         L.append(L[-1 ]+ 1)
>         n -= 1
>     return
>
>
> def main(nthreads=2, nmutations=2):
>     global L
>     L = [0]
>     threads = [myThread(nmutations) for i in range(nthreads)]

You can drop the myThread class and instead instantiate threading.Thread 
directly:

threads = [threading.Thread(target=mutate, args=(nmutations,))
    for i in range(nthreads)]
> Firstly, is it true that the statement
>
> L.append(L[-1 ]+ 1)
>
> is not atomic, that is the thread might evaluate L[-1] and then yield,
> allowing another thread to mutate L, before incrementing and appending?

That is indeed true. If the code were all run sequentially (setting nthreads to 
1), the last element in the final list would be equal to nthreads*nmutations, 
and you can mess around with the numbers to find out exactly how far short it 
is.

Python guarantees that certain primitive operations (such as the list append 
itself) won't be broken, but anything that involves application code can be 
split up. So you can be confident that you'll end up with a list of integers 
and not a segfault, but they might not even be consecutive.

> Secondly, the original code printed the list at the end of a thread's run
> method to examine the state of the list.  I don't think this would work
> quite as expected, because the thread might yield after mutating the list
> but before printing, so the list could have been mutated before the print
> was executed.  Is there a way to display the state of the list before any
> further mutations take place?

At the end of one thread's run, there are other threads still running. So 
you're correct again; another thread could change the list. What you could 
possibly do mess with locking, but before I advise that, I'd have to see what 
you're trying to accomplish - toy examples are hard to mess with. Bear in mind 
that simple code like this can't actually benefit from threads, as Python (or 
at least, CPython) won't run two of them at once.

> (Disclaimer: I understand that sanity, mutable global state and threads are
> unlikely bedfellows and so promise never to try anything like this in
> production code).

Well, I lost my sanity shortly after becoming a programmer, so I've happily 
used mutable globals in threaded programs. :) You can use them quite happily as 
long as you know what you're doing. But if you find yourself tossing in heaps 
of locks to try to reclaim your sanity, you may want to consider asyncio 
instead. Instead of locking and unlocking to say "don't yield here", you 
explicitly say "yield here". The downside is that you have to have *everything* 
cope with that - and not everything can. (For a long time, it was easy to 
establish a socket connection asynchronously, but hard to look up 
"www.python.org" without potentially stalling.) They're two models that can 
both be used to solve a lot of the same problems.

By and large, your analysis is correct. Have fun threading! :)

ChrisA