Threading

[Thomas Jensen]

David Bolen <db3l at fitlinxx.com> wrote in
news:ur8r024it.fsf at ctwd0143.fitlinxx.com: 

> Thomas Jensen <thomasNO at SPAM.obscure.dk> writes:

> When in doubt, that's the right approach.  You might be able to
> eeek a little more performance out of hand tuning some "safe"
> scenarios, but I don't think the potential downsides are worth it
> in general. 

I guess you're right :-)

>> Is there any "rule-of-thumb" (or doc) regarding which operations
>> are atomic, for example is the following safe: 
> 
> An operation would be atomic with respect to Python interpreter
> threads if it executes while holding onto the GIL (global
> interpreter lock).  Generally speaking this must (a) be something
> written in C, either as part of the interpreter or an extension
> module, and (b) not release the GIL to provide for threading
> execution around a long running or blocked activity, such as I/O.

Ahh, the GIL, thought I'd heard about such a beast.

> Generally, it's just safer to lock more conservatively.

I'll do that.

>> thread 1:
>> mylist.reverse()
>> 
>> thread 2:
>> for item in mylist: ...
>> 
>> (silly example perhaps)
> 
> This doesn't appear as it would be safe if the goal of thread 2 was
> to iterate through every item in the list consistently.  If mylist

That would most probably be the goal.

> was a builtin list type object (and at least in the CPython
> implementation) then you are guaranteed that once it begins, the
> reverse() method call in thread 1 will run to completion before any
> other Python thread continues execution.  What you aren't
> guaranteed is how that reverse call's timing will occur with
> respect to thread 2. 
> 
> So if thread 1 managed to transfer control to the reverse() method
> prior to thread 2 hitting that for loop, you'd get a consistent
> (reverse) list.
> 
> But if thread 1 didn't make it into the call until thread 2 was
> partially through the loop, then the list would be reversed, and it
> would be equivalent to having Python code that mutated the list
> within the for loop, in which case behavior is non-deterministic
> (at least in general, since it depends on the type of mutation).
> 
> Since the mutation in this case keeps the list the same length but
> just reverses the contents, I expect you'd have entries processed
> in the for loop up until the thread context switch, and then you'd
> get the entries in the remaining positions of the list but for the
> reversed copy.

That would probably not be the goal.

> I'd definitely protect something like this - you've clearly got
> higher level processing (the iteration over the list) that needs
> the list to remain intact during the work.  Either that, or protect
> the act of making a copy of the list, and then iterate over that
> copy.  The copying is one of those potential tuning points - I'd
> have to double check, but I bet that an operation like "newlist =
> mylist[:]" would be atomic, so you could probably get away with not
> protecting that.  At least until mylist happened to end up a user
> class with its own __getslice__.

Thank you for the detailed explanation, I guess the reason I asked was 
that I didn't quite know how the GIL worked.
I thought that perhaps the for loop would lock the specific list for 
modification. The more I think about it, the more unreasonable it 
seems. Especially since the lock is global, it would not only block 
access to the list.

I'll keep up the paranoia :-)

-- 
Best Regards
Thomas Jensen