how to write thread-safe module ? and pytz

[Stuart Bishop]

nicolas_riesch wrote:
> Does someone know if the module pytz
> (http://sourceforge.net/projects/pytz/) is thread-safe ?
> I have not seen it explicitely stated, and just wanted to be sure, as I
> want to use it.

pytz is thread safe.

> That's because in the file pytz/tzinfo.py, I see global variables
> _timedelta_cache, _datetime_cache, _ttinfo_cache, which are
> dictionnaries and are used as cache.

> I always thought that you must protect shared data with locks when
> multithreading, but I don't see any lock anywhere in pytz.
> However, pytz seems to work well with multiple threads creating various
> timezone objects at the same time.
> I don't understand where the trick is, that allows multiple threads to
> access this module without any locking and that all this seems to work
> without any problem...

Thanks to the global interpreter lock, with the Python builtin types you
only need to maintain a lock if there is a race condition, or if you care
about the race condition. For example, the following is thread safe code:

>>> from threading import Thread
>>> import time
>>> stack = []
>>> stack2 = []
>>> def doit(i):
...     stack.append(i)
...     time.sleep(0.1)
...     stack2.append(stack.pop())
...
>>> threads = [Thread(target=doit, args=(i,)) for i in range(0,100)]
>>> for t in threads: t.start()
...
>>> for t in threads: t.join()
...
>>> len(stack2)
100
>>> stack2
[99, 95, 98, 94, 93, 97, 92, 91, 96, 88, 87, 86, 85, 84, 83, 90, 79, 78, 77,
76, 74, 73, 72, 71, 70, 75, 82, 81, 80, 89, 69, 67, 66, 65, 64, 68, 60, 59,
58, 57, 56, 55, 63, 62, 61, 49, 54, 53, 52, 51, 46, 45, 44, 50, 48, 47, 29,
28, 35, 34, 33, 43, 42, 41, 40, 39, 38, 32, 37, 31, 30, 36, 27, 26, 25, 24,
23, 22, 21, 20, 19, 18, 17, 12, 16, 15, 14, 13, 11, 10, 9, 8, 7, 6, 4, 3, 2,
1, 0, 5]

Note that the value being appended to 'stack2' might not be the value that
was appended to 'stack' in any particular thread - in this case, we don't
care (but is the sort of thing you might need to watch out for).

In the code you mention in pytz, there *is* a race condition. However, if
this condition occurs the side effects are so trivial as to not worry about
locking. ie. if a number of threads call memorized_timedelta(seconds=60)
simultaneously, there is a slight chance that each thread will get a
different timedelta instance. This is extremely unlikely, and the rest of
the code doesn't care at all. If pytz compared the timedeltas using 'is'
instead of '==' at any point, it would be a bug (but it doesn't, so it isn't).

So you can write thread safe Python code without locks provided you are
using the builtin types, and keep a close eye out for race conditions. This
might sound error prone, but it is quite doable provided the critical areas
that are accessing shared objects are kept isolated, short and simple.

Here is an thread unsafe example. Here the mistake is made that the length
of stack will not change after checking it. Also because we don't use the
atomic stack.pop(), two threads might add the same value to stack2:

>>> from threading import Thread
>>> import time
>>> stack = range(0, 50)
>>> stack2 = []
>>> def doit():
...     if len(stack) > 0:
...         stack2.append(stack[-1])
...         time.sleep(0.1)
...         del stack[-1]
...
>>> threads = [Thread(target=doit) for i in range(0, 100)]
>>> for t in threads: t.start()
...
Exception in thread Thread-249:
Traceback (most recent call last):
  File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/usr/lib/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<stdin>", line 5, in doit
IndexError: list assignment index out of range



-- 
Stuart Bishop <stuart at stuartbishop.net>
http://www.stuartbishop.net/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20050814/3809faf0/attachment.sig>