Seeking advice on locking iterators

[Gonçalo Rodrigues]

On Fri, 15 Nov 2002 15:12:15 +0000, Gonçalo Rodrigues
<op73418 at mail.telepac.pt> wrote:

>On Thu, 14 Nov 2002 19:29:49 +0100, Ype Kingma <ykingma at accessforall.nl>
>wrote:
>
>>Gonçalo Rodrigues schreef:
>>
>>> Hi,
>>> 
>>> My problem is the following: I have a top object Application, roughly
>>> corresponding to the main thread, that spawns some child threads -
>>> objects on their own sake, deriving from the Thread class in the
>>> threading module. They communicate when they need to via queues in the
>>> usual way. So far so nice.
>>> 
>>> The problem is that those same child objects expose some public
>>> attributes (or properties, or methods, does not matter) that the
>>> Application object may need to change/call and doing the change via the
>>> queue mechanism is conceptually wrong (and a real PITA). The solution is
>>> obvious: wrap the attributes with locks around it. What is not so
>>> obvious is when these attributes are complex objects themselves, in
>>> particular they are iterables. In order to call iter() on these objects
>>> safely I coded the following helper class:
>>> 
>>> #WARNING: Brittle - use this at your own risk.
>>> class TIter(object):
>>>     """The TIter helper class, wrapping an iterator for thread safe
>>> acess."""
>>> 
>>>     def __init__(self, lock, iterator):
>>>         super(TIter, self).__init__(lock, iterator)
>>>         self.__lock = lock
>>>         self.__iterator = iter(iterator)
>>>         #Acquire lock => You cannot change the underlying structure
>>>         #while iter not exhausted.
>>>         self.__lock.acquire()
>>>         self.__acquired = 1
>>> 
>>>     #Iterator protocol.
>>>     def __iter__(self):
>>>         return self
>>> 
>>>     def next(self):
>>>         try:
>>>             ret = self.__iterator.next()
>>>         except StopIteration:
>>>             self.__lock.release()
>>>             self.__acquired = 0
>>>             raise StopIteration
>>>         else:
>>>             return ret
>>> 
>>>     def __del__(self):
>>>         if self.__acquired:
>>>             self.__lock.release()
>>> 
>>> The idea is that for the wrapping of a given attribute one has an
>>> associated lock (a reentrant lock, usually) that is passed to the ITer
>>> constructor. I have to ensure two things, though:
>>> 
>>> * The lock is acquired while the iterator is alive.
>>> * The lock is released when the iterator is disposed of.
>>> 
>>> So, finally, my question is: Is the above enough to ensure this? Can I
>>
>>In general, no. Some reasons:
>>One can never be sure from the code shown when the __del__ method will be 
>>called, since this depends on when the last reference to the iterator goes 
>>out of scope.
>>Although CPython more or less guarantees that the __del__ method will be 
>>called quite soon after the last reference goes out of scope, the language 
>>by itself does not make such a guarantee. In Jython, you are at the mercy
>>of a separate a garbage collecting thread in the JVM to call the __del__ 
>>method, ie. it might take more than a few million CPU cycles before the 
>>__del__ method is called.
>>
>>> be sure that __del__ is called if the iterator is garbage collected
>>> before being exhausted? Or is there a better (e.g. safer) mechanism for
>>> this?
>>
>>
>>One safe way is:
>>
>>yourObject.lock()
>>try:
>>    # use an iterator on yourObject.
>>finally:
>>    yourObject.unlock()
>>    # and don't use the iterator again.
>>
>>Ie. I don't see the point of locking the iterator, as you seem to want to
>>lock an object, ie. one the attributes you mentioned above.
>>
>
>I want to lock the iterator because I want the wrapped object to have
>the same interface as the object it wraps.
>
>>The lock() and unlock() methods need to be added to the class of yourObject 
>>for this to work.
>
>Anyway I think I have found a solution: The problem lies in the fact
>that the object I am iterating over and the iterator itself can be
>different objects => while the iterator acquires the lock it may get
>garbage collected before it releases it. Therefore I just wrap the
>object in a way that the wrapper is its own iterator and the problem
>disappears.
>>

Actually the problem does not disappear completely since the iterator
itself may still get confused if some thread in the middle of some
iterating decides to alter the structure. A bit like the
altering-the-structure-of-an-iterable-while-iterating-over-it problem.
And for *that reason* I do have to add lock/unlock methods. Oh well.
Just goes to show the importance of the Queue module when dealing with
threads...

>>Have fun,
>
>I am. Python *is* fun.
>
>>Ype
>
>With my best regards,
>G. Rodrigues

With my best regards,
G. Rodrigues