deque and thread-safety

[Christophe Vandeplas]

Ok sorry for the mail,
I found the solution by using deque.count(url) that's available
starting from python 2.7


On Fri, Oct 12, 2012 at 2:40 PM, Christophe Vandeplas
<christophe at vandeplas.com> wrote:
> Hello,
>
> I have a question about deque and thread-safety.
>
> My application has multiple threads running concurrently and doing the
> same action (downloading pages)
> To know what has already been downloaded I created the variable:
>   seen = deque('', 1000)   (keeps list of max 1000 urls in memory)
>
> In one place of the code I do:  seen.append(url)
>
> And another place:
> def seenPage()
>   if url in seen:
>     return True
>   return False
>
> From the documentation I understand that deques are thread-safe:
>> Deques are a generalization of stacks and queues (the name is pronounced “deck”
>> and is short for “double-ended queue”). Deques support thread-safe, memory
>> efficient appends and pops from either side of the deque with approximately the
>> same O(1) performance in either direction.
>
> It seems that appending to deques is indeed thread-safe, but not
> iterating over them.
> I've seen a similar, but different, situation here:
> http://comments.gmane.org/gmane.comp.python.devel/85487
>
> Forgetting the above url, and considering my situation this behavior
> screws up the concept I need:
> - Keeping a thread-safe collection of seen urls,
> - Being able to check if something is in that collection
> - No need to clean the collection to prevent the memory from filling up
>
> So I know I could work around this problem by using a lock.
> But then I don't only need to use the lock around the iterator, but
> also around the append(), but that defeats the purpose of deque being
> thread-safe.
>
> In short, what's your advice:
>
> 1/ build a lock around the .append() and the iterator. Using the
> already-existing lock in the deque. But HOW?
>
> 1/ simply build a lock around the .append() and the iterator.
> Defeating the build-in thread-safety.
>
> 2/ use another collection that does what I need
>
>
> Thanks for your expertise.
>
> Christophe