deque and thread-safety

Christophe Vandeplas christophe at vandeplas.com
Fri Oct 12 08:40:21 EDT 2012


Hello,

I have a question about deque and thread-safety.

My application has multiple threads running concurrently and doing the
same action (downloading pages)
To know what has already been downloaded I created the variable:
  seen = deque('', 1000)   (keeps list of max 1000 urls in memory)

In one place of the code I do:  seen.append(url)

And another place:
def seenPage()
  if url in seen:
    return True
  return False

>From the documentation I understand that deques are thread-safe:
> Deques are a generalization of stacks and queues (the name is pronounced “deck”
> and is short for “double-ended queue”). Deques support thread-safe, memory
> efficient appends and pops from either side of the deque with approximately the
> same O(1) performance in either direction.

It seems that appending to deques is indeed thread-safe, but not
iterating over them.
I've seen a similar, but different, situation here:
http://comments.gmane.org/gmane.comp.python.devel/85487

Forgetting the above url, and considering my situation this behavior
screws up the concept I need:
- Keeping a thread-safe collection of seen urls,
- Being able to check if something is in that collection
- No need to clean the collection to prevent the memory from filling up

So I know I could work around this problem by using a lock.
But then I don't only need to use the lock around the iterator, but
also around the append(), but that defeats the purpose of deque being
thread-safe.

In short, what's your advice:

1/ build a lock around the .append() and the iterator. Using the
already-existing lock in the deque. But HOW?

1/ simply build a lock around the .append() and the iterator.
Defeating the build-in thread-safety.

2/ use another collection that does what I need


Thanks for your expertise.

Christophe



More information about the Python-list mailing list