Persistent Queue implementations?

[Aahz]

In article <aufnif$f5$1 at baldur.whoi.edu>,
Karl A. Krueger <kkrueger at example.edu> wrote:
>Aahz <aahz at pythoncraft.com> wrote:
>> 
>> If re-processing already completed items isn't a problem, you could have
>> a separate thread that monitors the queue and checkpoints every X items.
>> It's less transparent in principle, but if the queue hides the
>> checkpoint thread from other threads, it doesn't look that way from the
>> outside.
>
>That's an interesting design.  Here's another (surely not original):
>
>When a queue is created, it's given a timeout interval.
>
>When an item is enqueued, it gets assigned a serial number or other
>magic cookie.
>
>When an item is dequeued by a client, the client gets back the tuple
>(item, cookie) ... and the item is not deleted from the queue, it's just
>marked "handed out" with a timestamp.  (Meanwhile, other clients can get
>things off the queue, and the queue is considered "empty" if all items
>have been handed out.)
>
>When the client thread is finished processing the item, it calls a
>release method with the cookie, and the item is actually deleted from
>the queue.
>
>A daemon thread enforces the timeout -- every timeout interval, it runs
>through the queue and clears the "handed out" flags from items that have
>been open for that amount of time or more.  (If everything was handed
>out before, this also clears the emptiness sema.)
>
>This is obviously no good if the queue must be processed in strict
>order.  It should be OK for a multithreaded mail filter, which is one of
>the applications I'm thinking of using this with.  :)

Instead of keeping the object in the queue, stick it in a holding dict
that also gets pickled.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I disrespectfully agree."  --SJM