Persistent Queue implementations?

[Kevin Altis]

I think you should be able to do what you want with a subclass of the Queue
class using the pickle module. For example, here's a test in the shell just
using pickle with get() and put() from the Queue instance.

>>> import Queue
>>> q = Queue.Queue()
>>> q.put(["spam", "spam", "spam", "eggs", "spam"])
>>> import pickle
>>> path = 'queustorage.pickle'
>>> if not q.empty():
...     fp = open(path, 'w')
...     pickle.dump(q.get(), fp)
...     fp.close()
...
>>> fp = open(path)
>>> q.put(pickle.load(fp))
>>> fp.close()
>>> q.get()
['spam', 'spam', 'spam', 'eggs', 'spam']
>>>

So, in the __init__ method of your Queue subclass you'll need to have a
try/except block that opens the filename you pass into the init and in the
except just initialize a new Queue to deal with when the file doesn't exist.
You'll need save and load methods for your subclass which basically just do
what I've shown above. One problem will be deciding when and if you want to
delete the pickle file. You probably just want to delete the pickle when
you've cleanly exited your app. You could also change the semantics of the
init so that whether the file is loaded or not is dependent on a flag.

A better solution might be to simply pickle the Queue instance and avoid a
special class. That would avoid having to change the contents with get and
put.

ka

"Karl A. Krueger" <kkrueger at example.edu> wrote in message
news:au7o7a$6fo$1 at baldur.whoi.edu...
> Has anyone implemented a persistent queue class?  I'm looking for
> something like the Queue module -- synchronized, threading-safe, and
> other nice things like that -- but which keeps copies of the queue
> elements in a synced file (or files) on disk.
>
> I'd like to be able to do this:
>
> # in initialization code
> q = PersistentQueue("/var/local/lib/queuefile")
>
> # in a thread
> q.put(["spam", "spam", "spam", "eggs", "spam"])
>
> ... and then, if the program exits and gets restarted ...
>
> q = PersistentQueue("/var/local/lib/queuefile")
> item = q.get()
>
> ... and get back my "spam" list.
>
> I'm not sure if this makes more sense as a subclass of Queue ... or as
> another module that implements Queue's interface, but is built on top of
> a shelf or some similar source of persistence, with added semaphores to
> make it thread-safer.
>
> (Presumably, items on such a queue can be limited to those which can be
> serialized sanely.  However, in the case of the program *not* having
> exited and restarted between a put() and a get(), I'd like for the queue
> to be handing back references to the same objects I gave it, not copies.)
>
> --
> Karl A. Krueger <kkrueger at example.edu>
> Woods Hole Oceanographic Institution
> Email address is spamtrapped.  s/example/whoi/
> "Outlook not so good." -- Magic 8-Ball Software Reviews