Cost of Queue.put

[Gabriel Genellina]

On 18 mar, 01:13, James <rent.lupin.r... at gmail.com> wrote:

> Basically, what I need is a multi-process safe persistent quick queue.
> The arrangement I had was a simple XML-RPC service running on the web
> server which the various web server threads POST the relevant search
> engine updates to. These updates were added to a persistent queue
> built on top of Queue.Queue and cPickle, and there was a separate
> thread in the XML-RPC server actually adding the updates to the search
> engine.
>
> However, the CPU consumption of the XML-RPC server seems to grow
> pretty much linearly with the length of the persistent queue. Isn't
> Queue.put O(1)?

Maybe, but pickle.dumps is very likely O(n) and dominates. Why don't
you use a database? Just insert and delete records with each get and
put.

--
Gabriel Genellina