Question regarding Queue object

[Terry]

On Apr 29, 4:32 pm, bock... at virgilio.it wrote:
> On 27 Apr, 12:27, Terry <terry.yin... at gmail.com> wrote:
>
>
>
> > Hello!
>
> > I'm trying to implement a message queue among threads using Queue. The
> > message queue has two operations:
> > PutMsg(id, msg) #  this is simple, just combine the id and msg as one
> > and put it into the Queue.
> > WaitMsg(ids, msg) # this is the hard part
>
> > WaitMsg will get only msg with certain ids, but this is not possible
> > in Queue object, because Queue provides no method to peek into the
> > message queue and fetch only matched item.
>
> > Now I'm using an ugly solution, fetch all the messages and put the not
> > used ones back to the queue. But I want a better performance. Is there
> > any alternative out there?
>
> > This is my current solution:
>
> >     def _get_with_ids(self,wait, timeout, ids):
> >         to = timeout
> >         msg = None
> >         saved = []
> >         while True:
> >             start = time.clock()
> >             msg =self.q.get(wait, to)
> >             if msg and msg['id'] in ids:
> >                 break;
> >             # not the expecting message, save it.
> >             saved.append(msg)
> >             to = to - (time.clock()-start)
> >             if to <= 0:
> >                 break
> >         # put the saved messages back to the queue
> >         for m in saved:
> >             self.q.put(m, True)
> >         return msg
>
> > br, Terry
>
> Wy put them back in the queue?
> You could have a defaultdict with the id as key and a list of
> unprocessed messages with that id as items.
> Your _get_by_ids function could first look into the unprocessed
> messages for items with that ids and then
> look into the queue, putting any unprocessed item in the dictionary,
> for later processing.
> This should improve the performances, with a little complication of
> the method code (but way simpler
> that implementing your own priority-based queue).
>
> Ciao
> -----
> FB

Yes, this will improve the performance. And I can see there's a
problem in my current implementation. The order of the message might
be changed if I put the saved message back to the end of the queue.
This may cause some confusion later, though I don't want to depend too
much on the message orders.

And you remind me one thing -- I need to implement 'priority' for
messages, so that the message with highest priority will tend to be
fetched first. OMG, this is going to be much more complicated then I
have expected.

Thanks for your suggestion. And I hope this will also work when I move
to stackless.