queue versus list

[MRAB]

On 2020-03-19 20:08, duncan smith wrote:
> Hello,
>        I have generator code along the following lines,
> 
> 
> Q = queue.Queue()
> Q.put(x)
> while not Q.empty():
>      x = Q.get()
>      if <some condition that depends on x>:
>          yield x
>      else:
>        Q.put(<some value that depends on x>)
> 
> 
> If I change it to,
> 
> 
> Q = []
> Q.append(x)
> for x in Q:
>      if <some condition that depends on x>:
>          yield x
>      else:
>        Q.append(<some value that depends on x>)
> 
> 
> then it runs approximately 3 times more quickly. I seem to remember
> (from somewhere) that the language specification doesn't guarantee that
> the latter will continue to work, but that it's unlikely that future
> implementations will break it. Apart from that, is there any good reason
> why I shouldn't use the list? Is there another approach that might avoid
> the overhead of using a queue? Results from profiling the two
> implementations below in case it makes anything clearer. TIA.
> 
[snip]
A number of points:

1. I'm not sure it's a good idea to mutate a list while iterating over it.

2. The example with the list retains all of the results, whereas the one 
with the queue consumes them.

3. Queues are thread-safe because they're used for communication between 
threads, but lists aren't thread-safe; that might be something to do 
with the difference.

4. If it doesn't need to be thread-safe, why not try deques instead?