Seeking an example on using Queue to update variable while threading

[Kushal Kumaran]

On Thu, Jul 28, 2011 at 11:13 AM, Danny Wong (dannwong)
<dannwong at cisco.com> wrote:
> Hi Python experts,
>        I'm trying to use a dict structure to store and update information from X number of threads. How do I share this dict structure between threads? I heard of using a queue, but I'm not familiar with how it works. Does anyone have an example of using a queue to store variables/dicts between threads?
>

With a queue, you would not actually share data.  One of your threads
will "own" the data, and the other threads will send messages to the
owner to update the data.  These messages will be sent on a queue,
which the owner thread is waiting on.  A dummy example follows:

import threading
import time
import random
import queue
import collections
import logging
import sys

def generator_func(q, name):
    for i in range(1000):
        time.sleep(random.randint(1, 10))
        value = random.randint(1, 100)
        logging.info('generated {}'.format(value))
        q.put(value)
    logging.info('finished')
    q.put(None)

def counter_func(q, num_generators):
    counts = collections.defaultdict(int)
    finished = 0
    while True:
        item = q.get()
        if item is None:
            finished += 1
            if finished == num_generators:
                return
        logging.info('received {}'.format(item))
        counts[item] += 1

def main():
    num_generators = 50

    logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                        format="%(asctime)s %(threadName)s %(message)s")

    q = queue.Queue()
    counter = threading.Thread(name='counter', target=counter_func,
                               args=(q, num_generators))
    counter.start()

    generators = []
    for i in range(num_generators):
        generators.append(threading.Thread(name='generator-{:03}'.format(i),
                                           target=generator_func, args=(q, i)))
        generators[-1].start()

    counter.join()
    for g in generators:
        g.join()

if __name__ == '__main__':
    main()

The "counter" thread runs the counter_func function.  This function
waits on a queue and updates a hash when it receives an item on the
queue.  The "generator" threads run the generator_func function, which
puts random values into the queue at random intervals.  So, in a
sense, the "generator" threads update the hash by sending messages to
the "counter" thread.

If you run this example, you might see output like this:

2011-07-28 21:23:40,445 generator-001 generated 55
2011-07-28 21:23:40,448 generator-026 generated 38
2011-07-28 21:23:40,448 generator-015 generated 62
2011-07-28 21:23:40,448 generator-009 generated 90
2011-07-28 21:23:40,459 counter received 55
2011-07-28 21:23:40,455 generator-039 generated 72
2011-07-28 21:23:40,460 counter received 38
2011-07-28 21:23:40,460 counter received 90
2011-07-28 21:23:40,460 counter received 62
2011-07-28 21:23:40,461 counter received 72
2011-07-28 21:23:41,447 generator-012 generated 12
2011-07-28 21:23:41,448 generator-029 generated 49
2011-07-28 21:23:41,448 generator-023 generated 11
2011-07-28 21:23:41,449 counter received 12
2011-07-28 21:23:41,449 generator-041 generated 91
2011-07-28 21:23:41,449 generator-049 generated 20
2011-07-28 21:23:41,449 generator-042 generated 73
2011-07-28 21:23:41,450 counter received 49
2011-07-28 21:23:41,451 counter received 11
2011-07-28 21:23:41,451 counter received 91
2011-07-28 21:23:41,452 counter received 20
2011-07-28 21:23:41,452 counter received 73
2011-07-28 21:23:41,461 generator-039 generated 85
2011-07-28 21:23:41,462 counter received 85
...


-- 
regards,
kushal