how to start thread by group?
Gabriel Genellina
gagsl-py2 at yahoo.com.ar
Tue Oct 7 00:37:21 EDT 2008
En Mon, 06 Oct 2008 11:24:51 -0300, <bieffe62 at gmail.com> escribió:
> On 6 Ott, 15:24, oyster <lepto.pyt... at gmail.com> wrote:
>> my code is not right, can sb give me a hand? thanx
>>
>> for example, I have 1000 urls to be downloaded, but only 5 thread at
>> one time
> I would restructure my code with someting like this ( WARNING: the
> following code is
> ABSOLUTELY UNTESTED and shall be considered only as pseudo-code to
> express my idea of
> the algorithm (which, also, could be wrong:-) ):
Your code creates one thread per url (but never more than MAX_THREADS
alive at the same time). Usually it's more efficient to create all the
MAX_THREADS at once, and continuously feed them with tasks to be done. A
Queue object is the way to synchronize them; from the documentation:
<code>
from Queue import Queue
from threading import Thread
num_worker_threads = 3
list_of_urls = ["http://foo.com", "http://bar.com",
"http://baz.com", "http://spam.com",
"http://egg.com",
]
def do_work(url):
from time import sleep
from random import randrange
from threading import currentThread
print "%s downloading %s" % (currentThread().getName(), url)
sleep(randrange(5))
print "%s done" % currentThread().getName()
# from this point on, copied almost verbatim from the Queue example
# at the end of http://docs.python.org/library/queue.html
def worker():
while True:
item = q.get()
do_work(item)
q.task_done()
q = Queue()
for i in range(num_worker_threads):
t = Thread(target=worker)
t.setDaemon(True)
t.start()
for item in list_of_urls:
q.put(item)
q.join() # block until all tasks are done
print "Finished"
</code>
--
Gabriel Genellina
More information about the Python-list
mailing list