threaded urllib.urlretrieve() confusion

Michael Mell mike at nthwave.net
Thu May 9 12:54:34 EDT 2002


I'm new to threads and am confounded by what seems to be happening.

The code below wants to spawn X threads per second, each thread
containing a call to  doRequest(). Immediately after each threaded call
to doRequest(), the request should begin. However, what seems to be
happening is that all the spawning happens, then all the threads begin.

It seems that all the threads are spawned in  # SPAWN THREADS before and
request in # DO REQUEST initiates.

How do I get each doRequest() to begin its process as soon as I call
    thread.start_new_thread (doRequest, (idStr, ))

TIA
mike

Here is the code snippet. If you have threads, it should run on your
system exactly as-is.
==================
import time, thread, urllib

base = "http://192.168.1.3/index.html"
query = {'dataID':'27346'}
data = urllib.urlencode(query)

totalRequests = 6
requestsPerSecond = 2

active = [] # each item in active represents one active thread
results = [] # we'll store the results for later

def storeResult(r):
    global active, results
    active.pop() # one less thread active
    results.append(r)

def doRequest(id): # DO REQUEST
    result = {'startTime': time.time() }
    print 'start',id, result['startTime']
    result['response'] = urllib.urlretrieve(base, '', '', data) # URL
RETRIEVE
    storeResult(result)
    return 1

processStart = time.time()
print 'processStart', processStart

for i in range(totalRequests/requestsPerSecond): # SPAWN THREADS
    for j in range(requestsPerSecond):
        active.append(i) # a new thread is active
        idStr = 'thread second '+`i` +' id '+ `j`
        print 'spawn '+ idStr
        thread.start_new_thread (doRequest, (idStr, )) # START NEW
THREAD
    print 'sleep', i
    time.sleep(1.0) # SLEEP pause the main process before starting the
next batch

while active: # WAIT until the last thread has finished
    pass

processFinish = time.time()
print 'processFinish', processFinish

tElapse = processFinish - processStart
print 'tElapse', tElapse

==========================
--
mike[at]nthwave.net
llemekim         YahooIM
415.455.8812     voice
419.735.1167     fax







More information about the Python-list mailing list