Speed up with threads

[Chris Liechti]

Rhymes <raims at dot.com> wrote in 
news:58vqku4lm7tii2n35ip7f3tlidirqb01sp at 4ax.com:
> On Sun, 04 Aug 2002 12:23:50 GMT, Syver Enstad
> <syver-en+usenet at online.no> wrote:
> 
>>You'll have to make the two classes share the same Queue, maybe like
>>this:
>>
>>In the ctor for the main class, or an init method of some kind:
>>self._queue = Queue.Queue()
>>
>>in a loop creating the Scanners:
>>self._scanners.append(Scanner(self._queue))
> 
> I don't understand how link scanners (the list) with the queue...
> 
> this is my whole unstable code (the queue version):
> 
> import socket as sk
> import sys
> import threading, Queue
> 
> #MAX_THREADS = 50
> queue = Queue.Queue()
> scanners = []
> 
> def usage():
>     print "\npyScan 0.1a - Rhymes (rhymes at myself.com)"
>     print "usage: pyScan <host> [start port] [end port]"
> 
> class Scanner(threading.Thread):
>     def __init__(self):     
>         threading.Thread.__init__(self)
>         # build up the socket obj
>         self.sd = sk.socket(sk.AF_INET, sk.SOCK_STREAM)
> 
>     def run(self):
>         try:
>             host, port = queue.get()
>             # connect to the given host:port
>             self.sd.connect((host, port))
>             print "%s:%d OPEN" % (host, port)
>             self.sd.close()

i'm not sure if you can close a socket and connect afterwards again... i 
would create the socket object in the loop. but mayve an expert can speak 
up here.

>         except: pass

be careful with try: ... except: it might hide useful exception for 
debugging... usually you should only catch specific exceptions. in this 
case you want to keep the thread running it's ok, but i sugest adding a 
"traceback.print_exc()" in the except part.
 
> class pyScan:
>     def __init__(self, args=[]):
>         self.args = args
>         self.start, self.stop = 1, 1024
>         self.host = ""       
> 
>         # check the arguments
>         if len(self.args) == 4:
>             self.host = self.args[1]
>             try:
>                 self.start = int(self.args[2])
>                 self.stop = int(self.args[3])
>             except ValueError:
>                 usage()
>                 return
>             if self.start > self.stop:
>                 usage()
>                 return
>         elif len(self.args) == 2:
>             self.host = self.args[1]
>         else:
>             usage()
>             return
> 
>         # create the scanners
>         for i in range(5):
>             scanners.append(Scanner())

you need to start the threads. in the commented out lines (snipped) you 
used start() but here you forgot it. it's also a good idea to make daemon 
threads, so that the program exits when the main thread dies. daemon 
threads don't keep the process alive, oposed to normal threads.

    	s = Scanner()
    	s.setDaemon(1)
    	s.start()
    	scanners.append(s)

> 
>         try:
>             sk.gethostbyname(self.host)
>         except:
>             print "hostname '%s' unknown" % self.host
> 
>         self.scan(self.host, self.start, self.stop)
> 
>     def scan(self, host, start, stop):
>         self.port = start
>         while self.port <= stop:
>                 self.port += 1
>                 queue.put((host, self.port))

that can be written as list comprehension or with map(), e.g.:
    	[queue.put((host,port)) for port in range(start, stop)]

> def main():
>     pyScan(sys.argv)

i don't see a point for the class "pyScan". just put the relevant code 
directly in the main function. a class makes sense to capsulate a threadl 
like in the Scanner above and it of course very useful in many places. but 
here you just call one function after an other, hidden in the __init__...

as we created daemon threads, we must now wait for completion of the job by 
polling the queue's size:

    	while queue.qsize():
    	    	time.sleep(0.1)

the sleep is very important. without it, the loop would eat up all the CPU 
cycles, what you certainly don't want...
any way when the queue is empty, the main thread dies and the process 
exits.

> if __name__ == "__main__":
>     main()

take it as construtive critique ;-)

chris

-- 
Chris <cliechti at gmx.net>