Urllib(1/2) how to open multiple client sockets?

[Gerhard Häring]

ShashiGowda wrote:
> Hey there i made a script to download all images from a web site but
> it runs damn slow though I have a lot of bandwidth waiting to be used
> please tell me a way to use urllib to open many connections to the
> server to download many pics simultaneously.... Any off question
> suggestions are also ok...

"Simultaneously" means using multiple threads (or processes).

You could create worker threads that do the downloading (use the 
threading module for this) and communicate with them via a queue (use 
the Queue module for this).

For example the main thread would then do the HTML parsing and push the 
image URLs to the worker threads via the queue.

Each worker thread then polls the queue and downloads the images it gets.

One issue is how to terminate the multithreaded application. A simple 
approach is to use a sentinel value to push to the queue to signal the 
worker threads to quit. Something like "QUIT" will do. Be sure to push 
at least as many sentinel values as you have worker threads, then.

-- Gerhard