Concurrent threads to pull web pages?

Gilles Ganault nospam at nospam.com
Thu Oct 1 05:28:09 EDT 2009


Hello

	I recently asked how to pull companies' ID from an SQLite database,
have multiple instances of a Python script download each company's web
page from a remote server, eg. www.acme.com/company.php?id=1, and use
regexes to extract some information from each page.

I need to run multiple instances to save time, since each page takes
about 10 seconds to be returned to the script/browser.

Since I've never written a multi-threaded Python script before, to
save time investigating, I was wondering if someone already had a
script that downloads web pages and save some information into a
database.

Thank you for any tip.



More information about the Python-list mailing list