Simple thread pools

Jacob Friis lists at debpro.webcom.dk
Fri Nov 5 09:39:19 EST 2004


Diez B. Roggisch wrote:
>>I need to download 150000 files several times every day.
>>How would you solve that?
> 
> How do you download them?

HTTP.

> What sizes?

15K (approx)

> filetypes?

XML: RSS or Atom.

> Can they be packed beforehand?

Yes, if the web server I download from support GZIP.

> Do you even have enough bandwidth?

Yes.

> From how many servers do you query them?

6712 (approx)

> As you can see, there are more than a few questions - so how about
> elaborating a bit on the task you have to accomplish?

I am building an online aggregator.

Thanks,
Jacob



More information about the Python-list mailing list