Simple thread pools
Jacob Friis
lists at debpro.webcom.dk
Fri Nov 5 09:39:19 EST 2004
Diez B. Roggisch wrote:
>>I need to download 150000 files several times every day.
>>How would you solve that?
>
> How do you download them?
HTTP.
> What sizes?
15K (approx)
> filetypes?
XML: RSS or Atom.
> Can they be packed beforehand?
Yes, if the web server I download from support GZIP.
> Do you even have enough bandwidth?
Yes.
> From how many servers do you query them?
6712 (approx)
> As you can see, there are more than a few questions - so how about
> elaborating a bit on the task you have to accomplish?
I am building an online aggregator.
Thanks,
Jacob
More information about the Python-list
mailing list