practical limits of urlopen()

webcomm ryandw at gmail.com
Sat Jan 24 12:17:10 EST 2009


Hi,

Am I going to have problems if I use urlopen() in a loop to get data
from 3000+ URLs?  There will be about 2KB of data on average at each
URL.  I will probably run the script about twice per day.  Data from
each URL will be saved to my database.

I'm asking because I've never opened that many URLs before in a loop.
I'm just wondering if it will be particularly taxing for my server.
Is it very uncommon to get data from so many URLs in a script?  I
guess search spiders do it, so I should be able to as well?

Thank you,
Ryan



More information about the Python-list mailing list