Repeatedly crawl website every 1 min

[Steve D'Aprano]

On Thu, 11 May 2017 12:18 pm, liyucun2012 at gmail.com wrote:

> Hi Everyone,
> 
> Thanks for stoping by. I am working on a feature to crawl website content
> every 1 min. I am curious to know if there any good open source project
> for this specific scenario.

I agree with Iuri: crawling a website every minute is abuse. Unless it is
your own website, once a month is more appropriate -- and even then, you
should be very careful to restrict the rate at which you make requests.



-- 
Steve
Emoji: a small, fuzzy, indistinct picture used to replace a clear and
perfectly comprehensible word.