ANN: botlib 0.1 !

[Michal Wallace (sabren)]

On Wed, 12 Apr 2000, Oleg Broytmann wrote:

> least two - I want a threading robot, but much more than that I want an
> asyncore-based robot. I haven't done anything yet; it seems I need to
> rewrite the entire urllib to use asyncore, and it is hard work. Anyway I

Yes, I came to the same conclusion.. You have to rewrite just about
everything.. Though I'm thinking urllib2 can make the job it easier.
Anyway, I decided to take a stab at it, and used the example in the
asyncore docs to create something that does a good chunk of the job:

-----------------------------
announcing botlib 0.1!!!!!!!!
-----------------------------

The botlib module contains two classes: UniBot and MultiBot.

UniBot fetches a single URL.  It's http only, no authentication, no
redirects, etc, BUT: it allows you to specify a timeout down to
fractions of a second.

MultiBot fetches multiple URLS at once. You tell it which urls to
get, and which functions to call for each one once the content is
fetched.

Here's two quick examples:

    #####################

    import botlib

    bot = botlib.UniBot()
    content = bot.fetch('http://www.python.org/')

    #####################


and:

    #########################

    import botlib

    def callback(url, contents, extras):
        print url, 'is this big:', len(contents)

    queen = botlib.MultiBot()
    queen.fetch('http://www.python.org/', callback)
    queen.fetch('http://www.linkwatcher.com/', callback)
    queen.fetch('http://www.sabren.com/', callback)
    queen.loop()
    
    #########################


Anyway, it's far from perfect... The methods and stuff might change
completely, especially once more of the urllib2 features get worked
in... but it is a start.

check it out at: http://www.linkwatcher.com/src/botlib.py

Feedback welcome!

Cheers,

- Michal
-------------------------------------------------------------------------
http://www.manifestation.com/         http://www.linkwatcher.com/metalog/
-------------------------------------------------------------------------