[BangPypers] HarvestMan with eventlets

[gnuyoga]

Anand Balachandran Pillai wrote:
> Hi,
>
>      Continuing from my previous post, the context of which was eventlet,
> here is the reason why I was asking about it.
>
> I have been planning to rewrite HarvestMan using a non-blocking approach
> rather than the current design of achieving concurrency using multiple
> threads (which do not scale very well in Python).
>
> I have been evaluating twisted for sometime for doing this, but I have not
> been able to wrap my brain around twisted. Also the fact that twisted
> requires an explicit event loop call is not very good and makes it difficult
> to write an application which has its own separate event loop.
>
> Eventlets look promising because it is implemented using coroutines without
> an explicit event loop call, instead the event loop is run transparently when
> required. Also due to its design, it can be easily merged with Stackless, which
> I think it is already on the way to.
>
> Moreover, the sample code shows a crawler example, so there it is!
>
> I am looking at creating another implementation of HarvestMan based on
> eventlets rather than threads. This will run parallel to the regular HarvestMan.
> The plan is to write a crawler that scales very well w.r.t the number
> of requests
> and websites.
>
> Let me know if someone would like to be part of this.
>
> Thanks
>
>   
+ 1

am a g8 fan of stackless. will read about eventlet and get back 2 u.

- sree



-- 
http://picasaweb.google.com/gnuyoga

Each soul is potentially divine. The goal is to manifest the divine by controlling nature, internal or external. Do this by work or worship or psychic control or philosophy by one or more, or all of these and be free.