[Baypiggies] Web Crawler/Backend Engineer - San Francisco, CA

[Alec Flett]

>
>
>
> What experience do folks have with event based frameworks in python?
>
>

On the HTTP server side, Twisted is by far the most pervasive, but Tornado,
which came out last year, has had a lot of positive responses, and I've
heard of more and more people using it. In general the trade off with these
is you have is that you have to think more about parallelism even during a
single http request because your control flow changes dramatically - learn
about Deferreds to learn more.

The bonus is that you get that parallelism automatically, very cheaply, once
you buy into the framework... but the drawback is that that because your
control flow is no longer stack-based, WSGI generally goes out the window...
so it's virtually impossible to reuse all those great WSGI middleware
packages out there.

Stackless has an excellent solution for this which is to make it look like
your control flow follows the stack, and thus you can actually use WSGI
middleware.

Why is the "automatic parallelism" I mentioned above important? At least
where I'm working, a single incoming HTTP request is fulfilled by the
appserver making several (2-100) internal HTTP requests to internal
services. If *those* requests can be made in parallel, then you can really
get very high throughput.

Our solution was actually to run Twisted inside of Pylons, but that's a
longer discussion...

Alec


> I was thinking I was going to have to write my own, but I stumbled upon
> "circuits" today and then read through some of the "twisted" doc and was
> surprised to find that it closely mirrored my own thoughts.
>
> Are there others?  Are any of the implementations definitive either
> positively or negatively?
>
> --rich
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20100204/9fa2b5a3/attachment.htm>