[Baypiggies] Asynchronous web tasks

[Shannon -jj Behrens]

Alec, if I had to do what you did at Metaweb again today, I'd probably use
gevent, and I'd run Pylons on top of gevent (aka gunicorn).  However, I
think your use case was a little bit different than a standard queuing
problem.  You were trying to execute multiple queries in parallel within a
single request, whereas most people who use queues are using them to queue
up requests to be done after the web request is done.

I still think your Twisted magic at Metaweb was amazing :)

Best Regards,
-jj

On Mon, Feb 6, 2012 at 10:31 AM, Alec Flett <alecf at flett.org> wrote:

> A little late to the party but...
>
> On Thu, Feb 2, 2012 at 8:28 AM, Meenal Pant <meenalpant at gmail.com> wrote:
>
>> I use Django in production and now have a need to perform asynchronous
>> backend tasks that
>> a. Are some scheduled jobs
>> b. Jobs triggered via a user input via Django frontend.
>>
>> I have been thinking of using twisted or message queues (RabbitMQ and
>> celery) to achieve this.  From my research so far:
>>
>> 1. Twisted allows both a and b though the learning curve with twisted is
>> steeper
>> 2. RabbitMQ and celery is easier to use, allows a but does it also allow
>> b?
>>
>
>
> I actually did this with Twisted inside of Pylons a few years ago - this
> was before all these queuing systems were popular.
>
> While everything actually did work great, I can describe drawbacks of
> using Twisted for this specific case right off the bat:
>
> 1) the architecture of Twisted has very tight bindings to a
> single-threaded process model - meaning that everything wants to use the
> global reactor. This means if you have a multi-threaded server, you have to
> keep thread-local reactors around and monkeypatch bits of Twisted to get it
> to behave
> 2) It's a little complicated to keep track of all the outstanding
> deferreds to make sure you return your response after all of them complete
> - basically you want a "wait for all deferred to resolve" but there isn't a
> single global version of that for twisted - there are various permutations
> of inspecting the reactor, but ultimately you have to do the accounting
> yourself..
>
> Overall I wish there was a system that matched the existing request/thread
> model that I was using (Pylons) rather than bolting on a secondary system.
> At the time there weren't any other options, but I'm sure now there are a
> bunch of better alternatives
>
> Alec
>
>
>
>> I am looking for some opinions /recommendations to use one of  these two
>> technologies or something else altogether ?
>> Thanks
>> Meenal
>>
>> _______________________________________________
>> Baypiggies mailing list
>> Baypiggies at python.org
>> To change your subscription options or unsubscribe:
>> http://mail.python.org/mailman/listinfo/baypiggies
>>
>
>
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>



-- 
In this life we cannot do great things. We can only do small things with
great love. -- Mother Teresa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20120206/3878dfe1/attachment.html>