[Baypiggies] Asynchronous web tasks

[Meenal Pant]

 Thanks everyone! I think I will go with celery and RabbitMQ.
Meenal

On Mon, Feb 6, 2012 at 4:29 PM, Shannon -jj Behrens <jjinux at gmail.com>wrote:

> Alec, if I had to do what you did at Metaweb again today, I'd probably use
> gevent, and I'd run Pylons on top of gevent (aka gunicorn).  However, I
> think your use case was a little bit different than a standard queuing
> problem.  You were trying to execute multiple queries in parallel within a
> single request, whereas most people who use queues are using them to queue
> up requests to be done after the web request is done.
>
> I still think your Twisted magic at Metaweb was amazing :)
>
> Best Regards,
> -jj
>
> On Mon, Feb 6, 2012 at 10:31 AM, Alec Flett <alecf at flett.org> wrote:
>
>> A little late to the party but...
>>
>> On Thu, Feb 2, 2012 at 8:28 AM, Meenal Pant <meenalpant at gmail.com> wrote:
>>
>>> I use Django in production and now have a need to perform asynchronous
>>> backend tasks that
>>> a. Are some scheduled jobs
>>> b. Jobs triggered via a user input via Django frontend.
>>>
>>> I have been thinking of using twisted or message queues (RabbitMQ and
>>> celery) to achieve this.  From my research so far:
>>>
>>> 1. Twisted allows both a and b though the learning curve with twisted is
>>> steeper
>>> 2. RabbitMQ and celery is easier to use, allows a but does it also allow
>>> b?
>>>
>>
>>
>> I actually did this with Twisted inside of Pylons a few years ago - this
>> was before all these queuing systems were popular.
>>
>> While everything actually did work great, I can describe drawbacks of
>> using Twisted for this specific case right off the bat:
>>
>> 1) the architecture of Twisted has very tight bindings to a
>> single-threaded process model - meaning that everything wants to use the
>> global reactor. This means if you have a multi-threaded server, you have to
>> keep thread-local reactors around and monkeypatch bits of Twisted to get it
>> to behave
>> 2) It's a little complicated to keep track of all the outstanding
>> deferreds to make sure you return your response after all of them complete
>> - basically you want a "wait for all deferred to resolve" but there isn't a
>> single global version of that for twisted - there are various permutations
>> of inspecting the reactor, but ultimately you have to do the accounting
>> yourself..
>>
>> Overall I wish there was a system that matched the existing
>> request/thread model that I was using (Pylons) rather than bolting on a
>> secondary system. At the time there weren't any other options, but I'm sure
>> now there are a bunch of better alternatives
>>
>> Alec
>>
>>
>>
>>> I am looking for some opinions /recommendations to use one of  these two
>>> technologies or something else altogether ?
>>> Thanks
>>> Meenal
>>>
>>> _______________________________________________
>>> Baypiggies mailing list
>>> Baypiggies at python.org
>>> To change your subscription options or unsubscribe:
>>> http://mail.python.org/mailman/listinfo/baypiggies
>>>
>>
>>
>> _______________________________________________
>> Baypiggies mailing list
>> Baypiggies at python.org
>> To change your subscription options or unsubscribe:
>> http://mail.python.org/mailman/listinfo/baypiggies
>>
>
>
>
> --
> In this life we cannot do great things. We can only do small things with
> great love. -- Mother Teresa
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20120206/1ef59714/attachment-0001.html>