Twisted 100% event driven or hybrid?

[exarkun at twistedmatrix.com]

On 05:48 pm, mcfletch at vrplumber.com wrote:
>exarkun at twistedmatrix.com wrote:
>>On 05:55 am, jacopo.pecci at gmail.com wrote:
>...
>>>results to be ready, I collect and process them. From now on I don 
>>>19t
>>>need a the system to be event drive any more, the processing should
>>>occur only on the master machine, following a deterministic flow.
>>>As soon as finished I am ready to start again to resubmit the models
>>>for recalculation and so on. This should go on forever.
>
>Jean-Paul is obviously far more authoritative on the "twisted way" than
>I am, so if he says you can just run your synchronous operation in- 
>situ,
>that's probably the way to go, but IIRC there's a
>reactor.deferToThread() function which can run your synchronous code
>"off to the side", while allowing the twisted code to continue to
>process incoming operations.  Thus you'd do something like:
>
>def process( dataset ):
>    dl = [ remote_call( x ) for x in dataset]
>    dl = defer.DeferredList( dl )
>    def on_all_results( results ):
>        reactor.deferToThread( sync_process, (results,)).addCallback(
>process )
>    return dl.addCallback( on_all_results )
>
>(I'm typing all of that from the distance of a few years of memory
>decay, so take it as "loosely this, with the proper function names and
>the like").  Threads aren't really part of the "twisted way" in my
>understanding, but they can be used if necessary AFAIK, and they will
>let your application remain responsive to network events during the
>processing.

Yep, you're correct here Mike (except it's 
`twisted.internet.defer.deferToThread` rather than 
`twisted.internet.reactor.deferToThread`).  If it is safe to call 
`sync_process` in a thread, then this may be a good approach as well, 
and it will free up the reactor to continue to respond to events 
(assuming `sync_process` plays nicely - ie, is written in Python or is 
an extension that releases the GIL, of course).

In my post, I was trying to highlight the idea that there's not really 
anything special going on in a Twisted program.  You can choose to block 
if you wish, if the consequences (events go unserviced for a while) are 
acceptable to you.

Jean-Paul