[Python-ideas] The async API of the future: Twisted and Deferreds

[Guido van Rossum]

On Fri, Oct 12, 2012 at 4:39 PM, Richard Oudkerk <shibturn at gmail.com> wrote:
> On 12/10/2012 11:11pm, Guido van Rossum wrote:
>>
>> Using Futures and generator coroutines, I would do it as follows. I'm
>> hypothesizing that for every blocking API foo() there is a
>> corresponding non-blocking API foo_async() with the same call
>> signature, and returning a Future whose result is what the synchronous
>> API returns (and raises what the synchronous call would raise, if
>> there's an error). These are the conventions I use in NDB. I'm also
>> inventing a @task decorator.
>>
>>   @task
>>   def view_paste_async(request, filekey):
>>      # Create Futures -- no yields!
>>      f1 = Pastes.objects.get_async(key=filekey) # This won't raise
>>      f2 = loader.get_template_async('pastebin/error.html')
>>      f3 = loader.get_template_async('pastebin/paste.html')
>>
>>      try:
>>          fileinfo= yield f1
>>      except DoesNotExist:
>>          t = yield f2
>>          return HttpResponse(t.render(Context(dict(error='File does not exist'))))
>>
>>      f = yield open_async(fileinfo.filename)
>>      fcontents = yield f.read_async()
>>      t = yield f3
>>      return HttpResponse(t.render(Context(dict(file=fcontents))))
>
>
> So would the futures be registered with the reactor as soon as they are
> created, or only when they are yielded?  I can't see how there can be any
> "concurrency" if they don't start till they are yielded.  It would be like
> doing
>
>    t1 = Thread(target=f1)
>    t2 = Thread(target=f2)
>    t3 = Thread(target=f3)
>    t1.start()
>    t1.join()
>    t2.start()
>    t2.join()
>    t3.start()
>    t3.join()
>
> But if the futures are registered immediately with the reactor then does
> that mean there is a singleton reactor?  That seems rather inflexible.

I don't think it follows that there can only be one reactor if they
are registered immediately. There could be a notion of "current
reactor" maintained in thread-local context; moreover it could depend
on the reactor that made the callback that caused the current task to
run. The reactor could also be chosen by the code that made the
Future. (Though I'm not immediately sure how that would work in the
yield-from scenario -- but I'm sure there's a way.)

FWIW, in NDB there is one event loop per thread; separate threads are
handling separate requests and are completely independent. Also, in
NDB there's some code that turns Futures into actual RPCs that runs
only once there are no more immediately runnable tasks. I think that
in general such behaviors are up to the reactor implementation for the
platform though, and should not directly be reflected in the reactor
API.

-- 
--Guido van Rossum (python.org/~guido)