[Python-ideas] async/await in Python (C API ?)

[M.-A. Lemburg]

On 20.04.2015 17:17, Yury Selivanov wrote:
> 
> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote:
>> On 20.04.2015 15:40, Yury Selivanov wrote:
>>> Hi Marc-Andre,
>>>
>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote:
>>>> Hi Yury,
>>>>
>>>> do you have any plans of also exposing an easy to use C API
>>>> for async and await ?
>>>>
>>>> I think this would be useful to e.g. implement asynchronous
>>>> database access. Most of those adapters are written in C and
>>>> need so a C API would help a lot with this.
>>> I think one way to have a convenient C API is to implement
>>> a Future-like object in C -- an object with __await__ method
>>> that should return an iterator, which should also be
>>> implemented in C and provide a way to attach callback
>>> or to *pipeline* C functions.
>>>
>>> This way C code should seamlessly integrate with asyncio.
>>>
>>> I doubt that we have time to include this kind of API in
>>> the PEP, but it can be developed as a separate library.
>>>
>>> Your thoughts?
>> I'm just sketching here, since I don't have much experience
>> with writing async code (I'm using threads and multiple
>> processes instead) and it probably shows ;-)
>>
>>  From the C side of things, I think using callbacks
>> and a way to register/unregister polling functions
>> would be great to have.
>>
>> The C code could then run like this in polling mode:
>>
>> long_running_query(...) {
>>      /* Start long running operation... creating a handle for it */
>>      PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval);
>>      PyAsync_ReturnAwait();
>> }
>>
>> mypolling_fct(int handle) {
>>      /* Check completion of operation */
>>      if (completed) {
>>          /* Fetch results into value */
>>          PyAsync_UnregisterPoll(mypolling_fct, handle);
>>          PyAsync_ReturnReady(value);
>>      }
>>      else
>>          PyAsync_ReturnContinue();
>> }
>>
>> and like this in callback mode:
>>
>> long_running_query(...) {
>>      /* Create a handle for a long running operation... */
>>      PyAsync_RegisterCallback(myquery_done, handle, timeout);
>>      /* Start long running operation... */
>>      PyAsync_ReturnAwait();
>> }
>>
>> myquery_done(int handle, PyObject *value) {
>>      PyAsync_UnregisterCallback(myquery_done, handle);
>>      PyAsync_ReturnReady(value);
>> }
>>
>> Both mechanisms will likely have to move the long running
>> operation into a separate (blocking) thread, unless there's also
>> a way to register an event loop selector somewhere and the C lib
>> your interfacing to provides this mechanism as well.
>>
> 
> I see what you want to do and why now.
> 
> However, I can't imagine how this could be implemented
> without asyncio event loop being implemented in C.
> Coroutines from PEP 492 aren't an indepenedant concept,
> they require a library/framework with an event loop
> to make them work.
> 
> The only way to go right now is to get 'get_event_loop()'
> and 'loop.call_soon()' working in C using Python C API.
> 
> From that point you can implement Future object in C,
> see 'asyncio/futures.py'; Future._schedule_callbacks,
> Future.set_result and Future.__iter__ methods in
> particular.
> 
> With all that, you can easily return your future object
> from C methods and everything should work just fine.
> 
> I know this isn't ideal, and we actually should start a
> separate discussion on how we can improve things on
> our asyncio C side.

I'm not expecting this to be ready anytime soon,
just wanted to bring this up, since it may be something
that has to be considered in the async/await API design
early rather than later to not make it too complex to
add later on.

I'm not sure, but going through the Python API from C
can potentially result in the code to reenter which sounds
like it could cause some unwanted race conditions. A C API
would avoid this.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 20 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/