Unified async/sync interface

[Ian Kelly]

On Sat, Mar 31, 2018 at 7:44 PM, Demian Brecht <demianbrecht at gmail.com> wrote:
> I might be entirely off my face, but figured I'd ask anyways given I
> haven't figured out a clean solution to this problem myself yet:
>
> I'm trying to write a REST API client that supports both async and
> synchronous HTTP transports (initially requests and aiohttp). So far,
> I've tried a few approaches with `async def` but, of course, you always
> run into the issue with the top level API that should either be async
> or not, but can't be redefined on the fly (AFAICT). I'm also trying to
> limit code duplication. I know I could write async and sync versions of
> each method and then instantiate through composition based on the
> transport layer but I'd like to avoid that if possible. As an
> simplified example of what I'm after:
>
> [code snipped]
>
> Is this perhaps possible through asyncio.coroutine decorators
> (admittedly, I haven't looked yet, I've had async/await blinders on)?
> Are there any projects that anyone's aware of that does anything
> similar that I could use as an example?
>
> The overall point of what I'm trying to achieve here is to write a
> client that's supported from 2.7 to 3.x and supports /either/
> synchronous or asynchronous transports (not both at the same time).

asyncio doesn't exist for 2.7 unless you're using trollius, which I
understand is an incomplete port and also currently unmaintained. Even
with trollius, I think that aiohttp is still going to restrict you to
3.5.3+. Unless your plan is to support synchronous transports only for
2.7. You definitely can't use "async def" with 2.7.

To answer your question though, it sounds like you need your
get_something method to return either the deserialized data if the
transport is synchronous, or an awaitable that will resolve to the
deserialized data when asynchronous. Since you don't want to implement
the logic twice, I suggest breaking down the logic into transforms
that can be applied generically, something like this (untested):

    def get_something(self):
        # Agnosticly get either synchronously fetched data
        # or an async awaitable.
        data = self.transport.get('https://example.com')
        return self._transform(data, json.loads)

    def _transform(self, value, func):
        # Transform value using func either synchronously or
        # asynchronously, depending on the transport regime.
        if self.transport.is_async():
            async def await_and_transform(awaitable):
                new_value = func(await awaitable)
                if inspect.isawaitable(new_value):
                    # In case func might also be async
                    return await new_value
                else:
                    return new_value

            return await_and_transform(value)
        else:
            return func(value)

Note that if the async framework might be something other than asyncio
then the above code probably wouldn't work. In that case perhaps it
would be better to let the transport define the async transformation
logic.