[Python-ideas] async: feedback on EventLoop API

[Nick Coghlan]

On Tue, Dec 18, 2012 at 10:40 AM, Guido van Rossum <guido at python.org> wrote:

> I see. That's a fundamentally different API style, and one I'm less
> familiar with. DelayedCall isn't meant to be that at all -- it's just
> meant to be this object that (a) is sortable by time (needed for
> heapq) and (b) can be cancelled (useful functionality in general). I
> expect that at least one of the reasons for libuv etc. to do it their
> way is probably that the languages are different -- Python has keyword
> arguments to pass options, while C/C++ must use something else.
>
> Anyway, Handler sounds like a pretty good name. Let me think it over.
>

Is DelayedCall a subclass of Future, like Task? If so, FutureCall might
work.

>>> * It would be nice to be a way to call a callback once per loop
> iteration.
> >>>   An example here is dispatching in libdbus. The easiest way to do
> this is
> >>>   to call dbus_connection_dispatch() every iteration of the loop (a
> more
> >>>   complicated way exists to get notifications when the dispatch status
> >>>   changes, but it is edge triggered and difficult to get right).
> >>>
> >>>   This could possibly be implemented by adding a "repeat" argument to
> >>>   call_soon().
> >>
> >> Again, I'd rather introduce a new method. What should the semantics
> >> be? Is this called just before or after we potentially go to sleep, or
> >> at some other point, or at the very top or bottom of run_once()?
> >
> > That is a good question. Both libuv and libev have both options. The
> > one that is called before we go to sleep is called a "Prepare"
> > handler, the one after we come back from sleep a "Check" handler. The
> > libev documentation has some words on check and prepare handlers here:
> >
> >
> http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#code_ev_prepare_code_and_code_ev_che
> >
> > I am not sure both are needed, but i can't oversee all the consequences.
>
> I'm still not convinced that both are needed. However they are easy to
> add, so if the need really does arise in practical use I am fine with
> evolving the API that way. Until then, let's stick to KISS.
>


> > * libev uses a ~60 second timeout by default. This reason is subtle.
> > Libev supports a wall-clock time event that fires when a certain
> > wall-clock time has passed. Having a non-infinite timeout will allow
> > it to pick up changes to the system time (e.g. by NTP), which would
> > change when the wall-clock timer needs to run.
> >
> > * libuv does not have a wall-clock timer and uses an infinite timeout.
>
> I've not actually ever seen a use case for the wall-clock timer, so
> I've taken it out.
>

If someone really does want a wall-clock timer with a given granularity, it
can be handled by adding a repeating timer with that granularity (with the
obvious consequences for low power modes).


> >> But multiple callbacks per FD seems a different issue -- currently
> >> add_reader() just replaces the previous callback if one is already
> >> set. Since not every event loop can support this, I'm not sure it
> >> ought to be in the PEP, and making it optional sounds like a recipe
> >> for trouble (a library that depends on this may break subtly or only
> >> under pressure). Also, what's the use case? If you really need this
> >> you are free to implement a mechanism on top of the standard in user
> >> code that dispatches to multiple callbacks -- that sounds like a small
> >> amount of work if you really need it, but it sounds like an attractive
> >> nuisance to put this in the spec.
> >
> > A not-so-good use case are libraries like libdbus that don't document
> > their assumptions regarding this. For example, i have to provide an
> > "add watch" function that creates a new watch (a watch is just a
> > generic term for an FD event that can be read, write or read|write). I
> > have observed that it only ever sets one read and one write watch per
> > FD.
> >
> > If we go for one reader/writer per FD, then it's probably fine, but it
> > would be nice if code that does install multiple readers/writers per
> > FD would get an exception rather than silently updating the callback.
> > The requirement could be that you need to remove the event before you
> > can add a new event for the same FD.
>
> That makes sense. If we wanted to be fancy we could have several
> different APIs: add (must not be set), set (may be set), replace (must
> be set). But I think just offering the add and remove APIs is nicely
> minimalistic and lets you do everything else with ease. (I'll make the
> remove API return True if it did remove something, False otherwise.)
>

Perhaps the best bet would be to have the standard API allow multiple
callbacks, and emulate that on systems which don't natively support
multiple callbacks for a single event?

Otherwise, I don't see how an event loop could efficiently expose access to
the multiple callback APIs without requiring awkward fallbacks in the code
interacting with the event loop. Given that the natural fallback
implementation is reasonably clear (i.e. a single callback that calls all
of the other callbacks), why force reimplementing that on users rather than
event loop authors?

Related, the protocol/transport API design may end up needing to consider
the gather/scatter problem (i.e. fanning out data from a single transport
to multiple consumers, as well as feeding data from multiple producers into
a single underlying transport). Actual *implementations* of such tools
shouldn't be needed in the standard suite, but at least understanding how
you would go about writing multiplexers and demultiplexers can be a good
test of a stacked I/O design.

> Just enabling/disabling these events is a bit more friendly to the
> > programmer IMHO than having to cancel and recreate them when needed.
>
> The methods on the Transport class take care of this at a higher
> level: pause() and resume() to suspend reading, and the write() method
> takes care of buffering and so on.
>

And the main advantage of handling that at a higher level is that suitable
buffering designs are going to be transport specific.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121218/ccaae0ef/attachment.html>