[Python-ideas] The async API of the future: Twisted and Deferreds
Laurens Van Houtven
_ at lvh.cc
Sat Oct 13 21:13:09 CEST 2012
I quite like IOStream's interface, actually. If that's part of the
transport layer, how do you prevent from having duplicating its behavior
(read_until etc)? If there's just another separate object that would be the
ITransport in twisted, I think the difference is purely one of labeling.
On Sat, Oct 13, 2012 at 8:54 PM, Ben Darnell <ben at bendarnell.com> wrote:
> On Sat, Oct 13, 2012 at 10:49 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
> > Interesting. That's certainly a nice API, but that then again
> (read_until)
> > sounds like something I'd implement using dataReceived... You know,
> > read_until clears the buffer, logs the requested callback. data_received
> > adds something to the buffer, and checks if it triggered the (one of
> the?)
> > registered callbacks.
>
> Right, that's how IOStream is implemented internally. The
> transport/protocol split works a little differently in Tornado:
> IOStream is implemented something like a Protocol subclass, but we
> consider it a part of the transport layer. The "protocols" are
> arbitrary classes that don't share any particular interface, but
> instead just call methods on the IOStream.
>
> -Ben
>
> >
> > Of course, I may just be rusted in my ways and trying to implement
> > everything in terms of things I know (then again, that might be just
> what's
> > needed when you're trying to make a useful general API).
> >
> > I guess it's time for me to go deep-diving into Tornado :)
> >
> >
> > On Sat, Oct 13, 2012 at 7:27 PM, Ben Darnell <ben at bendarnell.com> wrote:
> >>
> >> On Sat, Oct 13, 2012 at 10:18 AM, Laurens Van Houtven <_ at lvh.cc> wrote:
> >> > What calls on_headers in this example? Coming from twisted, that seems
> >> > like
> >> > dataReceived's responsibility, but given your introductory paragraph
> >> > that's
> >> > not actually what goes on here?
> >>
> >> The IOStream does, after send_request calls
> >> stream.read_until("\r\n\r\n", on_headers). Inside IOStream, there is
> >> a _handle_read method that is registered with the IOLoop and fills up
> >> a buffer. When the read condition is satisfied the IOStream calls
> >> back into application code.
> >>
> >> -Ben
> >>
> >> >
> >> >
> >> > On Sat, Oct 13, 2012 at 7:07 PM, Ben Darnell <ben at bendarnell.com>
> wrote:
> >> >>
> >> >> On Fri, Oct 12, 2012 at 11:14 PM, Antoine Pitrou <
> solipsis at pitrou.net>
> >> >> wrote:
> >> >> > On Fri, 12 Oct 2012 15:11:54 -0700
> >> >> > Guido van Rossum <guido at python.org> wrote:
> >> >> >>
> >> >> >> > 2. Method dispatch callbacks:
> >> >> >> >
> >> >> >> > Similar to the above, the reactor or somebody has a handle
> on
> >> >> >> > your
> >> >> >> > object, and calls methods that you've defined when events happen
> >> >> >> > e.g. IProtocol's dataReceived method
> >> >> >>
> >> >> >> While I'm sure it's expedient and captures certain common patterns
> >> >> >> well, I like this the least of all -- calling fixed methods on an
> >> >> >> object sounds like a step back; it smells of the old Java way
> >> >> >> (before
> >> >> >> it had some equivalent of anonymous functions), and of asyncore,
> >> >> >> which
> >> >> >> (nearly) everybody agrees is kind of bad due to its insistence
> that
> >> >> >> you subclass its classes. (Notice how subclassing as the prevalent
> >> >> >> approach to structuring your code has gotten into a lot of
> discredit
> >> >> >> since 1996.)
> >> >> >
> >> >> > But how would you write a dataReceived equivalent then? Would you
> >> >> > have
> >> >> > a "task" looping on a read() call, e.g.
> >> >> >
> >> >> > @task
> >> >> > def my_protocol_main_loop(conn):
> >> >> > while <some_condition>:
> >> >> > try:
> >> >> > data = yield conn.read(1024)
> >> >> > except ConnectionError:
> >> >> > conn.close()
> >> >> > break
> >> >> >
> >> >> > I'm not sure I understand the problem with subclassing. It works
> fine
> >> >> > in Twisted. Even in Python 3 we don't shy away from subclassing,
> for
> >> >> > example the IO stack is based on subclassing RawIOBase,
> >> >> > BufferedIOBase,
> >> >> > etc.
> >> >>
> >> >> Subclassing per se isn't a problem, but requiring a single
> >> >> dataReceived method per class can be awkward. Many protocols are
> >> >> effectively state machines, and modeling each state as a function can
> >> >> be cleaner than a big if/switch block in dataReceived. For example,
> >> >> here's a simplistic HTTP client using tornado's IOStream:
> >> >>
> >> >> from tornado import ioloop
> >> >> from tornado import iostream
> >> >> import socket
> >> >>
> >> >> def send_request():
> >> >> stream.write("GET / HTTP/1.0\r\nHost:
> >> >> friendfeed.com\r\n\r\n")
> >> >> stream.read_until("\r\n\r\n", on_headers)
> >> >>
> >> >> def on_headers(data):
> >> >> headers = {}
> >> >> for line in data.split("\r\n"):
> >> >> parts = line.split(":")
> >> >> if len(parts) == 2:
> >> >> headers[parts[0].strip()] = parts[1].strip()
> >> >> stream.read_bytes(int(headers["Content-Length"]),
> on_body)
> >> >>
> >> >> def on_body(data):
> >> >> print data
> >> >> stream.close()
> >> >> ioloop.IOLoop.instance().stop()
> >> >>
> >> >> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
> >> >> stream = iostream.IOStream(s)
> >> >> stream.connect(("friendfeed.com", 80), send_request)
> >> >> ioloop.IOLoop.instance().start()
> >> >>
> >> >>
> >> >> Classes allow and encourage broader interfaces, which are sometimes a
> >> >> good thing, but interact poorly with coroutines. Both twisted and
> >> >> tornado use separate callbacks for incoming data and for the
> >> >> connection being closed, but for coroutines it's probably better to
> >> >> just treat a closed connection as an error on the read. Futures (and
> >> >> yield from) give us a nice way to do that.
> >> >>
> >> >> -Ben
> >> >> _______________________________________________
> >> >> Python-ideas mailing list
> >> >> Python-ideas at python.org
> >> >> http://mail.python.org/mailman/listinfo/python-ideas
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > cheers
> >> > lvh
> >> >
> >
> >
> >
> >
> > --
> > cheers
> > lvh
> >
>
--
cheers
lvh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121013/c2084236/attachment.html>
More information about the Python-ideas
mailing list