[IPython-dev] Coordinating the XREQ and SUB channels

[Brian Granger]

Evan,

On Wed, Jul 14, 2010 at 2:21 PM, Evan Patterson <epatters at enthought.com> wrote:
> On Wed, Jul 14, 2010 at 2:10 PM, Brian Granger <ellisonbg at gmail.com> wrote:
>>
>> On Wed, Jul 14, 2010 at 10:56 AM, Evan Patterson <epatters at enthought.com>
>> wrote:
>> > Hi guys,
>> >
>> > I've been making decent progress at connecting my FrontendWidget to a
>> > KernelManager. I have, however, encountered one fairly serious problem:
>> > since the XREQ and SUB channels of the KernelManager are in separate
>> > threads, there is no guarantee about the order in which signals are
>> > emitted.
>> > I'm finding that 'execute_reply' signals are frequently emitted *before*
>> > all
>> > the output signals have been emitted.
>>
>> Yes, that is definitely possible and we really don't have control over
>> it.  Part of the difficulty is that the SUB/SUB channel does buffering
>> of stdout/stderr (just like sys.stdout/sys.stderr).  While it will
>> make your application logic more difficult, I think this is something
>> fundamental we have to live with.  Also, I wouldn't be surprised if
>> the same were true of the regular python shell because of the
>> buffering of stdout.
>
> I'm not sure it's fair to call this problem fundamental (if we ignore the
> corner case of the threads in the kernel). After all, output and execution
> completion happen in a very predictable order in the kernel; it's only our
> use of multiple frontend-side channel threads that has complicated the
> issue.

But this leaves out one of the main causes of unpredictability in a
distributed system:  network and IO latency.  I  our architecture,
this occurs when we ask 0MQ to send a message.  At that point, it is
up to 0MQ, the OS Kernel, the network stack (including routers, etc.)
to get deliver the messages in the best way they can.  In a
multi-channel model like this, there is simple no promise that the
order in which we send messages is the order in which tey will arrive.
 This is a fundamental issue that I consider a feature of our current
architecture, because, in my experience, if you artificially try to
impose determinacy on network traffic you end up with extremely
awkward error handing.

> In a regular same-process shell, this wouldn't be a problem because you
> would simply flush stdout before writing the new prompt. It makes sense to
> be able to request a flush here, I think. A 'flush' in this case would just
> consist of the making the SubChannel thread active, so that its event loop
> would pick up whatever it needs to. I believe calling time.sleep(0) once in
> the XReqChannel before sending an execute reply will be sufficient. The
> latency introduced should be negligible. I'll experiment with this.

OK, I think this is worth a shot.

>>
>> > It seems to me that we should be enforcing, to the extent that we can
>> > (i.e.
>> > ignoring threads in the kernel for now), the assumption that when
>> > 'execute_reply' is signaled, all output has been signaled. Is this
>> > reasonable?

Because of the networking issues, I don't think so.

>> I don't think so.  I would write the frontend to allow for arbitrary
>> timing of the execute_reply and the SUB messages.  You will have to
>> use the parent_id information to order things properly in the GUI.
>> Does this make sense.  I think if we try to impose the timing of the
>> signals, we will end up breaking the model or introducing extra
>> latencies.  Let us know if you have questions.  I know this will
>> probably be one of the more subtle parts of the frontend logic.
>
> Yes, this is something that will be quite difficult to get right. For
> frontend implementors who are interested only in console-style interaction,
> it doesn't make sense for them to have worry about this.

Definitely hard to get right and terminal based frontends will
definitely need something like flush.  Let's see how it goes with this
approach.

Brian

> Evan
>
>>
>> Cheers,
>>
>> Brian
>>
>>
>>
>>
>> > Evan
>> >
>>
>>
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com