[IPython-dev] Input transformation API (IPEP 2 again)

[Thomas Kluyver]

Our input handling actually does two intertwined things: telling when a
block of input is complete and ready to execute, and transforming IPython's
special syntax into regular Python calls. They're intertwined because we
need to handle the special syntax to tell when it's complete.

The different frontends use these in different ways:

- The plain terminal sends lines to InputSplitter as they're entered, and
when InputSplitter says the block is complete, it's sent to run_cell to
execute.
- The Qt console currently does the same thing in the frontend, and sends
the cell to the kernel when it's ready to execute. Users can override the
automatic 'is it complete' using ctrl/shift+return.
- The notebook has no 'is it complete' detection, and simply sends the raw
cell to the notebook when the user executes it.

There are a couple of problems I'd like to address:

- In the line-oriented frontends, transformations are actually done twice:
once for the 'is it complete' detection, and again when run_cell is called.
This is inefficient and confusing.
- The Qt console currently relies on having a frontend copy of part of the
kernel's code. If you add a new input transformer in the kernel, the
frontend won't know about it. This also means you can't write a
line-oriented frontend with 'is it complete' detection in anything but
Python - which I think is a problem, although Brian disagrees.

I'm proposing these changes, which I'll implement in a new pull request
once we've finally dealt with #2447:

1. Add an optional transformed_cell parameter to run_cell(). In the
terminal, the transformed code from InputSplitter will be passed in,
avoiding the need to run those transformations again.

2. Add a new message type to the protocol, is_complete_request &
is_complete_reply. When the Qt console (or another line-oriented frontend)
gets a line of input, it will make an is_complete_request to decide whether
it should accept another line of input, or send the code for execution. The
InputSplitter logic in the Qt console would be removed. There are two
possible APIs:
  a. Send the entire block entered so far on each request to ask if it's
complete (stateless).
  b. Send each line at a time, and have a reset signal to indicate a new
cell. This is more efficient, because the kernel doesn't repeatedly process
the same lines. But it's stateful, so it's more complex to manage.

My preference is for a. In a line-oriented frontend, you're often running
single-line commands anyway, in which case it's no less efficient. Keeping
the API simple seems more important.

Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20130331/401c940a/attachment.html>