[IPython-dev] New hooks suggestion: line buf, cursor pos

[Brian Granger]

>> 1.  The IPython parser+prefilter needs to be block based and the line
>> based logic need to be separated out into a separate Frontend class.
>
> We can make it block-based by handling it block-at-the time, instead
> of line-at-a-time, by just splitting the lines and running the lines
> through prefilter function. I am not a big fan of moving prefiltering
> to frontend classes, so can't comment on that.

Yes, the prefilter is easy to make block based.  But, the parser stuff
is quite a bit more involved and that is the more important part.  I
agree with you that the prefilter (like the parser) should be in the
core, not the frontend.  What *does* need to be in the frontend (once
the core is block based) is all the extra logic (for line based
interfaces) that handles line based input - prompts, when is a block
complete, etc.

>> 2.  Everything must be refactored in a way that the frontend can run
>> in a different process.
>>
>> This means that the frontend must be a separate class that talks to
>> the core through a clean well defined API that *can't* be hacked like
>> the current ipapi.IP hack that goes on.
>
> The ipapi.IP hacks are mostly there because of the unwillingness to
> add to the api (especially regarding something unimportant). Frontends
> shouldn't really need it. There is nothing hugely important that
> mandates its use anyway, so I won't expect you to hit any problems.

This one is definitely easier.  But, I think we should expand the
ipapi interface if this is the reason we haven't.

>> 3.  We need a way of capturing stdout/stderr/stdin that allows things
>> to be done over a network connection.  See IPython.kernel.core for how
>> we do this now, but this is not even sufficient as 1) it doesn't
>> handle stdin and 2) it only returns stdout/stderr when a block is done
>> executing.  Thus, the following won't do what you expect:
>>
>> for i in range(10):
>>    time.sleep(1)
>>    print i
>>
>> (it prints is all at the end rather than asynchronously).
>
> The stdin is the bigger problem, but normal stdout capturing should
> take care of the partial stdout problem rather easily. Can't you just
> send the data over the network connection?

For stdout we can, but making is really asynchronous is not trivial,
but it is doable.  Stdin will be a pain in the @$$.

>> 4.  The core needs to be threadsafe and allow Twisted to be run in a
>> thread.  This is needed because currently (for example in the new
>> TextMate bundle) we are calling the core (ipapi.runlines) in a very
>> thread-unsafe manner.  This will require rethinking from the ground up
>> how we handle the threaded shells and GUI integration.
>
> Threaded GUI shells can probably be ignored in the future, since it's
> questionable whether they are even required.

In what sense?  Obviously, people will continue to want to use GUIs in
conjunction with IPython, correct?

>>
>> 5.  The output of certain things like tracebacks and other colored
>> text, needs to be represented in a way (i.e., without ASCI coloring)
>> that is independent of the terminal.  The best choices are something
>> like XML or Python dicts.  Otherwise, non terminal based frontends
>> will have a though time coloring such things.
>
> It's actually quite easy to color that stuff, as it's already done in
> various gui frontends. If I were you, I wouldn't give this particular
> issue much thought.

Nice, what do you think the best way of handling this is?  Just
convert the ASCI colors to whatever the frontend needs?  What about
web based frontends?

>> configured/customized on the fly.  While the hook system is better
>> than nothing, my feeling is that it exists to simply cover up the
>> nasty, monolithic design of iplib.  kernel.core.interpreter has the
>> beginnings of a cleaner design that is much more modular an loosely
>> coupled.
>
> But some kind of hook system will still be needed, so let's not remove
> it before replacement is ready.

Absolutely!!!  This is one of the big benefits of having only one code
base now - every merge into trunk that we do must keep everything in a
completely working and tested state.  That is why we it is so
important for *everything* to have tests.  Obviously, we will have to
break the API at times, but that will be a deliberate decision to
improve IPython.

Brian