[IPython-dev] New hooks suggestion: line buf, cursor pos

[Brian Granger]

> So what's missing from the IPython core that you would like to use?
> Perhaps there is something we should add to ipapi? IPython core should
> be pretty easy to extend in the fashion that allows it to do whatever
> it is you need to integrate your other stuff with it.

Good questions.

1.  The IPython parser+prefilter needs to be block based and the line
based logic need to be separated out into a separate Frontend class.

Currently, both the prefilter and parser operate in a line based
fashion.  Thus, even for your nice block based Qt frontend, it is not
*really* block based.  Most importantly, you have to have multiple
newlines to end a block of code.  That is not valid Python though.

This logic in currently in runsource, runcode, push, raw_input and friends.

We have a true block based parser in the execute_python method here:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/1152?file_id=interpreter.py-20080606193734-psuinzq9276j4ah1-42

We also have the basics of a frontend here:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/files/1152?file_id=frontend-20080611225339-bjp8mnnd2tnxj9n5-1

The default prefilter should still be lines based though (once the
parsing has happened, but it should be possible to have a completely
block based prefilter.

2.  Everything must be refactored in a way that the frontend can run
in a different process.

This means that the frontend must be a separate class that talks to
the core through a clean well defined API that *can't* be hacked like
the current ipapi.IP hack that goes on.

3.  We need a way of capturing stdout/stderr/stdin that allows things
to be done over a network connection.  See IPython.kernel.core for how
we do this now, but this is not even sufficient as 1) it doesn't
handle stdin and 2) it only returns stdout/stderr when a block is done
executing.  Thus, the following won't do what you expect:

for i in range(10):
    time.sleep(1)
    print i

(it prints is all at the end rather than asynchronously).

4.  The core needs to be threadsafe and allow Twisted to be run in a
thread.  This is needed because currently (for example in the new
TextMate bundle) we are calling the core (ipapi.runlines) in a very
thread-unsafe manner.  This will require rethinking from the ground up
how we handle the threaded shells and GUI integration.

5.  The output of certain things like tracebacks and other colored
text, needs to be represented in a way (i.e., without ASCI coloring)
that is independent of the terminal.  The best choices are something
like XML or Python dicts.  Otherwise, non terminal based frontends
will have a though time coloring such things.

6.  Because the frontend and the core can be run in different
processes, we need to rethink the usage of hooks and how IPython is
configured/customized on the fly.  While the hook system is better
than nothing, my feeling is that it exists to simply cover up the
nasty, monolithic design of iplib.  kernel.core.interpreter has the
beginnings of a cleaner design that is much more modular an loosely
coupled.

7.  The core needs to have a super clean API that is based only on
methods, not attributes.  Otherwise it is difficult to propagate
things over a network connection.  We have a sketch of such an API in
kernel.core.interpreter.

8.  As you mention, readline needs to be factored out, but when this
is done, we need to create the frontend that is a completely separate
class.

My challenge is that I don't know the code base of the existing core
very well.  Plus there are some really subtle things going on -
especially with threads.  It will take me a long time to get up to
speed on these things so I an make the needed changes.  But, I may end
up doing this.  I would love to get your feedback on these things.

Cheers,

Brian



>> * What do you think the main challenges are? (I have my own list, but
>> I am curious what you have in mind).
>
> I don't really see the challenges, but that depends on your set of
> requirements. I would "just do it" and see what breaks.
>
>> * What do you think the best approach is to move forward on all of this stuff?
>
> Again, it depends on what you mean by "all this stuff". Personally, I
> would just use normal IPython as the engine, with necessary output
> trapping as needed. If you list the problems you are concerned about,
> I can give you more specific answers.
>
> --
> Ville M. Vainio
> http://tinyurl.com/vainio
>