[IPython-dev] Sage customizations

[Fernando Perez]

[ Finishing an old draft... ]

On Tue, Sep 11, 2012 at 2:24 PM, Jason Grout
<jason-sage at creativetrax.com> wrote:
> Fernando recently posted that we should be able to do the Sage
> customizations to ipython via just config changes and extensions.  As
> I'm getting deeper into what we do, I see the following obstacles.
> Should we be able to do these changes from config changes and extensions?

Perhaps I should have hedged slightly: you should be able to do *most*
of Sage as config/extensions.  It's obviously possible that you will
come up with something that you guys want to do that we decide isn't
worth another config/extension point, or that proves more practical
via subclassing.  But at least bringing these points up here will let
us understand those boundaries and draw them deliberately, rather than
have them be whatever the vagaries of historical evolution dictate.

> * interactiveshell system_raw function: Sage messes with the
> LD_LIBRARY_PATH so that programs included in Sage work correctly.
> However, if we're calling out to a system program installed outside of
> Sage, we need to reset the LD_LIBRARY_PATH.  So right now we override
> system_raw to set the correct LD_LIBRARY_PATH if we don't recognize the
> command as a sage-provided command.  What if we could have pre- and
> post- system_raw hooks, where we could run these commands?  I can
> imagine that these would also be useful if you wanted to always set a
> current directory for raw system commands, etc.  I suppose the same
> issue applies to the system_piped command.

I don't know if adding more hooks is the right answer; interaction
with the OS is a messy affair, partly because there are subtle
differences between platforms and the results aren't the same
depending on whether you use os.system, subprocess.* or pexpect.  In
ipython, over the years all the code to manage that has grown a little
weedy and scattered (though there have been cleanups, so it's not a
complete mess either).

What I'd like to do is to have an IPEP where we rationalize the entire
design of how we call to the OS: we define the various special
syntaxes for it explicitly, indicate which tools are used under the
hood to to the job, etc.  Having such a document lay out the
boundaries of this problem explicitly, would make it much easier then
to consider what the best solutions for user customizations are.
Perhaps in the end it will be hooks, perhaps not.  But I'd like to see
the whole chessboard in one shot, including a listing of the various
needs (such as yours) before adding more knobs to tweak.

> * SageTerminalApp inherits from TerminalIPythonApp and overrides the
> crash handler to print out custom text for reporting crashes to Sage.
> Is there a way to get the app, given the shell?  If there was, I could
> override this as part of an extension, but I'm not sure if that's the
> most elegant way to do things.  Having a SageTerminalApp also lets us
> set the name to be "Sage", and just seems more natural in some ways.

apps are meant to be per-process singletons, so you can get it this way:

In [1]: from IPython.frontend.terminal.ipapp import TerminalIPythonApp

In [2]: TerminalIPythonApp.instance()
Out[2]: <IPython.frontend.terminal.ipapp.TerminalIPythonApp at 0x7f6095cc3190>


> Here are several sticky points that seem like they should be easier (or
> at least less controversial) to solve
>
> * transformations are stateless (they only depend on a single line, and
> shouldn't store state) --- what is the rationale behind this?  For
> example, we'd like to do a dedenting transformation (useful if we are
> pasting lots of text, after we turn off autoindenting, or even if we are
> pasting a lot of single commands with autoindenting).  Of course, to do
> a dedenting transformation, we need to know the indentation of the first
> line, so it is not stateless.  Looking at the IPython code, it would be
> trivial to pass the line number in with the line string to transformations.

The rationale was simplicity: if transformations are stateless and
purely functional, they can be tested in isolation and composed in
different ways without side effects.  State means potential for side
effects, weird behavior arising from seemingly innocuous problems,
etc.  Obviously some times state *is* necessary in a problem, and
objects are one approach to manage state with some degree of control
and containment.  But they also raise the complexity bar significantly
for a design.

Over the years, I've become a big fan of keeping software designs as
purely functional as possible, layering object orientation and state
only in a controlled manner.  In this case, we should look precisely
at what information would be necessary to have for your use cases, and
think about the simplest design to achieve that.  I'm not convinced
that turning the transformations system into a big state machine is a
price worth paying.

> * oinspect -- we need to monkey-patch ipython to replace these functions:
>
> IPython.core.oinspect.getdoc = sageinspect.sage_getdoc
> IPython.core.oinspect.getsource = sagedoc.my_getsource
> IPython.core.oinspect.getargspec = sageinspect.sage_getargspec
>
> because these functions are explicitly imported and used in various
> places in IPython.  Rather, could IPython use, say, something tied to an
> inspector object attached to the shell or something?  Then we could just
> set those attributes.

Well, but we already use an object, it's called a module :) I'm only
half-joking here: you're asking us to artificially bundle a group of
functions that otherwise have no reason to share state whatsoever,
whose signature doesn't need a 'self' anywhere, so that you can then
grab that object and modify it.  But python offers precisely one
abstraction for grouping related functions that otherwise need to
share no state together: the module!  And it's an object, whose
attributes can be set by users who need to modify the default behavior
of the object.  I don't see the point in creating another layer so
that you can set your changes in a new object instead of the module,
frankly.


We certainly want to make the customization and building of derivative
systems as clean as possible, so this kind of discussion is very
important.  We'll have to iterate to find a balanced design, and in
the end it may be that for something as complex as Sage a subclass is
still the right approach and not simply a rich profile.  I hope this
helps!

Cheers,

f