[IPython-dev] !commands in the zmq model

[Fernando Perez]

Howdy,

I've spent hours on getting a better implementation of

!cmd

in the multiprocess model, that:

1. shows output as it happens
2. is cleanly interruptible.

We can interrupt subprocesses fine with our current
(subprocess.Popen-based) implementation, but we can't get their output
cleanly as it happens.  So in the usual:

    for i in range(200):
        print i,
        sys.stdout.flush()
        time.sleep(0.01)
    print 'done!'

We only see *all* the numbers at the very end. I haven't found any way
around this with subprocess, and neither did our old ipythonx
implementation done by Gael; as best as I understand popen it simply
can't be done: in pipe mode, the C stdio library does block-buffered
io on all process execution and there's no clean way I can find, to
read from a pipe with a timeout.

But the good news is that I think I have an implementation that will
work on *nix (linux/mac), using pexpect.  On Windows we'll have to
fall back to subprocess.popen(), with its limitations.

In addition, the pexpect implementation gives us the benefit of
correctly formatted 'ls' output, reverting the surprise Brian had
initially from ls being only on one column.  Pexpect creates a proper
pseudo-tty, and knows how to read from it very intelligently.

It's a shame it doesn't exist on windows, but it's apparently a very
non-trivial task to port it (I remember hearing William Stein several
times comment on how good it would be for Sage to have pexpect on
Windows, and knowing them, if it was doable they would have already
done it).

I may not finish this today, I'm too exhausted, but if anyone knows
this type of problem well, pitch in.  I'm sure I'll make good use of
any help...

Cheers,

f

ps - for reference, the current implementation is along the lines of:

from __future__ import print_function

import sys
import pexpect

self = get_ipython()

if 1:
    def system( cmd):
        cmd = self.var_expand(cmd, depth=2).strip()
        sh = '/bin/bash'
        timeout = 0.05 # seconds
        pcmd = '%s -c %r' % (sh, cmd)
        try:
            child = pexpect.run(pcmd, logfile=sys.stdout)

            ## child = pexpect.spawn(sh, ['-c', cmd])
            ## while True:
            ##     res = child.expect([pexpect.TIMEOUT, pexpect.EOF], timeout)
            ##     if res==0:
            ##         #pass
            ##         print(child.before, end='')
            ##     elif res==1:
            ##         break

        except KeyboardInterrupt:
            print('\nInterrupted command: %r.' % cmd, file=sys.stderr)

            #return child