Configuring the REPL's tab completion

[Chris Angelico]

On Mon, Mar 11, 2019 at 9:38 PM Thomas Jollans <tjol at tjol.eu> wrote:
>
> On 10/03/2019 15.20, Chris Angelico wrote:
> > I have absolutely no idea how to do this or even where to go looking,
> > so I'd appreciate a starting pointer :)
> >
> > When you're in the Python REPL (just the basic core one, not IDLE or
> > anything), you can tab-complete global and built-in names, attributes
> > of known objects, etc. But quoted strings work kinda weirdly - they
> > try to tab-complete a global or keyword:
> >
> > Python 3.8.0a0 (heads/master:8b9c33ea9c, Nov 20 2018, 02:18:50)
> > [GCC 6.3.0 20170516] on linux
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> "in
> > in      input(  int(
> >>>> "input("
> > 'input('
> >
> > I typed "in and hit tab twice, then typed p and hit tab, enter. It
> > filled in the function name *input*, added an open parenthesis... and
> > then closed the quote. Which doesn't make a lot of sense, but then,
> > tab completing globals and keywords inside a text string doesn't make
> > that much sense either.
> >
> > What would be more useful would be tab-completing file names from the
> > current directory.
> >
> > open("Foo<tab>
> >
> > to fill in the name of a file. Sure, not every quoted string is a file
> > name (in fact, very few are), but I don't know of anything else that
> > would feel natural. (Also, Pike's REPL behaves this way, so presumably
> > it's of use to more people than me.)
> >
> > Where would I start looking to try to make this happen? Doesn't
> > necessarily have to be pushed upstream as a core Python feature; I'm
> > guessing this can probably be done in sitecustomize.py. Anyone have
> > tutorials on messing with tab completion? There's not a lot of info in
> > the rlcompleter module docs.
>
> I had a quick look at the code -
>
> rlcompleter sets itself as the completer
> <https://github.com/python/cpython/blob/master/Lib/rlcompleter.py#L200>
> using readline.set_completer
> <https://docs.python.org/3.6/library/readline.html#readline.set_completer>
>
> I imagine you could write your own completer, and call set_completer
> again after rlcompleter has been imported.

Hmm. Been tinkering with this a bit, and it's not easy to get info
about whether the word to be completed is inside a quoted string.
There MAY be more info available if I use the third-party 'rl' module
[1], but on playing with that, I managed to bomb Python hard with a
SIGABRT. Not promising.

I get the feeling that this ought to be really really easy, yet
there's nothing obvious saying "do this, you can interact with the
completer that way". Maybe the problem is that the CPython core
"readline" module simply doesn't have the functionality I want, but if
that's the case, I'll need to figure out why on earth my simple
probing of rl.readline is crashing hard. It'd be way better if I could
work entirely with the stdlib.

So here's my current exploration:

class Completer(rlcompleter.Completer):
    def complete(self, text, state):
        ret = super().complete(text, state)
        txt = readline.get_line_buffer()[readline.get_begidx():readline.get_endidx()]
        print("complete(%r, %r) -> %r [%r]" % (text, state, ret, txt),
file=tracer)
        tracer.flush()
        return ret

readline.set_completer(Completer().complete)

With this, I can inspect the current buffer and find the parts of it
that are being flagged by the completer. (txt and text should always
have the same value.) Figuring out whether something is inside a
quoted string is actually a bit of a hard problem, but maybe I'll
cheat and say "if you're tab-completing a word immediately after a
quote character, it's a file name". I think coping with triple-quoted
strings across multiple input lines is so hard that it's not worth
attempting, but it'd be nice to be able to correctly handle
single-line strings.

This really does feel like it ought to be a lot simpler, though. GNU
Readline has file name facilities built-in, I believe; it certainly
has quoted string detection (see, for instance, the way it autocloses
a string with current Python behaviour). I must be missing something
here.

[1] https://pythonhosted.org/rl/index.html

ChrisA