[IPython-dev] Input transformation rework - input requested

[Jason Grout]

It would be good to post separately to the Sage list. The code has changed
since I was last in it, but I think these are the relevant bits:

Register transforms:
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/ipython_extension.py#L504-L513

Preparsing: Defined at
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L379,
actual implementation at
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/preparse.py

Prompts:
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L416

There's also another transformation, it seems:
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L468

Thanks,

Jason




On Mon, May 28, 2018 at 5:27 AM Thomas Kluyver <takowl at gmail.com> wrote:

> Thanks Aaron. If working with valid Python syntax is sufficient, the API
> will probably just be functions that take and return a string, leaving it
> up to you how you tokenise/parse the code. We only need to do something
> more complex if third-party transformations need to integrate with our own
> ones to produce valid Python.
>
> Does anyone know what Sage does? Jason mentioned that they might also use
> input transformations.
>
> On 28 May 2018 at 08:13, Aaron Meurer <asmeurer at gmail.com> wrote:
>
>> I believe you're right.
>>
>> I suppose another potential transform would be to allow x^2 to
>> represent x**2. This would have be done at the token level since ^ has
>> a different precedence than **.
>>
>> Only operating on valid Python seems fine. If someone wants to
>> implement a DSL they should just write a kernel.
>>
>> Have you considered using something like parso, which can manipulate
>> an AST and tokens losslessly?
>>
>> Aaron Meurer
>>
>> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <takowl at gmail.com>
>> wrote:
>> > Thanks Aaron. I am aware that Sympy uses input transformation, but it
>> would
>> > be good to check some more details of exactly how:
>> >
>> > 1. I believe wrapping integers and defining symbols can both be done
>> with
>> > AST transformations. Let me know if I'm missing something.
>> > 2. Wrapping floats can't, because the string form is discarded before
>> we get
>> > the AST. But this is still an operation that changes one syntactically
>> valid
>> > piece of Python code to another, right? I'm considering limiting the
>> API to
>> > only operate on already valid Python code, and using private APIs to
>> handle
>> > IPython's special syntax (like %magics).
>> > 3. Are there any other transformations that Sympy does (or wants to do)
>> > besides these three?
>> >
>> > On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:
>> >>
>> >> I'm sure you're already aware, but we use them for SymPy for some
>> >> (optional) transformers that make Python more symbolic friendly
>> >> (auto-replace undefined variables with Symbols, wrap integer literals
>> >> with Integer so that exact rational numbers work, and so on). The AST
>> >> transformer is nice but limiting. We can't use it to wrap float
>> >> literals with higher precision for instance because Python drops the
>> >> precision of floats in the AST.
>> >>
>> >> There are already some open issues about some limitations we've seen
>> >> (https://github.com/sympy/sympy/issues/14440 and
>> >> https://github.com/ipython/ipython/issues/10893). I'm not fully
>> >> updated on the specifics of what the limitations were, but from what I
>> >> remember, there were issues with IPython doing things line-by-line.
>> >>
>> >> I don't know how the new code is architected, but it would be nice if
>> >> the transformers could just access the raw input from IPython, and
>> >> various transformers could be plugged in to that to make things
>> >> simpler if desired (the AST transformer could be one such instance).
>> >>
>> >> At the very least, if you want to test the new design, you could look
>> >> at translating our transformers in SymPy (including the work in
>> >> progress float transformer at
>> >> https://github.com/sympy/sympy/pull/13300).
>> >>
>> >> Aaron Meurer
>> >>
>> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
>> >> wrote:
>> >> > Thomas this is great news! I think that part of the code base could
>> >> > benefit
>> >> > from being simpler and more extensible. Thanks for tackling this!
>> >> >
>> >> > Sent from my iPhone
>> >> >
>> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com>
>> wrote:
>> >> >
>> >> > Hi all,
>> >> >
>> >> > For IPython 7, I'm planning a rework of the input transformation
>> >> > framework:
>> >> > https://github.com/ipython/ipython/pull/11041
>> >> >
>> >> > The new framework is - I hope - simpler than the one it replaces, and
>> >> > will
>> >> > hopefully have fewer weird corner cases. But it's still a pretty
>> complex
>> >> > beast, and I don't feel like it's a good platform for people to add
>> more
>> >> > transformations to. One option to improve this is to make a smaller
>> >> > extensible API, restricting the kind of things that third party code
>> can
>> >> > do.
>> >> >
>> >> > I don't think there's ever been much third party code extending
>> >> > IPython's
>> >> > input transformation, but I'd like to find out about the code that
>> does.
>> >> > So
>> >> > if you're aware of a project that does use our input transformation
>> API:
>> >> >
>> >> > - What project?
>> >> > - What does it want to transform?
>> >> > - Can you point me to the code?
>> >> > - What fallback options would it have if IPython didn't support the
>> >> > transformation it wanted?
>> >> >
>> >> > This is specifically about text transformations; AST transformations
>> are
>> >> > not
>> >> > changed.
>> >> >
>> >> > Thanks,
>> >> > Thomas
>> >> >
>> >> > _______________________________________________
>> >> > IPython-dev mailing list
>> >> > IPython-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > IPython-dev mailing list
>> >> > IPython-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >> >
>> >> _______________________________________________
>> >> IPython-dev mailing list
>> >> IPython-dev at python.org
>> >> https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at python.org
>> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at python.org
>> https://mail.python.org/mailman/listinfo/ipython-dev
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180528/317794ee/attachment.html>