From takowl at gmail.com  Sun May 27 16:19:00 2018
From: takowl at gmail.com (Thomas Kluyver)
Date: Sun, 27 May 2018 22:19:00 +0200
Subject: [IPython-dev] Input transformation rework - input requested
Message-ID: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>

Hi all,

For IPython 7, I'm planning a rework of the input transformation framework:
https://github.com/ipython/ipython/pull/11041

The new framework is - I hope - simpler than the one it replaces, and will
hopefully have fewer weird corner cases. But it's still a pretty complex
beast, and I don't feel like it's a good platform for people to add more
transformations to. One option to improve this is to make a smaller
extensible API, restricting the kind of things that third party code can do.

I don't think there's ever been much third party code extending IPython's
input transformation, but I'd like to find out about the code that does. So
if you're aware of a project that does use our input transformation API:

- What project?
- What does it want to transform?
- Can you point me to the code?
- What fallback options would it have if IPython didn't support the
transformation it wanted?

This is specifically about text transformations; AST transformations are
not changed.

Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180527/d9e8439b/attachment.html>

From ellisonbg at gmail.com  Sun May 27 16:50:46 2018
From: ellisonbg at gmail.com (Brian Granger)
Date: Sun, 27 May 2018 13:50:46 -0700
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
Message-ID: <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>

Thomas this is great news! I think that part of the code base could benefit from being simpler and more extensible. Thanks for tackling this!

Sent from my iPhone

> On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> 
> Hi all,
> 
> For IPython 7, I'm planning a rework of the input transformation framework:
> https://github.com/ipython/ipython/pull/11041
> 
> The new framework is - I hope - simpler than the one it replaces, and will hopefully have fewer weird corner cases. But it's still a pretty complex beast, and I don't feel like it's a good platform for people to add more transformations to. One option to improve this is to make a smaller extensible API, restricting the kind of things that third party code can do.
> 
> I don't think there's ever been much third party code extending IPython's input transformation, but I'd like to find out about the code that does. So if you're aware of a project that does use our input transformation API:
> 
> - What project?
> - What does it want to transform?
> - Can you point me to the code?
> - What fallback options would it have if IPython didn't support the transformation it wanted?
> 
> This is specifically about text transformations; AST transformations are not changed.
> 
> Thanks,
> Thomas
> 
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180527/63131738/attachment.html>

From asmeurer at gmail.com  Sun May 27 18:02:32 2018
From: asmeurer at gmail.com (Aaron Meurer)
Date: Sun, 27 May 2018 16:02:32 -0600
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
Message-ID: <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>

I'm sure you're already aware, but we use them for SymPy for some
(optional) transformers that make Python more symbolic friendly
(auto-replace undefined variables with Symbols, wrap integer literals
with Integer so that exact rational numbers work, and so on). The AST
transformer is nice but limiting. We can't use it to wrap float
literals with higher precision for instance because Python drops the
precision of floats in the AST.

There are already some open issues about some limitations we've seen
(https://github.com/sympy/sympy/issues/14440 and
https://github.com/ipython/ipython/issues/10893). I'm not fully
updated on the specifics of what the limitations were, but from what I
remember, there were issues with IPython doing things line-by-line.

I don't know how the new code is architected, but it would be nice if
the transformers could just access the raw input from IPython, and
various transformers could be plugged in to that to make things
simpler if desired (the AST transformer could be one such instance).

At the very least, if you want to test the new design, you could look
at translating our transformers in SymPy (including the work in
progress float transformer at
https://github.com/sympy/sympy/pull/13300).

Aaron Meurer

On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com> wrote:
> Thomas this is great news! I think that part of the code base could benefit
> from being simpler and more extensible. Thanks for tackling this!
>
> Sent from my iPhone
>
> On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com> wrote:
>
> Hi all,
>
> For IPython 7, I'm planning a rework of the input transformation framework:
> https://github.com/ipython/ipython/pull/11041
>
> The new framework is - I hope - simpler than the one it replaces, and will
> hopefully have fewer weird corner cases. But it's still a pretty complex
> beast, and I don't feel like it's a good platform for people to add more
> transformations to. One option to improve this is to make a smaller
> extensible API, restricting the kind of things that third party code can do.
>
> I don't think there's ever been much third party code extending IPython's
> input transformation, but I'd like to find out about the code that does. So
> if you're aware of a project that does use our input transformation API:
>
> - What project?
> - What does it want to transform?
> - Can you point me to the code?
> - What fallback options would it have if IPython didn't support the
> transformation it wanted?
>
> This is specifically about text transformations; AST transformations are not
> changed.
>
> Thanks,
> Thomas
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>

From takowl at gmail.com  Mon May 28 01:50:33 2018
From: takowl at gmail.com (Thomas Kluyver)
Date: Mon, 28 May 2018 07:50:33 +0200
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
 <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>
Message-ID: <CAOvn4qiKjHDbUS1OPeRHDU_kO79gn2rt19NFzjgygGfEw2YriQ@mail.gmail.com>

Thanks Aaron. I am aware that Sympy uses input transformation, but it would
be good to check some more details of exactly how:

1. I believe wrapping integers and defining symbols can both be done with
AST transformations. Let me know if I'm missing something.
2. Wrapping floats can't, because the string form is discarded before we
get the AST. But this is still an operation that changes one syntactically
valid piece of Python code to another, right? I'm considering limiting the
API to only operate on already valid Python code, and using private APIs to
handle IPython's special syntax (like %magics).
3. Are there any other transformations that Sympy does (or wants to do)
besides these three?

On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:

> I'm sure you're already aware, but we use them for SymPy for some
> (optional) transformers that make Python more symbolic friendly
> (auto-replace undefined variables with Symbols, wrap integer literals
> with Integer so that exact rational numbers work, and so on). The AST
> transformer is nice but limiting. We can't use it to wrap float
> literals with higher precision for instance because Python drops the
> precision of floats in the AST.
>
> There are already some open issues about some limitations we've seen
> (https://github.com/sympy/sympy/issues/14440 and
> https://github.com/ipython/ipython/issues/10893). I'm not fully
> updated on the specifics of what the limitations were, but from what I
> remember, there were issues with IPython doing things line-by-line.
>
> I don't know how the new code is architected, but it would be nice if
> the transformers could just access the raw input from IPython, and
> various transformers could be plugged in to that to make things
> simpler if desired (the AST transformer could be one such instance).
>
> At the very least, if you want to test the new design, you could look
> at translating our transformers in SymPy (including the work in
> progress float transformer at
> https://github.com/sympy/sympy/pull/13300).
>
> Aaron Meurer
>
> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
> wrote:
> > Thomas this is great news! I think that part of the code base could
> benefit
> > from being simpler and more extensible. Thanks for tackling this!
> >
> > Sent from my iPhone
> >
> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> >
> > Hi all,
> >
> > For IPython 7, I'm planning a rework of the input transformation
> framework:
> > https://github.com/ipython/ipython/pull/11041
> >
> > The new framework is - I hope - simpler than the one it replaces, and
> will
> > hopefully have fewer weird corner cases. But it's still a pretty complex
> > beast, and I don't feel like it's a good platform for people to add more
> > transformations to. One option to improve this is to make a smaller
> > extensible API, restricting the kind of things that third party code can
> do.
> >
> > I don't think there's ever been much third party code extending IPython's
> > input transformation, but I'd like to find out about the code that does.
> So
> > if you're aware of a project that does use our input transformation API:
> >
> > - What project?
> > - What does it want to transform?
> > - Can you point me to the code?
> > - What fallback options would it have if IPython didn't support the
> > transformation it wanted?
> >
> > This is specifically about text transformations; AST transformations are
> not
> > changed.
> >
> > Thanks,
> > Thomas
> >
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at python.org
> > https://mail.python.org/mailman/listinfo/ipython-dev
> >
> >
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at python.org
> > https://mail.python.org/mailman/listinfo/ipython-dev
> >
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180528/b022c7d3/attachment-0001.html>

From asmeurer at gmail.com  Mon May 28 02:13:28 2018
From: asmeurer at gmail.com (Aaron Meurer)
Date: Mon, 28 May 2018 00:13:28 -0600
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAOvn4qiKjHDbUS1OPeRHDU_kO79gn2rt19NFzjgygGfEw2YriQ@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
 <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>
 <CAOvn4qiKjHDbUS1OPeRHDU_kO79gn2rt19NFzjgygGfEw2YriQ@mail.gmail.com>
Message-ID: <CAKgW=6KmcqbuxUy2_6jQohBmR-HAFp6mQ6HN33MCfDHfpZ+k6Q@mail.gmail.com>

I believe you're right.

I suppose another potential transform would be to allow x^2 to
represent x**2. This would have be done at the token level since ^ has
a different precedence than **.

Only operating on valid Python seems fine. If someone wants to
implement a DSL they should just write a kernel.

Have you considered using something like parso, which can manipulate
an AST and tokens losslessly?

Aaron Meurer

On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> Thanks Aaron. I am aware that Sympy uses input transformation, but it would
> be good to check some more details of exactly how:
>
> 1. I believe wrapping integers and defining symbols can both be done with
> AST transformations. Let me know if I'm missing something.
> 2. Wrapping floats can't, because the string form is discarded before we get
> the AST. But this is still an operation that changes one syntactically valid
> piece of Python code to another, right? I'm considering limiting the API to
> only operate on already valid Python code, and using private APIs to handle
> IPython's special syntax (like %magics).
> 3. Are there any other transformations that Sympy does (or wants to do)
> besides these three?
>
> On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:
>>
>> I'm sure you're already aware, but we use them for SymPy for some
>> (optional) transformers that make Python more symbolic friendly
>> (auto-replace undefined variables with Symbols, wrap integer literals
>> with Integer so that exact rational numbers work, and so on). The AST
>> transformer is nice but limiting. We can't use it to wrap float
>> literals with higher precision for instance because Python drops the
>> precision of floats in the AST.
>>
>> There are already some open issues about some limitations we've seen
>> (https://github.com/sympy/sympy/issues/14440 and
>> https://github.com/ipython/ipython/issues/10893). I'm not fully
>> updated on the specifics of what the limitations were, but from what I
>> remember, there were issues with IPython doing things line-by-line.
>>
>> I don't know how the new code is architected, but it would be nice if
>> the transformers could just access the raw input from IPython, and
>> various transformers could be plugged in to that to make things
>> simpler if desired (the AST transformer could be one such instance).
>>
>> At the very least, if you want to test the new design, you could look
>> at translating our transformers in SymPy (including the work in
>> progress float transformer at
>> https://github.com/sympy/sympy/pull/13300).
>>
>> Aaron Meurer
>>
>> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
>> wrote:
>> > Thomas this is great news! I think that part of the code base could
>> > benefit
>> > from being simpler and more extensible. Thanks for tackling this!
>> >
>> > Sent from my iPhone
>> >
>> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com> wrote:
>> >
>> > Hi all,
>> >
>> > For IPython 7, I'm planning a rework of the input transformation
>> > framework:
>> > https://github.com/ipython/ipython/pull/11041
>> >
>> > The new framework is - I hope - simpler than the one it replaces, and
>> > will
>> > hopefully have fewer weird corner cases. But it's still a pretty complex
>> > beast, and I don't feel like it's a good platform for people to add more
>> > transformations to. One option to improve this is to make a smaller
>> > extensible API, restricting the kind of things that third party code can
>> > do.
>> >
>> > I don't think there's ever been much third party code extending
>> > IPython's
>> > input transformation, but I'd like to find out about the code that does.
>> > So
>> > if you're aware of a project that does use our input transformation API:
>> >
>> > - What project?
>> > - What does it want to transform?
>> > - Can you point me to the code?
>> > - What fallback options would it have if IPython didn't support the
>> > transformation it wanted?
>> >
>> > This is specifically about text transformations; AST transformations are
>> > not
>> > changed.
>> >
>> > Thanks,
>> > Thomas
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at python.org
>> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at python.org
>> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at python.org
>> https://mail.python.org/mailman/listinfo/ipython-dev
>
>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>

From takowl at gmail.com  Mon May 28 08:25:45 2018
From: takowl at gmail.com (Thomas Kluyver)
Date: Mon, 28 May 2018 14:25:45 +0200
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAKgW=6KmcqbuxUy2_6jQohBmR-HAFp6mQ6HN33MCfDHfpZ+k6Q@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
 <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>
 <CAOvn4qiKjHDbUS1OPeRHDU_kO79gn2rt19NFzjgygGfEw2YriQ@mail.gmail.com>
 <CAKgW=6KmcqbuxUy2_6jQohBmR-HAFp6mQ6HN33MCfDHfpZ+k6Q@mail.gmail.com>
Message-ID: <CAOvn4qguM=GTfBAbOP9_HLa=Tivg5MW7DRbUd5Rpu8ABPzqRDg@mail.gmail.com>

Thanks Aaron. If working with valid Python syntax is sufficient, the API
will probably just be functions that take and return a string, leaving it
up to you how you tokenise/parse the code. We only need to do something
more complex if third-party transformations need to integrate with our own
ones to produce valid Python.

Does anyone know what Sage does? Jason mentioned that they might also use
input transformations.

On 28 May 2018 at 08:13, Aaron Meurer <asmeurer at gmail.com> wrote:

> I believe you're right.
>
> I suppose another potential transform would be to allow x^2 to
> represent x**2. This would have be done at the token level since ^ has
> a different precedence than **.
>
> Only operating on valid Python seems fine. If someone wants to
> implement a DSL they should just write a kernel.
>
> Have you considered using something like parso, which can manipulate
> an AST and tokens losslessly?
>
> Aaron Meurer
>
> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> > Thanks Aaron. I am aware that Sympy uses input transformation, but it
> would
> > be good to check some more details of exactly how:
> >
> > 1. I believe wrapping integers and defining symbols can both be done with
> > AST transformations. Let me know if I'm missing something.
> > 2. Wrapping floats can't, because the string form is discarded before we
> get
> > the AST. But this is still an operation that changes one syntactically
> valid
> > piece of Python code to another, right? I'm considering limiting the API
> to
> > only operate on already valid Python code, and using private APIs to
> handle
> > IPython's special syntax (like %magics).
> > 3. Are there any other transformations that Sympy does (or wants to do)
> > besides these three?
> >
> > On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:
> >>
> >> I'm sure you're already aware, but we use them for SymPy for some
> >> (optional) transformers that make Python more symbolic friendly
> >> (auto-replace undefined variables with Symbols, wrap integer literals
> >> with Integer so that exact rational numbers work, and so on). The AST
> >> transformer is nice but limiting. We can't use it to wrap float
> >> literals with higher precision for instance because Python drops the
> >> precision of floats in the AST.
> >>
> >> There are already some open issues about some limitations we've seen
> >> (https://github.com/sympy/sympy/issues/14440 and
> >> https://github.com/ipython/ipython/issues/10893). I'm not fully
> >> updated on the specifics of what the limitations were, but from what I
> >> remember, there were issues with IPython doing things line-by-line.
> >>
> >> I don't know how the new code is architected, but it would be nice if
> >> the transformers could just access the raw input from IPython, and
> >> various transformers could be plugged in to that to make things
> >> simpler if desired (the AST transformer could be one such instance).
> >>
> >> At the very least, if you want to test the new design, you could look
> >> at translating our transformers in SymPy (including the work in
> >> progress float transformer at
> >> https://github.com/sympy/sympy/pull/13300).
> >>
> >> Aaron Meurer
> >>
> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
> >> wrote:
> >> > Thomas this is great news! I think that part of the code base could
> >> > benefit
> >> > from being simpler and more extensible. Thanks for tackling this!
> >> >
> >> > Sent from my iPhone
> >> >
> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > For IPython 7, I'm planning a rework of the input transformation
> >> > framework:
> >> > https://github.com/ipython/ipython/pull/11041
> >> >
> >> > The new framework is - I hope - simpler than the one it replaces, and
> >> > will
> >> > hopefully have fewer weird corner cases. But it's still a pretty
> complex
> >> > beast, and I don't feel like it's a good platform for people to add
> more
> >> > transformations to. One option to improve this is to make a smaller
> >> > extensible API, restricting the kind of things that third party code
> can
> >> > do.
> >> >
> >> > I don't think there's ever been much third party code extending
> >> > IPython's
> >> > input transformation, but I'd like to find out about the code that
> does.
> >> > So
> >> > if you're aware of a project that does use our input transformation
> API:
> >> >
> >> > - What project?
> >> > - What does it want to transform?
> >> > - Can you point me to the code?
> >> > - What fallback options would it have if IPython didn't support the
> >> > transformation it wanted?
> >> >
> >> > This is specifically about text transformations; AST transformations
> are
> >> > not
> >> > changed.
> >> >
> >> > Thanks,
> >> > Thomas
> >> >
> >> > _______________________________________________
> >> > IPython-dev mailing list
> >> > IPython-dev at python.org
> >> > https://mail.python.org/mailman/listinfo/ipython-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > IPython-dev mailing list
> >> > IPython-dev at python.org
> >> > https://mail.python.org/mailman/listinfo/ipython-dev
> >> >
> >> _______________________________________________
> >> IPython-dev mailing list
> >> IPython-dev at python.org
> >> https://mail.python.org/mailman/listinfo/ipython-dev
> >
> >
> >
> > _______________________________________________
> > IPython-dev mailing list
> > IPython-dev at python.org
> > https://mail.python.org/mailman/listinfo/ipython-dev
> >
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180528/07c1bca6/attachment-0001.html>

From jason at jasongrout.org  Mon May 28 11:41:53 2018
From: jason at jasongrout.org (Jason Grout)
Date: Mon, 28 May 2018 08:41:53 -0700
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAOvn4qguM=GTfBAbOP9_HLa=Tivg5MW7DRbUd5Rpu8ABPzqRDg@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
 <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>
 <CAOvn4qiKjHDbUS1OPeRHDU_kO79gn2rt19NFzjgygGfEw2YriQ@mail.gmail.com>
 <CAKgW=6KmcqbuxUy2_6jQohBmR-HAFp6mQ6HN33MCfDHfpZ+k6Q@mail.gmail.com>
 <CAOvn4qguM=GTfBAbOP9_HLa=Tivg5MW7DRbUd5Rpu8ABPzqRDg@mail.gmail.com>
Message-ID: <CAPDWZHxcM42yTSXVApkZgATQG0yb6QDZ+eN7AuBoYWAE5cS9CA@mail.gmail.com>

It would be good to post separately to the Sage list. The code has changed
since I was last in it, but I think these are the relevant bits:

Register transforms:
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/ipython_extension.py#L504-L513

Preparsing: Defined at
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L379,
actual implementation at
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/preparse.py

Prompts:
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L416

There's also another transformation, it seems:
https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L468

Thanks,

Jason




On Mon, May 28, 2018 at 5:27 AM Thomas Kluyver <takowl at gmail.com> wrote:

> Thanks Aaron. If working with valid Python syntax is sufficient, the API
> will probably just be functions that take and return a string, leaving it
> up to you how you tokenise/parse the code. We only need to do something
> more complex if third-party transformations need to integrate with our own
> ones to produce valid Python.
>
> Does anyone know what Sage does? Jason mentioned that they might also use
> input transformations.
>
> On 28 May 2018 at 08:13, Aaron Meurer <asmeurer at gmail.com> wrote:
>
>> I believe you're right.
>>
>> I suppose another potential transform would be to allow x^2 to
>> represent x**2. This would have be done at the token level since ^ has
>> a different precedence than **.
>>
>> Only operating on valid Python seems fine. If someone wants to
>> implement a DSL they should just write a kernel.
>>
>> Have you considered using something like parso, which can manipulate
>> an AST and tokens losslessly?
>>
>> Aaron Meurer
>>
>> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <takowl at gmail.com>
>> wrote:
>> > Thanks Aaron. I am aware that Sympy uses input transformation, but it
>> would
>> > be good to check some more details of exactly how:
>> >
>> > 1. I believe wrapping integers and defining symbols can both be done
>> with
>> > AST transformations. Let me know if I'm missing something.
>> > 2. Wrapping floats can't, because the string form is discarded before
>> we get
>> > the AST. But this is still an operation that changes one syntactically
>> valid
>> > piece of Python code to another, right? I'm considering limiting the
>> API to
>> > only operate on already valid Python code, and using private APIs to
>> handle
>> > IPython's special syntax (like %magics).
>> > 3. Are there any other transformations that Sympy does (or wants to do)
>> > besides these three?
>> >
>> > On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:
>> >>
>> >> I'm sure you're already aware, but we use them for SymPy for some
>> >> (optional) transformers that make Python more symbolic friendly
>> >> (auto-replace undefined variables with Symbols, wrap integer literals
>> >> with Integer so that exact rational numbers work, and so on). The AST
>> >> transformer is nice but limiting. We can't use it to wrap float
>> >> literals with higher precision for instance because Python drops the
>> >> precision of floats in the AST.
>> >>
>> >> There are already some open issues about some limitations we've seen
>> >> (https://github.com/sympy/sympy/issues/14440 and
>> >> https://github.com/ipython/ipython/issues/10893). I'm not fully
>> >> updated on the specifics of what the limitations were, but from what I
>> >> remember, there were issues with IPython doing things line-by-line.
>> >>
>> >> I don't know how the new code is architected, but it would be nice if
>> >> the transformers could just access the raw input from IPython, and
>> >> various transformers could be plugged in to that to make things
>> >> simpler if desired (the AST transformer could be one such instance).
>> >>
>> >> At the very least, if you want to test the new design, you could look
>> >> at translating our transformers in SymPy (including the work in
>> >> progress float transformer at
>> >> https://github.com/sympy/sympy/pull/13300).
>> >>
>> >> Aaron Meurer
>> >>
>> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
>> >> wrote:
>> >> > Thomas this is great news! I think that part of the code base could
>> >> > benefit
>> >> > from being simpler and more extensible. Thanks for tackling this!
>> >> >
>> >> > Sent from my iPhone
>> >> >
>> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com>
>> wrote:
>> >> >
>> >> > Hi all,
>> >> >
>> >> > For IPython 7, I'm planning a rework of the input transformation
>> >> > framework:
>> >> > https://github.com/ipython/ipython/pull/11041
>> >> >
>> >> > The new framework is - I hope - simpler than the one it replaces, and
>> >> > will
>> >> > hopefully have fewer weird corner cases. But it's still a pretty
>> complex
>> >> > beast, and I don't feel like it's a good platform for people to add
>> more
>> >> > transformations to. One option to improve this is to make a smaller
>> >> > extensible API, restricting the kind of things that third party code
>> can
>> >> > do.
>> >> >
>> >> > I don't think there's ever been much third party code extending
>> >> > IPython's
>> >> > input transformation, but I'd like to find out about the code that
>> does.
>> >> > So
>> >> > if you're aware of a project that does use our input transformation
>> API:
>> >> >
>> >> > - What project?
>> >> > - What does it want to transform?
>> >> > - Can you point me to the code?
>> >> > - What fallback options would it have if IPython didn't support the
>> >> > transformation it wanted?
>> >> >
>> >> > This is specifically about text transformations; AST transformations
>> are
>> >> > not
>> >> > changed.
>> >> >
>> >> > Thanks,
>> >> > Thomas
>> >> >
>> >> > _______________________________________________
>> >> > IPython-dev mailing list
>> >> > IPython-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > IPython-dev mailing list
>> >> > IPython-dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >> >
>> >> _______________________________________________
>> >> IPython-dev mailing list
>> >> IPython-dev at python.org
>> >> https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > IPython-dev mailing list
>> > IPython-dev at python.org
>> > https://mail.python.org/mailman/listinfo/ipython-dev
>> >
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at python.org
>> https://mail.python.org/mailman/listinfo/ipython-dev
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180528/317794ee/attachment.html>

From takowl at gmail.com  Tue May 29 09:19:13 2018
From: takowl at gmail.com (Thomas Kluyver)
Date: Tue, 29 May 2018 15:19:13 +0200
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAPDWZHxcM42yTSXVApkZgATQG0yb6QDZ+eN7AuBoYWAE5cS9CA@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <7C1E24D8-C754-4C79-986E-C3AD5C1109CA@gmail.com>
 <CAKgW=6+xnLEio=CGz3hubTOYfosnjgOVUZDBiOqvjx=2ShmqEQ@mail.gmail.com>
 <CAOvn4qiKjHDbUS1OPeRHDU_kO79gn2rt19NFzjgygGfEw2YriQ@mail.gmail.com>
 <CAKgW=6KmcqbuxUy2_6jQohBmR-HAFp6mQ6HN33MCfDHfpZ+k6Q@mail.gmail.com>
 <CAOvn4qguM=GTfBAbOP9_HLa=Tivg5MW7DRbUd5Rpu8ABPzqRDg@mail.gmail.com>
 <CAPDWZHxcM42yTSXVApkZgATQG0yb6QDZ+eN7AuBoYWAE5cS9CA@mail.gmail.com>
Message-ID: <CAOvn4qinoQ9LxYPf9aof3nOg3RxXrO7c2iBwktL1wdvX88c4Yw@mail.gmail.com>

Looking at those links, Sage definitely does not restrict itself to valid
Python syntax. However, if there's only one downstream doing that, I think
it's reasonable for us to simplify the API and let Sage handle its own
preparsing. Does anyone know of any other projects which use input
transformations?

More specifically, I propose that we offer three transformation APIs:

1. Before handling our special syntax, to strip formatting like prompts
(accepts a cell as a string, returns transformed string)
2. After handling our special syntax, when the result should be valid
Python syntax, for things like wrapping floats (string -> string)
3. AST transformations (AST -> AST)

If projects like Sage want to, they could still transform invalid tokens in
hook 2, but their special syntax may interact unpredictably with our own.
Alternatively, they could copy and modify our transformation machinery to
handle their own special syntax. Either way, it avoids us having to
maintain a complex transformation interface which we may need to break
again.

Can I ask someone who's got a foot in the Sage camp to ping them about this
on their own lists?

Thomas

On 28 May 2018 at 17:41, Jason Grout <jason at jasongrout.org> wrote:

> It would be good to post separately to the Sage list. The code has changed
> since I was last in it, but I think these are the relevant bits:
>
> Register transforms: https://github.com/sagemath/sage/blob/
> 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/
> ipython_extension.py#L504-L513
>
> Preparsing: Defined at https://github.com/sagemath/sage/blob/
> 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L379,
> actual implementation at https://github.com/sagemath/sage/blob/
> 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/preparse.py
>
> Prompts: https://github.com/sagemath/sage/blob/
> 854f9764d14236110b8d7f7b35a7d52017e044f8/src/sage/repl/interpreter.py#L416
>
> There's also another transformation, it seems:
> https://github.com/sagemath/sage/blob/854f9764d14236110b8d7f7b35a7d5
> 2017e044f8/src/sage/repl/interpreter.py#L468
>
> Thanks,
>
> Jason
>
>
>
>
> On Mon, May 28, 2018 at 5:27 AM Thomas Kluyver <takowl at gmail.com> wrote:
>
>> Thanks Aaron. If working with valid Python syntax is sufficient, the API
>> will probably just be functions that take and return a string, leaving it
>> up to you how you tokenise/parse the code. We only need to do something
>> more complex if third-party transformations need to integrate with our own
>> ones to produce valid Python.
>>
>> Does anyone know what Sage does? Jason mentioned that they might also use
>> input transformations.
>>
>> On 28 May 2018 at 08:13, Aaron Meurer <asmeurer at gmail.com> wrote:
>>
>>> I believe you're right.
>>>
>>> I suppose another potential transform would be to allow x^2 to
>>> represent x**2. This would have be done at the token level since ^ has
>>> a different precedence than **.
>>>
>>> Only operating on valid Python seems fine. If someone wants to
>>> implement a DSL they should just write a kernel.
>>>
>>> Have you considered using something like parso, which can manipulate
>>> an AST and tokens losslessly?
>>>
>>> Aaron Meurer
>>>
>>> On Sun, May 27, 2018 at 11:50 PM, Thomas Kluyver <takowl at gmail.com>
>>> wrote:
>>> > Thanks Aaron. I am aware that Sympy uses input transformation, but it
>>> would
>>> > be good to check some more details of exactly how:
>>> >
>>> > 1. I believe wrapping integers and defining symbols can both be done
>>> with
>>> > AST transformations. Let me know if I'm missing something.
>>> > 2. Wrapping floats can't, because the string form is discarded before
>>> we get
>>> > the AST. But this is still an operation that changes one syntactically
>>> valid
>>> > piece of Python code to another, right? I'm considering limiting the
>>> API to
>>> > only operate on already valid Python code, and using private APIs to
>>> handle
>>> > IPython's special syntax (like %magics).
>>> > 3. Are there any other transformations that Sympy does (or wants to do)
>>> > besides these three?
>>> >
>>> > On 28 May 2018 at 00:02, Aaron Meurer <asmeurer at gmail.com> wrote:
>>> >>
>>> >> I'm sure you're already aware, but we use them for SymPy for some
>>> >> (optional) transformers that make Python more symbolic friendly
>>> >> (auto-replace undefined variables with Symbols, wrap integer literals
>>> >> with Integer so that exact rational numbers work, and so on). The AST
>>> >> transformer is nice but limiting. We can't use it to wrap float
>>> >> literals with higher precision for instance because Python drops the
>>> >> precision of floats in the AST.
>>> >>
>>> >> There are already some open issues about some limitations we've seen
>>> >> (https://github.com/sympy/sympy/issues/14440 and
>>> >> https://github.com/ipython/ipython/issues/10893). I'm not fully
>>> >> updated on the specifics of what the limitations were, but from what I
>>> >> remember, there were issues with IPython doing things line-by-line.
>>> >>
>>> >> I don't know how the new code is architected, but it would be nice if
>>> >> the transformers could just access the raw input from IPython, and
>>> >> various transformers could be plugged in to that to make things
>>> >> simpler if desired (the AST transformer could be one such instance).
>>> >>
>>> >> At the very least, if you want to test the new design, you could look
>>> >> at translating our transformers in SymPy (including the work in
>>> >> progress float transformer at
>>> >> https://github.com/sympy/sympy/pull/13300).
>>> >>
>>> >> Aaron Meurer
>>> >>
>>> >> On Sun, May 27, 2018 at 2:50 PM, Brian Granger <ellisonbg at gmail.com>
>>> >> wrote:
>>> >> > Thomas this is great news! I think that part of the code base could
>>> >> > benefit
>>> >> > from being simpler and more extensible. Thanks for tackling this!
>>> >> >
>>> >> > Sent from my iPhone
>>> >> >
>>> >> > On May 27, 2018, at 1:19 PM, Thomas Kluyver <takowl at gmail.com>
>>> wrote:
>>> >> >
>>> >> > Hi all,
>>> >> >
>>> >> > For IPython 7, I'm planning a rework of the input transformation
>>> >> > framework:
>>> >> > https://github.com/ipython/ipython/pull/11041
>>> >> >
>>> >> > The new framework is - I hope - simpler than the one it replaces,
>>> and
>>> >> > will
>>> >> > hopefully have fewer weird corner cases. But it's still a pretty
>>> complex
>>> >> > beast, and I don't feel like it's a good platform for people to add
>>> more
>>> >> > transformations to. One option to improve this is to make a smaller
>>> >> > extensible API, restricting the kind of things that third party
>>> code can
>>> >> > do.
>>> >> >
>>> >> > I don't think there's ever been much third party code extending
>>> >> > IPython's
>>> >> > input transformation, but I'd like to find out about the code that
>>> does.
>>> >> > So
>>> >> > if you're aware of a project that does use our input transformation
>>> API:
>>> >> >
>>> >> > - What project?
>>> >> > - What does it want to transform?
>>> >> > - Can you point me to the code?
>>> >> > - What fallback options would it have if IPython didn't support the
>>> >> > transformation it wanted?
>>> >> >
>>> >> > This is specifically about text transformations; AST
>>> transformations are
>>> >> > not
>>> >> > changed.
>>> >> >
>>> >> > Thanks,
>>> >> > Thomas
>>> >> >
>>> >> > _______________________________________________
>>> >> > IPython-dev mailing list
>>> >> > IPython-dev at python.org
>>> >> > https://mail.python.org/mailman/listinfo/ipython-dev
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > IPython-dev mailing list
>>> >> > IPython-dev at python.org
>>> >> > https://mail.python.org/mailman/listinfo/ipython-dev
>>> >> >
>>> >> _______________________________________________
>>> >> IPython-dev mailing list
>>> >> IPython-dev at python.org
>>> >> https://mail.python.org/mailman/listinfo/ipython-dev
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > IPython-dev mailing list
>>> > IPython-dev at python.org
>>> > https://mail.python.org/mailman/listinfo/ipython-dev
>>> >
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at python.org
>>> https://mail.python.org/mailman/listinfo/ipython-dev
>>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at python.org
>> https://mail.python.org/mailman/listinfo/ipython-dev
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180529/ca51dc5c/attachment.html>

From lkb.teichmann at gmail.com  Wed May 30 04:23:41 2018
From: lkb.teichmann at gmail.com (Martin Teichmann)
Date: Wed, 30 May 2018 10:23:41 +0200
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
Message-ID: <CAK9R32QO_fB70PD0SFFNDYrcdbFq3qBfRCXrdUdJj16T0+5t8g@mail.gmail.com>

Hi Thomas, Hi List,

years ago I have been working on some input transformation, which, I
think, would be a good idea to have in mind when reworking them, it's
at https://github.com/tecki/ipython-yf

The idea was at the time to support python's asyncio on a command
line, such that you can copy-and-paste code you're trying to test into
a command line, and it runs! At the time, that still meant to support
yield from on a command line, nowadays supporting await would be
needed as well.

But as we're already at it, I guess in the future Jupyter and IPython
will slowly converge into a asyncio world. The underlying zmq library
is already asyncio compatible, and if I remember correctly there is
also already some code in IPython which plays with asyncio event
loops.

In order to make things suitable for asyncio, it is a good idea to
seperate out blocking from non-blocking code. Currently, IPython is
not written at all like that, and I would not ask for it to be changed
as it is a lot of work, but if you're already working on it, keeping
it in mind may be fine.

Greetings

Martin

From bussonniermatthias at gmail.com  Wed May 30 10:07:46 2018
From: bussonniermatthias at gmail.com (Matthias Bussonnier)
Date: Wed, 30 May 2018 07:07:46 -0700
Subject: [IPython-dev] Input transformation rework - input requested
In-Reply-To: <CAK9R32QO_fB70PD0SFFNDYrcdbFq3qBfRCXrdUdJj16T0+5t8g@mail.gmail.com>
References: <CAOvn4qgF+s9-Wz=DmCU2ey4V1XBt9UF_GEMu9p5w2nCNk_8LYQ@mail.gmail.com>
 <CAK9R32QO_fB70PD0SFFNDYrcdbFq3qBfRCXrdUdJj16T0+5t8g@mail.gmail.com>
Message-ID: <CANJQusVfeXnKw1uWU2E04NMD9NMHeA3Y-Mr9d7MPT-hcz3mUrQ@mail.gmail.com>

Hi Martin,

Could you have a look at https://github.com/ipython/ipython/pull/11155,
which does some work to allow `await  coroutine` without the need for
magics ?
-- 
Matthias


On Wed, 30 May 2018 at 01:24, Martin Teichmann <lkb.teichmann at gmail.com>
wrote:

> Hi Thomas, Hi List,
>
> years ago I have been working on some input transformation, which, I
> think, would be a good idea to have in mind when reworking them, it's
> at https://github.com/tecki/ipython-yf
>
> The idea was at the time to support python's asyncio on a command
> line, such that you can copy-and-paste code you're trying to test into
> a command line, and it runs! At the time, that still meant to support
> yield from on a command line, nowadays supporting await would be
> needed as well.
>
> But as we're already at it, I guess in the future Jupyter and IPython
> will slowly converge into a asyncio world. The underlying zmq library
> is already asyncio compatible, and if I remember correctly there is
> also already some code in IPython which plays with asyncio event
> loops.
>
> In order to make things suitable for asyncio, it is a good idea to
> seperate out blocking from non-blocking code. Currently, IPython is
> not written at all like that, and I would not ask for it to be changed
> as it is a lot of work, but if you're already working on it, keeping
> it in mind may be fine.
>
> Greetings
>
> Martin
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at python.org
> https://mail.python.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20180530/25e65f36/attachment.html>