[Numpy-discussion] Syntax Improvement for Array Transpose

Wed Jun 26 17:51:41 EDT 2019

On Wed, 2019-06-26 at 17:22 -0400, Marten van Kerkwijk wrote:
> Hi Ralf,
> 
> I realize you feel strongly that this whole thread is rehashing
> history, but I think it is worth pointing out that many seem to
> consider that the criterion for allowing backward incompatible
> changes, i.e., that "existing code is buggy or is consistently
> confusing many users", is actually fulfilled here.
> 
> Indeed, this appears true to such an extent that even those among the
> steering council do not agree: while the topic of this thread was
> about introducing *new* properties (because in the relevant issue I
> had suggested to Steward it was not possible to change .T), it was
> Eric who brought up the question whether we shouldn't just change
> `.T` after all. And in the relevant issue, Sebastian noted that "I am
> not quite convinced that we cannot change .T (at least in the sense
> of deprecation) myself", with Chuck chiming in that "I don't recall
> being in opposition, and I also think the current transpose is not
> what we want."
> 
> That makes three of your fellow steering council members who are not
> sure, despite all the previous discussions (of which Chuck surely has
> seen most - sorry, Chuck!).
> 
> It seems to me the only sure way in which we can avoid future
> discussions is to actually address the underlying problem. E.g., is
> the cost of deprecating & changing .T truly that much more than even
> having this discussion?
> 

To me, I think what we have here is simply that if we want to do it, it
will be an uphill battle. And an uphill battle may mean that we have to
write something close to an NEP. Including seeing how much code blows
up, e.g. by providing an environment variable switchable behaviour or
so. I think it would be better to approach it from that side: What is
necessary to be convincing enough?

The problem of going from one behaviour to another (at least without an
many-year waiting period) is real though, it not uncommon to leave
scripts lying around for 5 years...

So in that sense, I would agree that to really switch behaviour (not
just error), it would need extremely careful analysis, which may not be
feasible. OTOH, some other options, such as a new name or deprecations
(or warning) do not have such fundamental problems. Quite honestly, I
am not sure that deprecating `.T` completely for high dimensions is
much more painful then e.g. the move of factorial in scipy, which
forced me to modify a lot of my scripts (ok, its search+replace instead
of replacing one line).

We could go further of course, and say we do a "painful major" release
at some point with things like py3k warnings and all. But we probably
need more good reasons than a `.T`, and in-person discussions before
even considering it.

Best,

Sebastian

> All the best,
> 
> Marten
> 
> 
> On Wed, Jun 26, 2019 at 4:18 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
> > 
> > On Wed, Jun 26, 2019 at 10:04 PM Kirill Balunov <
> > kirillbalunov at gmail.com> wrote:
> > > Only concerns #4 from Ilhan's list.
> > > 
> > > ср, 26 июн. 2019 г. в 00:01, Ralf Gommers <ralf.gommers at gmail.com
> > > >:
> > > > [....]
> > > > 
> > > > Perhaps not full consensus between the many people with
> > > > different opinions and interests. But for the first one, arr.T
> > > > change: it's clear that this won't happen.
> > > > 
> > > 
> > > To begin with, I must admit that I am not familiar with the
> > > accepted policy of introducing changes to NumPy. But I find it
> > > quite nonconstructive just to say - it will not happen. What then
> > > is the point in the discussion?
> > > 
> > 
> > There has been a *very* long discussion already, and several others
> > on the same topic before. There are also long-standing ways of
> > dealing with backwards compatibility - e.g. what Matthew said is
> > not new, it's an agreed upon way of working. 
> > http://www.numpy.org/neps/nep-0023-backwards-compatibility.html
> > lists some principles. That NEP is not yet accepted (it needs rework), but it gives a good idea of what does and does not go.
> >  
> > >  
> > > > Between Juan's examples of valid use, and what Stephan and
> > > > Matthew said, there's not much more to add. We're not going to
> > > > change correct code for minor benefits.
> > > > 
> > > 
> > > I fully agree that any feature can find its use, valid or not is
> > > another question. Juan did not present these examples, but I will
> > > allow myself to assume that it is more correct to describe what
> > > is being done there as a permutation, and not a transpose. In
> > > addition, in the very next sentence, Juan adds that "These could
> > > be easily changed to .transpose() (honestly they probably
> > > should!)"
> > > 
> > > > We're not going to change correct code for minor benefits. 
> > > 
> > > It's fair, I personally have no preferences in both cases, the
> > > most important thing for me is that in the 2d case it works
> > > correctly. To be honest, until today, I thought that `.T` will
> > > raise for` ndim > 2`. At least that's what my experience told me.
> > > For example in
> > > 
> > >     Matlab - Error using  .' Transpose on ND array is not
> > > defined. Use PERMUTE instead.
> > > 
> > >     Julia - transpose not defined for Array(Float64, 3). Consider
> > > using permutedims for higher-dimensional arrays.
> > > 
> > >     Sympy - raise ValueError("array rank not 2") 
> > > 
> > > Here, I agree with the authors that, to begin with, `transpose`
> > > is not the best name, since in general it doesn’t fit as an any
> > > mathematical definition (of course it will depend on what we take
> > > as an element) or a definition from linear algebra. Thus the name
> > > `transpose` only leads to confusion. 
> > > 
> > > For a note about another suggestion - `.T` to mean a transpose of
> > > the last two dimensions, in Mathematica authors for some reason
> > > did the opposite (personally, I could not understand why they
> > > made such a choice :) ):
> > > 
> > >     Transpose[list]
> > >         transposes the first two levels in list. 
> > > 
> > > >     I feel strongly that we should have the following policy:
> > > > 
> > > >     * Under no circumstances should we make changes that mean
> > > > that correct
> > > >     old code will give different results with new Numpy.
> > > > 
> > > 
> > > I find this overly strict rules that do not allow to evolve. I
> > > completely agree that a silent change in behavior is a disaster,
> > > that changing behavior (if it is not an error) in the same minor
> > > version (1.X.Y) is not acceptable, but I see no reason to extend
> > > this rule for a major version bump (2.A.B.),  especially if it
> > > allows something to improve.
> > > 
> > 
> > I'm sorry, you'll have to live with this rule. We've had lots of
> > discussion about this rule in many concrete cases. When existing
> > code is buggy or is consistently confusing many users, we can
> > discuss. But in general changing old code to do something else is a
> > terrible idea.
> > 
> > > I would see such a rough version of a roadmap of change (I
> > > foresee my loneliness in this :)) Also considering this comment  
> > > 
> > > >     Personally I would find any divergence between a.T and
> > > > a.transpose()
> > > >     to be rather surprising.
> > > > 
> > > 
> > > it will be as follows:
> > > 
> > > 1. in 1.18 add the `.permute` method to the array, with the same
> > > semantics as `.transpose`.  
> > > 2. Starting from 1.18, emit  `FutureWarning`, `
> > > DeprectationWarning` for `.transpose` and advise replacing it
> > > with `.permute`.
> > > 3. Starting from 1.18 for `.T` with` ndim> 2`, emit a
> > > `FutureWarning`, with a note that in future versions the behavior
> > > will change.
> > > 4. In version 2, remove the `.transpose` and change the behavior
> > > for `.T`.
> > > 
> > 
> > This is simply not enough. Many users will skip versions when
> > upgrading. There must be an exceptionally good reason to change
> > numerical results, and this simply is not one.
> > 
> > Cheers,
> > Ralf
> > 
> > 
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190626/c3beb1a2/attachment.sig>