Multiple inheritance, super() and changing signature

[Steven D'Aprano]

On Wed, 1 Jun 2016 02:10 am, Nagy Lc3a1szlc3b3 Zsolt wrote:

> Today I come across this problem for the N+1st time. Here are some
> classes for the example:


A couple of comments... if you're using Python 2, then you may be having
trouble because none of the classes shown below inherit from object. They
are all "old-style" classes. super needs at least one newstyle class to
work correctly.

In Python 3, that will be automatic and you don't need to worry about it.


[...] 
> class BootstrapDesktop(BootstrapWidget, BaseDesktop):
>     def __init__(self, appserver, session):
>         # there is NOTHING else here, it just connects bootstrap widget
> implementation with desktop methods
>         super(BootstrapDesktop, self).__init__(appserver, session)

The correct way to do that is to simply not define an __init__ method at
all.


> Here is a part of the hierarchy:
> 
> Observable      AppServerSessionMixin
>          \      /
>           \    /
>           Widget (__init__ with two parameters is first introduced here)
>              |
>            VisualWidget
>              |          
>           BootstrapWidget BaseDesktop
>                       \   /
>                  BootstrapDesktop

For this to actually work, it needs to start:

          object
          /    \
         /      \
Observable      AppServerSessionMixin
         \      /
          ... continues as above...


or something similar.



> The problem is obvious: 

Heh :-) There's nothing obvious about multiple inheritance problems.


> because of the method resolution order, the 
> super call in Observable.__init__ will try to call BaseDesktop.__init__
> without arguments, but that constructor has needs to arguments. The
> arguments are lost when Widget calls super() without them.

Raymond Hettinger gives an excellent presentation where he describes various
problems with MI and gives solutions for them. I think this might be it:

http://pyvideo.org/video/1094/the-art-of-subclassing-0

He also discusses super here:

https://rhettinger.wordpress.com/2011/05/26/super-considered-super/

You should also read the provocatively named "Super Considered Harmful"
here:

https://fuhm.net/super-harmful/

Despite the title, after a long discussion on the Python-Dev mailing list,
the author eventually admitted that in fact you do need to use super, and
that it works better than the alternatives. (Unfortunately, apart from a
partial change to the title, he didn't update the essay.) Nevertheless, MI
is always difficult, and he makes some good points in the essay.


[...]
> There is a single class (Widget) that has changed the signature of the
> constructor. 

Is this your code? Then simply fix Widget. MI in Python is cooperative: all
the classes have to be designed for MI. It seems that Widget is not.
(Possibly I have misdiagnosed the problem, and the fault lies elsewhere.)


> I was thinking about cutting the MRO list somehow into two 
> parts. It would be very nice to tell Python to create a separate MRO
> list for classes above and below the Widget class. Classes below Widget
> would not see any method above Widget, and that would allow me to use
> super() anywhere in the class hierarchy below Widget. I would only have
> to do an explicit call when the method signature is changed. That would
> save me a lot of parameter retyping and refactoring.

Even if you got your wish, you couldn't use it until you're using Python 3.6
or higher.


> In fact, I could never safely use super() in any case when the method
> signature has changed. When signatures are changed, method resolution
> must be done manually anyway.

No, that it incorrect. See Raymond Hettinger's talk and essay.


> Here is a proposal:
> 
> * method signature changes can be detected syntactically, relatively
> easily 

I don't think they can, at least not reliably.


> * raise an exception when super() is called in a method has changed 
> the signature (from any of base class(es))

That's overly strict. As Raymond shows, it is easy to deal with changing
method signatures in *cooperative* classes.


> * when a method changes signature, then split the MRO automatically into
> two parts. The tail of the MRO "below" the signature-changing-class
> should end in the method that has changed signature, and that method
> would be responsible for calling (or not calling) the method(s) with the
> same name inherited from base classes.

This sounds confusing and harder to get right than the current. Consider
that means that, in the worst case, you could split the MRO into two, then
two again, then two again, then two again... trying to debug this, or
understand it, would be a nightmare.

The MRO used by Python is a well-known linearisation studied in great
detail. It is proven to be effective and *correct*. Your proposal would
break that.

Perhaps you are unaware that manually calling the superclass method does not
work correctly in cases of multiple inheritance? You end up either missing
classes, and not calling their method, or calling them twice. That's why
you need to use a proper linearisation algorithm, as used by super.

See this explanation of C3 linearisation here:

https://www.python.org/download/releases/2.3/mro/


> Of course, this would require Python to calculate an MRO that depends on
> the name of the method, and not just the class hierarchy.

I don't even know what that means.


> And also the 
> signatures of the methods, but those can also be changed dynamically. So
> this may be a bad idea after all.



-- 
Steven