[Python-3000] [Python-Dev] PEP 367: New Super

[Guido van Rossum]

On 5/25/07, Tim Delaney <timothy.c.delaney at gmail.com> wrote:
> Bah - this should have gone to Pyton-3000 too, since it's discussing the
> PEP.

My fault; I started sending you feedback that only went to you, Calvin
and the PEP editors. I've added python-3000 at python.org back here.

> Guido van Rossum wrote:
>
> > - This seems to be written from the POV of introducing it in 2.6.
> > Perhaps the PEP could be slightly simpler if it could focus just on
> > Py3k? Then it's up to the 2.6 release managers to decide if and how to
> > backport it.
>
> That was my original intention, but it was assigned a non-Py3k PEP number,
> so I presumed I'd missed an email where you'd decided it should be for 2.6.
>
> We should probably change the PEP number if it's to be targetted at Py3K
> only.

Maybe. There are a bunch of PEPs that were originally proposed before
the Py3k work started but that are now slated for inclusion in 3.0. I
don't think we should renumber all of those.

> > - Why not make super a keyword, instead of just prohibiting assignment
> > to it? (I'm planning to do the same with None BTW in Py3k -- I find
> > the "it's a name but you can't assign to it" a rather silly business
> > and hardly "the simplest solution".)
>
> That's currently an open issue - I'm happy to make it a keyword - in which
> case I think the title should be changed to "super as a keyword" or
> something like that.

As it was before. :-)

What's the argument against?

> > - "Calling a static method or normal function that accesses the name
> > super will raise a TypeError at runtime." This seems too vague. What
> > if the function is nested within a method? Taking the specification
> > literally, a nested function using super will have its own preamble
> > setting super, which would be useless and wrong.
>
> I'd thought I'd covered that with "This name behaves
> identically to a normal local, including use by inner functions via a cell,
> with the following exceptions:", but re-reading it it's a bit clumsy.
>
> The intention is that functions that do not have access to a 'super' cell
> variable will raise a TypeError. Only methods using the keyword 'super' will
> have a preamble.
>
> Th preamble will only be added to functions/methods that cause the 'super'
> cell to exist i.e. for CPython have 'super' in co.cellvars. Functions that
> just have 'super' in co.freevars wouldn't have the preamble.

I think it's still too vague. For example:

class C:
  def f(s):
    return 1
class D(C):
  pass
def f(s):
  return 2*super.f()
D.f = f
print(D().f())

Should that work? I would be okay if it didn't, and if the super
keyword is only allowed inside a method that is lexically inside a
class. Then the second definition of f() should be a (phase 2)
SyntaxError.

Was it ever decided whether the implicitly bound class should be:

- the class object as produced by the class statement (before applying
class decorators);
- whatever is returned by the last class decorator (if any); or
- whatever is bound to the class name at the time the method is invoked?

I've got a hunch that #1 might be more solid; #3 seems asking for trouble.

There's also the issue of what to do when the method itself is
decorated (the compiler can't know what the decorators mean, even for
built-in decorators like classmethod).

> > - "For static methods and normal functions, <class> will be None,
> > resulting in a TypeError being raised during the preamble." How do you
> > know you're in this situation at run time? By the time the function
> > body is entered the knowledge about whether this was a static or
> > instance method is lost.
>
> The preamble will not technically be part of the function body - it occurs
> after unpacking the parameters, but before entering the function body, and
> has access to the C-level variables of the function/method object. So the
> exception will be raised before entering the function body.
>
> The way I see it, during class construction, a C-level variable on the
> method object would be bound to the (decorated?) class. This really needs to
> be done as the last step in class construction if it's to bind to the
> decorated class - otherwise it can be done as the methods are processed.

We could make the class in question a fourth attribute of the (poorly
named) "bound method" object, e.g. im_class_for_super (im_super would
be confusing IMO). Since this is used both by instance methods and by
the @classmethod decorator, it's just about perfect for this purpose.
(I would almost propose to reuse im_self for this purpose, but that's
probably asking for subtle backwards incompatibilities and not worth
it.)

Then when we're calling a bound method X (bound either to an instance
or to a class, depending on whether it's an instance or class method),
*if* the im_class_for_super is set, and *if* the function (im_func)
has a "free variable" named 'super', *then* we evaluate
__builtin__.__super__(X.im_class_for_super, X.im_self) and bind it to
that variable. If there's no such free variable, we skip this step.
This step could be inserted in call_function() in Python/ceval.c in
the block starting with "if (PyMethod_check(func) && ...)". It also
needs to be inserted into method_call() in Objects/classobject.c, in
the toplevel "else" block. (The ceval version is a speed hack, it
inlines the essence of method_call().)

Now we need to modify the compiler, as follows (assume super is a keyword):

- Consider three types of scopes, which may be nested: the outermost
(module or exec) scope, class scope, and function scope. The latter
two can be nested arbitrarily.

- The super keyword is only usable in an expression (it becomes an
alternative for 'atom' in the grammar). It can not be used as an
assignment target (this is a phase 2 SyntaxError) nor in a nonlocal
statement.

- The super keyword is only allowed in a function that is contained in
a class (directly or nested inside another function). It is not
allowed directly in a class, nor in the outermost scope.

- If a function contains a valid use of super, add a free variable
named 'super' to the function's set of free variables.

- If the function is nested inside another function (not in a class),
add the same free variable to that outer function too, and so on,
until a function is reached that is nested in a class, not in a
function.

- All *uses* of the super keyword are turned into references to this
free variable.

I think this should work; it mostly uses existing machinery; it is
explainable using existing mechanisms.

If a function using super is somehow called without going through the
binding of super, it will just get the normal error message when super
is used:

NameError: free variable 'super' referenced before assignment in enclosing scope

IMO that's good enough; it's pretty hard to produce such a call.

> I was thinking that by binding that variable to Py_None for static methods
> it would allow someone to do the following:
>
> def modulefunc(self):
>     pass
>
> class A(object):
>     def func(self):
>         pass
>
>     @staticmethod
>     def staticfunc():
>         pass
>
> class B(object):
>     func = A.func
>     staticfunc = A.staticfunc
>     outerfunc = modulefunc
>
> class C(object):
>     outerfunc = B.outerfunc
>
> but that's already going to cause problems when you call the methods - they
> will be being called with instances of the wrong type (raising a TypeError).

I don't see any references to super in that example -- what's the relevance?

> So now I think both static methods and functions should just have that
> variable left as NULL. Trying to get __super__(NULL) will throw a TypeError.

See my proposal above. It differs slightly in that the __super__ call
is made only when the class is not NULL. On the expectation that a
typical function that references super uses it exactly once per call
(that would be by far the most common case I expect) this is just
fine. In my proposal the 'super' variable contains whatever
__super__(<class>, <inst>) returned, rather than <class> which you
seem to be proposing here.

> > - The reference implementation (by virtue of its bytecode hacking)
> > only applies to CPython. (I'll have to study it in more detail later.)
>
> Yep, and it has quite a few limitations. I'd really like to split it out
> from the PEP itself, but I'm not sure where I should host it.

Submit it as a patch to SourceForge and link to it from the PEP (I did
this for PEP 3119). If you still care about it -- I'm also okay with
just having it in the subversion archives.

> > I'll probably come up with more detailed feedback later. Keep up the
> > good work!!
>
> Now I've got to find the time to try implementing it. Neal has said he's
> willing to help, but I want to give it a go myself.

Great (either way) !

PS if you like my proposal, feel free to edit it into shape for the PEP.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)