[Python-3000] Abilities / Interfaces

[Phillip J. Eby]

Before delving into any specific points, let me attempt to explain my 
assumptions, briefly, because I suspect that you and I do not mean the same 
thing by the words "generic function", and thus you perceive me to be 
proposing to change things that I am in fact proposing we *not* change.

To me, Python already has generic functions, including len(), iter(), 
getattr(), hasattr(), operator.getitem(), etc. etc.  Anything based on 
__special__ methods is in my view, *already* a "generic function", because 
it can take various types, and new types can be created that have 
specialized behavior.  The implementation of 'len()' is *open to extension 
for new types*, and therefore it is "generic".

That being the case, my position is that we should provide language support 
for defining other kinds of generic functions that are on an equal footing 
with the builtins in terms of language support.  __special__-based generics 
are adequate for quite a lot of purposes, registry-based ones are useful 
for many others, and advanced special cases have their uses as well.

This doesn't seem particularly radical to me, no more so than say, 
type/class unification.  And it's far less ambitious than type/class 
unification was, from an implementation point of view, because I don't 
propose to change the implementation of anything that already exists!

*Nothing* in this position relies upon RuleDispatch or even the simple 
multiple-dispatch generic function implementation you prototyped earlier 
this year.  Nor is there anything head-exploding about __special__ methods, 
as far as I am aware.  :)

So I'm at a loss to see any radicalness here.

What I have said that seems to be controversial, is the idea that we could 
do without interfaces.  However, we actually *already have interfaces based 
on generics*.

Consider 'iter()', for example, which can be viewed as adapting an object 
to the "iteration interface" and returning an object supporting iteration.

This is an *example* of what I am proposing: i.e., using generic operations 
to define interfaces.  And again, it's already supported by Python!

So how is this radical?  Everything I've stated up to this point in this 
message, is *exactly what we have today*!

The only things I propose to *add*, would be:

1. functions like 'addmethod()' and 'hasmethod()', themselves to be 
generics in the style of iter() or len().  Whether they are implemented 
using registries or __special__ methods is of no consequence.

2. a 'defop' syntax as shorthand for 'addmethod()', such that e.g.:

     import operator

     class Foo:
         defop iter(self):
             ...
         defop len(self):
             ...
         defop operator.getitem(self, key):
             ...

would produce exactly the same results as the same code using __special__ 
methods, except that it could be expanded to add methods for arbitrary 
*new* operations not defined by the language.

With these two things, Python would have enough foundation to allow any 
number of interface, adaptation, or generic function-based frameworks to 
run on top of it.  And, instead of each framework having to teach people 
new __special__ methods or provide different decorators, they could all 
simply say, "here are the generic functions or interfaces we use".

I submit that this is more than sufficient to cover the 80-90% of common 
use cases that don't need abstract interfaces at all, or which can simply 
use a generic function to stand in as an interface, the way 'iter' can 
stand in for the "iteration interface".

And, in the same way that 'def __iter__(self): return self' means "I 
implement iteration", doing "defop IFoo(self): return self" would mean, "I 
implement IFoo".  (Note, by the way, that common usage of interface 
adaptation in Python is already to call IFoo(anObject), so this is *also* 
no change to current usage!)

So, from my POV, this is actually a very modest proposal thus far.

Now I'll answer those specific points you brought up that aren't covered in 
the above.


At 06:07 PM 11/21/2006 -0800, Guido van Rossum wrote:
>are downright off-putting? What on earth is "when_object(iter)"
>supposed to mean? HCI indeed! :-)

Simplegeneric allows you to define methods for either types or 
instances.  The doc is at http://cheeseshop.python.org/pypi/simplegeneric


> > So, all I am proposing is that we:
> >
> >     * provide a standard GF implementation for the simple cases
> >     * ...that is extensible to allow others to handle the more complex
> > cases, by having a generic API for manipulating generic functions
>
>I'm all for this. I think it would be more successful if it had
>support for interfaces.

Sure.  See my explanation above for why I think that this *does* support 
interfaces.


> > So that the "one obvious way" to create new generics is to use the standard
> > GF implementation unless you need something else.
>
>I'd rather not try to teach people that they can't define a __len__
>method any more but must update half a dozen generic functions in
>order to create a new sequence type.

Implementing __len__ (or invoking 'addmethod(len,...)', or 'defop 
len(self):...') will still be sufficient to make a type len-able.  So code 
that uses len() will "just work" in that case.  Same thing for 
operator.getitem() or its [] shorthand syntax.


>Overriding __special__ methods works fine for the most part.

We can certainly keep the implementation of built-in generics based on 
__special__ methods.  My example code was intended to demonstrate that it's 
possible even in today's Python to have a uniform interface to defining 
such methods -- not that it's necessarily a desirable way to do it, given 
the absence of dedicated syntax like 'defop' for operator (operation?) 
overloading.

Even *with* a 'defop' syntax, I would still not propose we eliminate 
__special__ methods, as they are very fast when implemented as C slots.  I 
would just say that, like 'apply()', they would no longer be the optimum 
way to do something for which there is a syntactical shortcut.  Note that 
__special__ methods can be misspelled, and the resulting error can be hard 
to find.  Misspell a 'defop', and the failure is immediate.  (Of course, 
you could still defop the *wrong* function, but it's still an improvement.)


>Sure, I see a use case for defining an operation that the class
>author(s) did *not* foresee, but that's the exception rather than the
>rule.

Right, at least in the sense that team development is exceptional for 
Python.  In "enterprisey" and other team development scenarios (e.g. Zope, 
Chandler, etc.), it's valuable to have separation between domain model code 
and presentation code.  IIRC, the driving use cases for adaptation in Zope 
3 were to achieve this separation.

In other words, even if you *can* define the methods, that doesn't mean 
it's a good idea to, if you are trying to maximize reuse.  But I freely 
admit that Zope and PEAK at least are exceptional situations: Zope Corp. 
(and my group at Verio during the time I initially created PEAK), were both 
effectively consulting organizations gaining cost-effectiveness through 
reuse of framework code.  This is not exactly the most common use of 
Python...  unless of course you're in an "enterprise" shop or consulting 
organization.


> >      addmethod(iter, somefunc, sometype)
>
>I don't actually understand what you want this example to mean. What
>is 'iter' supposed to be? A generic function?

The iter builtin, as an example of the uniform addmethod() being applicable 
to any generic function, including existing builtin ones.  Although I guess 
you already know that from what you read beyond that point.


> > would actually work by doing 'sometype.__iter__ = somefunc' under the
> > hood.
>
>How would addmethod know this?

By addmethod itself being a generic function.


>At least one poster has already remarked that the head-exploding
>capacity of generic functions is greater than that of metaclasses. I
>think that's a gross exaggeration, but calling it "simple" and "easy
>to use" is an exaggeration too. It's quite deep (otherwise we'd seen
>it in Java already ;-).

We've already seen it in Python, actually.  Generic operations of this 
nature are at the very soul of the language; we simply provide syntactic 
shorthand for many of them, and builtins for the rest.

So, adding an explicit GF API seems more like type/class unification to me 
(or adding the callable __class__ hook), than a sea change in the nature of 
Python itself.  That is, a nice improvement in uniformity and 
user-extensibility, rather than any actual change.


>How should it be extensible? Please explain this without making any
>use of examples from RuleDispatch.

Implement the API using generic functions.  If you have an addmethod(), 
make it a generic.  If you have a 'hasmethod()', make it generic.

(Please note that 'generic' here refers to the extensibility of the 
function, not to a requirement that it be implemented via a registry!  I 
don't care if we have to give generic functions __addmethod__ and 
__hasmethod__ specials.  The point is merely that there should be *some* 
way to extend the core API.)


>What's a generic function type? How does one create one?

Implement a callable object that can be passed to addmethod(), hasmethod(), 
etc.


>OK, I take it back. My brain just exploded. it *is* worse than
>metaclasses. Please find enclosed the bits of brain that I scraped of
>my monitors. :-)

Ironically, it's *you* who gave me this idea, although I don't think you 
realized it at the time.  It was back when you were prototyping the 
multiple-dispatch implementation using a tuple of types -- something you 
said made me realize that we could have a generic API for manipulating 
generics, and thus allow hypothetical Zope and PEAK generics to live 
alongside Python-provided generics as equal citizens.  I then went away and 
made a proof of concept, before coming back to explode your head.


>Poof. You disappear in a cloud of orange smoke.

I guess now it's my turn not to understand what something means.  :)

I was merely trying to show that my idea is trivially 
implementable.  simplegeneric is only about 100 lines of Python added to 
what I wrote.  Throw in a 'defop' syntax that calls addmethod(), and we're 
done.

What I suppose is not obvious to anyone else from what I've written, is 
that this would:

1. allow competing type systems to coexist and even interoperate with a 
minimum of fuss
2. provide a *definition-time checkable* alternative to __special__ method 
names
3. ...while still allowing __specials__ to be used for fast lookup under 
the hood
4. allow for a "blessed" interface mechanism to either be chosen based on 
actual uses, OR
5. avoid the need for having a "blessed" interface mechanism at all, if GF 
introspection or adaptation suffices


>That's an interesting thought but it works with or without generic
>functions -- it's just "ability algebra". I think I've played with
>"type algebra" using similar ideas in a blog entry once.

Sure -- the difference is that I'm suggesting using the operation objects 
themselves as the basic unit of that algebra.

>Not really, IMO. The underlying problem is that standard type
>hierarchy defies capturing it in interfaces *BECAUSE INTERFACES DIDN'T
>EXIST WHEN THEY WERE CREATED*.

That's a reasonable hypothesis.  You might find it works out okay for 
things with few operations (like sequences and "mappings") but not so well 
for things with lots of utility methods ("list-like", "file-like", 
"dict-like" etc.).  I guess we'll see how it works out in practice.


> > The advanced abilities (per-instance/on-the-fly and multi-operation tests)
> > will likely affect performance and/or simplicity of the default
> > implementation.  A purely type-based system can be implemented efficiently,
> > because declaring an interface can simultaneously add a concrete type
> > registration for all affected generics.  (And registering an
> > interface-linked method in a generic function can pull in all the concrete
> > classes.)
>
>This seems to argue *for* interfaces?

I was pointing out that if you really need interfaces, you can have them, 
without needing to create a distinct interface object.  As you yourself 
pointed out in your blog, an "interface" can just be a generic function 
that returns an object supporting that interface, for a given input 
object.  In this sense, "iter" is an interface, because when you call it on 
an object, you get back an object that supports the "iter" interface.

Thus, I don't see much need to create a special notion of interfaces as a 
first-class citizen.  In the common case, a generic "adapting" function 
(like iter) is sufficient, even if the "interface" includes multiple 
methods (like __iter__ and next()).


> > Per-instance tests, however, increase code complexity.  Generally speaking,
> > RuleDispatch does per-instance tests after class tests, but the way it gets
> > reasonable performance is by building decision trees beforehand to manage
> > the possible tests and avoid overlap.  If it actually had to call
> > individual methods, or call a method more than once (ob.has_ability(Foo),
> > ob.has_ability(Bar), etc.)  it would be considerably slower to select an
> > option.
>
>That would make more sense as Foo.implemented_by(ob), right?

Um, I guess.  I was emphasizing the dynamic/per-instance aspect of the 
issue.  Whichever way you do it, unless you have a well-defined implication 
or inheritance hierarchy between interfaces, you have the problem of 
ambiguity between opaque interfaces.


> > Also -- and this is the real kicker -- what do you do if more than one
> > ability applies?  Now we have to have precedence rules for what's more
> > specific.  The advantage of defining an ability as a set of one or more
> > applicable generic functions is that precedence is comparatively
> > straightforward: supersets take precedence over subsets, and overlaps are
> > ambiguous.  You also have some possibility of being able to implement this
> > by registration-time checks, without needing to build a dispatch tree 
> at all.
>
>Couldn't ability inheritance serve the same purpose?

Sure.  The above was part of my argument against *dynamically-determined* 
abilities as a core feature.


> > In short, I think it's adding interface introspection that's the radical
> > move, where generic functions are just a bit of polish that brings a bit
> > more order to what Python already has.  In contrast, interfaces are a
> > foreign religion imported from other languages, while the roots of the
> > generic function faith (len, iter, etc.) have already been in place since
> > the dawn of time!  :)
>
>Python has a long tradition of borrowing ideas from other languages,
>and mostly very successful.

Of course -- and generic functions have a successful tradition in many 
other languages too, from Common Lisp to Haskell, that IIUC goes back much 
further than "interfaces" as such do.  I was pointing out that you had 
*already* borrowed this tradition many years ago, whether you realized it 
or not.  You just "hid" it by using __special__ method names instead of 
registries.  ;-)