magic names in python

[Lenard Lindstrom]

Steven D'Aprano wrote:
> On Tue, 05 Jun 2007 18:08:31 +0000, Lenard Lindstrom wrote:
> 
>> Steven D'Aprano wrote:
>>> On Mon, 04 Jun 2007 22:19:35 +0000, Lenard Lindstrom wrote:
>>>
>>>> What is "magic" about __init__ and __repr__? They are identifiers just 
>>>> like "foo" or "JustAnotherClass". They have no special meaning to the 
>>>> Python compiler. The leading and trailing double underscores represent 
>>>> no special incantation. It is just a naming convention.
>>> That's not quite true, as you point out:
>> Disassemble the code object of a class statement. "__init__" is just an 
>> identifier, a Python name. So there is no "magic" here.
> 
> It is not JUST an identifier, it is an identifier with a special, dare I
> say *magic*, meaning.
> 
> I suspect we have different ideas of what a "magic name" is. You seem to
> be under the impression that magic names are a bad thing, and that
> therefore one has to minimize their number. Hence you are glossing over
> the differences and emphasizing the similarities between methods (say) foo
> and __init__.
> 

No, I don't consider magic in a programming language a bad thing. 
Overloaded operators is one feature I like in a language. Using special, 
or magic, methods to do this is elegant. And it would appear we are 
working with the same definition as regarding methods: a magic name 
method is an identifier recognized by the compiler as a special case. So 
a C++ constructor is a magic name. But by the same definition Python 
lacks magic methods. They have nothing to do with the compiler. They are 
a run-time feature.

> I don't consider "magic name" in this content to be anything to be ashamed
> of, so I'm not concerned about downplaying the differences. Yes, there are
> significant similarities between __init__ and random methods you write
> yourself, but the differences are equally, if not more, significant.
> 

I'm not contesting that some identifiers have special meaning or are 
treated differently under certain conditions. Nor am I trying to hide 
the fact. But in the context of Python I don't find it remarkable. The 
Python type system is quite abstract, and class behavior is kept 
separate from the byte code interpreter. It is in the built-in types 
that special methods are given significance. And through these special 
methods one has access to the inner workings of Python's type system and 
modify the special behavior. So if that's magic, then it is a very 
approachable magic.

> 
> 
>>>> So a number of method names like __init__ and __repr__ have a 
>>>> pre-defined usage. 
>>> That makes them magic, in the best possible way.
>> I was careful to use the word "usage". Maybe I should have used 
>> "protocol" or "specification" here. A method named "write" is understood 
>> to have a particular definition and purpose. Is it "magic"? 
> 
> It could be, depending on the context.
> 
> If we're discussing file objects compared to str objects, then file
> objects have a method "write" and str objects don't, and the method is
> just another method.
> 
> But in the context of some third function, which takes a file-like object
> with a "write" method, then yes it is magic, because any type of object
> will work so long as it has a method "write" with the right semantics. Not
> very much magic, but a little. 
> 

Okay.

> 
>> These 
>> methods are simply callbacks. And yes, these callbacks make Python flexible.
>>
>>>
>>>> In every other respect they are just normal methods. 
>>> That is *almost* true. Or, to put it another way, that is wrong. Some of
>>> the double-underscore magic methods are automatically called on the class
>>> instead of the instance, bypassing normal inheritance. 
>>>
>> They exist as Python functions in the class dictionary. They have no 
>> hidden flags that distinguish them from other class level function 
>> attributes.
> 
> They don't need special flags, because the Python compiler itself already
> knows about the existence of them and takes special actions with them.

By compiler I mean the parser and byte code generator, not the 
interpreter or built-in objects. And the compiler knows nothing about 
special Python methods. Neither does the byte code interpreter. By 
defining ones own meta-meta class one can make __init__ and __new__ 
unremarkable.

> 
> [snip]
> 
> 
>> 
 >> [snip]
 >> In CPython an extension type can be written from
>> scratch that treats special methods exactly as a new-style class does. 
>> Or an extension type can implement a completely novel approach. The 
>> point is, at some level, the machinery which defines special method 
>> "magic" is accessible to the programmer. So is it really "magic" or 
>> advanced technology?
> 
> Does it matter? As Clarke said, any sufficiently advanced technology is
> indistinguishable from magic... or to put it another way, as Larry Niven
> did, any sufficiently advanced magic is indistinguishable from technology.
> 
> Since I don't actually believe in magic, as in "wishing makes it so", of
> course it is technology. But most Python programmers aren't capable of, or
> have any interest in, writing extension types in C or hacking the compiler. 
> 

If certain magic behaviors depend on the compiler then that magic may be 
unavailable to the programmer. In Python assignment to None is now a 
syntax error. I would consider this real magic since Python has no way 
to declare compile time constants. But I can turn __build__ into a 
special method that is called at class creation. Special methods are not 
strictly limited to the ones already defined. And yes, in the sense that 
magic means using obscure language features to define new behavior, this 
is magic. Finally, nothing I have mentioned so far requires "hacking the 
compiler".

> 
>> This comes down to the original posting not defining a "magic name". It 
>> does not mention what is so objectionable about __init__ and __repr__. I 
>> am claiming they are not as "magical" as they may first appear.
> 
> I don't believe the Original Poster considers there is anything
> objectionable about magic names like __init__ and __repr__, and even if he
> does, I certainly don't.
> 
I am not claiming that __init__ or __repr__ are objectionable. I am 
suggesting that calling them "magic names" in the context of Python is 
an exaggeration and overlooks the true power of the language.

> Magic names in this context are those methods which have pre-defined
> meanings to the compiler. That is mostly double-underscore methods like
> __init__, but also methods like iterator.next(). If you ask yourself "What
> methods do I have to over-ride to make a customized sub-class of a
> built-in type?", the answer will include mostly magic names. If you ask
> "What methods do I have to write for my class to work with Python
> operators?", the answer will also include magic names like __add__,
> __mul__, and __xor__.
> 
I assumed you considered magic names as any pre-defined identifiers that 
have a special meaning within the language. I am guessing that's how 
others use the term. In this respect I agree that special methods are 
magic names.

But in a programming language labeling a special case as "magic" is not 
descriptive. It says nothing about what makes the special case unusual 
or noteworthy. Is the quality unique to certain pre-defined identifiers? 
Does it make use of obscure language features? Is it a glaring 
inconsistency in an otherwise uniform language?

So I would say to anyone encountering Python magic for the first time to 
reserve judgment until you understanding it. Then it may not seem so 
peculiar after all. Don't make assumptions based on other languages.

--
Lenard Lindstrom
<len-l at telus.net>