[Python-ideas] function defaults and an empty() builtin

[Steven D'Aprano]

On Sat, 21 May 2011 04:15:18 am you wrote:
> On 2011-05-20, at 17:03 , Nick Coghlan wrote:
> > I share Steve's puzzlement as the intended use case.
> >
> > To get value from the magic empty immutable list, you will have to
> > explicitly test that calling your function with the default value
> > does the right thing.
>
> Why is that? The value of the empty immutable list (there's nothing
> magic to it) would be an eternal assertion that an incorrect behavior
> (trying to mutate the default parameter) can not be introduced in the
> function.

It's not the caller's responsibility to avoid mangling the internals of 
the function. It is the function's responsibility to avoid exposing 
those internals.

This suggestion seems crazy to me. Let me try to explain from the point 
of view of the caller. Suppose I call func and get a list back:

x = func(a)

So I can treat x as a list, because that's what it is:

x.append(None)

But if I fail to pass an argument, and the default empty() is used, I 
get something that looks like a list:

y = func()
hasattr(y, "append")  # returns True

but blows up when I try to use it:

y.append(None)  # raise an exception

All because the function author doesn't want me modifying the return 
result. And why does the author care what I do with the result? Because 
he's exposing the default function value in such a way that the caller 
can mangle it.

If it causes problems when the function returns the default value, stop 
returning the default value! Don't push the burden onto the caller by 
dropping a landmine into their code.

Now you can "fix" this, for some definition of "fix", by documenting the 
fact that not passing the argument will result in something other than 
a list:

"If you don't pass an argument, and use the default, then you will get 
back an immutable empty sequence that has the same API as a list but 
that will raise an exception if you try to mutate it."

This is downright awful API design. As the caller, I simply don't care 
about the function author's difficulties in ensuring that the default 
value is not modified. That's Not My Problem. Fix your own buggy code. 
(Not that it is actually difficult: the idiom for mutable default 
values is two simple lines.)

What Is My Problem is that rather than fix his function, the author has 
dumped the problem in my lap. Now I have this immutable empty sequence 
that is useless to me. I either have to detect it and change it myself:

result = func(*args)  # args could be empty
if result is empty():
    # Fix stupid design flaw in func
    result = []


or I have to remember to never, under any circumstances, call func() 
without supplying an argument.


[...]
> There are 17 functions or methods with list default parameters and
> 133 with dict default parameters in the Python standard library.
>
> Surely some of them legitimately make use of a mutable default
> parameter as some kind of process-wide cache or accumulator, but
> I would doubt the majority does (why would SMTP.sendmail need to
> accumulate data in its mail_options parameter across runs?)
>
> Do you know for sure that no mutation of these 150+ parameters will
> ever be introduced, that all of these functions and methods are
> sufficiently tested, called often enough that the introduction of
> a mutation of the default parameter in themselves or one of their
> callees would *never* be able to pass muster?

Fine, you've discovered 150 potentially buggy functions in the standard 
library.

If the authors didn't remember to use the default=None idiom in their 
functions, what makes you think that they'd remember to use 
default=empty() instead?

This suggested idiom is counterproductive. The function author doesn't 
save any work -- he still has to remember not to write default=[] in 
his functions. The author's burden is increased, because now he has to 
choose between three idioms instead of two:

# use this when default is like a cache
default=[]  

# use this when you need to mutate default within the function
default=None  
if default is None:
    default = []

# use this when you want to return the default value but don't want 
# the caller to mutate it
default=empty()


And the caller's burden is increased, because now he has to deal with 
this immutable list instead of a real list.


-- 
Steven D'Aprano