[Python-ideas] Adding an optional function argument to all() and any() builtins

[Steven D'Aprano]

Andy Buckley wrote:
> I may be missing a standard idiom, but it seems to me that the any(...)
> and all(...) builtins are unnecessarily limited by requiring that the
> iterable they take as an argument is already in a form suitable for the
> intended kind of boolean comparison.

Yes, just like every function requires its input to already be in a form 
suitable for whatever the function expects to do. Just as any() and 
all() expect their argument to be truth-values (not necessarily True or 
False, but values to be inspected for their truthness), so sum() expects 
input consisting of numbers. Would you expect to be able to write this?

mylist = ["a12", "b13", "c14", "d15"]
sum(mylist, convert=lambda s: int(s[1:]))

I hope not. Converting its input into a suitable format is not the 
responsibility of sum. sum's responsibility begins and ends with adding 
the input, any conversions should be done either externally:

mylist = ["a12", "b13", "c14", "d15"]
newlist = [int(s[1:]) for s in mylist]
sum(newlist)

or lazily using a generator expression as the argument to sum:

mylist = ["a12", "b13", "c14", "d15"]
sum(int(s[1:]) for s in mylist)

There are variations on this using built-in map, itertools.imap, and 
similar.

The problems with shifting the responsibility to any and all (or sum) 
instead of keeping it with the caller include:

* It adds complexity to the API for all and any, and that makes the API 
harder to learn and harder to remember.

* It duplicates functionality: now two more functions are expected to 
know how to accept a second argument and call it against the input 
before testing.

* It can lead to "Jack of All Trades, Master of None" functions:

oven.roast(
     chicken, add_stuffing="sage and onion",
     serve=["pre-dinner drinks", "appetizer", "main course", "desert"],
     wash_pots_afterward=False
     )

Why is the oven being asked to prepare the stuffing and serve the food? 
Now this is an extreme case, but any time you consider adding extra 
responsibility to a function, you're taking a step in that direction.

There should be a clear benefit to the caller before accepting these 
extra costs. For example, list.sort(), sorted(), min() and max() now all 
take a key argument which is superficially similar to your suggested 
"test" argument. You would need to demonstrate similar benefit.


> Most of the time, when I want to check that any or all of a collection
> matches some test criterion, my iterable is not already in a valid form
> to pass to any() or all(), and so I either have to explicitly re-code a
> slightly modified version of the builtin, or wastefully use map() to
> apply my test to *all* the items in the list. 

Re-coding all and any is usually a big No. That's almost always the 
wrong solution.

In Python 3, map becomes a lazy iterator, which means it no longer 
applies the test to every item up front. In Python 2, the simple 
alternative is itertools.imap, which saves you from re-coding map:

def lazymap(func, iterable):
     # Don't use this, use itertools.imap
     for obj in iterable:
         yield func(obj)


But probably the best solution is to use a generator expression.

# Like a list comp, except with round brackets.
genexp = (mytestfunction(x) for x in mylist)
any(genexp)


This becomes even easier when you embed the generator expression 
directly in the function call:

any(mytestfunction(x) for x in mylist)

If mytestfunction is simple enough, you can do it in-place without 
needing to write a function:

any(x >= 42 for x in mylist)



[...]
> The "test" keyword arg is perhaps not the best name, I admit, but no
> other jumps prominently to mind ("match", "fn"?).

When you have difficulty thinking of a name which clearly and concisely 
describes what the argument does, that's often a good hint that you are 
dumping too much responsibility onto the function.

Having said that, I'd suggest that an appropriate name might be the same 
name used by sort and friends: key.

-0 on the idea -- I don't find it compelling or necessary, but nor do I 
hate it.



-- 
Steven