[Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging)

[Eric V. Smith]

On 2/18/2017 2:25 AM, Steven D'Aprano wrote:
> On Fri, Feb 17, 2017 at 11:37:04AM -0500, Mark E. Haase wrote:
>
>> Python has two string formatting mini-languages.
>
> Four. % string formatting, .format method, f-strings, string.Template
> strings.
>
> But who's counting? :-)

Technical correctness is the best kind! But string.Template has no 
formatting language, and str.format and f-strings are identical as far 
as what they do with format specifiers (by using the underlying 
.__format__() machinery). [0]

>> Both allow formatting
>> flags, for example in "%03d", the "0" (zero) is a flag that means to pad
>> with leading zeroes. I propose to add a string format flag to both
>> mini-languages that explicitly says, "this argument is callable, and its
>> *return value* should be formatted."
>
> The % string format codes are intentionally limited to more-or-less the
> similar codes available in C, and won't be given any newer functionality.

I agree it would be difficult to add this to %-formatting, which is 
unfortunate. The most often cited need for this feature is for logging.

Maybe someone smarter than me could devise an interface to logging which 
supports .format() formatting, and we could add the callable flag to 
.format().

Maybe return a different type of logger from logging.getLogger that 
supports .format?

> It's really only f-strings and .format method that this change could be
> applied to.

I'd suggest adding an explicit conversion flag to the .format() string 
parser. You'll recall the existing flags are !a, !s, and !r. We could 
add a !c which calls its argument (with zero parameters) before 
formatting it. For example:

'{!c:%Y-%m-%d}'.format(datetime.datetime.now)
'2017-02-18'

This would have the advantage that the format specifier would apply to 
the result of the callable.

If logging were made smart enough (or a new interface added), then you 
could write:
logger.info('{!c:%Y-%m-%d} error at {}', datetime.datetime.now, lineno)

Then .now() would never be called unless the logging module needed the 
value. (I realize .now() isn't a great example for calling a function in 
the future. Substitute any expensive function.)

> I see three problems:
>
> (1) It will be a bug magnet. People will accidently write
>
>     logging.debug('%03d %C03d', 1, expensive())
>
>
> and then not only will their code still be slow, but they'll have to
> debug mysterious
>
>     TypeError: 'int' object is not callable
>
> exceptions, but only *sometimes*. Most insideously, these Heisenbugs
> will only occur when they turn the log level all the way up to
> debugging, which will crash their program *before* logging the error!

I think this is lessened with my proposal to use !c, but it's definitely 
still an issue. Test your logging!

> (2) It requires the expensive calculation to be wrapped in a lambda:
>
>     logging.debug('%03d %C03d', 1, lambda: nth_prime(10**8) + 1)
>
> which I guess is kind of Python's way of spelling a thunk, but people
> don't like using lambda for that.

This doesn't bother me so much.

> (3) It is useless for delaying evaluation of something that isn't going
> to be converted into a string.

True. But that's the most common use case I see presented, and it's much 
easier to reason about and implement than some of the discussions going 
on in other threads.

Just for completeness, I'd add !c to the f-string parser. Like !s, !r, 
and !a, it would not be needed [1]. Why write:
f'{!c:function}'
when you could write:
f'{function()}'
But, it can't hurt to be consistent.

Eric.

[0]: there is one slight difference in how str.format and f-strings 
handle expressions, but their format specifiers are identical
https://www.python.org/dev/peps/pep-0498/#differences-between-f-string-and-str-format-expressions

[1]: https://www.python.org/dev/peps/pep-0498/#id45