[Python-ideas] Draft PEP on string interpolation

[Nick Coghlan]

On 23 August 2015 at 14:09, Guido van Rossum <guido at python.org> wrote:
> On Sat, Aug 22, 2015 at 6:37 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Right, but that's where I came to the conclusion that the lack of
>> arbitrary interpolation support ends up making PEP 498 actively
>> dangerous, as string interpolation based substitution ends up being so
>> much prettier than doing things right. Compare:
>>
>>     os.system(f"echo {filename}")
>>     subprocess.call(f"echo {filename}")
>>     subprocess.call(["echo", filename])
>>
>> Even in that simple case, the two unsafe approaches are much nicer to
>> read, and as the command line gets more complex, the safe version gets
>> harder and harder to read relative to the unsafe ones.
>
>
> That reasoning is perverse, and feels disingenuous.

Yeah, professional paranoia produces a weird way of looking at the world :)

The key change in my thinking relative to a couple of years ago has
been that it's no longer the things that throw surprising exceptions
that cause me the most concern, but rather those that *appear* to
work, but are actually hiding a dangerous latent defect. These are the
situations where a developer (or reviewer) has to "just know" that the
apparently obvious way to do something is actually problematic, and
that's where we get security vulnerabilities.

Having the preferred interpolation syntax produce a non-string object
by default provides us with the opportunity to consider on an
interface by interface basis whether we want to:

* require callers to prerender interpolation templates (the default)
* implicitly render interpolation templates with the default renderer
(by calling str on the input, which many APIs do already)
* define and use a custom renderer for interpolation templates

This wouldn't prevent folks from doing the wrong thing -
os.system(str(i"echo $filename")) is just as dangerous from a code
injection perspective as os.system(f"echo {filename}"). The difference
lies in the appearance of the *fixed* code, where
subprocess.call(i"echo $filename") would be just as readable as the
os.system version, while f-strings don't help with any case that
requires a custom renderer in order to do the right thing.

That way, when a security linter picks up a problematic call like
os.system(str(i"echo $filename")), the solution it suggests can be
just as easy to read as the original.

> Which reminds me of your one-time attempts to make call parentheses
> optional, so we could have print be a function and yet be able to write
>
>   print x, y

Yeah, that comparison occurred to me as well. It's one of the reasons
I kept looking for a way to do custom interpolation using a normal
function call instead of needing a new binary operator :)

>> However, I'm now coming full circle back to the idea of making this a
>> string prefix, so that would instead look like:
>>
>>     subprocess.call($"echo $filename")
>>
>> The trick would be to make interpolation lazy *by default* (preserving
>> the triple of the raw template string, the parsed fields, and the
>> expression values), and put the default rendering in the resulting
>> object's *__str__* method.
>
> That's a clever idea. But I expect it will make interpolation much less
> convenient, because every recipient will have to call str(). The elegance of
> PEP 498 is that the recipient doesn't have to do or know anything special,
> because the result is *just* a string object.

Right, although eager rendering with i-strings just involves calling
"str" at the point of definition.

Another alternative would be to combine the two ideas, and have
i-strings be an implementation detail of f-strings, with f"echo
$filename" being a highly optimised version of str(i"echo $filename")
that avoids the need for a builtin name lookup (modulo whichever
substitution field syntax you eventually choose).

>> That description is probably as clear as mud, though, so back to the
>> PEP I go! :)
>
> I recommend taking a break first.

Aye, having got PEP 501 back to a place where *I* like it again, I'll
leave it alone for a while. The pace of iteration this weekend was
because I kept discovering aspects I didn't like myself, and coming up
with related improvements.

> Or maybe sample the recent activity in
> datetime-sig instead. :-)

Minstrels (singing): Brave Sir Robin ran away, bravely ran away away ,
when danger reared its ugly head, he bravely turned his tail and
fled... ;)

Cheers,
Nick.

P.S. For folks not familiar with that last reference:
http://www.montypython.net/scripts/bravesir.php :)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia