[Python-checkins] [3.12] gh-102856: Add changes related to PEP 701 in 3.12 What's New docs (GH-104824) (#104847)
pablogsal
webhook-mailer at python.org
Wed May 24 06:00:41 EDT 2023
https://github.com/python/cpython/commit/a5c0ef87a1c7b08d9c7407a705b3751d9e0b3638
commit: a5c0ef87a1c7b08d9c7407a705b3751d9e0b3638
branch: 3.12
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: pablogsal <Pablogsal at gmail.com>
date: 2023-05-24T11:00:34+01:00
summary:
[3.12] gh-102856: Add changes related to PEP 701 in 3.12 What's New docs (GH-104824) (#104847)
gh-102856: Add changes related to PEP 701 in 3.12 What's New docs (GH-104824)
(cherry picked from commit c45701e9ef004a523ebb28f3be902b3cf2cf7a9b)
Co-authored-by: Marta Gómez Macías <mgmacias at google.com>
Co-authored-by: Pablo Galindo Salgado <Pablogsal at gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra at gmail.com>
files:
M Doc/whatsnew/3.12.rst
diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst
index 4ff90664bb79..27fbb21c3342 100644
--- a/Doc/whatsnew/3.12.rst
+++ b/Doc/whatsnew/3.12.rst
@@ -66,6 +66,10 @@ Summary -- Release highlights
.. PEP-sized items next.
+New grammar features:
+
+* :pep:`701`: Syntactic formalization of f-strings
+
New typing features:
* :pep:`688`: Making the buffer protocol accessible in Python
@@ -136,22 +140,70 @@ Improved Error Messages
New Features
============
-* Add :ref:`perf_profiling` through the new
- environment variable :envvar:`PYTHONPERFSUPPORT`,
- the new command-line option :option:`-X perf <-X>`,
- as well as the new :func:`sys.activate_stack_trampoline`,
- :func:`sys.deactivate_stack_trampoline`,
- and :func:`sys.is_stack_trampoline_active` APIs.
- (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
- with contributions from Gregory P. Smith [Google] and Mark Shannon
- in :gh:`96123`.)
-* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`,
- have a new a *filter* argument that allows limiting tar features than may be
- surprising or dangerous, such as creating files outside the destination
- directory.
- See :ref:`tarfile-extraction-filter` for details.
- In Python 3.14, the default will switch to ``'data'``.
- (Contributed by Petr Viktorin in :pep:`706`.)
+.. _whatsnew312-pep701:
+
+PEP 701: Syntactic formalization of f-strings
+---------------------------------------------
+
+:pep:`701` lifts some restrictions on the usage of f-strings. Expression components
+inside f-strings can now be any valid Python expression including backslashes,
+unicode escaped sequences, multi-line expressions, comments and strings reusing the
+same quote as the containing f-string. Let's cover these in detail:
+
+* Quote reuse: in Python 3.11, reusing the same quotes as the containing f-string
+ raises a :exc:`SyntaxError`, forcing the user to either use other available
+ quotes (like using double quotes or triple quotes if the f-string uses single
+ quotes). In Python 3.12, you can now do things like this:
+
+ >>> songs = ['Take me back to Eden', 'Alkaline', 'Ascensionism']
+ >>> f"This is the playlist: {", ".join(songs)}"
+ 'This is the playlist: Take me back to Eden, Alkaline, Ascensionism
+
+ Note that before this change there was no explicit limit in how f-strings can
+ be nested, but the fact that string quotes cannot be reused inside the
+ expression component of f-strings made it impossible to nest f-strings
+ arbitrarily. In fact, this is the most nested f-string that could be written:
+
+ >>> f"""{f'''{f'{f"{1+1}"}'}'''}"""
+ '2'
+
+ As now f-strings can contain any valid Python expression inside expression
+ components, it is now possible to nest f-strings arbitrarily:
+
+ >>> f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}"
+ '2'
+
+* Multi-line expressions and comments: In Python 3.11, f-strings expressions
+ must be defined in a single line even if outside f-strings expressions could
+ span multiple lines (like literal lists being defined over multiple lines),
+ making them harder to read. In Python 3.12 you can now define expressions
+ spaning multiple lines and include comments on them:
+
+ >>> f"This is the playlist: {", ".join([
+ ... 'Take me back to Eden', # My, my, those eyes like fire
+ ... 'Alkaline', # Not acid nor alkaline
+ ... 'Ascensionism' # Take to the broken skies at last
+ ... ])}"
+ 'This is the playlist: Take me back to Eden, Alkaline, Ascensionism'
+
+* Backslashes and unicode characters: before Python 3.12 f-string expressions
+ couldn't contain any ``\`` character. This also affected unicode escaped
+ sequences (such as ``\N{snowman}``) as these contain the ``\N`` part that
+ previously could not be part of expression components of f-strings. Now, you
+ can define expressions like this:
+
+ >>> print(f"This is the playlist: {"\n".join(songs)}")
+ This is the playlist: Take me back to Eden
+ Alkaline
+ Ascensionism
+ >>> print(f"This is the playlist: {"\N{BLACK HEART SUIT}".join(songs)}")
+ This is the playlist: Take me back to Eden♥Alkaline♥Ascensionism
+
+See :pep:`701` for more details.
+
+(Contributed by Pablo Galindo, Batuhan Taskaya, Lysandros Nikolaou, Cristián
+Maureira-Fredes and Marta Gómez in :gh:`102856`. PEP written by Pablo Galindo,
+Batuhan Taskaya, Lysandros Nikolaou and Marta Gómez).
.. _whatsnew312-pep709:
@@ -223,6 +275,24 @@ See :pep:`692` for more details.
Other Language Changes
======================
+* Add :ref:`perf_profiling` through the new
+ environment variable :envvar:`PYTHONPERFSUPPORT`,
+ the new command-line option :option:`-X perf <-X>`,
+ as well as the new :func:`sys.activate_stack_trampoline`,
+ :func:`sys.deactivate_stack_trampoline`,
+ and :func:`sys.is_stack_trampoline_active` APIs.
+ (Design by Pablo Galindo. Contributed by Pablo Galindo and Christian Heimes
+ with contributions from Gregory P. Smith [Google] and Mark Shannon
+ in :gh:`96123`.)
+
+* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`,
+ have a new a *filter* argument that allows limiting tar features than may be
+ surprising or dangerous, such as creating files outside the destination
+ directory.
+ See :ref:`tarfile-extraction-filter` for details.
+ In Python 3.14, the default will switch to ``'data'``.
+ (Contributed by Petr Viktorin in :pep:`706`.)
+
* :class:`types.MappingProxyType` instances are now hashable if the underlying
mapping is hashable.
(Contributed by Serhiy Storchaka in :gh:`87995`.)
@@ -543,6 +613,14 @@ tkinter
like ``create_*()`` methods.
(Contributed by Serhiy Storchaka in :gh:`94473`.)
+tokenize
+--------
+
+* The :mod:`tokenize` module includes the changes introduced in :pep:`701`. (
+ Contributed by Marta Gómez Macías and Pablo Galindo in :gh:`102856`.)
+ See :ref:`whatsnew312-porting-to-python312` for more information on the
+ changes to the :mod:`tokenize` module.
+
types
-----
@@ -687,6 +765,11 @@ Optimizations
* Speed up :class:`asyncio.Task` creation by deferring expensive string formatting.
(Contributed by Itamar O in :gh:`103793`.)
+* The :func:`tokenize.tokenize` and :func:`tokenize.generate_tokens` functions are
+ up to 64% faster as a side effect of the changes required to cover :pep:`701` in
+ the :mod:`tokenize` module. (Contributed by Marta Gómez Macías and Pablo Galindo
+ in :gh:`102856`.)
+
CPython bytecode changes
========================
@@ -1130,6 +1213,8 @@ Removed
Iceape, Firebird, and Firefox versions 35 and below (:gh:`102871`).
+.. _whatsnew312-porting-to-python312:
+
Porting to Python 3.12
======================
@@ -1201,6 +1286,40 @@ Changes in the Python API
that may be surprising or dangerous.
See :ref:`tarfile-extraction-filter` for details.
+* The output of the :func:`tokenize.tokenize` and :func:`tokenize.generate_tokens`
+ functions is now changed due to the changes introduced in :pep:`701`. This
+ means that ``STRING`` tokens are not emitted any more for f-strings and the
+ tokens described in :pep:`701` are now produced instead: ``FSTRING_START``,
+ ``FSRING_MIDDLE`` and ``FSTRING_END`` are now emitted for f-string "string"
+ parts in addition to the appropriate tokens for the tokenization in the
+ expression components. For example for the f-string ``f"start {1+1} end"``
+ the old version of the tokenizer emitted::
+
+ 1,0-1,18: STRING 'f"start {1+1} end"'
+
+ while the new version emits::
+
+ 1,0-1,2: FSTRING_START 'f"'
+ 1,2-1,8: FSTRING_MIDDLE 'start '
+ 1,8-1,9: OP '{'
+ 1,9-1,10: NUMBER '1'
+ 1,10-1,11: OP '+'
+ 1,11-1,12: NUMBER '1'
+ 1,12-1,13: OP '}'
+ 1,13-1,17: FSTRING_MIDDLE ' end'
+ 1,17-1,18: FSTRING_END '"'
+
+ Aditionally, there may be some minor behavioral changes as a consecuence of the
+ changes required to support :pep:`701`. Some of these changes include:
+
+ * Some final ``DEDENT`` tokens are now emitted within the bounds of the
+ input. This means that for a file containing 3 lines, the old version of the
+ tokenizer returned a ``DEDENT`` token in line 4 whilst the new version returns
+ the token in line 3.
+
+ * The ``type`` attribute of the tokens emitted when tokenizing some invalid Python
+ characters such as ``!`` has changed from ``ERRORTOKEN`` to ``OP``.
+
Build Changes
=============
More information about the Python-checkins
mailing list