[Python-checkins] [3.10] Clarify the supported cases in the tokenize module (GH-105569) (#105575)
pablogsal
webhook-mailer at python.org
Fri Jun 9 12:01:06 EDT 2023
https://github.com/python/cpython/commit/45de31db9cc9be945702f3a7ca35bbb9f98476af
commit: 45de31db9cc9be945702f3a7ca35bbb9f98476af
branch: 3.10
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: pablogsal <Pablogsal at gmail.com>
date: 2023-06-09T16:00:59Z
summary:
[3.10] Clarify the supported cases in the tokenize module (GH-105569) (#105575)
Co-authored-by: Pablo Galindo Salgado <Pablogsal at gmail.com>
files:
M Doc/library/tokenize.rst
diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 11f569df2e7cd..a903bff1da8f5 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type. The exact
type can be determined by checking the ``exact_type`` property on the
:term:`named tuple` returned from :func:`tokenize.tokenize`.
+
+.. warning::
+
+ Note that the functions in this module are only designed to parse
+ syntactically valid Python code (code that does not raise when parsed
+ using :func:`ast.parse`). The behavior of the functions in this module is
+ **undefined** when providing invalid Python code and it can change at any
+ point.
+
Tokenizing Input
----------------
More information about the Python-checkins
mailing list