[Python-Dev] Backward incompatible change about docstring AST

[Serhiy Storchaka]

27.02.18 15:37, INADA Naoki пише:
> Of course, this change was backward incompatible.
> Tools reading/writing docstring via AST will be broken by this change.
> For example, it broke PyFlakes, and PyFlakes solved it already.
> 
> https://github.com/PyCQA/pyflakes/pull/273

Other examples:

coveragepy: https://bitbucket.org/ned/coveragepy/commits/99176232199b
pytest: https://github.com/pytest-dev/pytest/pull/2870


> Last week, Mark Shannon reported issue about this backward incompatibility.
> As he said, this change losted lineno and column of docstring from AST.

While losing lineno and column is a loss, it is not so large. There are 
existing issues with docstring position.

1. CPython and PyPy set different position for multiline strings. PyPy 
sets the position of the start of string, but CPython sets the position 
of the end of the string. A program that utilizes the docstring position 
needs to handle both of these cases.

2. Usually the position of the docstring is used for determining the 
absolute position of some fragments in the docstring (for example 
doctests). But since the literal string can contain \n and escaped 
newlines, and this information is lost in AST, the position of the 
fragment can be determined accurately. This is just a best effort 
approximation.

3. You can determine an approximate position of the docstring by 
positions of preceding or following nodes.