[issue32911] Doc strings no longer stored in body of AST

[Nick Coghlan]

Nick Coghlan <ncoghlan at gmail.com> added the comment:

After looking at potential implementation approaches, I believe the following would be the ASDL code generator changes needed to present the docstring as part of the body in the Python API, while keeping it as a separate attribute in the C API:

* in ast2obj, add code in the converter postamble to inject the docstring as body[0]
* in obj2ast, add code in the converter preamble to pop and discard body[0] when docstring is not None
* in ast_type_init, add code to either inject the docstring as body[0] (if the docstring is set), or else to extract and set the docstring based on body[0]

Since CPython's own AST optimizer works at the C API layer, we then wouldn't need to revert any of the changes that relied on the separation of the docstring out to its own attribute.

At the Python level, some differences would still be visible though:

- there'd be a node for the docstring in the AST, but the value would still be optimised out of the resulting code object
- the first line number of code objects would still change to be that of the first non-docstring line
- the synthesised docstring node would always claim to be the line before the first non-docstring line (since escape characters mean we can't just count line feeds in the docstring)

Given the complexity of the required changes to the ASDL code generator, and the Python level API differences that would still remain, I think Serhiy's full reversion patch is going to be the more reliable option at this late stage of the release process.

While it isn't nice for the folks that already adapted their code to cope with the earlier beta releases, it's the lowest risk approach that lets us get 3.7.0 out the door without unintended consequences, while allowing the issue to be revisited for 3.8 with greater awareness of the potential backwards compatibility and code migration issues.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32911>
_______________________________________