[Python-ideas] Retain string form of AST Numbers

[Nick Coghlan]

On 5 April 2014 18:09, Mark Dickinson <dickinsm at gmail.com> wrote:
> On Sat, Apr 5, 2014 at 4:50 AM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> If the Num nodes also had a .str member with the unparsed string ("1",
>> "2.000", and "3.0j"), it would be possible to influence their
>> interpretation with an AST transform - for instance, turning them into
>> decimal.Decimal construction calls, or wrapping them in something that
>> automatically keeps track of significant figures (which would mean
>> that "2.000" is considered more precise than a mere "2.0" or "2.").
>>
>> It shouldn't be too difficult to do, and it could simply be ignored in
>> most cases; any sort of change to the actual number could leave the
>> original string out of sync with it, which isn't a problem.
>>
>> Thoughts?
>
>
> I like it. Apart from the AST transformation possibilities opened up, I have
> some AST-based style checkers that would have benefited from access to the
> raw numeric strings.
>
> So as long as there's no significant adverse affect on total compilation
> time or space requirements, +1 from me.

Also somewhat tangentially related to this issue about being able to
render particular args differently in a signature:
http://bugs.python.org/issue16801

I think the discussion on http://bugs.python.org/issue10769 suggests a
broader philosophical discussion may be appropriate, since it makes a
big difference to what is in scope for the ast module if its purpose
is expanded from "easier to manipulate intermediate representation
between source and bytecode" to "supports lossless source to source
transformations".

Articles like http://time-loop.tumblr.com/post/47664644/python-ast-preserving-whitespace-and-comments
suggest to me that we may need to start pushing people *away* from the
ast module and add a lib2to3 inspired API specifically for lossless
transformations based on the *actual* grammar of the running Python
version (the grammar 2to3 itself uses is a bit mixed up, because it
needs to tolerate Python 2 code as input, and it currently doesn't
handle "yield from" expressions - the latter isn't really an issue in
practice for 2/3 transition related use cases, since hybrid 2/3 code
can't use "yield from" anyway).

(A Google search for "Python AST preserve comments" provides other
interesting links on the topic of source->source transformations that
aren't lossy the way those based on the ast module inevitably are)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia