[Python-Dev] Assign(expr* targets, expr value) - why targetS?

[anatoly techtonik]

On Fri, Nov 22, 2013 at 3:29 PM, Benjamin Peterson <benjamin at python.org> wrote:
> 2013/11/22 anatoly techtonik <techtonik at gmail.com>:
>> On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>> 2013/11/15 anatoly techtonik <techtonik at gmail.com>:
>>>> On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin at python.org> wrote:
>>>>> 2013/11/12 anatoly techtonik <techtonik at gmail.com>:
>>>>>> On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin at python.org> wrote:
>>>>>>> 2013/11/10 anatoly techtonik <techtonik at gmail.com>:
>>>>>>>> http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
>>>>>>>>
>>>>>>>> In Assign(expr* targets, expr value), why the first argument is a list?
>>>>>>>
>>>>>>> x = y = 42
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Speaking of this ASDL. `expr* targets` means that multiple entities of
>>>>>> `expr` under the name 'targets' can be passed to Assign statement.
>>>>>> Assign uses them as left value. But `expr` definition contains things
>>>>>> that can not be used as left side assignment targets:
>>>>>>
>>>>>>     expr = BoolOp(boolop op, expr* values)
>>>>>>          | BinOp(expr left, operator op, expr right)
>>>>>>          ...
>>>>>>          | Str(string s) -- need to specify raw, unicode, etc?
>>>>>>          | Bytes(bytes s)
>>>>>>          | NameConstant(singleton value)
>>>>>>          | Ellipsis
>>>>>>
>>>>>>          -- the following expression can appear in assignment context
>>>>>>          | Attribute(expr value, identifier attr, expr_context ctx)
>>>>>>          | Subscript(expr value, slice slice, expr_context ctx)
>>>>>>          | Starred(expr value, expr_context ctx)
>>>>>>          | Name(identifier id, expr_context ctx)
>>>>>>          | List(expr* elts, expr_context ctx)
>>>>>>          | Tuple(expr* elts, expr_context ctx)
>>>>>>
>>>>>> If I understand correctly, this is compiled into C struct definitions
>>>>>> (Python-ast.c), and there is a code to traverse the structure, but
>>>>>> where is code that validates that the structure is correct? Is it done
>>>>>> on the first level - text file parsing, before ASDL is built? If so,
>>>>>> then what is the role of this ADSL exactly that the first step is
>>>>>> unable to solve?
>>>>>
>>>>> Only valid expression targets are allowed during AST construction. See
>>>>> set_expr_context in ast.c.
>>>>
>>>> Oh my. Now there is also CST in addition to AST. This stuff -
>>>> http://docs.python.org/devguide/ - badly needs diagrams about data
>>>> transformation toolchain from Python source code to machine
>>>> execution instructions. I'd like some pretty stuff, but raw blogdiag
>>>> hack will do the job http://blockdiag.com/en/blockdiag/index.html
>>>>
>>>> There is no set_expr_context in my copy of CPython code, which
>>>> seems to be some alpha of Python 3.4
>>>
>>> It's actually called set_context.
>>
>> Ok. So what is the process?
>>
>>  SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST -->
>> --> AST --> BYTECODE
>>
>> Is that right?
>
> I don't know what sentence stream is, but otherwise looks right.

I mean that when you have TOKENS, you need to validate that their
order is valid and throw errors to console if it is not. Like every sentence
should have certain word order to make sense, you won't be able to
construct CST from tokens that are placed in wrong order. Right?

Or is CST itself a tree of tokens, which position is validated when the
tree is transformed to AST?


FWIW, I've modified my astdump project to make it easy to explore
Python AST and experiment with it. This dataset shows how to create
coverage for AST transformations (visitors)
https://bitbucket.org/techtonik/astdump/src/tip/dataset/?at=default