[Python-ideas] Error handling for unknown Unicode characters (was Re: allow `lambda' to be spelled λ)

[Terry Reedy]

[Apologies for the previous premature send]
On 7/21/2016 3:41 AM, Nick Coghlan wrote:
> On 21 July 2016 at 15:08, Rustom Mody <rustompmody at gmail.com> wrote:
>> My “wrongheaded” was (intended) quite narrow and technical:
>>
>> - The embargo on non-ASCII everywhere in the language except identifiers
>> (strings
>>   and comments obviously dont count as “in” the language
>> - The opening of identifiers to large swathes of Unicode widens as you say
>>   hugely the surface area of attack
>>
>> This was solely the contradiction I was pointing out.
>
> OK, thanks for the clarification, and my apologies for jumping on you.
> I can be a bit hypersensitive on this topic, as my day job sometimes
> includes encouraging commercial redistributors and end users to stop
> taking community volunteers for granted and instead help find ways to
> ensure their work is sustainable :)

Like reporting errors on the tracker or at least python-list or even 
here, instead of only in books, articles, posts, other forum questions. 
I found a few unreported errors on SO questions (and have fixed a couple).

> As it is, I think there are some possible checks that could be added
> to the code generator pipeline to help clarify matters:
>
> - for the "invalid character" error message, we should be able to
> always report both the printed symbol *and* the ASCII hex escape,
> rather than assuming the caret will point to the correct place
> - the caret positioning logic for syntax errors needs to be checked to
> see if it's currently counting encoded UTF-8 bytes instead of code
> points (as that will consistently do the wrong thing on a correctly
> configured UTF-8 terminal)
> - (more speculatively) when building the symbol table, we may be able
> to look for identifiers referenced in a namespace that are not NKFC
> equivalent, but nevertheless qualify as Unicode confusables, and emit
> a SyntaxWarning (this is speculative, as I'm not sure what degree of
> performance hit would be associated with it)
>
> As far as Danilo's observation regarding the CPython code generator
> always emitting SyntaxError and SyntaxWarning (regardless of which
> part of the code generation actually failed) goes,

It is not true. https://bugs.python.org/issue25733
All these are (supposedly) possible:
SyntaxError -- obvious
NameError -- ?, already caught in code module
OverflowError-- ?, already caught in code module
SystemError  - 22 nested for loops ('deeply nested blocks')
TypeError -- chr(0), 2.7
ValueError -- chr(0), 3.x; bytes(0), 2.7

Emitting SystemError was changed in 27514.

I wish Danilo's observation were true.  This issue is about the fact 
that the code module and IDLE do not catch all possible compile errors 
because there was no documented list until I compiled the above, which 
may still not be complete.

> I wouldn't be
> opposed to our getting more precise about that by defining additional
> subclasses, but one of the requirements would be for documentation in
> https://docs.python.org/devguide/compiler.html or helper functions in
> the source to clearly define "when working on <this> part of the code
> generation pipeline, raise <that> kind of error if something goes
> wrong".

Or I wish that the compile doc gave a complete list.

-- 
Terry Jan Reedy