[issue21765] Idle: make 3.x HyperParser work with non-ascii identifiers.

[Terry J. Reedy]

Terry J. Reedy added the comment:

I did not open another issue yet because I did not want to split the general discussion of parsing 3.x unicode-based python. We might also need another issue for idlelib/PyParse, and that might need to come first.

What I think is that Idle should have at 1 definition of 'identifier' and not a least 3, in 3 separate places ;-). Hyperparser seems like the most appropriate place if there is to be more than str.isidentifier.

That is not to say that equivalents versions might be needed for different uses. ColorDelegator, as it is, needs an re version to be combined with other regexes. I have not looked yet at what UndoDelegator does with its character classification function and whether str.isidentifier could be used.  Hyperparsar seems the hardest since it parses backwards.

I do not consider this issue to be the highest priority, at the moment, but we have collected enough info with help from Ezio and Martin to do some experiments.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21765>
_______________________________________