[issue21765] Idle: make 3.x HyperParser work with non-ascii identifiers.

[Tal Einat]

Tal Einat added the comment:

I'm attaching a patch which really fixes this issue, along with additional tests for idlelib.HyperParser.

I did indeed have to fix PyParse as well. I got it working with re.subn() as Martin suggested, but the performance was much worse (between 100x and 1000x slower according to my timings). So I came up with the clever solution of passing a defaultdict to str.translate(), with the default value being ord('x'). That works as expected and is much faster than the regexp.

Finally, since the defaultdict is kept around as long as IDLE is running, I decided to avoid having it grow continually and consume memory unnecessarily. So I wrote a simple Mapping class, which wraps a normal dict and uses a custom default value instead of None, ord('x') in this case. Works like a charm :)

I haven't written tests for PyParse since it currently doesn't have any tests at all. But from the HyperParser tests and from some manual testing, it seems to be working as expected.

----------
Added file: http://bugs.python.org/file35959/taleinat.20140716.IDLE_HyperParser_unicode_ids.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21765>
_______________________________________