[issue27112] tokenize.__all__ list is incomplete

[Jacek Kołodziej]

Jacek Kołodziej added the comment:

> I would lean toward ensuring the test fails if someone adds a new implementation detail without an underscore prefix. It is also good to be explicit that the ISTERMINAL() etc functions are special cases.

Original patch meets these requirements. I've updated it with moving the test__all__ method to TestMisc class as suggested (tokenize_all.v2.patch).

I'm also attaching the alternative version (tokenize_all.v2.1.patch) that uses self.assertCountEqual instead of support.check__all__ and whitelisting as Serhiy suggested; this version of test doesn't meet the requirements above.

Yes, neither one challenge the tok_name (#25324) problem, I'm not really sure whether is should, though. I'll try to solve it with separate patch if I find some time.

----------
Added file: http://bugs.python.org/file44374/tokenize_all.v2.1.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27112>
_______________________________________