[Python-Dev] Treating tokenize.Untokenizer as private

[Terry Reedy]

I am working through the multiple bugs afflicting tokenize.untokenize, 
which is described in the tokenize doc and has an even longer docstring. 
While the function could be implemented as one 70-line function, it 
happens to be implemented as a 4-line wrapper for a completely 
undocumented (Untokenizer class with 4 methods. (It is unmentioned in 
the doc and there are currently no docstrings.)

I view the class as a private implementation detail and would like to 
treat it as such, and perhaps even rename it _Untokenizer to make that 
clear. The issue arises in #9974. It appears that a fix may require the 
addition of an instance attribute or .add_whitespace parameter. If there 
is objection to treating the whole class as private, I would at least 
like to treat add_whitespace as the private helper that it is. There is 
no reason to call it directly except for testing. Otherwise, it could 
just as well have been left inline at the one call site.

-- 
Terry Jan Reedy