[issue9969] tokenize: add support for tokenizing 'str' objects

[Nick Coghlan]

Nick Coghlan <ncoghlan at gmail.com> added the comment:

Possible approach (untested):

def get_tokens(source):
    if hasattr(source, "encode"):
        # Already decoded, so bypass encoding detection
        return _tokenize(io.StringIO(source).readline, None)
    # Otherwise attempt to detect the correct encoding
    return tokenize(io.BytesIO(source).readline)

----------
nosy: +ncoghlan

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9969>
_______________________________________