[issue16221] tokenize.untokenize() "compat" mode misses the encoding when using an iterator

[Eric Snow]

New submission from Eric Snow:

While traversing the passed iterable, untokenize() will go to "compatibility mode" one it hits a 2-tuple instead of a 5-tuple (coming from the iterable).  That token will not go through the normal steps that other tokens do.  Most critically, if that token is the ENCODING token, the source is not encoded to that encoding at the end (even though it should have been).  I'll have a patch up in a moment (for the tip of default).

There are a couple of other issues at hand that I will bring up separately.

----------
assignee: eric.snow
messages: 172848
nosy: eric.snow
priority: normal
severity: normal
stage: test needed
status: open
title: tokenize.untokenize() "compat" mode misses the encoding when using an iterator
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16221>
_______________________________________