[New-bugs-announce] [issue16224] tokenize.untokenize() misbehaves when moved to "compatiblity mode"

[Eric Snow]

New submission from Eric Snow:

When tokenize.untokenize() encounters a 2-tuple, it moves to compatibility mode, where only the token type and string are used from that point forward.  There are two closely related problems:

* when the iterable is a sequence, the portion of the sequence prior to the 2-tuple is traversed a second time under compatibility mode.
* when the iterable is an iterator, the first 2-tuple encountered is essentially gobbled up (see issue16221).

Either an explicit "iterable = iter(iterable)" or "iterable = list(iterable)" should happen at the very beginning of Untokenizer.untokenize().  If the former, Untokenizer.compat() should be fixed to not treat that first token differently.  If the latter, self.tokens should be cleared at the beginning of Untokenizer.compat().

I'll put up a patch with the second option when I get a chance.

----------
assignee: eric.snow
components: Library (Lib)
messages: 172851
nosy: eric.snow
priority: normal
severity: normal
stage: test needed
status: open
title: tokenize.untokenize() misbehaves when moved to "compatiblity mode"
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16224>
_______________________________________