Controlling a generator the pythonic way

[Thomas Lotze]

Thomas Lotze wrote:

> Does anybody here have a third way of dealing with this?

Sleeping a night sometimes is an insightful exercise *g*

I realized that there is a reason why fiddling with the pointer from
outside the generator defeats much of the purpose of using one. The
implementation using a simple method call instead of a generator needs
to store some internal state variables on an object to save them for the
next call, among them the pointer and a tokenization mode.

I could make the thing a generator by turning the single return
statement into a yield statement and adding a loop, leaving all the
importing and exporting of the pointer intact - after all, someone might
reset the pointer between next() calls.

This is, however, hardly using all the possibilities a generator allows.
I'd rather like to get rid of the mode switches by doing special things
where I detect the need for them, yielding the result, and proceeding as
before. But as soon as I move information from explicit (state variables
that can be reset along with the pointer) to implicit (the point where
the generator is suspended after yielding a token), resetting the
pointer will lead to inconsistencies.

So, it seems to me that if I do want to use generators for any practical
reason instead of just because generators are way cool, they need to be
instantiated anew each time the pointer is reset, for simple consistency
reasons.

Now a very simple idea struck me: If one is worried about throwing away
a generator as a side-effect of resetting the tokenization pointer, why
not define the whole tokenizer as not being resettable? Then the thing
needs to be re-instantiated very explicitly every time it is pointed
somewhere. While still feeling slightly awkward, it has lost the threat
of doing unexpected things.

Does this sound reasonable?

-- 
Thomas