Controlling a generator the pythonic way

[Kent Johnson]

Thomas Lotze wrote:
> Mike Meyer wrote:
> What worries me about the approach of changing state before making a
> next() call instead of doing it at the same time by passing a parameter is
> that the state change is meant to affect only a single call. The picture
> might fit better (IMO) if it didn't look so much like working around the
> fact that the next() call can't take parameters for some technical reason.

I suggest you make the tokenizer class itself into an iterator. Then you can define additional next() methods with additional parameters. You could wrap an actual generator for the convenience of having multiple yield statements. For example (borrowing Peter's PdfTokenizer):

class PdfTokenizer:
    def __init__(self, ...):
        # set up initial state
        self._tokenizer = _getTokens()

    def __iter__(self):
        return self

    def next(self, options=None):
        # set self state according to options, if any
        n = self._tokenizer.next()
        # restore default state
        return n

    def nextIgnoringSpace(self):
        # alterate way of specifying variations
        # ...

    def _getTokens(self):
        while whatever:
            yield token

    def seek(self, newPosition):
        # change state here

Kent