Controlling a generator the pythonic way

[Peter Hansen]

Thomas Lotze wrote:
> I can see two possibilities to do this: either the current file position
> has to be read from somewhere (say, a mutable object passed to the
> generator) after each yield, or a new generator needs to be instantiated
> every time the tokenizer is pointed to a new file position.
>... 
> Does anybody here have a third way of dealing with this? Otherwise,
> which ugliness is the more pythonic one?

The third approach, which is certain to be cleanest for this situation, 
is to have a custom class which stores the state information you need, 
and have the generator simply be a method in that class.  There's no 
reason that a generator has to be a standalone function.

class PdfTokenizer:
     def __init__(self, ...):
         # set up initial state

     def getTokens(self):
         while whatever:
             yield token

     def seek(self, newPosition):
         # change state here

# usage:
pdf = PdfTokenizer('myfile.pdf', ...)
for token in pdf.getTokens():
     # do stuff...

     if I need to change position:
         pdf.seek(...)

Easy as pie! :-)

-Peter