ka-ping yee tokenizer.py

[Fredrik Lundh]

Karl Kobata wrote:

> I have enjoyed using ka-ping yee’s tokenizer.py.  I would like to 
> replace the readline parameter input with my own and pass a list of 
> strings to the tokenizer.  I understand it must be a callable object and 
> iteratable but it is obvious with errors I am getting, that this is not 
> the only functions required.

not sure I can decipher your detailed requirements, but to use Python's 
standard "tokenize" module (written by ping) on a list, you can simple 
do as follows:

     import tokenize

     program = [ ... program given as list ... ]

     for token in tokenize.generate_tokens(iter(program).next):
         print token

another approach is to turn the list back into a string, and wrap that 
in a StringIO object:

     import tokenize
     import StringIO

     program = [ ... program given as list ... ]

     program_buffer = StringIO.StringIO("".join(program))

     for token in tokenize.generate_tokens(program_buffer.readline):
         print token

</F>