Tokenizing a string

[Michael Dartt]

I've got a string I'd like to tokenize, but it's not in a file, and it'd
be rather inefficient to write it to a file just to tokenize it.  Is
there any function I can use to pass this string to
tokenize.tokenize()?  

I could probably use re in this situation, but I'd really rather not, as
the eat() function I'm working with is really nifty, making it much more
convenient to just use tokenize().

Thanks in advance for your help!


--Mike