string-based tokenizer?

[The Blue Wizard]

I was looking for a quick way to tokenize the Python and Python-like
expressions, and I found the tokenizer.py.  It is quite nice, but there
is one little ugly fact:  it is dependent on datastream.  I want to be able
to just type  tokenize('a cute string known as 3+4')  and get a series of
tokens directly from it.  I hope I would not have to resort to recoding the
tokenizer.py...any solution would be appreciated.

One obvious advantage is that it allows looping, like this:
if __name__ == "__main__":
    done = 0
    prompt = ">>> "
    while not done:
        print prompt,
        line = readline()
        done = tokenize(line)
        prompt = "... "

Thanks for any help!

The Blue Wizard