My first Python program -- a lexer

[Thomas Mlynarczyk]

Arnaud Delobelle schrieb:

> Adding to John's comments, I wouldn't have source as a member of the
> Lexer object but as an argument of the tokenise() method (which I would
> make public).  The tokenise method would return what you currently call
> self.result.  So it would be used like this.

>>>> mylexer = Lexer(tokens)
>>>> mylexer.tokenise(source)
>>>> mylexer.tokenise(another_source)

At a later stage, I intend to have the source tokenised not all at once, 
but token by token, "just in time" when the parser (yet to be written) 
accesses the next token:

     token = mylexer.next( 'FOO_TOKEN' )
     if not token: raise Exception( 'FOO token expected.' )
     # continue doing something useful with token

Where next() would return the next token (and advance an internal 
pointer) *if* it is a FOO_TOKEN, otherwise it would return False. This 
way, the total number of regex matchings would be reduced: Only that 
which is expected is "tried out".

But otherwise, upon reflection, I think you are right and it would 
indeed be more appropriate to do as you suggest.

Thanks for your feedback.

Greetings,
Thomas

-- 
Ce n'est pas parce qu'ils sont nombreux à avoir tort qu'ils ont raison!
(Coluche)