Parser needed.

[Skybuck Flying]

I made it way too difficult on myself with that stupid dictionary bs...

What I really wanted was to know if the ref was already in the reflist.

Turns out python has a really nice simple operation for that:

   if not (Ref in EntityRef):
    EntityRef.append(Ref)

DONE ! =D

No need for dictionary.

Anyway... I am trying a more robust parser... because my own parser right 
now didn't work out for new inputs.

It almost worked except for first item... don't know what problem was.... 
maybe just this...

But I'll try and do it the usually way.

"Tokenize", "Parse" etc.

It's significantly slower though.

Maybe an idea for a new kind of parser could be:

Find all recgonized token keywords in one go... and stuff all find indexes 
into a list... and then perhaps sort the list...

and then use the sorted list... to return token found at that position...

and then start processing like that...

Benefit of this idea is less characters to compare... and could be done with 
just integer compare... or even lookup table processing.

Added benefit is still in-order which is nice.

For now I won't do it... cause I am not sure it would be an improvemt.

Another idea which might be an improvement is.

Parallel searching... ofcourse that might not be possible... though... multi 
core does it exist and threading too.

But to mimic parallel searching and to prevent problems.

A sliding window approach could be taken.

And perhaps items found that way in a certain buffer or so and then still 
added to processing list or so...

which kinda mimics parallel search... but just uses data cache nicely.

Though it's somewhat of a complex method... I will avoid it for now.

The split() routine is real nice... to get rid of fudd/white space... and 
just tokens which is nice.

So for now I will use that as my tokenizer ;) =D

and bracket level counting and sections and stuff like that yeah...

Bye,
  Skybuck.