[Python-Dev] UTF-8 Decoder

[Antoine Pitrou]

Jeroen Ruigrok van der Werven <asmodai <at> in-nomine.org> writes:
> 
> So on medium and large datasets the decoder of Bjoern is very interesting,
> but the tiny case (just Bjoern's name) is quite a tad bit slower. The other
> cases seems more typical of what the average use in Python would be.

Keep in mind what the datasets are:

« The large buffer is a April 2009 Hindi Wikipedia article XML dump, the medium
buffer Markus Kuhn's UTF-8-demo.txt, and the tiny buffer my name »

It would be interesting to test with mostly ASCII data to see what that gives.
Now the good thing is that, even with wildly non-ASCII data, our current decoder
is very efficient.

Regards

Antoine.