[Web-SIG] simplejson 2.0.0 released, much faster.

[Bob Ippolito]

On Sat, Sep 27, 2008 at 2:10 PM, Arnar Birgisson <arnarbi at gmail.com> wrote:
> On Sat, Sep 27, 2008 at 22:13, Bob Ippolito <bob at redivi.com> wrote:
>> If you give it unicode input, it will decode to unicode. Basically it
>> scans through the str until it finds non-ASCII, escape, or end quote.
>> If it finds the end quote first it will just allocate a new string
>> with exactly that day, which is super fast since it's just an alloc
>> and copy.
>>
>> It will of course always decode everything containing non-ASCII
>> characters or any escape sequences to unicode. It is not currently
>> configurable. It was done for performance, but also does produce nicer
>> looking repr output because you don't have so many 'u' characters to
>> look at :) Given the way str works in Python 2.x it should not be an
>> incompatible change except for doctests... and I guess code that
>> explicitly checks for unicode and doesn't know what to do with str,
>> but that would be weird.
>
> The reason I asked was because I've had problems even with pure-ASCII
> strs when mixed with unicode objects in some DB-API drivers, working
> with filesystems on the OS-X and others. A "solution" was to have
> everything in unicode.

I've never seen a pure ASCII str cause problems. I've seen pure ASCII
unicode cause problems in stupid ways though, because not all Python C
code that handles text can handle unicode. Dumb stuff like this bites
me all the time in Genshi templates (where all string literals are
unicode):

>>> datetime.datetime.now().strftime(u'%Y')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: strftime() argument 1 must be str, not unicode

Any operation involving a str and unicode should up-convert to
unicode, and regardless of the defaultencoding a pure ASCII str will
properly get handled (at least in Python 2.5, I don't remember what
2.4 did)... e.g. ''.join(['', u'foo']) returns u'foo'

> Since that means the string given to simplejson to decode will be a
> unicode string anyways, so in that case there's no problem :)

If you can prove that there is an actual problem I'm sure I can come
up with a flag that would ensure unicode, but the implementation would
probably be just translation of the input document to unicode before
decoding ;)

-bob