pickle broken: can't handle NaN or Infinity under win32

[Grant Edwards]

On 2005-06-22, Scott David Daniels <Scott.Daniels at Acm.Org> wrote:

>> I finally figured out why one of my apps sometimes fails under
>> Win32 when it always works fine under Linux: Under Win32, the
>> pickle module only works with a subset of floating point
>> values.  In particular the if you try to dump/load an infinity
>> or nan value, the load operation chokes:
>
> There is no completely portable way to do this.

Python deals with all sorts of problems for which there is no
completely portable solution.  Remember: "practicality beats purity."

> Any single platform can have a solution, but (since the C
> standards don't address how NaNs and Infs are represented)
> there is not a good portable way to do the pickle / unpickle.

Likewise, there is no completely portable python
implimentation.  Any single platform can have a Python
implimentation, but since the C standards don't address a
universal standard for "a computer" there is not a good
portable way to do Python.  I guess we'd better give up on
Python.  :)

> It is nice the exception is raised, since at one point it was
> not (and a simple 1.0 was returned).

That would be even worse.

>> [NaN and Infinity are prefectly valid (and extremely useful)
>> floating point values, and not using them would require huge
>> complexity increases in my apps (not using them would probably
>> at least triple the amount of code required in some cases).]
>
> You could check to see if the Python 2.5 pickling does a better
> job.  Otherwise, you've got your work cut out for you.

Fixing it is really quite trivial.  It takes less than a dozen
lines of code.  Just catch the exception and handle it.

    def load_float(self):
        s = self.readline()[:-1]
        try:
            f = float(s)
        except ValueError:
            s = s.upper()
            if s in ["1.#INF", "INF"]:
                f = 1e300*1e300
            elif s in ["-1.#INF", "-INF"]:
                f = -1e300*1e300
            elif s in ["NAN","1.#QNAN","QNAN","1.#IND","IND","-1.#IND"]:
                f = -((1e300*1e300)/(1e300*1e300))
            else:
                raise ValueError, "Don't know what to do with "+`s`
        self.append(f)

Obviously the list of accepted string values should be expanded
to include other platforms as needed.  The above example
handles Win32 and glibc (e.g. Linux).

Even better, add that code to float().

-- 
Grant Edwards                   grante             Yow!  Is the EIGHTIES
                                  at               when they had ART DECO
                               visi.com            and GERALD McBOING-BOING
                                                   lunch boxes??