[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

[Perry Greenfield]

Hi Joe,

We'll look into it. This is a general problem with pickles (and one  
reason I've been hesitant to avoid using them like save files). I  
wonder if there is a better solution than that. In this case we had to  
clean out the previous numarray interface.

Perry

On Nov 10, 2010, at 7:24 PM, Joe Harrington wrote:

> My research group uses Python pickles to save data as it goes through
> our pipeline (.npy and .npz do not save objects, and neither does HDF,
> etc.).  These need to be loadable forever, as we often compare work to
> work done much earlier.  Some of the objects we save contain pyfits
> header objects.  Pickles have to import all classes used in the
> pickled objects before they load, and we are getting an ImportError
> about NP_pyfits.  The file NP_pyfits.py existed in stsci_python 2.8
> but is gone in 2.10.  The pickles refer to this object explicitly:
>
> ....
> sS'photchan'
> p494
> I3
> sS'header'
> p495
> (ipyfits.NP_pyfits
> Header
> p496
> (dp497
> S'_hdutype'
> p498
> cpyfits.NP_pyfits
> PrimaryHDU
> p499
> sS'ascard'
> p500
> ccopy_reg
> _reconstructor
> p501
> (cpyfits.NP_pyfits
> CardList
> p502
> c__builtin__
> list
> p503
> (lp504
> g501
> (cpyfits.NP_pyfits
> Card
> p505
> c__builtin__
> object
> p506
> NtRp507
> (dp508
> S'_valuestring'
> p509
> S'T'
> ....
>
> Is there any way to make our pickles readable again, other than
> running the old version of pyfits forever?  Can you provide a pickle
> converter that replaces the old names in the file with whatever is
> new?
>
> Please (everyone, not just STScI) be aware of this issue going
> forward.  Pickles are the only way we know of to save objects.  You
> can add things to your classes, but if you change what they import (or
> otherwise break pickle), nobody can restore your class across
> releases.
>
> Thanks,
>
> --jh--