[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

[Thomas Robitaille]

Hi Joe,

> Also, if you know of *any* other way to save an object, please say.

If you don't have too many different object types, and if you really want long-term retrieval, then you may want to consider actually not using pickles, but for example in the case of FITS headers, you could really just use the toTxtFile and fromTxtFile methods to save and read from ASCII. I can see how pickling can be useful for some instances, but I think FITS headers are definitely one example where there is not much gain in using pickles over plain old ASCII. You can always write your own 'pickle' module which would decide how to deal with various datatypes, and use a plain ASCII write/read for pyfits.Header objects.

Cheers,

Tom

> It seems pretty clear to me that to load objects from any kind of save
> file, you have to import the classes of the object and any objects it
> contains that are not standard Python objects.  So even if we had
> other methods for saving, they would have the same problem as pickle.
> But we have to be able to save objects!  Perhaps saving the definitions
> of the types rather than importing them would be the way to go?  I bet
> there's a long thread about this somewhere.
> 
> --jh--
> 
> On Thu, 11 Nov 2010 14:38:38 -0500, Perry Greenfield <perry at stsci.edu> wrote:
> 
>> We'll look into it. This is a general problem with pickles (and one  
>> reason I've been hesitant to avoid using them like save files). I  
>> wonder if there is a better solution than that. In this case we had to  
>> clean out the previous numarray interface.
>> 
>> Perry
>> 
>> On Nov 10, 2010, at 7:24 PM, Joe Harrington wrote:
>> 
>>> My research group uses Python pickles to save data as it goes through
>>> our pipeline (.npy and .npz do not save objects, and neither does HDF,
>>> etc.).  These need to be loadable forever, as we often compare work to
>>> work done much earlier.  Some of the objects we save contain pyfits
>>> header objects.  Pickles have to import all classes used in the
>>> pickled objects before they load, and we are getting an ImportError
>>> about NP_pyfits.  The file NP_pyfits.py existed in stsci_python 2.8
>>> but is gone in 2.10.  The pickles refer to this object explicitly:
>>> 
>>> ....
>>> sS'photchan'
>>> p494
>>> I3
>>> sS'header'
>>> p495
>>> (ipyfits.NP_pyfits
>>> Header
>>> p496
>>> (dp497
>>> S'_hdutype'
>>> p498
>>> cpyfits.NP_pyfits
>>> PrimaryHDU
>>> p499
>>> sS'ascard'
>>> p500
>>> ccopy_reg
>>> _reconstructor
>>> p501
>>> (cpyfits.NP_pyfits
>>> CardList
>>> p502
>>> c__builtin__
>>> list
>>> p503
>>> (lp504
>>> g501
>>> (cpyfits.NP_pyfits
>>> Card
>>> p505
>>> c__builtin__
>>> object
>>> p506
>>> NtRp507
>>> (dp508
>>> S'_valuestring'
>>> p509
>>> S'T'
>>> ....
>>> 
>>> Is there any way to make our pickles readable again, other than
>>> running the old version of pyfits forever?  Can you provide a pickle
>>> converter that replaces the old names in the file with whatever is
>>> new?
>>> 
>>> Please (everyone, not just STScI) be aware of this issue going
>>> forward.  Pickles are the only way we know of to save objects.  You
>>> can add things to your classes, but if you change what they import (or
>>> otherwise break pickle), nobody can restore your class across
>>> releases.
>>> 
>>> Thanks,
>>> 
>>> --jh--
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy