[AstroPy] new pyfits version deletes NP_pyfits, breaking pickle

[Erik Tollerud]

> Erik, thanks for the unpickler code.  We saw the note about those but
> don't offhand know how to use it to fix the problem.

Actually,  I looked at this a bit further (anticipating that I will
also encounter this problem when I next upgrade...), and have attached
a python file that appears to do the conversion.  Either call the
function unpickle_old_pyfits to load your pyfits file, or use it as a
command line script to save a new version of the file (appended with
'.new') that has the same content, but unpickles correctly. Perry,
perhaps this can be included somewhere in pyfits?



> Also, if you know of *any* other way to save an object, please say.

A solution for large amounts of data might be databases.  The only
actual object database (e.g. the entries are all actually objects
rather than translating back and forth between SQL tables) is ZODB
(http://www.zodb.org/).  I know it has explicit support for migrating
databases from an old class definition to a new one, but I've never
actually used that functionality, so I don't know much about it.

There's also lots of stuff out there to map python objects to SQL
databases (I think SQLAlchemy is the most popular), but that of course
requires figuring out how to make your object look like an SQL table.


> contains that are not standard Python objects.  So even if we had
> other methods for saving, they would have the same problem as pickle.

Yeah, this is in inherent problem... If you write the class yourself,
you can override __setstate__ and __getstate__ to manually fix up the
old classes to have whatever new things they need.  But that doesn't
help much if it's an external module that you didn't write.

The whole idea of having the object data and the classes defined
separately (inherent in python) I suppose forces this issue.  There is
probable some way to use the imp and inspect modules to inject old
versions of classes into the stream of pickled objects, but it's
probably a lot of work to do, and wouldn't work if you have any
C-based extensions.


-- 
Erik Tollerud
-------------- next part --------------
A non-text attachment was scrubbed...
Name: np_unpickle.py
Type: text/x-python
Size: 1380 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/astropy/attachments/20101111/438fceeb/attachment.py>