[AstroPy] {SunPy} Re: Unable to Save Compressed HDU

[Christoph Deil]

Hi Harsh,

Pickles should never be used to share data.

As mentioned in the red warning at the top of https://docs.python.org/3/library/pickle.html <https://docs.python.org/3/library/pickle.html> they are unsafe.

It’s easy to make a pickle that when loaded erases the whole hard drive, or worse, basically downloading a pickle from the internet and calling pickle.loads gives the pickle author complete control over your computer.

Still, I tried loading your pickle in a virtual machine, but got this error:
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath’
I’m not sure, I think pickles just aren’t compatible across Python versions, or Numpy versions.
So that’s another reason not to use pickles to share data.

Can you put the header in a text file (header.txt), and for the data, just call numpy.ones or numpy.random to get a Numpy data array that shows the issue? You might have to set the Numpy array dtype explicitly to reproduce the issue. The issue cannot be dependent on the exact values in your data array, very likely it is just dependent on the dtype of the data and the options you pass to the FITS write call.

Christoph

> On 30. Jan 2019, at 18:03, Harsh Mathur <harshmathur.1990 at gmail.com> wrote:
> 
> Hi,
> 
> Please find the steps to reproduce:
> 
> 1. Download the data and header pickled from here (https://drive.google.com/drive/folders/10V7ruXr3As9dD5o416VLTh_ZIMAH3POw?usp=sharing <https://drive.google.com/drive/folders/10V7ruXr3As9dD5o416VLTh_ZIMAH3POw?usp=sharing>)
> 
> 2. Python code:
> 
> import pickle
> import sunpy.io.fits
> from astropy.io <http://astropy.io/> import fits
> data = pickle.loads(open('fits_data.data', 'rb').read())
> header = pickle.loads(open('header.data', 'rb').read())
> sunpy.io.fits.write('sample.fits', data, header, hdu_type=fits.CompImageHDU)
> 
> 3. Before saving we can also confirm the data is correct by
> 
> import matplotlib.pyplot as plt
> plt.imshow(data, cmap='gray')
> plt.show()
> 
> Attached is the plt.show() output.
> 
> Thanks for looking into this.
> 
> Regards
> Harsh Mathur
> harshmathur.1990 at gmail.com <mailto:harshmathur.1990 at gmail.com>
> 
> “Perseverance is the hard work you do after you get tired of doing the hard work you already did."
> 
> 
> 
> On Wed, Jan 30, 2019 at 2:29 PM Christoph Deil <deil.christoph at googlemail.com <mailto:deil.christoph at googlemail.com>> wrote:
> 
> 
>> On 30. Jan 2019, at 09:41, Harsh Mathur <harshmathur.1990 at gmail.com <mailto:harshmathur.1990 at gmail.com>> wrote:
>> 
>> Hi,
>> I have a fits file which normally takes around 132 MB. I want to save it with compressed HDU.
>> 
>> I tried both with astropy and sunpy, but for some reason it is not working. To be specific, it is saving empty HDU (i.e. the grid is only filled with nans).
>> 
>> Here's what I did with astropy:
>> 
>> from astropy.io <http://astropy.io/> import fits
>> header_object = self._get_header_object(header)
>> comp_hdu = fits.CompImageHDU(data=data, header=header_object)
>> comp_hdu.writeto(filename, output_verify='fix')
>> 
>> Here's what I did with sunpy:
>> import sunpy.io.fits
>> sunpy.io.fits.write(
>>             filename,
>>             data,
>>             header,
>>             hdu_type=fits.CompImageHDU
>>         )
>> 
>> I tried to debug using ipdb and checked just before the save instruction, the data and header are present and not empty, just that its not saving. 
>> 
>> The worst thing is, there is no error for me debug. Any help is appreciated.
>> 
>> Thank you in advance.
>> 
>> Regards
>> Harsh Mathur
>> harshmathur.1990 at gmail.com <mailto:harshmathur.1990 at gmail.com>
>> 
>> “Perseverance is the hard work you do after you get tired of doing the hard work you already did."
>> 
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at python.org <mailto:AstroPy at python.org>
>> https://mail.python.org/mailman/listinfo/astropy <https://mail.python.org/mailman/listinfo/astropy>
> 
> 
> Hi Harsh,
> 
> Can you try to create a standalone test case that we can run to try and reproduce / debug the issue?
> E.g. make up some data using numpy.ones or numpy.random, provide a code snippet we can just copy & paste & run.
> 
> Also please mention your Numpy and Astropy version number.
> 
> It’s probably not your exact issue, but there is this issue https://github.com/astropy/astropy/issues/2150 <https://github.com/astropy/astropy/issues/2150>
> which causes issues sometimes, but only when writing certain Numpy arrays and when using compression.
> 
> Without an example to reproduce, it’s likely hard to help figure out exactly what goes wrong in your case.
> 
> Christoph
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org <mailto:AstroPy at python.org>
> https://mail.python.org/mailman/listinfo/astropy <https://mail.python.org/mailman/listinfo/astropy>
> 
> -- 
> You received this message because you are subscribed to the Google Groups "SunPy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sunpy+unsubscribe at googlegroups.com <mailto:sunpy+unsubscribe at googlegroups.com>.
> To post to this group, send email to sunpy at googlegroups.com <mailto:sunpy at googlegroups.com>.
> Visit this group at https://groups.google.com/group/sunpy <https://groups.google.com/group/sunpy>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> <Screenshot 2019-01-30 at 10.32.25 PM.png>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20190201/35881986/attachment-0001.html>