[AstroPy] memmap for RICE-compressed 3D fits?

[Attie, Raphael (GSFC-6710)[UNIVERSITIES SPACE RESEARCH ASSOCIATION]]

Hello,

As a regular users of large RICE-compressed 3D fits files, I have to get the 2D-slices in them. These 3D fits files are time series of images from SDO. The ones i’m using are typically [700 MB - 2 GB], with ZCMPTYPE= ‘RICE_1  ‘, XTENSION=‘BINTABLE’, and keep the extension .fits (not .fits.gz or alike).
I would like to use memory mapping so I do not need to load the whole cube before accessing a 2D frame. I used to do it in C with CFITSIO, with custom mex function for Matlab. It was working fine but I’m now  transitioning to Python.
With astropy.fits.open(), I see the option of using memmap = true. I never tried this option before on uncompressed 3D fits but using it on RICE-compressed 3D cube does not seem to make any difference with respect to the default (memmap = false). I test that by simply looking at the RAM usage of the python processes in each case. For reproducing the issue, I provide the codes and link to the data down below.

So my questions are:

1) Is memory mapping supported for such RICE-compressed 3D fits files?

2) If it is supported, could my code down below loading the data misusing astropy.fits?

3) If it is not supported, what are the alternatives to do it within Python?


Python code for loading the 1st slice of the data cube:

from astropy.io<http://astropy.io> import fits

hdu = fits.open(file, ignore_missing_end=True, memmap=True)
hdu.verify('silentfix')
slice    = hdu[1].data[1, :, :]

Dropbox link to the 3D FITS file (also accessible to non-dropbox users):
https://www.dropbox.com/s/t0se5aum5hwyvff/mtrack_20101126_170034_TAI_20101127_170034_TAI_LambertCylindrical_continuum.fits?dl=0



Thank you for your help,

Raphael


- - - - - - - - - -- - - - -- - - - -- - - - -- - - - -- -
Dr. Raphael Attié (GSFC-6710)
NASA Postdoctoral fellow (NPP)
NASA/Goddard Space Flight Center
Universities Space Research Association
Office 041: 301-286-0360
Cell: 301-631-4954
Email: raphael.attie at nasa.gov<mailto:raphael.attie at nasa.gov>







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20170911/97ac8071/attachment.html>