[AstroPy] writing FITS files fast

[Erin Sheldon]

Ivan, I've opened an issue for you here

https://github.com/esheldon/fitsio/issues/41

I encourage you to get a github account so you can follow the progress.
-e

On 4/21/15, Ivan Zolotukhin <ivan.zolotukhin at gmail.com> wrote:
> Hi,
>
>> fitsio does support 64-bit.
>
> Indeed, I was probably using it wrong, but maybe its error message can
> be improved not to cause a confusion in similar cases, even though my
> use case is somewhat weird. Here's the minimal test case I had
> problems with (tested in numpy 1.8.1 and 1.9.2, fitsio 0.9.7):
>
> ```
> import numpy as np
> from fitsio import FITS
>
> fits = FITS('test.fits', 'rw')
> arr = [1L, 2L]
> nparr = np.array(arr)
> nulls = np.equal(arr, None)
> nparr[nulls] = -1 # or whatever your FITS null value is
> #arr = nparr.astype('int64') # uncomment this line and comment the
> next one to make it work
> arr = np.array(nparr, dtype='int64')
> fits.write({'test': arr})
> fits.close()
> ```
>
> This results in:
>
> ```
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> /home/user/<ipython-input-92-b693c45fa56d> in <module>()
> ----> 1 fits.write({'test': arr})
>
> /usr/local/lib/python2.7/dist-packages/fitsio/fitslib.pyc in
> write(self, data, units, extname, extver, compress, tile_dims, header,
> names, table_type, **keys)
>     446                              extname=extname, extver=extver,
> header=header,
>     447                              names=names,
> --> 448                              table_type=table_type)
>     449
>     450
>
> /usr/local/lib/python2.7/dist-packages/fitsio/fitslib.pyc in
> write_table(self, data, table_type, names, formats, units, extname,
> extver, header)
>     714             self[-1]._update_info()
>     715
> --> 716         self[-1].write(data,names=names)
>     717
>     718     def create_table_hdu(self, data=None, dtype=None,
>
> /usr/local/lib/python2.7/dist-packages/fitsio/fitslib.pyc in
> write(self, data, **keys)
>    1506                 firstrow=keys.get('firstrow',0)
>    1507                 self._FITS.write_columns(self._ext+1,
> nonobj_colnums, nonobj_arrays,
> -> 1508                                          firstrow=firstrow+1)
>    1509
>    1510         # writing the object arrays always occurs the same way
>
>
> TypeError: Unsupported numpy table datatype 9
> ```
>
> I'll explain why I needed this seemingly excessive numpy operations. I
> have a list of long integers as an input and I need to check it for
> None values (even though the test case does not have it), that's why I
> make numpy array of objects `nparr` first and set Nones in it to a
> dedicated integer value (let me know if there's a way to avoid this in
> case your input has missing values). Then I naively expected that
> doing `np.array(nparr, dtype='int64')` casts my array from numpy array
> of objects to the array of long integers. I still do not get this
> numpy magics, but if I change the type casting to the line which is
> commented above `arr = nparr.astype('int64')` the fitsio error is
> gone. I initially thought this weird problem has something to do with
> int64 because changing data type in question to int32 also makes this
> error disappear.
>
> As for performance, with fitsio I'm now bottlenecked by my SQL
> database which is how I expect things to happen with complex data
> models. This is still not the case for astropy 1.0.2, which
> bottlenecks my FITS writing procedure. I can prepare a test case if
> someone is interested.
>
> --
> With best regards,
>  Ivan
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>


-- 
Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com