[AstroPy] PyFITS: Appending tables with variable length array...?

[Taro Sato]

On Mon, May 9, 2011 at 11:38 AM, Erik Bray <embray at stsci.edu> wrote:
> Hi Taro,
>
> On 05/06/2011 07:05 PM, Taro Sato wrote:
>>
>> On Fri, May 6, 2011 at 5:56 PM, Erik Bray<embray at stsci.edu>  wrote:
>>>
>>> Here's a workaround that worked for me:
>>>
>>> for idx in range(len(t1[1].columns)):
>>>     col = t1[1].columns[idx]
>>>     if col.format[0] == 'P':
>>>         hdu.columns[idx].array = np.resize(col.array, nr)
>>>         hdu.data._convert[idx] = hdu.columns[idx].array
>>>     hdu.data.field(idx)[nr1:] = t2[1].data.field(idx)
>>>
>> Thanks very much for your response, Erik.  Unfortunately it doesn't
>> work for me quite yet...
>>
>> I see what you are doing, but for some reason the column doesn't
>> remain a _VLF object after setting _convert.  (Before setting _convert
>> I confirm it's still a _VLF.)  For me, right after _convert setting it
>> becomes a flat np.array of int32 and it complains about me trying to
>> set an array element with a sequence.
>>
>> Any idea how to work around this?
>
> Sorry I couldn't get back to you sooner.  I see what you're saying; I might
> have made a mistake in writing down the sample code.  There are maybe two
> things to try here:
>
> Where it assigns to hdu.data._convert[idx], instead try:
>
> hdu.data._convert[idx] = None
>
> Or if that doesn't work you can manually wrap the array in a _VLF object:
>
> hdu.data._convert[idx] = pyfits.core._VLF(hdu.columns[idx].array)
>
> I created a ticket for this bug here:
> http://trac6.assembla.com/pyfits/ticket/54
>
> Thanks, and I hope that works.
>
> Erik
>
>


Erik -- Thanks for your response (and absolutely no need for apology
since I wasn't expecting a tech support anything like that...).    I
think I managed to get it to work for me, but I need to do a tiny bit
more of dirty hacking of the private parts...  I love/hate that Python
doesn't hide anything...  So indecent...

I needed to assign _VLF to _convert[idx], but unfortunately that
somehow messed up the structure of _VLF by flattening the original
_VLF array (into an array of int, float, etc.).  Within PyFITS it
appears _VLF is always assumed to be an array of arrays (i.e., 2D) so
I had to manually maintain it that way.  Here's the code

#### BEGIN ####
import pyfits as pf

...

t1 = pf.open(fits1)
t2 = pf.open(fits2)

nr1 = t1[1].data.shape[0]
nr2 = t2[1].data.shape[0]
nr = nr1 + nr2
hdu = pf.new_table(t1[1].columns, nrows=nr)
for idx in range(len(t1[1].columns)):
    col = t1[1].columns[idx]
    if col.format[0] == 'P':
        orig = t1[1].data.field(idx)[:nr1]
        hdu.columns[idx].array = np.resize(col.array, nr)
        hdu.data._convert[idx] = pf.core._VLF(hdu.columns[idx].array)
        hdu.data._convert[idx][:nr1] = orig
   hdu.data.field(idx)[nr1:] = t2[1].data.field(idx)

...
#### END ####

For now it's working for me.  This post is just for completeness.
Hope it gets implemented more elegantly in PyFITS.

Cheers,
Taro