[AstroPy] Moving Ahead with Raw Image Conversion

[Jim Vickroy]

At the risk of stating the obvious, you should only be using numpy (not 
numarray).  Numarray was a competing package that pyfits previously 
supported.  Numarray is deprecated and the recent releases of pyfits 
have complete support for numpy.

-- jv

Wayne Watson wrote:
> Howdy, I think I see the ccdsoft problem. I looked at a fit (fits) file 
> I produced with my CCD camera using it, and it produces a uint16 
> (BITPIX=16). I just made the change you suggested, and the resulting 
> file shows BITPIX=8, but CCDSoft liked it. Apparently, it objected to 
> the 32 bit image earlier, which probably gave rise to the black image. 
> The former way produced a file of 1,204KB, and the current file produced 
> sized with the simple replacement produced a file of 304KB. I used 
> dtype='uint8' instead of 'B'.
>
> Somewhere along the line I confused myself with the numarray use in the 
> FITS manual, and apparently its look-alike notion ( as seen via arange 
> and reshape) in numpy. Maybe they are equivalent in every respect. I was 
> trying to figure out how to get Python to put together a 'matrix' 
> (mathematical sense) to feed it to data, but it looks like reshape is 
> the ticket. reshape seems missing from numarray (not mentioned in the 
> fits manual), so maybe there is a difference. I'm not sure the pyfits 
> manual ever showed how to make a 'matrix' and stuff it into the data 
> portion of a FITS header (maybe that's what section 2.2.3 is about, 
> which interestingly has the same title as 2.2.2).
>
> I think we are pretty much, successfully, done here. Allowing the meteor 
> program I'm adding new features to be able to use FITS will be a very 
> useful change. Some exposure to pyfits and numpy helps too from the 
> coding aspects. Of course, if I decide to add features that allow FITS 
> files into the program, say, for IP (img processing) work, that'll 
> require some unraveling of the data part of the fits file for 
> PIL/Tinter. Later though.
>
> Thanks for the help.
>
> Peter Erwin wrote:
>   
>> Hi Wayne,
>>
>> Anne Archibald pointed out a much more concise and elegant way to read in
>> and convert the raw file, using a function in Numpy.  So you could 
>> replace the
>> following original code:
>>
>>     
>>> # get raw data from .raw file:
>>> raw_file=open('sent_internal.raw','rb')
>>> raw_image=raw_file.read()
>>> raw_file.close()
>>>
>>> # convert raw data to Python list of integers (1-D):
>>> rawim_intarray = []
>>> for x in raw_image:
>>>   # convert byte to integer ('B' = treat data as single unsigned byte)
>>>   newint = struct.unpack('B', x)[0]
>>>   rawim_intarray.append(newint)
>>>
>>> # convert Python list of integers into numpy array of integers (note: 
>>> still 1-D at this point)
>>> rawim_numpy = numpy.array(rawim_intarray)
>>>       
>> with this:
>>
>> rawim_numpy = numpy.fromfile('sent_internal.raw', dtype='B')
>>
>> [note: dtype='B' is identical to dtype=numpy.uint8, which Anne used in 
>> her example]
>>
>> and then continue as before.
>>
>> This has the added benefit of producing a smaller output file (300 KB vs
>> 1.2 MB), since the numpy array is now explicitly in 8-bit-integer form
>> (rather than the 32-bit-integer form that my code created), and so
>> Pyfits will automatically save it as 8-bit integers (you can check this
>> by looking at the header keyword BITPIX).
>>
>>
>> You're right, the raw image is a sequence of (standard) 8-bit bytes.  
>> Well, all
>> computer files are streams of bytes; the question is how to interpret 
>> them.  Since
>> you said earlier that it was supposed to be 8-bit values, the idea is 
>> then to
>> interpret each individual byte as an unsigned "integer" value (values 
>> from 0 to 255);
>> that's what the "dtype='B'" (or "dtype=numpy.uint8") option passed to 
>> numpy.fromfile()
>> does.  (My previous code did the same thing, but then assigned each 
>> value to a
>> standard Python integer, which is 4 bytes long; perfectly workable, 
>> but a bit
>> wasteful!).
>>
>> Anyway, when I do this, I can see an image in DS9 (a circular aperture 
>> with ragged
>> edges and an overall horizontal gradient inside the aperture).  So I'm 
>> pretty sure
>> it's working.  (I've attached a jpeg saved from DS9 to show what the 
>> image looks like.)
>>
>> I don't think I can help you with this "ccdsoft" program; I'm not 
>> familiar with it.
>> If it's a Windows program, then it may expect all FITS files to end in 
>> ".fit" instead
>> of ".fits", just because Windows traditionally expects files to have 
>> 3-letter extensions.
>>
>>
>> As for overwriting an existing FITS file, try this:
>>
>> hdu.writeto("existing_file_name.fit", clobber=True)
>>
>> cheers,
>>
>> Peter
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>>
>>
>> On Apr 14, 2009, at 6:28 AM, Wayne Watson wrote:
>>
>>     
>>> Hi, Peter. It looks like I confused a few people with the hex. I had 
>>> removed it, but only after I placed the code in the msg. I was 
>>> playing with hex in the event I might need it somehow. int was fine 
>>> for my experimental purposes.
>>>
>>> Here's your code with a minor correction. I changed this line:
>>>  rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>> The difference is rawmin_numpy.
>>> I added a print at the end, and threw in two lines that attempt to 
>>> put header info into the final file.  Continuing below the code.
>>>
>>> =====================Start===============
>>> import struct, numpy, pyfits   # struct is part of the main Python 
>>> library
>>>
>>> # get raw data from .raw file:
>>> raw_file=open('sent_internal.raw','rb')
>>> raw_image=raw_file.read()
>>> raw_file.close()
>>>
>>> # convert raw data to Python list of integers (1-D):
>>> rawim_intarray = []
>>> for x in raw_image:
>>>   # convert byte to integer ('B' = treat data as single unsigned byte)
>>>   newint = struct.unpack('B', x)[0]
>>>   rawim_intarray.append(newint)
>>>
>>> # convert Python list of integers into numpy array of integers (note: 
>>> still 1-D at this point)
>>> rawim_numpy = numpy.array(rawim_intarray)
>>>
>>> # reshape numpy array into 2D form, using our knowledge of the 
>>> original image's
>>> # x and y sizes (kudos to Megan Sosey for pointing out how to do this)
>>> rawim_numpy_2d = numpy.reshape(rawim_numpy,(480, 640))
>>>
>>> # create Pyfits header-data unit:
>>> hdu = pyfits.PrimaryHDU(rawim_numpy_2d)
>>> hdu.header.update('LATOBS',"32:11:56")
>>> hdu.header.update('LNGOBS',"120:00:00")
>>>
>>> # (make any modifications to the header you might want to... e.g., 
>>> see Megan Sosey's
>>> # email of April 8 for examples)
>>>
>>> # ... and save the data to disk as a FITS file:
>>> hdu.writeto("test.fits")
>>> print "Finished. See test.fits"
>>> ===============End code============
>>>
>>> I'm not sure what the 'byte' stuff is about. It may be that it's not 
>>> needed. I think the original 'raw' file is byte oriented--rather than 
>>> int or whatever. Anyway,  I don't seem to have succeeded in getting 
>>> header info into the file. Sort of. If I use DS9, I can see the image 
>>> and the fits header with what I coded into it. Here's what I see with 
>>> ds9.
>>>
>>> *However, with my ccdsoft program, te image is black and the header 
>>> is not the same. Whoops. Interesting. I changed the file to test.fit 
>>> instead of test.fits, and the header info is there, but the image is 
>>> still black. The dimension are the same, 640x480. I'm attaching 
>>> test-ww.fits. I just renamed my test.fits. I think I'll check with 
>>> The Bisque (ccdsoft) to see what format they produce and use for the 
>>> raw image.
>>> Question. When I execute the program a second time, the 
>>> *hdu.writeto("test.fits") *will not write over the old file. Is there 
>>> an option to force the write?
>>>
>>> Well, this is all good progress. **  *
>>>
>>> -- 
>>>
>>>          Wayne Watson (Watson Adventures, Prop., Nevada City, CA)
>>>
>>>            (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time)
>>>             Obz Site:  39° 15' 7" N, 121° 2' 32" W, 2700 feet
>>>          All the neutrons, and protons in the human body occupy
>>>          a cube whose side is 5.52*10**-6 meters (tiny!). That
>>>          adds up to a 150 pound person. It's not a surprise that
>>>          we are mostly space. (Calculation by WTW)
>>>       
>> =============================================================
>> Peter Erwin                   Max-Planck-Insitute for Extraterrestrial
>> erwin at mpe.mpg.de              Physics, Giessenbachstrasse
>> tel. +49 (0)89 30000 3695     85748 Garching, Germany
>> fax  +49 (0)89 30000 3495     http://www.mpe.mpg.de/~erwin
>>
>>
>>
>>     
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20090414/bfb2e559/attachment.html>