From gr.maravelias at gmail.com Wed Feb 10 11:30:11 2016 From: gr.maravelias at gmail.com (Grigoris Maravelias) Date: Wed, 10 Feb 2016 17:30:11 +0100 Subject: [AstroPy] question on saving fits files Message-ID: <56BB6593.8070908@gmail.com> Hello! I have a problem with a rather simple process, but I cannot understand what I am doing wrong. I have an image in a fits file that i want to open and process its data values, and save it as another file (including some extra information on the header for what I did). So according to the astropy docs [1] I did the following: hdulist = fits.open('small.fits') hdu = hdulist[0] hd = hdu.header dt = hdu.data hd.append(('HISTORY', 'blabla '), bottom=True) print dt # check data before dt = dt*1000 print dt # and after hdulist.writeto('small_mod.fits') If I print dt before and after their modification (multiplied by 1000) I can see that they are indeed different (properly multiplied), but when I save the fits file I just see the original data values (however, the HISTORY line is added properly in the primary header). What am I missing here? Best Grigoris [1] http://docs.astropy.org/en/stable/io/fits/usage/headers.html From dburke.gw at gmail.com Wed Feb 10 11:42:40 2016 From: dburke.gw at gmail.com (Doug Burke) Date: Wed, 10 Feb 2016 16:42:40 +0000 Subject: [AstroPy] question on saving fits files In-Reply-To: <56BB6593.8070908@gmail.com> References: <56BB6593.8070908@gmail.com> Message-ID: Grigoris, If you change 'dt = dt * 1000' to 'dt *= 1000' do you get what you expect? Doug On Wed, Feb 10, 2016 at 11:30 AM Grigoris Maravelias < gr.maravelias at gmail.com> wrote: > Hello! > > I have a problem with a rather simple process, but I cannot understand > what I am doing wrong. I have an image in a fits file that i want to > open and process its data values, and save it as another file (including > some extra information on the header for what I did). So according to > the astropy docs [1] I did the following: > > hdulist = fits.open('small.fits') > hdu = hdulist[0] > hd = hdu.header > dt = hdu.data > > hd.append(('HISTORY', 'blabla '), bottom=True) > > print dt # check data before > dt = dt*1000 > print dt # and after > > hdulist.writeto('small_mod.fits') > > If I print dt before and after their modification (multiplied by 1000) I > can see that they are indeed different (properly multiplied), but when I > save the fits file I just see the original data values (however, the > HISTORY line is added properly in the primary header). > > What am I missing here? > > Best > Grigoris > > > [1] http://docs.astropy.org/en/stable/io/fits/usage/headers.html > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcio.melendez at gmail.com Wed Feb 10 12:06:23 2016 From: marcio.melendez at gmail.com (Marcio Melendez) Date: Wed, 10 Feb 2016 12:06:23 -0500 Subject: [AstroPy] question on saving fits files In-Reply-To: References: <56BB6593.8070908@gmail.com> Message-ID: The problem is that you just changed a variable, dt, and not your object hdu (the one that you are writing to). So you need to replace dt = dt*1000 with hdu.data = dt*1000. Alternatively you can create another object (if you want to keep hdu intact) by: hdu_new = pyfits.PrimaryHDU(dt) # where dt = hdu.data*1000 hdu_new.writeto('small_mod.fits') Remember to add clobber = True to your writeto if you want to overwrite the file. I hope this help, On Wed, Feb 10, 2016 at 11:42 AM, Doug Burke wrote: > > Grigoris, > > If you change 'dt = dt * 1000' to 'dt *= 1000' do you get what you expect? > > Doug > > On Wed, Feb 10, 2016 at 11:30 AM Grigoris Maravelias < > gr.maravelias at gmail.com> wrote: > >> Hello! >> >> I have a problem with a rather simple process, but I cannot understand >> what I am doing wrong. I have an image in a fits file that i want to >> open and process its data values, and save it as another file (including >> some extra information on the header for what I did). So according to >> the astropy docs [1] I did the following: >> >> hdulist = fits.open('small.fits') >> hdu = hdulist[0] >> hd = hdu.header >> dt = hdu.data >> >> hd.append(('HISTORY', 'blabla '), bottom=True) >> >> print dt # check data before >> dt = dt*1000 >> print dt # and after >> >> hdulist.writeto('small_mod.fits') >> >> If I print dt before and after their modification (multiplied by 1000) I >> can see that they are indeed different (properly multiplied), but when I >> save the fits file I just see the original data values (however, the >> HISTORY line is added properly in the primary header). >> >> What am I missing here? >> >> Best >> Grigoris >> >> >> [1] http://docs.astropy.org/en/stable/io/fits/usage/headers.html >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy >> > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -- Marcio B. Melendez, Ph.D JWST ISIM Optics Support Scientist Wyle Science, Technology and Engineering Group NASA Goddard Space Flight Center Greenbelt, MD 20771 Bldg. 29 Rm. T29-10, phone: 301-286-8517 -------------- next part -------------- An HTML attachment was scrubbed... URL: From iafe.leticia at gmail.com Wed Feb 10 12:07:42 2016 From: iafe.leticia at gmail.com (Leticia Rodriguez) Date: Wed, 10 Feb 2016 14:07:42 -0300 Subject: [AstroPy] question on saving fits files In-Reply-To: References: <56BB6593.8070908@gmail.com> Message-ID: Hi Grigoris, Try to open the file for writing like this: hdulist = fits.open('small.fits', mode='update') BR 2016-02-10 14:06 GMT-03:00 Marcio Melendez : > The problem is that you just changed a variable, dt, and not your object > hdu (the one that you are writing to). So you need to replace dt = > dt*1000 with hdu.data = dt*1000. Alternatively you can create another > object (if you want to keep hdu intact) by: > > hdu_new = pyfits.PrimaryHDU(dt) # where dt = hdu.data*1000 > hdu_new.writeto('small_mod.fits') > > Remember to add clobber = True to your writeto if you want to overwrite > the file. > > I hope this help, > > On Wed, Feb 10, 2016 at 11:42 AM, Doug Burke wrote: > >> >> Grigoris, >> >> If you change 'dt = dt * 1000' to 'dt *= 1000' do you get what you expect? >> >> Doug >> >> On Wed, Feb 10, 2016 at 11:30 AM Grigoris Maravelias < >> gr.maravelias at gmail.com> wrote: >> >>> Hello! >>> >>> I have a problem with a rather simple process, but I cannot understand >>> what I am doing wrong. I have an image in a fits file that i want to >>> open and process its data values, and save it as another file (including >>> some extra information on the header for what I did). So according to >>> the astropy docs [1] I did the following: >>> >>> hdulist = fits.open('small.fits') >>> hdu = hdulist[0] >>> hd = hdu.header >>> dt = hdu.data >>> >>> hd.append(('HISTORY', 'blabla '), bottom=True) >>> >>> print dt # check data before >>> dt = dt*1000 >>> print dt # and after >>> >>> hdulist.writeto('small_mod.fits') >>> >>> If I print dt before and after their modification (multiplied by 1000) I >>> can see that they are indeed different (properly multiplied), but when I >>> save the fits file I just see the original data values (however, the >>> HISTORY line is added properly in the primary header). >>> >>> What am I missing here? >>> >>> Best >>> Grigoris >>> >>> >>> [1] http://docs.astropy.org/en/stable/io/fits/usage/headers.html >>> >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> https://mail.scipy.org/mailman/listinfo/astropy >>> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy >> >> > > > -- > Marcio B. Melendez, Ph.D > JWST ISIM Optics Support Scientist > Wyle Science, Technology and Engineering Group > NASA Goddard Space Flight Center Greenbelt, MD 20771 > Bldg. 29 Rm. T29-10, phone: 301-286-8517 > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -- Leticia. Technical Contact NOVA - Virtual Observatory Argentina http://nova.conicet.gov.ar -------------- next part -------------- An HTML attachment was scrubbed... URL: From dburke.gw at gmail.com Wed Feb 10 12:37:18 2016 From: dburke.gw at gmail.com (Doug Burke) Date: Wed, 10 Feb 2016 17:37:18 +0000 Subject: [AstroPy] question on saving fits files In-Reply-To: References: <56BB6593.8070908@gmail.com> Message-ID: Apologies for the noise; the perils of sending an email from a long, technical meeting -Doug On Wed, Feb 10, 2016, 11:42 Doug Burke wrote: > > Grigoris, > > If you change 'dt = dt * 1000' to 'dt *= 1000' do you get what you expect? > > Doug > > On Wed, Feb 10, 2016 at 11:30 AM Grigoris Maravelias < > gr.maravelias at gmail.com> wrote: > >> Hello! >> >> I have a problem with a rather simple process, but I cannot understand >> what I am doing wrong. I have an image in a fits file that i want to >> open and process its data values, and save it as another file (including >> some extra information on the header for what I did). So according to >> the astropy docs [1] I did the following: >> >> hdulist = fits.open('small.fits') >> hdu = hdulist[0] >> hd = hdu.header >> dt = hdu.data >> >> hd.append(('HISTORY', 'blabla '), bottom=True) >> >> print dt # check data before >> dt = dt*1000 >> print dt # and after >> >> hdulist.writeto('small_mod.fits') >> >> If I print dt before and after their modification (multiplied by 1000) I >> can see that they are indeed different (properly multiplied), but when I >> save the fits file I just see the original data values (however, the >> HISTORY line is added properly in the primary header). >> >> What am I missing here? >> >> Best >> Grigoris >> >> >> [1] http://docs.astropy.org/en/stable/io/fits/usage/headers.html >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gr.maravelias at gmail.com Wed Feb 10 15:49:54 2016 From: gr.maravelias at gmail.com (Grigoris Maravelias) Date: Wed, 10 Feb 2016 21:49:54 +0100 Subject: [AstroPy] question on saving fits files In-Reply-To: References: <56BB6593.8070908@gmail.com> Message-ID: <56BBA272.6090507@gmail.com> Exactly! I forgot that what I was manipulating was just a variable and that the object hdu remained in memory unchanged. Now it works as it should be! Thanks Marcio! @Leticia : I have already tried that but it was not working for the above reason. @Doug : no problem! thanks anyway! On 02/10/2016 06:06 PM, Marcio Melendez wrote: > The problem is that you just changed a variable, dt, and not your > object hdu (the one that you are writing to). So you need to > replace dt = dt*1000 with hdu.data = dt*1000. Alternatively you can > create another object (if you want to keep hdu intact) by: > > hdu_new = pyfits.PrimaryHDU(dt) # where dt = hdu.data*1000 > > hdu_new.writeto('small_mod.fits') > > Remember to add clobber = True to your writeto if you want to > overwrite the file. > > I hope this help, > > On Wed, Feb 10, 2016 at 11:42 AM, Doug Burke > wrote: > > > Grigoris, > > If you change 'dt = dt * 1000' to 'dt *= 1000' do you get what you > expect? > > Doug > > On Wed, Feb 10, 2016 at 11:30 AM Grigoris Maravelias > > wrote: > > Hello! > > I have a problem with a rather simple process, but I cannot > understand > what I am doing wrong. I have an image in a fits file that i > want to > open and process its data values, and save it as another file > (including > some extra information on the header for what I did). So > according to > the astropy docs [1] I did the following: > > hdulist = fits.open('small.fits') > hdu = hdulist[0] > hd = hdu.header > dt = hdu.data > > hd.append(('HISTORY', 'blabla '), bottom=True) > > print dt # check data before > dt = dt*1000 > print dt # and after > > hdulist.writeto('small_mod.fits') > > If I print dt before and after their modification (multiplied > by 1000) I > can see that they are indeed different (properly multiplied), > but when I > save the fits file I just see the original data values > (however, the > HISTORY line is added properly in the primary header). > > What am I missing here? > > Best > Grigoris > > > [1] http://docs.astropy.org/en/stable/io/fits/usage/headers.html > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > > > > -- > Marcio B. Melendez, Ph.D > JWST ISIM Optics Support Scientist > Wyle Science, Technology and Engineering Group > NASA Goddard Space Flight Center Greenbelt, MD 20771 > Bldg. 29 Rm. T29-10, phone: 301-286-8517 > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.roberts.15 at ucl.ac.uk Thu Feb 11 05:26:25 2016 From: michael.roberts.15 at ucl.ac.uk (Roberts, Michael) Date: Thu, 11 Feb 2016 10:26:25 +0000 Subject: [AstroPy] Creating a mask file for .fits file Message-ID: Dear astropy community, I'm looking to create a mask file for a Swift .img fits format file, which has one dimensionless primary header and anywhere between 1 and 9 image headers (with dimensions). I am attempting to create a mask whereby if any value in the image is either a 0 or a NaN the new mask will have that corresponding pixel as 0, and for all other values the mask will have a value of 1. My attempted script is: import os from astropy.io import fits import numpy as np #This is the main function. def main(): #For all files in the current directory: for filename in sorted(os.listdir(os.getcwd())): #If the file is not an exposure map, skip this file and continue with the next file. if not filename.endswith("lss_new.img"): continue #Replace all NaNs and 0s by 0s and all other values with 1 in the large scale sensitivity maps. replace_mask(filename) #Function to replace all masked pixels by 0s. #Open the data and replace all masked pixels by 0s and all other values with 1. Save the header and the new data of the frame to a new hdu. Append this hdu to the new hdulist. def replace_pix(frame,mask,new_hdulist): data = frame.data header = frame.header data[mask_nan] = 0.0 data[mask_zero] = 0.0 data[mask_one] = 1.0 new_hdu = fits.ImageHDU(data,header) new_hdulist.append(new_hdu) #Function to replace all NaNs in the exposure map by 0s and to replace the corresponding pixels in the sky and large scale sensitivity map by 0s. def replace_mask(filename): #Print that all NaNs and Os will be replaced by 0s in the exposure map and that the corresponding pixels in the sky and large scale sensitivity map will also be replaced by 0s. print "All NaNs and Os will be replaced by 0s in " + filename + " and all other values will be set to zero." #Open the exposure map, the corresponding sky and large scale sensitivity map and copy the primary headers (extension 0 of hdulist) to new hdulists. hdulist_lss = fits.open(filename) new_hdu_header_lss = fits.PrimaryHDU(header=hdulist_lss[0].header) new_hdulist_lss = fits.HDUList([new_hdu_header_lss]) #For all frames in the image: Create the mask and run the function replace_pix. if np.isnan(hdulist_ex[i].data): mask_nan replace_pix(hdulist_lss[i],mask_nan,new_hdulist_lss) elif (hdulist_ex[i].data)=0: mask_zero replace_pix(hdulist_lss[i],mask_zero,new_hdulist_lss) else: mask_one replace_pix(hdulist_lss[i],mask_one,new_hdulist_lss) #Write the new hdulists to new images. new_hdulist_lss.writeto(filename.replace("lss_new.img","lss_mask.img")) #Print that all NaNs are replaced by 0s in the exposure map and that the corresponding pixels in the sky and large scale sensitivity map are also replaced by 0s. print "All NaNs and 0s are replaced by 0s in " + filename + " and all other corresponding pixels in the large scale sensitivity map are replaced by 1s." if __name__ == '__main__': main() Essentially, hopefully it is obvious, but it is failing around the if, elif, else statements (essentially I'm not defining the values correctly). If anyone would know the correct syntax for this I would be greatly appreciative. Many thanks, Michael Roberts -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.covino at brera.inaf.it Thu Feb 11 05:42:42 2016 From: stefano.covino at brera.inaf.it (Stefano Covino) Date: Thu, 11 Feb 2016 11:42:42 +0100 Subject: [AstroPy] slicing Message-ID: <6A5CBBAC-2684-4CA9-94E3-9A2C92E8ADB7@brera.inaf.it> Dear friends, let?s assume we have a data array read from a FITS file: data = getdata(?file.fits?) I need to do some computations on a subset of it, for instance: datf = dat[bb-size:bb+size,aa-size:aa+size] where aa and bb are x,y positions and size a fixed number (e.g. 10). So, the question. Is it possible that aa and bb are not simple integers but lists or arrays of integers? Thanks, Stefano From d.f.evans at keele.ac.uk Thu Feb 11 05:43:34 2016 From: d.f.evans at keele.ac.uk (Daniel Evans) Date: Thu, 11 Feb 2016 10:43:34 +0000 Subject: [AstroPy] Creating a mask file for .fits file In-Reply-To: References: Message-ID: I would've thought the definitions for the masks would be: mask_nan = np.isnan(hdulist_ex[i].data) mask_zero = hdulist_ex[i].data == 0.0 mask_one = np.logical_not(np.logical_and(mask_nan, mask_zero)) Which replace your "if" statements in replace_mask(), which aren't actually assigning any indices to the masks as far as I can see. Are hdulist_ex and hdulist_lss are meant to the same variable? I can't find where hdulist_ex is defined. You also need to change the definition of replace_pix(), as you pass it "mask", but the function refers to "mask_nan", "mask_zero", and "mask_one", which it doesn't know about - the simplest method would be to pass all three masks. Regards, Daniel On 11 February 2016 at 10:26, Roberts, Michael wrote: > Dear astropy community, > > > I'm looking to create a mask file for a Swift .img fits format file, which > has one dimensionless primary header and anywhere between 1 and 9 image > headers (with dimensions). > > > I am attempting to create a mask whereby if any value in the image is > either a 0 or a NaN the new mask will have that corresponding pixel as 0, > and for all other values the mask will have a value of 1. > > > My attempted script is: > > > import os > > from astropy.io import fits > > import numpy as np > > > #This is the main function. > > def main(): > > > #For all files in the current directory: > > for filename in sorted(os.listdir(os.getcwd())): > > > > #If the file is not an exposure map, skip this file and continue > with the next file. > > if not filename.endswith("lss_new.img"): continue > > > > #Replace all NaNs and 0s by 0s and all other values with 1 in the > large scale sensitivity maps. > > replace_mask(filename) > > > #Function to replace all masked pixels by 0s. > > #Open the data and replace all masked pixels by 0s and all other values > with 1. Save the header and the new data of the frame to a new hdu. Append > this hdu to the new hdulist. > > def replace_pix(frame,mask,new_hdulist): > > data = frame.data > > header = frame.header > > data[mask_nan] = 0.0 > > data[mask_zero] = 0.0 > > data[mask_one] = 1.0 > > new_hdu = fits.ImageHDU(data,header) > > new_hdulist.append(new_hdu) > > > #Function to replace all NaNs in the exposure map by 0s and to replace the > corresponding pixels in the sky and large scale sensitivity map by 0s. > > def replace_mask(filename): > > #Print that all NaNs and Os will be replaced by 0s in the exposure > map and that the corresponding pixels in the sky and large scale > sensitivity map will also be replaced by 0s. > > print "All NaNs and Os will be replaced by 0s in " + filename + " and > all other values will be set to zero." > > #Open the exposure map, the corresponding sky and large scale > sensitivity map and copy the primary headers (extension 0 of hdulist) to > new hdulists. > > hdulist_lss = fits.open(filename) > > new_hdu_header_lss = fits.PrimaryHDU(header=hdulist_lss[0].header) > > new_hdulist_lss = fits.HDUList([new_hdu_header_lss]) > > > > #For all frames in the image: Create the mask and run the function > replace_pix. > > if np.isnan(hdulist_ex[i].data): > > mask_nan > > replace_pix(hdulist_lss[i],mask_nan,new_hdulist_lss) > > elif (hdulist_ex[i].data)=0: > > mask_zero > > replace_pix(hdulist_lss[i],mask_zero,new_hdulist_lss) > > else: > > mask_one > > replace_pix(hdulist_lss[i],mask_one,new_hdulist_lss) > > > #Write the new hdulists to new images. > > new_hdulist_lss.writeto(filename.replace("lss_new.img","lss_mask.img" > )) > > > > #Print that all NaNs are replaced by 0s in the exposure map and that the > corresponding pixels in the sky and large scale sensitivity map are also > replaced by 0s. > > print "All NaNs and 0s are replaced by 0s in " + filename + " and all > other corresponding pixels in the large scale sensitivity map are replaced > by 1s." > > > if __name__ == '__main__': > > main() > > > Essentially, hopefully it is obvious, but it is failing around the if, > elif, else statements (essentially I'm not defining the values correctly). > If anyone would know the correct syntax for this I would be greatly > appreciative. > > > Many thanks, > > > Michael Roberts > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Thu Feb 11 06:39:45 2016 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Thu, 11 Feb 2016 06:39:45 -0500 Subject: [AstroPy] slicing In-Reply-To: <6A5CBBAC-2684-4CA9-94E3-9A2C92E8ADB7@brera.inaf.it> References: <6A5CBBAC-2684-4CA9-94E3-9A2C92E8ADB7@brera.inaf.it> Message-ID: On Thu, Feb 11, 2016 at 5:42 AM, Stefano Covino < stefano.covino at brera.inaf.it> wrote: > Dear friends, > > let?s assume we have a data array read from a FITS file: > > data = getdata(?file.fits?) > > I need to do some computations on a subset of it, for instance: > > datf = dat[bb-size:bb+size,aa-size:aa+size] > > where aa and bb are x,y positions and size a fixed number (e.g. 10). > > > > So, the question. Is it possible that aa and bb are not simple integers > but lists or arrays of integers? > Hi Stefano, You can get a good idea of what is possible with array indexing by starting with the numpy documentation: http://docs.scipy.org/doc/numpy-1.10.1/user/basics.indexing.html - Tom > > > Thanks, > Stefano > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From npkuin at gmail.com Thu Feb 11 06:45:41 2016 From: npkuin at gmail.com (Paul Kuin) Date: Thu, 11 Feb 2016 11:45:41 +0000 Subject: [AstroPy] slicing In-Reply-To: References: <6A5CBBAC-2684-4CA9-94E3-9A2C92E8ADB7@brera.inaf.it> Message-ID: I had this problem and ended up converting the indices to integers by rounding to the nearest (and making sure there we no NaNs ). On Thu, Feb 11, 2016 at 11:39 AM, Aldcroft, Thomas < aldcroft at head.cfa.harvard.edu> wrote: > > > On Thu, Feb 11, 2016 at 5:42 AM, Stefano Covino < > stefano.covino at brera.inaf.it> wrote: > >> Dear friends, >> >> let?s assume we have a data array read from a FITS file: >> >> data = getdata(?file.fits?) >> >> I need to do some computations on a subset of it, for instance: >> >> datf = dat[bb-size:bb+size,aa-size:aa+size] >> >> where aa and bb are x,y positions and size a fixed number (e.g. 10). >> >> >> >> So, the question. Is it possible that aa and bb are not simple integers >> but lists or arrays of integers? >> > > Hi Stefano, > > You can get a good idea of what is possible with array indexing by > starting with the numpy documentation: > > http://docs.scipy.org/doc/numpy-1.10.1/user/basics.indexing.html > > - Tom > > >> >> >> Thanks, >> Stefano >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy >> > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -- * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * * Dr. N.P.M. Kuin (n.kuin at ucl.ac.uk) phone +44-(0)1483 (prefix) -204211 (work) mobile +44(0)7806985366 skype ID: npkuin Mullard Space Science Laboratory ? University College London ? Holmbury St Mary ? Dorking ? Surrey RH5 6NT? U.K. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.covino at brera.inaf.it Thu Feb 11 06:52:38 2016 From: stefano.covino at brera.inaf.it (Stefano Covino) Date: Thu, 11 Feb 2016 12:52:38 +0100 Subject: [AstroPy] slicing In-Reply-To: References: <6A5CBBAC-2684-4CA9-94E3-9A2C92E8ADB7@brera.inaf.it> Message-ID: <94ECBEE3-7FE5-46C6-9169-8D0402607B6B@brera.inaf.it> > I had this problem and ended up converting the indices to integers by rounding to the nearest (and making sure there we no NaNs ). > > Hi Tom, Paul, thanks for your replies. I guess I was quite unclear, or the solution is really simple and I haven?t got it in the documentations. So, please my apologies for that. My problem, however, is how to avoid to loop on the components of the integer arrays aa and bb. Something like: for i,l in zip(aa,bb): datf = dat[l-size:l+size,i-size:i+size] of course works, only it is slow. I just wondered if there is a quicker method. In case I am not looking for a slice, but just at the value at pixel location (aa,bb) something like: datf = dat[bb,aa] works fine, giving me an array of values, as expected. Sorry again if the question is trivial. Bye, Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.roberts.15 at ucl.ac.uk Thu Feb 11 06:58:03 2016 From: michael.roberts.15 at ucl.ac.uk (Roberts, Michael) Date: Thu, 11 Feb 2016 11:58:03 +0000 Subject: [AstroPy] AstroPy Digest, Vol 113, Issue 3 In-Reply-To: References: Message-ID: Hi Daniel, You're quite right, frustratingly I have included some errors in my script. Here is the offending section again: for i in range(1,len(hdulist_lss)): if hdulist_lss[i].data=0: mask_zero replace_pix(hdulist_lss[i],mask_zero,new_hdulist_lss) else: mask_one replace_pix(hdulist_lss[i],mask_one,new_hdulist_lss) Essentially, my syntax is wrong at ^ in: hdulist_lss[i].data^=0 , and I will be honest: I don't know what the syntax should be. Essentially I want to say if a pixel in the image is 0, then data[mask_zero] = 0.0 (I know this seems odd setting all zeroes to equal zero but I required an else statement which then would say for all other values, 10, 34, 12120, 2344, whatever would then be equal to 1.) Hopefully that makes sense, or if you have a better idea I would be more than welcome to hear it, of course. Michael ________________________________________ From: AstroPy on behalf of astropy-request at scipy.org Sent: Thursday, February 11, 2016 11:40 AM To: astropy at scipy.org Subject: AstroPy Digest, Vol 113, Issue 3 Send AstroPy mailing list submissions to astropy at scipy.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.scipy.org/mailman/listinfo/astropy or, via email, send a message with subject or body 'help' to astropy-request at scipy.org You can reach the person managing the list at astropy-owner at scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of AstroPy digest..." Today's Topics: 1. slicing (Stefano Covino) 2. Re: Creating a mask file for .fits file (Daniel Evans) 3. Re: slicing (Aldcroft, Thomas) ---------------------------------------------------------------------- Message: 1 Date: Thu, 11 Feb 2016 11:42:42 +0100 From: Stefano Covino To: Astronomical Python mailing list Subject: [AstroPy] slicing Message-ID: <6A5CBBAC-2684-4CA9-94E3-9A2C92E8ADB7 at brera.inaf.it> Content-Type: text/plain; charset=utf-8 Dear friends, let?s assume we have a data array read from a FITS file: data = getdata(?file.fits?) I need to do some computations on a subset of it, for instance: datf = dat[bb-size:bb+size,aa-size:aa+size] where aa and bb are x,y positions and size a fixed number (e.g. 10). So, the question. Is it possible that aa and bb are not simple integers but lists or arrays of integers? Thanks, Stefano ------------------------------ Message: 2 Date: Thu, 11 Feb 2016 10:43:34 +0000 From: Daniel Evans To: Astronomical Python mailing list Subject: Re: [AstroPy] Creating a mask file for .fits file Message-ID: Content-Type: text/plain; charset="utf-8" I would've thought the definitions for the masks would be: mask_nan = np.isnan(hdulist_ex[i].data) mask_zero = hdulist_ex[i].data == 0.0 mask_one = np.logical_not(np.logical_and(mask_nan, mask_zero)) Which replace your "if" statements in replace_mask(), which aren't actually assigning any indices to the masks as far as I can see. Are hdulist_ex and hdulist_lss are meant to the same variable? I can't find where hdulist_ex is defined. You also need to change the definition of replace_pix(), as you pass it "mask", but the function refers to "mask_nan", "mask_zero", and "mask_one", which it doesn't know about - the simplest method would be to pass all three masks. Regards, Daniel On 11 February 2016 at 10:26, Roberts, Michael wrote: > Dear astropy community, > > > I'm looking to create a mask file for a Swift .img fits format file, which > has one dimensionless primary header and anywhere between 1 and 9 image > headers (with dimensions). > > > I am attempting to create a mask whereby if any value in the image is > either a 0 or a NaN the new mask will have that corresponding pixel as 0, > and for all other values the mask will have a value of 1. > > > My attempted script is: > > > import os > > from astropy.io import fits > > import numpy as np > > > #This is the main function. > > def main(): > > > #For all files in the current directory: > > for filename in sorted(os.listdir(os.getcwd())): > > > > #If the file is not an exposure map, skip this file and continue > with the next file. > > if not filename.endswith("lss_new.img"): continue > > > > #Replace all NaNs and 0s by 0s and all other values with 1 in the > large scale sensitivity maps. > > replace_mask(filename) > > > #Function to replace all masked pixels by 0s. > > #Open the data and replace all masked pixels by 0s and all other values > with 1. Save the header and the new data of the frame to a new hdu. Append > this hdu to the new hdulist. > > def replace_pix(frame,mask,new_hdulist): > > data = frame.data > > header = frame.header > > data[mask_nan] = 0.0 > > data[mask_zero] = 0.0 > > data[mask_one] = 1.0 > > new_hdu = fits.ImageHDU(data,header) > > new_hdulist.append(new_hdu) > > > #Function to replace all NaNs in the exposure map by 0s and to replace the > corresponding pixels in the sky and large scale sensitivity map by 0s. > > def replace_mask(filename): > > #Print that all NaNs and Os will be replaced by 0s in the exposure > map and that the corresponding pixels in the sky and large scale > sensitivity map will also be replaced by 0s. > > print "All NaNs and Os will be replaced by 0s in " + filename + " and > all other values will be set to zero." > > #Open the exposure map, the corresponding sky and large scale > sensitivity map and copy the primary headers (extension 0 of hdulist) to > new hdulists. > > hdulist_lss = fits.open(filename) > > new_hdu_header_lss = fits.PrimaryHDU(header=hdulist_lss[0].header) > > new_hdulist_lss = fits.HDUList([new_hdu_header_lss]) > > > > #For all frames in the image: Create the mask and run the function > replace_pix. > > if np.isnan(hdulist_ex[i].data): > > mask_nan > > replace_pix(hdulist_lss[i],mask_nan,new_hdulist_lss) > > elif (hdulist_ex[i].data)=0: > > mask_zero > > replace_pix(hdulist_lss[i],mask_zero,new_hdulist_lss) > > else: > > mask_one > > replace_pix(hdulist_lss[i],mask_one,new_hdulist_lss) > > > #Write the new hdulists to new images. > > new_hdulist_lss.writeto(filename.replace("lss_new.img","lss_mask.img" > )) > > > > #Print that all NaNs are replaced by 0s in the exposure map and that the > corresponding pixels in the sky and large scale sensitivity map are also > replaced by 0s. > > print "All NaNs and 0s are replaced by 0s in " + filename + " and all > other corresponding pixels in the large scale sensitivity map are replaced > by 1s." > > > if __name__ == '__main__': > > main() > > > Essentially, hopefully it is obvious, but it is failing around the if, > elif, else statements (essentially I'm not defining the values correctly). > If anyone would know the correct syntax for this I would be greatly > appreciative. > > > Many thanks, > > > Michael Roberts > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Thu, 11 Feb 2016 06:39:45 -0500 From: "Aldcroft, Thomas" To: Astronomical Python mailing list Subject: Re: [AstroPy] slicing Message-ID: Content-Type: text/plain; charset="utf-8" On Thu, Feb 11, 2016 at 5:42 AM, Stefano Covino < stefano.covino at brera.inaf.it> wrote: > Dear friends, > > let?s assume we have a data array read from a FITS file: > > data = getdata(?file.fits?) > > I need to do some computations on a subset of it, for instance: > > datf = dat[bb-size:bb+size,aa-size:aa+size] > > where aa and bb are x,y positions and size a fixed number (e.g. 10). > > > > So, the question. Is it possible that aa and bb are not simple integers > but lists or arrays of integers? > Hi Stefano, You can get a good idea of what is possible with array indexing by starting with the numpy documentation: http://docs.scipy.org/doc/numpy-1.10.1/user/basics.indexing.html - Tom > > > Thanks, > Stefano > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy ------------------------------ End of AstroPy Digest, Vol 113, Issue 3 *************************************** From michael.roberts.15 at ucl.ac.uk Sun Feb 14 07:47:59 2016 From: michael.roberts.15 at ucl.ac.uk (Roberts, Michael) Date: Sun, 14 Feb 2016 12:47:59 +0000 Subject: [AstroPy] Equivalent expression for SXPAR from IDL in python Message-ID: I'm hoping this question is relatively straightforward for those proficient with IDL and python and fits file handling (I'm using the astropy.fits module) I'm looking to find an equivalent to SXPAR (whose function is to obtain the value of a parameter in a FITS header) from IDL into python... http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro "sxpar.pro" - idlastro.gsfc.nasa.gov idlastro.gsfc.nasa.gov Apostrophes are stripped ; from strings. If the parameter is logical, 1b is ; returned for T, and 0b is returned for F. ; If Name was of ... This is the original code in IDL: nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') This is my conversion attempt into python using astropy.fits: from astropy.io import fits hima_sk_um2 = fits.open('') #Will contain the directory extension to the fits/img file I wish to open # Compute the size of the images (you can also do this manually rather than calling these keywords from the header): nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] Firstly, is this correct? And secondly, is this the best way to handle an extension value? Any advise would be warmly received. Many thanks, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevin.gullikson at gmail.com Sun Feb 14 09:37:12 2016 From: kevin.gullikson at gmail.com (Kevin Gullikson) Date: Sun, 14 Feb 2016 14:37:12 +0000 Subject: [AstroPy] Equivalent expression for SXPAR from IDL in python In-Reply-To: References: Message-ID: If all you want is the header, you can call header = fits.getheader(filename, extension_number) That is a bit simpler, and might be a bit faster too (not sure). On Sun, Feb 14, 2016 at 6:48 AM Roberts, Michael < michael.roberts.15 at ucl.ac.uk> wrote: > I'm hoping this question is relatively straightforward for those > proficient with IDL and python and fits file handling (I'm using > the astropy.fits module) > > > I'm looking to find an equivalent to SXPAR (whose function is to obtain > the value of a parameter in a FITS header) from IDL into python... > > > http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro > "sxpar.pro" - idlastro.gsfc.nasa.gov > > idlastro.gsfc.nasa.gov > Apostrophes are stripped ; from strings. If the parameter is logical, 1b > is ; returned for T, and 0b is returned for F. ; If Name was of ... > > This is the original code in IDL: > > nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') > > nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') > > This is my conversion attempt into python using astropy.fits: > > from astropy.io import fits > > > hima_sk_um2 = fits.open('') #Will contain the directory extension to the > fits/img file I wish to open > > > # Compute the size of the images (you can also do this manually rather > than calling these keywords from the header): > > nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] > > nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] > > Firstly, is this correct? And secondly, is this the best way to handle an > extension value? > > Any advise would be warmly received. > > Many thanks, > > Michael > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -- Kevin Gullikson PhD Candidate University of Texas Astronomy -------------- next part -------------- An HTML attachment was scrubbed... URL: From pebarrett at gmail.com Sun Feb 14 11:08:22 2016 From: pebarrett at gmail.com (Paul Barrett) Date: Sun, 14 Feb 2016 11:08:22 -0500 Subject: [AstroPy] Equivalent expression for SXPAR from IDL in python In-Reply-To: References: Message-ID: astropy.io.fits.open(filename)[0]['NAXIS1'] will give you the value of the NAXSI1 keyword in the primary header. In the Python FITS library, a FITS file is a list of headers and each header is a list of rows. The value of a keyword can be obtained by specifying the header name or index, and the keyword name as show above. On Sun, Feb 14, 2016 at 7:47 AM, Roberts, Michael < michael.roberts.15 at ucl.ac.uk> wrote: > I'm hoping this question is relatively straightforward for those > proficient with IDL and python and fits file handling (I'm using > the astropy.fits module) > > > I'm looking to find an equivalent to SXPAR (whose function is to obtain > the value of a parameter in a FITS header) from IDL into python... > > > http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro > "sxpar.pro" - idlastro.gsfc.nasa.gov > > idlastro.gsfc.nasa.gov > Apostrophes are stripped ; from strings. If the parameter is logical, 1b > is ; returned for T, and 0b is returned for F. ; If Name was of ... > > This is the original code in IDL: > > nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') > > nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') > > This is my conversion attempt into python using astropy.fits: > > from astropy.io import fits > > > hima_sk_um2 = fits.open('') #Will contain the directory extension to the > fits/img file I wish to open > > > # Compute the size of the images (you can also do this manually rather > than calling these keywords from the header): > > nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] > > nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] > > Firstly, is this correct? And secondly, is this the best way to handle an > extension value? > > Any advise would be warmly received. > > Many thanks, > > Michael > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.roberts.15 at ucl.ac.uk Sun Feb 14 11:12:52 2016 From: michael.roberts.15 at ucl.ac.uk (Roberts, Michael) Date: Sun, 14 Feb 2016 16:12:52 +0000 Subject: [AstroPy] AstroPy Digest, Vol 113, Issue 5 In-Reply-To: References: Message-ID: That's excellent, that is similar to what I have done. But works. So how would I go about creating a new image file with the same image dimesions (which are given by NAXIS1 x-axis and NAXIS2 y-axis)? Many thanks, Michael ________________________________________ From: AstroPy on behalf of astropy-request at scipy.org Sent: Sunday, February 14, 2016 4:08 PM To: astropy at scipy.org Subject: AstroPy Digest, Vol 113, Issue 5 Send AstroPy mailing list submissions to astropy at scipy.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.scipy.org/mailman/listinfo/astropy or, via email, send a message with subject or body 'help' to astropy-request at scipy.org You can reach the person managing the list at astropy-owner at scipy.org When replying, please edit your Subject line so it is more specific than "Re: Contents of AstroPy digest..." Today's Topics: 1. Equivalent expression for SXPAR from IDL in python (Roberts, Michael) 2. Re: Equivalent expression for SXPAR from IDL in python (Kevin Gullikson) 3. Re: Equivalent expression for SXPAR from IDL in python (Paul Barrett) ---------------------------------------------------------------------- Message: 1 Date: Sun, 14 Feb 2016 12:47:59 +0000 From: "Roberts, Michael" To: "astropy at scipy.org" Subject: [AstroPy] Equivalent expression for SXPAR from IDL in python Message-ID: Content-Type: text/plain; charset="iso-8859-1" I'm hoping this question is relatively straightforward for those proficient with IDL and python and fits file handling (I'm using the astropy.fits module) I'm looking to find an equivalent to SXPAR (whose function is to obtain the value of a parameter in a FITS header) from IDL into python... http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro "sxpar.pro" - idlastro.gsfc.nasa.gov idlastro.gsfc.nasa.gov Apostrophes are stripped ; from strings. If the parameter is logical, 1b is ; returned for T, and 0b is returned for F. ; If Name was of ... This is the original code in IDL: nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') This is my conversion attempt into python using astropy.fits: from astropy.io import fits hima_sk_um2 = fits.open('') #Will contain the directory extension to the fits/img file I wish to open # Compute the size of the images (you can also do this manually rather than calling these keywords from the header): nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] Firstly, is this correct? And secondly, is this the best way to handle an extension value? Any advise would be warmly received. Many thanks, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Sun, 14 Feb 2016 14:37:12 +0000 From: Kevin Gullikson To: Astronomical Python mailing list Subject: Re: [AstroPy] Equivalent expression for SXPAR from IDL in python Message-ID: Content-Type: text/plain; charset="utf-8" If all you want is the header, you can call header = fits.getheader(filename, extension_number) That is a bit simpler, and might be a bit faster too (not sure). On Sun, Feb 14, 2016 at 6:48 AM Roberts, Michael < michael.roberts.15 at ucl.ac.uk> wrote: > I'm hoping this question is relatively straightforward for those > proficient with IDL and python and fits file handling (I'm using > the astropy.fits module) > > > I'm looking to find an equivalent to SXPAR (whose function is to obtain > the value of a parameter in a FITS header) from IDL into python... > > > http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro > "sxpar.pro" - idlastro.gsfc.nasa.gov > > idlastro.gsfc.nasa.gov > Apostrophes are stripped ; from strings. If the parameter is logical, 1b > is ; returned for T, and 0b is returned for F. ; If Name was of ... > > This is the original code in IDL: > > nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') > > nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') > > This is my conversion attempt into python using astropy.fits: > > from astropy.io import fits > > > hima_sk_um2 = fits.open('') #Will contain the directory extension to the > fits/img file I wish to open > > > # Compute the size of the images (you can also do this manually rather > than calling these keywords from the header): > > nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] > > nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] > > Firstly, is this correct? And secondly, is this the best way to handle an > extension value? > > Any advise would be warmly received. > > Many thanks, > > Michael > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -- Kevin Gullikson PhD Candidate University of Texas Astronomy -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 3 Date: Sun, 14 Feb 2016 11:08:22 -0500 From: Paul Barrett To: Astronomical Python mailing list Subject: Re: [AstroPy] Equivalent expression for SXPAR from IDL in python Message-ID: Content-Type: text/plain; charset="utf-8" astropy.io.fits.open(filename)[0]['NAXIS1'] will give you the value of the NAXSI1 keyword in the primary header. In the Python FITS library, a FITS file is a list of headers and each header is a list of rows. The value of a keyword can be obtained by specifying the header name or index, and the keyword name as show above. On Sun, Feb 14, 2016 at 7:47 AM, Roberts, Michael < michael.roberts.15 at ucl.ac.uk> wrote: > I'm hoping this question is relatively straightforward for those > proficient with IDL and python and fits file handling (I'm using > the astropy.fits module) > > > I'm looking to find an equivalent to SXPAR (whose function is to obtain > the value of a parameter in a FITS header) from IDL into python... > > > http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro > "sxpar.pro" - idlastro.gsfc.nasa.gov > > idlastro.gsfc.nasa.gov > Apostrophes are stripped ; from strings. If the parameter is logical, 1b > is ; returned for T, and 0b is returned for F. ; If Name was of ... > > This is the original code in IDL: > > nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') > > nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') > > This is my conversion attempt into python using astropy.fits: > > from astropy.io import fits > > > hima_sk_um2 = fits.open('') #Will contain the directory extension to the > fits/img file I wish to open > > > # Compute the size of the images (you can also do this manually rather > than calling these keywords from the header): > > nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] > > nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] > > Firstly, is this correct? And secondly, is this the best way to handle an > extension value? > > Any advise would be warmly received. > > Many thanks, > > Michael > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy ------------------------------ End of AstroPy Digest, Vol 113, Issue 5 *************************************** From evert.rol at gmail.com Sun Feb 14 11:33:34 2016 From: evert.rol at gmail.com (Evert Rol) Date: Mon, 15 Feb 2016 03:33:34 +1100 Subject: [AstroPy] AstroPy Digest, Vol 113, Issue 5 In-Reply-To: References: Message-ID: > So how would I go about creating a new image file with the same image dimesions (which are given by NAXIS1 x-axis and NAXIS2 y-axis)? You may not want to use the naxis1/2 keywords directly, but instead use the image shape of the original image: newdata = np.empty_like( hima_sk_um2[1].data) hdu = ImageHDU(newdata) # or PrimaryHDU(newdata) HDUList([hdu]).writeto('newfile.fits') The np._like functions (ones_like, zeros_like, empty_like) take an existing array and create a similar one (same dimensions and type) with ones, zeros or empty (random) values. The header is implicitly created in the call to ImageHDU or PrimaryHDU, and will have the right naxis etc information. > I'm hoping this question is relatively straightforward for those proficient with IDL and python and fits file handling (I'm using the astropy.fits module) > > > I'm looking to find an equivalent to SXPAR (whose function is to obtain the value of a parameter in a FITS header) from IDL into python... > > > http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro > > "sxpar.pro" - idlastro.gsfc.nasa.gov > idlastro.gsfc.nasa.gov > Apostrophes are stripped ; from strings. If the parameter is logical, 1b is ; returned for T, and 0b is returned for F. ; If Name was of ... > > > This is the original code in IDL: > > nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') > > nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') > > This is my conversion attempt into python using astropy.fits: > > from astropy.io import fits > > > hima_sk_um2 = fits.open('') #Will contain the directory extension to the fits/img file I wish to open > > > # Compute the size of the images (you can also do this manually rather than calling these keywords from the header): > > nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] > > nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] > > Firstly, is this correct? And secondly, is this the best way to handle an extension value? > > Any advise would be warmly received. > > Many thanks, > > Michael > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 2 > Date: Sun, 14 Feb 2016 14:37:12 +0000 > From: Kevin Gullikson > To: Astronomical Python mailing list > Subject: Re: [AstroPy] Equivalent expression for SXPAR from IDL in > python > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > If all you want is the header, you can call > > header = fits.getheader(filename, extension_number) > > That is a bit simpler, and might be a bit faster too (not sure). > > On Sun, Feb 14, 2016 at 6:48 AM Roberts, Michael < > michael.roberts.15 at ucl.ac.uk> wrote: > >> I'm hoping this question is relatively straightforward for those >> proficient with IDL and python and fits file handling (I'm using >> the astropy.fits module) >> >> >> I'm looking to find an equivalent to SXPAR (whose function is to obtain >> the value of a parameter in a FITS header) from IDL into python... >> >> >> http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro >> "sxpar.pro" - idlastro.gsfc.nasa.gov >> >> idlastro.gsfc.nasa.gov >> Apostrophes are stripped ; from strings. If the parameter is logical, 1b >> is ; returned for T, and 0b is returned for F. ; If Name was of ... >> >> This is the original code in IDL: >> >> nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') >> >> nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') >> >> This is my conversion attempt into python using astropy.fits: >> >> from astropy.io import fits >> >> >> hima_sk_um2 = fits.open('') #Will contain the directory extension to the >> fits/img file I wish to open >> >> >> # Compute the size of the images (you can also do this manually rather >> than calling these keywords from the header): >> >> nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] >> >> nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] >> >> Firstly, is this correct? And secondly, is this the best way to handle an >> extension value? >> >> Any advise would be warmly received. >> >> Many thanks, >> >> Michael >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy >> > -- > Kevin Gullikson > PhD Candidate > University of Texas Astronomy > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 3 > Date: Sun, 14 Feb 2016 11:08:22 -0500 > From: Paul Barrett > To: Astronomical Python mailing list > Subject: Re: [AstroPy] Equivalent expression for SXPAR from IDL in > python > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > astropy.io.fits.open(filename)[0]['NAXIS1'] will give you the value of the > NAXSI1 keyword in the primary header. > > In the Python FITS library, a FITS file is a list of headers and each > header is a list of rows. The value of a keyword can be obtained by > specifying the header name or index, and the keyword name as show above. > > > On Sun, Feb 14, 2016 at 7:47 AM, Roberts, Michael < > michael.roberts.15 at ucl.ac.uk> wrote: > >> I'm hoping this question is relatively straightforward for those >> proficient with IDL and python and fits file handling (I'm using >> the astropy.fits module) >> >> >> I'm looking to find an equivalent to SXPAR (whose function is to obtain >> the value of a parameter in a FITS header) from IDL into python... >> >> >> http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro >> "sxpar.pro" - idlastro.gsfc.nasa.gov >> >> idlastro.gsfc.nasa.gov >> Apostrophes are stripped ; from strings. If the parameter is logical, 1b >> is ; returned for T, and 0b is returned for F. ; If Name was of ... >> >> This is the original code in IDL: >> >> nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') >> >> nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') >> >> This is my conversion attempt into python using astropy.fits: >> >> from astropy.io import fits >> >> >> hima_sk_um2 = fits.open('') #Will contain the directory extension to the >> fits/img file I wish to open >> >> >> # Compute the size of the images (you can also do this manually rather >> than calling these keywords from the header): >> >> nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] >> >> nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] >> >> Firstly, is this correct? And secondly, is this the best way to handle an >> extension value? >> >> Any advise would be warmly received. >> >> Many thanks, >> >> Michael >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy >> >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > > ------------------------------ > > End of AstroPy Digest, Vol 113, Issue 5 > *************************************** > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy From michael.roberts.15 at ucl.ac.uk Sun Feb 14 11:57:47 2016 From: michael.roberts.15 at ucl.ac.uk (Roberts, Michael) Date: Sun, 14 Feb 2016 16:57:47 +0000 Subject: [AstroPy] Creating a new Image file in astropy with specified dimensions of another image file and then applying corrections Message-ID: I'm having a problem wit fits file manipulation in the astropy package, and I'm in need of some help. I'm trying to convert an old IDL script into python. I essentially want to take an image I have in fits file format, and create a new file I need to start inputing correction factors to and a new image which can then be used with the correction factors and the original image to produce a correction image. Each of these will have the same dimensions. This is where I am at thus far: import numpy as numpy from astropy.io import fits #URL: to M33_sum_epoch1_um2_norm.img : https://www.dropbox.com/s/q7e14uosaj5u9en/M33_sum_epoch1_um2_norm.img?dl=0 ima_sk_um2 = fits.open('/Users/UCL_Astronomy/Documents/UCL/PHASG199/M33_UVOT_sum/UVOTIMSUM/M33_sum_epoch1_um2_norm.img') # Compute the size of the images (you can also do this manually rather than calling these keywords from the header): nxpix_um2_ext1 = ima_sk_um2[1].data['NAXIS1'] # IDL: nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') nypix_um2_ext1 = ima_sk_um2[1].data['NAXIS2'] # IDL: nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') # Make a new image file to save the correction factors: corrfact_um2_ext1 = numpy.array(nxpix_um2_ext1,nypix_um2_ext1) hdu_corrfact = ImageHDU(corrfact_um2_ext1) HDUList([hdu_corrfact]).writeto('/Users/UCL_Astronomy/Documents/UCL/PHASG199/M33_UVOT_sum/UVOTIMSUM/M33_sum_epoch1_um2_corrfact.img') # Make a new image file to save the corrected image: ima_sk_coincorr_um2_ext1 = numpy.array(nxpix_um2_ext1,nypix_um2_ext1) hdu_coincorr = ImageHDU(ima_sk_coincorr_um2_ext1) HDUList([hdu_coincorr]).writeto('/Users/UCL_Astronomy/Documents/UCL/PHASG199/M33_UVOT_sum/UVOTIMSUM/M33_sum_epoch1_um2_coincorr.img') # Define the variables from Poole et al. (2008) "Photometric calibration of the Swift ultraviolet/optical telescope": alpha = 0.9842000 ft = 0.0110329 a1 = 0.0658568 a2 = -0.0907142 a3 = 0.0285951 a4 = 0.0308063 for i in range(nxpix_um2_ext1-1): #do begin for j in range(nypix_um2_ext1-1): #do begin if (numpy.less_equal(i, 4) | numpy.greater_equal(i, nxpix_um2_ext1-4) | numpy.less_equal(j, 4) | numpy.greater_equal(j, nxpix_um2_ext1-4)): #then begin #UVM2 corrfact_um2_ext1[i,j] == 0 ima_sk_coincorr_um2_ext1[i,j] == 0 else: xpixmin = i-4 xpixmax = i+4 ypixmin = j-4 ypixmax = j+4 #UVM2 ima_UVM2sum = total(ima_sk_um2[xpixmin:xpixmax,ypixmin:ypixmax]) xvec_UVM2 = ft*ima_UVM2sum fxvec_UVM2 = 1 + (a1*xvec_UVM2) + (a2*xvec_UVM2*xvec_UVM2) + (a3*xvec_UVM2*xvec_UVM2*xvec_UVM2) + (a4*xvec_UVM2*xvec_UVM2*xvec_UVM2*xvec_UVM2) Ctheory_UVM2 = - alog(1-(alpha*ima_UVM2sum*ft))/(alpha*ft) corrfact_um2_ext1[i,j] = Ctheory_UVM2*(fxvec_UVM2/ima_UVM2sum) ima_sk_coincorr_um2_ext1[i,j] = corrfact_um2_ext1[i,j]*ima_sk_um2[i,j] This is essentially a lot of muddling between IDL, python and astropy. In essence I want to have a new .img file that has the same dimensions as the original file but with the correction factors calculated in the loop as the new image array, and then the coincidence correction as the two multiplied together, so the correction factor times the image input. A dropbox link to the image file is in the script, so hopefully that should be self-containing. Hopefully there is someone here that knows how to guide me on this. Many thanks, Michael Roberts -------------- next part -------------- An HTML attachment was scrubbed... URL: From martinberoiz at gmail.com Mon Feb 15 10:00:18 2016 From: martinberoiz at gmail.com (Martin Beroiz) Date: Mon, 15 Feb 2016 09:00:18 -0600 Subject: [AstroPy] Equivalent expression for SXPAR from IDL in python In-Reply-To: References: Message-ID: Hello, This is how I get a single value of the header from astropy.io import fits naxis1 = fits.getval(?my filename.fits?, ?NAXIS1?) M. > On Feb 14, 2016, at 6:47 AM, Roberts, Michael wrote: > > I'm hoping this question is relatively straightforward for those proficient with IDL and python and fits file handling (I'm using the astropy.fits module) > > I'm looking to find an equivalent to SXPAR (whose function is to obtain the value of a parameter in a FITS header) from IDL into python... > > http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro > "sxpar.pro" - idlastro.gsfc.nasa.gov > idlastro.gsfc.nasa.gov > Apostrophes are stripped ; from strings. If the parameter is logical, 1b is ; returned for T, and 0b is returned for F. ; If Name was of ... > This is the original code in IDL: > nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') > nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') > This is my conversion attempt into python using astropy.fits: > from astropy.io import fits > > hima_sk_um2 = fits.open('') #Will contain the directory extension to the fits/img file I wish to open > > # Compute the size of the images (you can also do this manually rather than calling these keywords from the header): > nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] > nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] > Firstly, is this correct? And secondly, is this the best way to handle an extension value? > Any advise would be warmly received. > Many thanks, > Michael > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From bushouse at stsci.edu Mon Feb 15 10:14:46 2016 From: bushouse at stsci.edu (Howard Bushouse) Date: Mon, 15 Feb 2016 15:14:46 +0000 Subject: [AstroPy] Equivalent expression for SXPAR from IDL in python In-Reply-To: References: Message-ID: getval is a convenient shortcut for not having to separately open the FITS file or load the entire header and then search for a particular keyword. The default example below will retrieve a keyword from the primary header only. To get a value from a particular extension header use options like: fits.getval(?my filename.fits?, ?NAXIS?, 2) or fits.getval(?my filename.fits?, ?NAXIS?, ext=2) or fits.getval(?my filename.fits?, ?NAXIS?, extname=?sci?, extver=2) or fits.getval(?my filename.fits?, ?NAXIS?, (?sci?, 2)) -Howard On Feb 15, 2016, at 10:00 AM, Martin Beroiz > wrote: Hello, This is how I get a single value of the header from astropy.io import fits naxis1 = fits.getval(?my filename.fits?, ?NAXIS1?) M. On Feb 14, 2016, at 6:47 AM, Roberts, Michael > wrote: I'm hoping this question is relatively straightforward for those proficient with IDL and python and fits file handling (I'm using the astropy.fits module) I'm looking to find an equivalent to SXPAR (whose function is to obtain the value of a parameter in a FITS header) from IDL into python... http://idlastro.gsfc.nasa.gov/ftp/pro/fits/sxpar.pro "sxpar.pro" - idlastro.gsfc.nasa.gov idlastro.gsfc.nasa.gov Apostrophes are stripped ; from strings. If the parameter is logical, 1b is ; returned for T, and 0b is returned for F. ; If Name was of ... This is the original code in IDL: nxpix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS1') nypix_uw1_ext1 = sxpar(hima_sk_uw1_ext1,'NAXIS2') This is my conversion attempt into python using astropy.fits: from astropy.io import fits hima_sk_um2 = fits.open('') #Will contain the directory extension to the fits/img file I wish to open # Compute the size of the images (you can also do this manually rather than calling these keywords from the header): nxpix_um2_ext1 = hima_sk_um2[1].header['NAXIS1'] nypix_um2_ext1 = hima_sk_um2[1].header['NAXIS2'] Firstly, is this correct? And secondly, is this the best way to handle an extension value? Any advise would be warmly received. Many thanks, Michael _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From xaviercouvelard at gmail.com Mon Feb 15 13:29:34 2016 From: xaviercouvelard at gmail.com (Xavier Couvelard) Date: Mon, 15 Feb 2016 19:29:34 +0100 Subject: [AstroPy] [Astropy] python install prb. Message-ID: <56C2190E.1050101@gmail.com> Hello, I went through the tutorial https://python4astronomers.github.io/installation/python_install.html to install python, it went well, but at the end i have problem with matplotlib. In [7]: print(matplotlib.__version__) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 print(matplotlib.__version__) NameError: name 'matplotlib' is not defined If i follow anyway: In [9]: plt.plot(x, sin(x)) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 plt.plot(x, sin(x)) Does any one can help ? Thanks Xavier From trive at astro.su.se Mon Feb 15 16:08:21 2016 From: trive at astro.su.se (=?UTF-8?Q?Th=c3=b8ger_Emil_Rivera-Thorsen?=) Date: Mon, 15 Feb 2016 22:08:21 +0100 Subject: [AstroPy] [Astropy] python install prb. In-Reply-To: <56C2190E.1050101@gmail.com> References: <56C2190E.1050101@gmail.com> Message-ID: <56C23E45.6060507@astro.su.se> This means Matplotlib is not installed or not found by your Python installation. Can you give some more detailed information about which steps you followed, in which operatng system, etc.? On 02/15/2016 07:29 PM, Xavier Couvelard wrote: > Hello, > > I went through the tutorial > https://python4astronomers.github.io/installation/python_install.html > to install python, it went well, but at the end i have problem with > matplotlib. > > > In [7]: print(matplotlib.__version__) > --------------------------------------------------------------------------- > > NameError Traceback (most recent call > last) > in () > ----> 1 print(matplotlib.__version__) > > NameError: name 'matplotlib' is not defined > > If i follow anyway: > > In [9]: plt.plot(x, sin(x)) > --------------------------------------------------------------------------- > > NameError Traceback (most recent call > last) > in () > ----> 1 plt.plot(x, sin(x)) > > Does any one can help ? > > Thanks > > Xavier > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy From xaviercouvelard at gmail.com Mon Feb 15 18:00:51 2016 From: xaviercouvelard at gmail.com (Xavier Couvelard) Date: Tue, 16 Feb 2016 00:00:51 +0100 Subject: [AstroPy] [Astropy] python install prb. In-Reply-To: <56C23E45.6060507@astro.su.se> References: <56C2190E.1050101@gmail.com> <56C23E45.6060507@astro.su.se> Message-ID: <56C258A3.5070801@gmail.com> Hi , i kind of got that, but i have no clue of how i should install it. I am using an ubuntu 14.04 on 64b laptop. i have python2.7 on it. I followed all the steps of this webpage. I installed Anaconda2-2.5.0-Linux-x86_64.sh and then did: pip install --upgrade aplpy pip install --upgrade pyregion pip install --upgrade pyparsing i had many warning but all went well and finish with success messages. So i went for the recommended test; $ python -V Python 2.7.11 :: Anaconda 2.5.0 (64-bit) $ ipython -V 4.0.3 $ ipython --matplotlib Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec 6 2015, 18:08:32) Type "copyright", "credits" or "license" for more information. IPython 4.0.3 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. Using matplotlib backend: Qt4Agg import numpy import scipy import scipy.linalg import matplotlib.pyplot as plt In [5]: print(numpy.__version__) 1.10.4 In [6]: print(scipy.__version__) 0.17.0 In [7]: print(matplotlib.__version__) --------------------------------------------------------------------------- Traceback (most recent call last) in () ----> 1 print(matplotlib.__version__) NameError: name 'matplotlib' is not defined that's almost all what i have done up to now.... hope that will help you understand my problems. Regards Le 15/02/2016 22:08, Th?ger Emil Rivera-Thorsen a ?crit : > This means Matplotlib is not installed or not found by your Python > installation. > > Can you give some more detailed information about which steps you > followed, in which operatng system, etc.? > > > > On 02/15/2016 07:29 PM, Xavier Couvelard wrote: >> Hello, >> >> I went through the tutorial >> https://python4astronomers.github.io/installation/python_install.html >> to install python, it went well, but at the end i have problem with >> matplotlib. >> >> >> In [7]: print(matplotlib.__version__) >> --------------------------------------------------------------------------- >> >> NameError Traceback (most recent call >> last) >> in () >> ----> 1 print(matplotlib.__version__) >> >> NameError: name 'matplotlib' is not defined >> >> If i follow anyway: >> >> In [9]: plt.plot(x, sin(x)) >> --------------------------------------------------------------------------- >> >> NameError Traceback (most recent call >> last) >> in () >> ----> 1 plt.plot(x, sin(x)) >> >> Does any one can help ? >> >> Thanks >> >> Xavier >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy From mcraig at mnstate.edu Mon Feb 15 18:10:52 2016 From: mcraig at mnstate.edu (Matthew Craig) Date: Mon, 15 Feb 2016 23:10:52 +0000 Subject: [AstroPy] [Astropy] python install prb. In-Reply-To: <56C258A3.5070801@gmail.com> References: <56C2190E.1050101@gmail.com> <56C23E45.6060507@astro.su.se> <56C258A3.5070801@gmail.com> Message-ID: <8120D7F6-8189-4D49-A0FA-0FEAF30823F8@mnstate.edu> Hi Xavier, Thanks for the details, makes it much easier to track down the issue?which is an error in python4astronomers, not in what you are doing. This error: In [7]: print(matplotlib.__version__) --------------------------------------------------------------------------- Traceback (most recent call last) in () ----> 1 print(matplotlib.__version__) is coming because you imported matplotlib.pyplot but you didn?t import matplotlib itself. If you add to your list of imports this: import matplotlib then you should be able to print the version. I?ve opened an issue for this at https://github.com/python4astronomers/python4astronomers/issues Matt Craig schedule: http://physics.mnstate.edu/craig ?? Professor Department of Physics and Astronomy Minnesota State University Moorhead 1104 7th Ave S, Moorhead MN 56563 office: Hagen 307F phone: (218) 477-2439 fax: (218) 477-2290 On Feb 15, 2016, at 5:00 PM, Xavier Couvelard > wrote: Hi , i kind of got that, but i have no clue of how i should install it. I am using an ubuntu 14.04 on 64b laptop. i have python2.7 on it. I followed all the steps of this webpage. I installed Anaconda2-2.5.0-Linux-x86_64.sh and then did: pip install --upgrade aplpy pip install --upgrade pyregion pip install --upgrade pyparsing i had many warning but all went well and finish with success messages. So i went for the recommended test; $ python -V Python 2.7.11 :: Anaconda 2.5.0 (64-bit) $ ipython -V 4.0.3 $ ipython --matplotlib Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec 6 2015, 18:08:32) Type "copyright", "credits" or "license" for more information. IPython 4.0.3 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. Using matplotlib backend: Qt4Agg import numpy import scipy import scipy.linalg import matplotlib.pyplot as plt In [5]: print(numpy.__version__) 1.10.4 In [6]: print(scipy.__version__) 0.17.0 In [7]: print(matplotlib.__version__) --------------------------------------------------------------------------- Traceback (most recent call last) in () ----> 1 print(matplotlib.__version__) NameError: name 'matplotlib' is not defined that's almost all what i have done up to now.... hope that will help you understand my problems. Regards Le 15/02/2016 22:08, Th?ger Emil Rivera-Thorsen a ?crit : This means Matplotlib is not installed or not found by your Python installation. Can you give some more detailed information about which steps you followed, in which operatng system, etc.? On 02/15/2016 07:29 PM, Xavier Couvelard wrote: Hello, I went through the tutorial https://python4astronomers.github.io/installation/python_install.html to install python, it went well, but at the end i have problem with matplotlib. In [7]: print(matplotlib.__version__) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 print(matplotlib.__version__) NameError: name 'matplotlib' is not defined If i follow anyway: In [9]: plt.plot(x, sin(x)) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 plt.plot(x, sin(x)) Does any one can help ? Thanks Xavier _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From derek at astro.physik.uni-goettingen.de Mon Feb 15 18:12:58 2016 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 16 Feb 2016 00:12:58 +0100 Subject: [AstroPy] [Astropy] python install prb. In-Reply-To: <56C258A3.5070801@gmail.com> References: <56C2190E.1050101@gmail.com> <56C23E45.6060507@astro.su.se> <56C258A3.5070801@gmail.com> Message-ID: <8559DFD3-F1AE-4FEC-ABE8-47E80035F815@astro.physik.uni-goettingen.de> > On 16 Feb 2016, at 12:00 am, Xavier Couvelard wrote: > > $ ipython -V > 4.0.3 > $ ipython --matplotlib > Python 2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec 6 2015, 18:08:32) > Type "copyright", "credits" or "license" for more information. > > IPython 4.0.3 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object', use 'object??' for extra details. > Using matplotlib backend: Qt4Agg > > import numpy > import scipy > import scipy.linalg > import matplotlib.pyplot as plt > > > In [5]: print(numpy.__version__) > 1.10.4 > In [6]: print(scipy.__version__) > 0.17.0 > In [7]: print(matplotlib.__version__) > --------------------------------------------------------------------------- > Traceback (most recent call last) > in () > ----> 1 print(matplotlib.__version__) > > NameError: name 'matplotlib' is not defined > And you are not getting an error on ?import matplotlib.pyplot as plt?? Seems then like that tutorial is simply missing a line import matplotlib If you wanted everything imported automatically, you?d have to start up ipython ?pylab but I don?t know what philosophy the above tutorial is following, so perhaps best stick with it for now and just at that import statement. HTH Derek From ifriad at gmail.com Thu Feb 18 13:47:47 2016 From: ifriad at gmail.com (ihab riad) Date: Thu, 18 Feb 2016 21:47:47 +0300 Subject: [AstroPy] installation error Message-ID: Hi, I am trying to install astropy and I am getting the following error. ImportError: cannot import name version_info I found some information saying that I need to del ~/.astropy but I went looking for it and I couldn't locate this directory. Please help. Cheers Ihab -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.ray at nrl.navy.mil Fri Feb 19 08:51:46 2016 From: paul.ray at nrl.navy.mil (Paul Ray) Date: Fri, 19 Feb 2016 08:51:46 -0500 Subject: [AstroPy] Time custom formats In-Reply-To: References: Message-ID: <699E05F6-DE20-4D9E-92D9-54C7B6BE6F0E@nrl.navy.mil> Hi, I've been trying to figure out how to print an astropy.Time with a custom string format and have not been able to. In [2]: t = astropy.time.Time.now() In [3]: t.yday Out[3]: '2016:050:13:45:21.260' Now, what if I want the string '2016-050T13:45:21' instead? What I'd like is basically the astropy equivalent of the C library strftime(). I see there is a way to make a new subclass of TimeFormat, but I don't really want a whole new format, just a way to customize the string representation (possibly for input as well as output). Maybe all I need is an example of how to subclass TimeFormat to accomplish that. I had some trouble getting it to work in the little experimenting I did. Thanks for any tips, -- Paul From varunb at iucaa.in Mon Feb 22 00:16:24 2016 From: varunb at iucaa.in (Varun Bhalerao) Date: Mon, 22 Feb 2016 10:46:24 +0530 Subject: [AstroPy] Time custom formats In-Reply-To: References: Message-ID: <4B16789A-C0F9-4167-B9AC-AE2529B2EBD1@iucaa.in> An alternate way to do this: import astropy.time import astropy.units as u t = astropy.time.Time.now() tnew = t - 1*u.d print tnew.isot Not a full solution, but does what you need. -- Varun ____________________ - Varun Bhalerao - Office: A 214, IUCAA Ph: +91 20 2560 4214 www.iucaa.in/~varunb > On 20-Feb-2016, at 5:30 pm, astropy-request at scipy.org wrote: > > Send AstroPy mailing list submissions to > astropy at scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.scipy.org/mailman/listinfo/astropy > or, via email, send a message with subject or body 'help' to > astropy-request at scipy.org > > You can reach the person managing the list at > astropy-owner at scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of AstroPy digest..." > > > Today's Topics: > > 1. Time custom formats (Paul Ray) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 19 Feb 2016 08:51:46 -0500 > From: Paul Ray > To: astropy at scipy.org > Subject: [AstroPy] Time custom formats > Message-ID: <699E05F6-DE20-4D9E-92D9-54C7B6BE6F0E at nrl.navy.mil> > Content-Type: text/plain; charset=us-ascii > > Hi, > > I've been trying to figure out how to print an astropy.Time with a custom string format and have not been able to. > > In [2]: t = astropy.time.Time.now() > > In [3]: t.yday > Out[3]: '2016:050:13:45:21.260' > > Now, what if I want the string '2016-050T13:45:21' instead? > What I'd like is basically the astropy equivalent of the C library strftime(). > > I see there is a way to make a new subclass of TimeFormat, but I don't really want a whole new format, just a way to customize the string representation (possibly for input as well as output). Maybe all I need is an example of how to subclass TimeFormat to accomplish that. I had some trouble getting it to work in the little experimenting I did. > > Thanks for any tips, > > -- Paul > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > > > ------------------------------ > > End of AstroPy Digest, Vol 113, Issue 11 > **************************************** From aldcroft at head.cfa.harvard.edu Mon Feb 22 07:07:48 2016 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Mon, 22 Feb 2016 07:07:48 -0500 Subject: [AstroPy] Time custom formats In-Reply-To: <699E05F6-DE20-4D9E-92D9-54C7B6BE6F0E@nrl.navy.mil> References: <699E05F6-DE20-4D9E-92D9-54C7B6BE6F0E@nrl.navy.mil> Message-ID: Hi Paul, What you want is exactly a custom format, and in this case it's quite easy to get both input and output representation in the format you want. This is just copied from the existing TimeYearDayTime class and modified slightly. When you define the class like this it is automatically registered as an available format like the rest, including automatic input parsing when possible. from astropy.time import TimeISO class TimeYearDayTimeCustom(TimeISO): """ Year, day-of-year and time as "-T::". The day-of-year (DOY) goes from 001 to 365 (366 in leap years). For example, 2000-001T00:00:00.000 is midnight on January 1, 2000. The allowed subformats are: - 'date_hms': date + hours, mins, secs (and optional fractional secs) - 'date_hm': date + hours, mins - 'date': date """ name = 'yday_custom' subfmts = (('date_hms', '%Y-%jT%H:%M:%S', '{year:d}-{yday:03d}T{hour:02d}:{min:02d}:{sec:02d}'), ('date_hm', '%Y-%jT%H:%M', '{year:d}-{yday:03d}T{hour:02d}:{min:02d}'), ('date', '%Y-%j', '{year:d}-{yday:03d}')) In [4]: t = Time.now() In [5]: t.yday_custom Out[5]: '2016-053T12:01:41.663' In [6]: t2 = Time('2016-001T00:00:00') In [7]: t2.iso Out[7]: '2016-01-01 00:00:00.000' - Tom On Fri, Feb 19, 2016 at 8:51 AM, Paul Ray wrote: > Hi, > > I've been trying to figure out how to print an astropy.Time with a custom > string format and have not been able to. > > In [2]: t = astropy.time.Time.now() > > In [3]: t.yday > Out[3]: '2016:050:13:45:21.260' > > Now, what if I want the string '2016-050T13:45:21' instead? > What I'd like is basically the astropy equivalent of the C library > strftime(). > > I see there is a way to make a new subclass of TimeFormat, but I don't > really want a whole new format, just a way to customize the string > representation (possibly for input as well as output). Maybe all I need is > an example of how to subclass TimeFormat to accomplish that. I had some > trouble getting it to work in the little experimenting I did. > > Thanks for any tips, > > -- Paul > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.ray at nrl.navy.mil Mon Feb 22 07:30:46 2016 From: paul.ray at nrl.navy.mil (Paul Ray) Date: Mon, 22 Feb 2016 07:30:46 -0500 Subject: [AstroPy] AstroPy Digest, Vol 113, Issue 12 In-Reply-To: References: Message-ID: <3BA8C3EB-BF54-4621-B39D-0B8E597B5925@nrl.navy.mil> > On Feb 22, 2016, at 7:00 AM, wrote: > > An alternate way to do this: > > import astropy.time > import astropy.units as u > t = astropy.time.Time.now() > tnew = t - 1*u.d > print tnew.isot > > Not a full solution, but does what you need. Thanks for the reply, but that doesn?t do what I need. the isot format uses YYYY-MM-DD, not YYYY-DOY. And, my example was just to be able to ask the general question about making my own custom string output formats for Time objects, like you can do with strftime in unix. Thanks, ? Paul From michaelseifert04 at yahoo.de Tue Feb 23 18:25:05 2016 From: michaelseifert04 at yahoo.de (Michael Seifert) Date: Tue, 23 Feb 2016 23:25:05 +0000 (UTC) Subject: [AstroPy] Cosmology Recession Velocity References: <1895503505.14136141.1456269905260.JavaMail.yahoo.ref@mail.yahoo.com> Message-ID: <1895503505.14136141.1456269905260.JavaMail.yahoo@mail.yahoo.com> Hi, today I tried to determine cosmological recession velocities based on the paper of Davis, Lineweaver, 2001 (http://arxiv.org/pdf/astro-ph/0011070v2.pdf) with astropy.cosmology. But I'm not sure what I'm doing wrong (or if I just misunderstood their reasoning). But they said the recession velocity is just "v_rec = da(z)/dt * comoving distance(z)" and "da/dt = H(z)*a(z)" so the recession velocity should be with "cosmo = astropy.cosmology.WMAP9" and redshift "z=1000": v_rec = cosmo.H(z) * cosmo.scale_factor(z) * cosmo.comoving_distance(z) but this gives me 62 times the speed of light where it should only be between 2 and 4 (based on the paper). But just using v_rec = cosmo.H0 * cosmo.comoving_distance(z) gives me 3.25 * c which seems to fit their calculations. Is my formula or reasoning wrong or is there something strange in the comoving distance calculation of WMAP9? I hope this question is not too basic but I'm not very deep into cosmology and it just seemed strange to me. Many thanks in advance, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From trive at astro.su.se Tue Feb 23 19:14:12 2016 From: trive at astro.su.se (=?UTF-8?Q?Th=c3=b8ger_Emil_Rivera-Thorsen?=) Date: Wed, 24 Feb 2016 01:14:12 +0100 Subject: [AstroPy] Cosmology Recession Velocity In-Reply-To: <1895503505.14136141.1456269905260.JavaMail.yahoo@mail.yahoo.com> References: <1895503505.14136141.1456269905260.JavaMail.yahoo.ref@mail.yahoo.com> <1895503505.14136141.1456269905260.JavaMail.yahoo@mail.yahoo.com> Message-ID: <56CCF5D4.1090508@astro.su.se> 62 c at z=1000 does not sound completely off for the *current value*. I suppose the values between 2 and 4 c you get gggfrom Davis & Lineweaver is velocities at the time of emission? These two differ *strongly*. On 02/24/2016 12:25 AM, Michael Seifert wrote: > Hi, > > today I tried to determine cosmological recession velocities based on > the paper of Davis, Lineweaver, 2001 > (http://arxiv.org/pdf/astro-ph/0011070v2.pdf > ) with astropy.cosmology. > > But I'm not sure what I'm doing wrong (or if I just misunderstood > their reasoning). But they said the recession velocity is just "v_rec > = da(z)/dt * comoving distance(z)" and "da/dt = H(z)*a(z)" so the > recession velocity should be with "cosmo = astropy.cosmology.WMAP9" > and redshift "z=1000": > > v_rec = cosmo.H(z) * cosmo.scale_factor(z) * cosmo.comoving_distance(z) > > but this gives me 62 times the speed of light where it should only be > between 2 and 4 (based on the paper). But just using > > v_rec = cosmo.H0 * cosmo.comoving_distance(z) > > gives me 3.25 * c which seems to fit their calculations. Is my formula > or reasoning wrong or is there something strange in the comoving > distance calculation of WMAP9? > > I hope this question is not too basic but I'm not very deep into > cosmology and it just seemed strange to me. > > Many thanks in advance, > Michael > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From trive at astro.su.se Tue Feb 23 19:35:36 2016 From: trive at astro.su.se (=?UTF-8?Q?Th=c3=b8ger_Emil_Rivera-Thorsen?=) Date: Wed, 24 Feb 2016 01:35:36 +0100 Subject: [AstroPy] Cosmology Recession Velocity In-Reply-To: <56CCF5D4.1090508@astro.su.se> References: <1895503505.14136141.1456269905260.JavaMail.yahoo.ref@mail.yahoo.com> <1895503505.14136141.1456269905260.JavaMail.yahoo@mail.yahoo.com> <56CCF5D4.1090508@astro.su.se> Message-ID: <56CCFAD8.60000@astro.su.se> Sorry, that did not come out very clear. The two different values you find are the recession velocity at the time of emission and the recession velocity now. Since the Universe has expanded by a lot since then, a current value of 62 c seems reasonable. I have a notebook demonstrating some key Astropy features for students at our department, including this very calculation, here: http://nbviewer.jupyter.org/urls/ttt.astro.su.se/~trive/teaching/tools-package/Astropy-intro.ipynb (Scroll to the bottom) On 02/24/2016 01:14 AM, Th?ger Emil Rivera-Thorsen wrote: > 62 c at z=1000 does not sound completely off for the *current value*. > I suppose the values between 2 and 4 c you get gggfrom Davis & > Lineweaver is velocities at the time of emission? These two differ > *strongly*. > > > > On 02/24/2016 12:25 AM, Michael Seifert wrote: >> Hi, >> >> today I tried to determine cosmological recession velocities based on >> the paper of Davis, Lineweaver, 2001 >> (http://arxiv.org/pdf/astro-ph/0011070v2.pdf) with astropy.cosmology. >> >> But I'm not sure what I'm doing wrong (or if I just misunderstood >> their reasoning). But they said the recession velocity is just "v_rec >> = da(z)/dt * comoving distance(z)" and "da/dt = H(z)*a(z)" so the >> recession velocity should be with "cosmo = astropy.cosmology.WMAP9" >> and redshift "z=1000": >> >> v_rec = cosmo.H(z) * cosmo.scale_factor(z) * cosmo.comoving_distance(z) >> >> but this gives me 62 times the speed of light where it should only be >> between 2 and 4 (based on the paper). But just using >> >> v_rec = cosmo.H0 * cosmo.comoving_distance(z) >> >> gives me 3.25 * c which seems to fit their calculations. Is my >> formula or reasoning wrong or is there something strange in the >> comoving distance calculation of WMAP9? >> >> I hope this question is not too basic but I'm not very deep into >> cosmology and it just seemed strange to me. >> >> Many thanks in advance, >> Michael >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> https://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From michaelseifert04 at yahoo.de Tue Feb 23 20:05:00 2016 From: michaelseifert04 at yahoo.de (Michael Seifert) Date: Wed, 24 Feb 2016 01:05:00 +0000 (UTC) Subject: [AstroPy] Cosmology Recession Velocity In-Reply-To: <56CCFAD8.60000@astro.su.se> References: <56CCFAD8.60000@astro.su.se> Message-ID: <803723521.14012194.1456275900893.JavaMail.yahoo@mail.yahoo.com> Many thanks, your explanation is very good and the notebook even better. But I have another question because I think the problem arose because I misunderstood something fundamental here. They quoted in their paper that "present day recession velocities are shown" (defining it in a later paper as "a(z) = a(0)"). So is present day recession velocity the "current" recession velocity (62c) or the currently measurable recession velocity of light that reaches us (3.3c)? If that's beyond the scope of the mailing list, I apologize. Thanks again,Michael Th?ger Emil Rivera-Thorsen schrieb am 1:35 Mittwoch, 24.Februar 2016: Sorry, that did not come out very clear. The two different values you find are the recession velocity at the time of emission and the recession velocity now. Since the Universe has expanded by a lot since then, a current value of 62 c seems reasonable. I have a notebook demonstrating some key Astropy features for students at our department, including this very calculation, here: http://nbviewer.jupyter.org/urls/ttt.astro.su.se/~trive/teaching/tools-package/Astropy-intro.ipynb (Scroll to the bottom) On 02/24/2016 01:14 AM, Th?ger Emil Rivera-Thorsen wrote: 62 c at z=1000 does not sound completely off for the *current value*. I suppose the values between 2 and 4 c you get gggfrom Davis & Lineweaver is velocities at the time of emission? These two differ *strongly*. On 02/24/2016 12:25 AM, Michael Seifert wrote: Hi, today I tried to determine cosmological recession velocities based on the paper of Davis, Lineweaver, 2001 (http://arxiv.org/pdf/astro- ph/0011070v2.pdf) with astropy.cosmology. But I'm not sure what I'm doing wrong (or if I just misunderstood their reasoning). But they said the recession velocity is just "v_rec = da(z)/dt * comoving distance(z)" and "da/dt = H(z)*a(z)" so the recession velocity should be with "cosmo = astropy.cosmology.WMAP9" and redshift "z=1000": v_rec = cosmo.H(z) * cosmo.scale_factor(z) * cosmo.comoving_distance(z) but this gives me 62 times the speed of light where it should only be between 2 and 4 (based on the paper). But just using v_rec = cosmo.H0 * cosmo.comoving_distance(z) gives me 3.25 * c which seems to fit their calculations. Is my formula or reasoning wrong or is there something strange in the comoving distance calculation of WMAP9? I hope this question is not too basic but I'm not very deep into cosmology and it just seemed strange to me. Many thanks in advance, Michael _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy _______________________________________________ AstroPy mailing list AstroPy at scipy.org https://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From york at stsci.edu Thu Feb 25 14:56:11 2016 From: york at stsci.edu (Brian York) Date: Thu, 25 Feb 2016 19:56:11 +0000 Subject: [AstroPy] Question re. astropy.table Message-ID: Greetings, I have a number of situations where I'm reading data from a (rather large) table in order to add that data into a separate numpy array (sample code below): t = Table.read(name, format) x_locations = t['X'] y_locations = t['Y'] fluxes = t['FLUX'] image_data[y_locations, x_locations] += fluxes Now, this works well for relatively short tables (~100,000 rows), but sometimes I end up with considerably longer tables (~5-10 million rows), and there tends to be a fairly high memory overhead in loading the table in those circumstances (especially given that the table has more than just the three columns -- the above code is an example, not exactly what I'm doing). Is there any way to load a table N rows at a time? Or some other way to reduce the memory footprint? Thank you, -Brian York From te.pickering at gmail.com Thu Feb 25 16:10:22 2016 From: te.pickering at gmail.com (Timothy Pickering) Date: Thu, 25 Feb 2016 16:10:22 -0500 Subject: [AstroPy] Question re. astropy.table In-Reply-To: References: Message-ID: <78403AB6-3072-4DD1-A248-826E4794D5CE@gmail.com> for a use-case like this, it might be worth looking into using pandas: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking tim > On Feb 25, 2016, at 2:56 PM, Brian York wrote: > > Greetings, > > I have a number of situations where I'm reading data from a (rather large) > table in order to add that data into a separate numpy array (sample code > below): > > t = Table.read(name, format) > x_locations = t['X'] > y_locations = t['Y'] > fluxes = t['FLUX'] > > image_data[y_locations, x_locations] += fluxes > > Now, this works well for relatively short tables (~100,000 rows), but > sometimes I end up with considerably longer tables (~5-10 million rows), > and there tends to be a fairly high memory overhead in loading the table > in those circumstances (especially given that the table has more than just > the three columns -- the above code is an example, not exactly what I'm > doing). > > Is there any way to load a table N rows at a time? Or some other way to > reduce the memory footprint? > > Thank you, > -Brian York > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy ? +-------------------------------------------------------------------+ | T. E. Pickering, Ph.D. | Space Telescope Science Institute | | Systems Software Engineer | 3700 San Martin Dr. | | +1-520-305-9823 | Baltimore, MD, 21218, USA | | te.pickering at gmail.com | pickering at stsci.edu | +-------------------------------------------------------------------+ From stsci.perry at gmail.com Thu Feb 25 16:30:26 2016 From: stsci.perry at gmail.com (Perry Greenfield) Date: Thu, 25 Feb 2016 16:30:26 -0500 Subject: [AstroPy] Question re. astropy.table In-Reply-To: References: Message-ID: <90FCD27D-83B2-4605-AC16-55764C39788B@gmail.com> It may be possible to speed it up a lot if you can access the underlying table as a numpy recarray and take advantage of memory mapping as well. The downside of doing that is that you have to deal with logical column translations yourself (i.e., testing for ?T? or ?F?), and having to scale results if any columns use BSCALE or BZERO. But for what you show below, that probably isn?t the case. I wish I had time to try this out myself, but it would work something like this (untested). nprarr = tablehdu.data.view(np.recarray) Then you would access whatever columns you want. If it is really large, you may want to read in column data in subsets (e.g., only a few hundred thousand rows at a time) to copy to an in memory array for those data. Perhaps tomorrow I can give this a try (particularly if you can point me to an example file). Perry On Feb 25, 2016, at 2:56 PM, Brian York wrote: > Greetings, > > I have a number of situations where I'm reading data from a (rather large) > table in order to add that data into a separate numpy array (sample code > below): > > t = Table.read(name, format) > x_locations = t['X'] > y_locations = t['Y'] > fluxes = t['FLUX'] > > image_data[y_locations, x_locations] += fluxes > > Now, this works well for relatively short tables (~100,000 rows), but > sometimes I end up with considerably longer tables (~5-10 million rows), > and there tends to be a fairly high memory overhead in loading the table > in those circumstances (especially given that the table has more than just > the three columns -- the above code is an example, not exactly what I'm > doing). > > Is there any way to load a table N rows at a time? Or some other way to > reduce the memory footprint? > > Thank you, > -Brian York > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy From aldcroft at head.cfa.harvard.edu Thu Feb 25 18:59:56 2016 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Thu, 25 Feb 2016 18:59:56 -0500 Subject: [AstroPy] Question re. astropy.table In-Reply-To: References: Message-ID: Hi Brian, If your input file an ASCII data table then please see: https://github.com/astropy/astropy/issues/3334 There is a code snippet there that should work to read in a large table in chunks. Best, Tom On Thu, Feb 25, 2016 at 2:56 PM, Brian York wrote: > Greetings, > > I have a number of situations where I'm reading data from a (rather large) > table in order to add that data into a separate numpy array (sample code > below): > > t = Table.read(name, format) > x_locations = t['X'] > y_locations = t['Y'] > fluxes = t['FLUX'] > > image_data[y_locations, x_locations] += fluxes > > Now, this works well for relatively short tables (~100,000 rows), but > sometimes I end up with considerably longer tables (~5-10 million rows), > and there tends to be a fairly high memory overhead in loading the table > in those circumstances (especially given that the table has more than just > the three columns -- the above code is an example, not exactly what I'm > doing). > > Is there any way to load a table N rows at a time? Or some other way to > reduce the memory footprint? > > Thank you, > -Brian York > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > https://mail.scipy.org/mailman/listinfo/astropy > -------------- next part -------------- An HTML attachment was scrubbed... URL: From york at stsci.edu Fri Feb 26 12:51:32 2016 From: york at stsci.edu (Brian York) Date: Fri, 26 Feb 2016 17:51:32 +0000 Subject: [AstroPy] Question re. astropy.table In-Reply-To: Message-ID: On 2016/02/25 18:59 , "Aldcroft, Thomas" wrote: >Hi Brian, > > >If your input file an ASCII data table then please see: > https://github.com/astropy/astropy/issues/3334 > > > >There is a code snippet there that should work to read in a large table >in chunks. I've tried this, but I keep running into issues. The table is an ASCII IPAC format, and running the above code (with format='ipac') always gets me the following exception: Traceback (most recent call last): File "/Users/york/ssbvirt/ssbdev-osx/lib/python2.7/site-packages/celery/app/trac e.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/Users/york/JWST/pandeia/simulator.py", line 87, in __call__ return TaskBase.__call__(self, *args, **kwargs) File "/Users/york/ssbvirt/ssbdev-osx/lib/python2.7/site-packages/celery/app/trac e.py", line 438, in __protected_call__ return self.run(*args, **kwargs) File "/Users/york/JWST/pandeia/simulator.py", line 173, in observation observed_catalogues.extend(obs.addCatalogue(cat)) File "/Users/york/JWST/pandeia/stips/stips/observation_module/observation_module .py", line 184, in addCatalogue cats = self.instrument.addCatalogue(catalogue) File "/Users/york/JWST/pandeia/stips/stips/instruments/instrument.py", line 218, in addCatalogue cats.append(detector.addCatalogue(cat)) File "/Users/york/JWST/pandeia/stips/stips/astro_image/astro_image.py", line 348, in addCatalogue for i, t in enumerate(read_table(cat)): File "/Users/york/JWST/pandeia/stips/stips/utilities/utilities.py", line 109, in read_table chunk = ascii.read(lines, format=format, guess=False) File "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/ui.p y", line 339, in read dat = reader.read(table) File "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/core .py", line 1045, in read self.header.get_cols(self.lines) File "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/ipac .py", line 180, in get_cols raise ValueError('At least one header line beginning and ending with ' ValueError: At least one header line beginning and ending with delimiter required I have no idea of what's causing this (for reference, the opening lines of the table are below, and this table opens just fine if I do a Table.read(format='ascii.ipac') (except that it takes up ~4GB of memory for a typical stellar population): \ Internal Format Catalogue \ \ Parameters: \ \type="internal" \filter="F115W" \ | RA| DEC| FLUX| TYPE| N| Re| Phi|Ratio| ID| Notes| | double| double| double| char|char|char|char| char| long| char| | | | | | | | | | | | | null| null| null| null|null|null|null| null| null| null| 8.35744567067e-09 6.07703574397e-09 1.77083404307 point N/A N/A N/A N/A 1 None 1.82673672356e-10 -2.22541481358e-11 4.16120404729 point N/A N/A N/A N/A 2 None 1.67512697711e-08 -1.68243630672e-08 2.70787939571 point N/A N/A N/A N/A 3 None If anyone has any idea of how I'm using the code wrong, it would be much appreciated. -Brian From aldcroft at head.cfa.harvard.edu Sat Feb 27 07:11:21 2016 From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas) Date: Sat, 27 Feb 2016 07:11:21 -0500 Subject: [AstroPy] Question re. astropy.table In-Reply-To: References: Message-ID: Hi Brian, I fixed the original code in the github thread and it works on your example (also copied below). - Tom from astropy.io import ascii import os def read_chunks(filename, n_chunk, format='basic'): lines = [] for i, line in enumerate(open(filename, 'r')): lines.append(line.rstrip(os.linesep)) if i % n_chunk == n_chunk - 1: if i < n_chunk: # first chunk = ascii.read(lines, format=format, guess=False) names = chunk.colnames yield chunk else: chunk = ascii.read(lines, format='no_header', names=names, guess=False) yield chunk lines = [] if lines: yield ascii.read(lines, format='no_header', names=names, guess=False) for dat in read_chunks('ipac.dat', 20, format='ipac'): print(dat) On Fri, Feb 26, 2016 at 12:51 PM, Brian York wrote: > On 2016/02/25 18:59 , "Aldcroft, Thomas" > wrote: > > >Hi Brian, > > > > > >If your input file an ASCII data table then please see: > > https://github.com/astropy/astropy/issues/3334 > > > > > > > >There is a code snippet there that should work to read in a large table > >in chunks. > > I've tried this, but I keep running into issues. The table is an ASCII > IPAC format, and running the above code (with format='ipac') always gets > me the following exception: > > Traceback (most recent call last): > File > "/Users/york/ssbvirt/ssbdev-osx/lib/python2.7/site-packages/celery/app/trac > e.py", line 240, in trace_task > R = retval = fun(*args, **kwargs) > File "/Users/york/JWST/pandeia/simulator.py", line 87, in __call__ > return TaskBase.__call__(self, *args, **kwargs) > File > "/Users/york/ssbvirt/ssbdev-osx/lib/python2.7/site-packages/celery/app/trac > e.py", line 438, in __protected_call__ > return self.run(*args, **kwargs) > File "/Users/york/JWST/pandeia/simulator.py", line 173, in observation > observed_catalogues.extend(obs.addCatalogue(cat)) > File > "/Users/york/JWST/pandeia/stips/stips/observation_module/observation_module > .py", line 184, in addCatalogue > cats = self.instrument.addCatalogue(catalogue) > File "/Users/york/JWST/pandeia/stips/stips/instruments/instrument.py", > line 218, in addCatalogue > cats.append(detector.addCatalogue(cat)) > File "/Users/york/JWST/pandeia/stips/stips/astro_image/astro_image.py", > line 348, in addCatalogue > for i, t in enumerate(read_table(cat)): > File "/Users/york/JWST/pandeia/stips/stips/utilities/utilities.py", line > 109, in read_table > chunk = ascii.read(lines, format=format, guess=False) > File > "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/ui.p > y", line 339, in read > dat = reader.read(table) > File > "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/core > .py", line 1045, in read > self.header.get_cols(self.lines) > File > "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/ipac > .py", line 180, in get_cols > raise ValueError('At least one header line beginning and ending with ' > ValueError: At least one header line beginning and ending with delimiter > required > > > I have no idea of what's causing this (for reference, the opening lines of > the table are below, and this table opens just fine if I do a > Table.read(format='ascii.ipac') (except that it takes up ~4GB of memory > for a typical stellar population): > > \ Internal Format Catalogue > \ > \ Parameters: > \ > \type="internal" > \filter="F115W" > \ > | RA| DEC| FLUX| TYPE| N| Re| > Phi|Ratio| ID| Notes| > | double| double| double| char|char|char|char| > char| long| char| > | | | | | | | | > | | | > | null| null| null| null|null|null|null| > null| null| null| > 8.35744567067e-09 6.07703574397e-09 1.77083404307 point N/A N/A N/A > N/A 1 None > 1.82673672356e-10 -2.22541481358e-11 4.16120404729 point N/A N/A N/A > N/A 2 None > 1.67512697711e-08 -1.68243630672e-08 2.70787939571 point N/A N/A N/A > N/A 3 None > > > If anyone has any idea of how I'm using the code wrong, it would be much > appreciated. > > -Brian > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lakshmanan.meiyappan at gmail.com Mon Feb 29 16:13:12 2016 From: lakshmanan.meiyappan at gmail.com (Lakshmanan Meiyappan) Date: Tue, 1 Mar 2016 02:43:12 +0530 Subject: [AstroPy] GSoC'16 reg Message-ID: Hi, I'm Lakshmanan Meiyappan, 2nd year Engineering student from India, and I wish to contribute to AstroPy in this GSoC, can anyone guide me. Thank you, Lakshmanan Meiyappan -------------- next part -------------- An HTML attachment was scrubbed... URL: