From humufr at yahoo.fr Tue Apr 1 10:41:02 2008 From: humufr at yahoo.fr (humufr at yahoo.fr) Date: Tue, 1 Apr 2008 10:41:02 -0400 Subject: [AstroPy] [SciPy-user] FITS images with header-supplied axes? Message-ID: <200804011041.02702.humufr@yahoo.fr> Hello, I just want to discuss about a problem that our python astronomer have and the answer from Perry is a very good example of it. There are too many projects which are doing exactly the same thing and essentially because nothing is centralized or worst, in the case of pywcs, not advertised. For example Perry told us about pywcs developped by STSCI (it was the first time I saw any reference to this project) but Adam spoke about astlib ( http://astlib.sourceforge.net/ ) which are different package with exactly the same goal and I did myself something similar (even if it was fast and dirtier). I think it's time to try to identified the most important task that the astronomer are needing and try to centralized all the effort at the same place. The astropy mail list is probably a good start as the astropy.scipy.org website. Perhaps we can start to identified the need and desiderata. We will have too many but that some most important or urgent must be identified. As example, we clearly needed the pywcs and thanks to STSCI we have it now. We also need a package to plot our data, images etc. Matplotlib is very good but does have a major problem (at least for my point of view), it's slow, very slow for big array or can't even produce the image if the image is too big. ( If I remember a precedent discussion, the problem is mainly due to Agg.) After we need to know, if possible, for which project if someone is doing something, who and how to contact him/her. So interested people can eventually help him/her to do it. It's seems that STSCI is doing most of the work but I'm pretty sure that other people are willingly to help them to extend python to be the ideal tool for astronomer. Just my 2 cents, Nicolas Le Sunday 30 March 2008 10:54:04 Perry Greenfield, vous avez ?crit?: > On Mar 28, 2008, at 8:28 PM, Keflavich wrote: > > Is there any plotting routine in scipy / matplotlib that can plot a > > fits image with correct WCS coordinates on the axes? I know pyfits > > can load fits files, astLib has routines to interpret header > > coordinates, and I think you can make the axes different using > > > matplotlib transforms, but is there anything that puts all three > > together currently available? > > > > Thanks, > > Adam > > Well, we (STScI) recently wrapped WCSLIB to obtain a mapping function > between pixel and sky coordinates for python (you can find it as pywcs in > astrolib on scipy; that may have been what you were referring to). > > But I'm not sure you understand what you are asking for with regard to > matplotlib. The new transforms stuff should make it much easier to display > the sky coordinates in the interactive display. The axis labeling is a > different matter. Suppose your image (let's say it's 1Kx1K for the sake of > discussion) is rotated 45 degrees with regard to north (either way, it > doesn't really matter). What would you expect to see for axis labels? I > don't think it is at all obvious how people would want labeling to be done > along the edges of the image. I can imagine someone wanting axes or grids > superimposed on the image itself, but that's not quite the same thing. Do > you want the image rotated so that it is resampled on to RA and Dec and > displayed that way? > > In any event, no we haven't yet done anything to try to integrate all three > things. Among other things we wanted to make sure that the api for the wcs > info was suitable before doing a lot with it (and in the meantime, Mike is > working on rewriting drizzle which is taking a lot of his time). > > Perry > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.org > http://projects.scipy.org/mailman/listinfo/scipy-user From William.T.Bridgman at nasa.gov Tue Apr 1 13:51:46 2008 From: William.T.Bridgman at nasa.gov (Bridgman, William T.) Date: Tue, 1 Apr 2008 13:51:46 -0400 Subject: [AstroPy] [SciPy-user] FITS images with header-supplied axes? In-Reply-To: <200804011041.02702.humufr@yahoo.fr> References: <200804011041.02702.humufr@yahoo.fr> Message-ID: <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> I agree there is some lack of coordination on some of these projects. Such is the nature of Open Source. I think there is also an issue where some of the projects are overkill for what individuals might need for production-pipeline/ research/educational uses. Hence many of us start from scratch to keep the work compact. It's only later that we discover others working in similar directions. I obtained a copy of pywcs many months ago, trying to get heliographic coordinate systems installed. Other priorities intervened and by the time I got back to it, migration to numpy and changes in pyFITS seemed to break the previous version. I didn't get a chance to make any revisions operational. Would there be any interest in members of the list publishing a short description of what types of modules they are designing in their own work? It might be worthwhile for coordination & possible collaborations. My requirements for work projects are quite different from my recreational & educational python projects. Tom On Apr 1, 2008, at 10:41 AM, humufr at yahoo.fr wrote: > Hello, > > I just want to discuss about a problem that our python astronomer > have and the > answer from Perry is a very good example of it. There are too many > projects > which are doing exactly the same thing and essentially because > nothing is > centralized or worst, in the case of pywcs, not advertised. > > For example Perry told us about pywcs developped by STSCI (it was > the first > time I saw any reference to this project) but Adam spoke about > astlib ( http://astlib.sourceforge.net/ ) which are different > package with > exactly the same goal and I did myself something similar (even if > it was fast > and dirtier). > > I think it's time to try to identified the most important task that > the > astronomer are needing and try to centralized all the effort at the > same > place. The astropy mail list is probably a good start as the > astropy.scipy.org website. > > Perhaps we can start to identified the need and desiderata. We will > have too > many but that some most important or urgent must be identified. As > example, > we clearly needed the pywcs and thanks to STSCI we have it now. We > also need > a package to plot our data, images etc. Matplotlib is very good but > does have > a major problem (at least for my point of view), it's slow, very > slow for big > array or can't even produce the image if the image is too big. ( If I > remember a precedent discussion, the problem is mainly due to Agg.) > > After we need to know, if possible, for which project if someone is > doing > something, who and how to contact him/her. So interested people can > eventually help him/her to do it. It's seems that STSCI is doing > most of the > work but I'm pretty sure that other people are willingly to help > them to > extend python to be the ideal tool for astronomer. > > Just my 2 cents, > > Nicolas > > Le Sunday 30 March 2008 10:54:04 Perry Greenfield, vous avez ?crit : >> On Mar 28, 2008, at 8:28 PM, Keflavich wrote: >>> Is there any plotting routine in scipy / matplotlib that can plot a >>> fits image with correct WCS coordinates on the axes? I know pyfits >>> can load fits files, astLib has routines to interpret header >> >> coordinates, and I think you can make the axes different using >> >>> matplotlib transforms, but is there anything that puts all three >>> together currently available? >>> >>> Thanks, >>> Adam >> >> Well, we (STScI) recently wrapped WCSLIB to obtain a mapping function >> between pixel and sky coordinates for python (you can find it as >> pywcs in >> astrolib on scipy; that may have been what you were referring to). >> >> But I'm not sure you understand what you are asking for with >> regard to >> matplotlib. The new transforms stuff should make it much easier to >> display >> the sky coordinates in the interactive display. The axis labeling >> is a >> different matter. Suppose your image (let's say it's 1Kx1K for the >> sake of >> discussion) is rotated 45 degrees with regard to north (either >> way, it >> doesn't really matter). What would you expect to see for axis >> labels? I >> don't think it is at all obvious how people would want labeling to >> be done >> along the edges of the image. I can imagine someone wanting axes >> or grids >> superimposed on the image itself, but that's not quite the same >> thing. Do >> you want the image rotated so that it is resampled on to RA and >> Dec and >> displayed that way? >> >> In any event, no we haven't yet done anything to try to integrate >> all three >> things. Among other things we wanted to make sure that the api for >> the wcs >> info was suitable before doing a lot with it (and in the meantime, >> Mike is >> working on rewriting drizzle which is taking a lot of his time). >> >> Perry >> _______________________________________________ >> SciPy-user mailing list >> SciPy-user at scipy.org >> http://projects.scipy.org/mailman/listinfo/scipy-user > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://lists.astropy.scipy.org/mailman/listinfo/astropy **** Please Update my e-mail address to my *NEW* OneNASA address below **** -- Dr. William T."Tom" Bridgman Scientific Visualization Studio Global Science & Technology, Inc. NASA/Goddard Space Flight Center Email: William.T.Bridgman at nasa.gov Code 610.3 Phone: 301-286-1346 Greenbelt, MD 20771 FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ From rowen at u.washington.edu Tue Apr 1 16:32:42 2008 From: rowen at u.washington.edu (Russell E Owen) Date: Tue, 1 Apr 2008 13:32:42 -0700 Subject: [AstroPy] [SciPy-user] FITS images with header-supplied axes? In-Reply-To: <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> References: <200804011041.02702.humufr@yahoo.fr> <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> Message-ID: At 1:51 PM -0400 2008-04-01, Bridgman, William T. wrote: >I agree there is some lack of coordination on some of these >projects. Such is the nature of Open Source. > >I think there is also an issue where some of the projects are >overkill for what individuals might need for production-pipeline/ >research/educational uses. Hence many of us start from scratch to >keep the work compact. It's only later that we discover others >working in similar directions. > >I obtained a copy of pywcs many months ago, trying to get >heliographic coordinate systems installed. Other priorities >intervened and by the time I got back to it, migration to numpy and >changes in pyFITS seemed to break the previous version. I didn't get >a chance to make any revisions operational. > >Would there be any interest in members of the list publishing a short >description of what types of modules they are designing in their own >work? It might be worthwhile for coordination & possible >collaborations. My requirements for work projects are quite >different from my recreational & educational python projects. Good idea. I'm part of the team working on data processing for the LSST. This number crunching is done in C++ and the high level operations are done in python. This work will probably be primarily of interest to those processing a lot of data because it is pipeline-oriented. The final product will include: - basic types for images, masks and masked images - image linearization (bias subtraction, flat fielding, etc.) - wcs determination - source detection - image subtraction We intend it to be usable on a variety of data sources (not just LSST) since that's the only way to test it. Nonetheless it will probably take a bit of work to massage the data headers. As far as wcs goes: right now we use wcslib and have a limited python interface on it. I suspect if anything better came along we'd be happy to switch. (at least at the C++ level; I'm not sure about the Python level). -- Russell From cygnusxlist at mac.com Tue Apr 1 21:58:14 2008 From: cygnusxlist at mac.com (cygnusxlist at mac.com) Date: Tue, 1 Apr 2008 21:58:14 -0400 Subject: [AstroPy] Astronomical Python Projects (was Re: [SciPy-user] FITS images with header-supplied axes?) In-Reply-To: References: <200804011041.02702.humufr@yahoo.fr> <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> Message-ID: Since I'm at home now, I'll describe my 'recreational' python projects. I'm probably the really weird one on the list. ;^) My major use is in developing simple models/simulations & data analysis exercises which can be used in debunking crank astronomy & physics (see my .sig below). I'm using scipy/numpy/pyfits/PIL/PyX for generating graphics from simple models. Here's some recent examples: - electron-proton-neutron gas equation of state (neutron stars can't exist claims) - power spectra of simple cosmological models (redshift quantization claims) - simple galactic supernova distribution models (not enough SNR claims) - simple radiation transport simulations - simple E&M field configuration simulations (trying out FiPy for this) - simulated 'observations' based on various real & crackpot cosmological models - nuclear decay & nuclear reaction networks. For this project, I don't have a strong need for precision tools. I don't use PyRAF, or the more complex tools available. I want most of my projects to be easily reproducible by students and instructors at various levels. FV & DS9 have sufficed for viewing images generated by image-based models to unprocessed Hubble imagery. The frameworks for some of these might be of use to others, but I'm not sure the best way to package them. I've looked at some of the data-table readers promoted on the list, but have generally found them overkill for my needs (hence I re-invent the wheel for much of this). More recently, I've been working with the pyPOV wrapper to generate nice visualizations from models using the POVray ray-tracer. I do have an interest in a good implementation of WCS in case I want to overlay a model with actual data. I have a simple UTC time class but it is *not* a precision object (unsuitable for say, pulsar or x- ray timing observations). I'll post the work list tomorrow. Thanks, Tom On Apr 1, 2008, at 4:32 PM, Russell E Owen wrote: > At 1:51 PM -0400 2008-04-01, Bridgman, William T. wrote: >> >> Would there be any interest in members of the list publishing a short >> description of what types of modules they are designing in their own >> work? It might be worthwhile for coordination & possible >> collaborations. My requirements for work projects are quite >> different from my recreational & educational python projects. > > Good idea. > > I'm part of the team working on data processing for the LSST. This > number crunching is done in C++ and the high level operations are > done in python. This work will probably be primarily of interest to > those processing a lot of data because it is pipeline-oriented. > > The final product will include: > - basic types for images, masks and masked images > - image linearization (bias subtraction, flat fielding, etc.) > - wcs determination > - source detection > - image subtraction > > We intend it to be usable on a variety of data sources (not just > LSST) since that's the only way to test it. Nonetheless it will > probably take a bit of work to massage the data headers. > > As far as wcs goes: right now we use wcslib and have a limited python > interface on it. I suspect if anything better came along we'd be > happy to switch. (at least at the C++ level; I'm not sure about the > Python level). > > -- Russell > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://lists.astropy.scipy.org/mailman/listinfo/astropy -- Dealing with Creationism in Astronomy http://homepage.mac.com/cygnusx1 cygnusx1 at mac.com "They're trained to believe, not to know. Belief can be manipulated. Only knowledge is dangerous." --Frank Herbert, "Dune Messiah" -------------- next part -------------- An HTML attachment was scrubbed... URL: From William.T.Bridgman at nasa.gov Wed Apr 2 12:52:42 2008 From: William.T.Bridgman at nasa.gov (Bridgman, William T.) Date: Wed, 2 Apr 2008 12:52:42 -0400 Subject: [AstroPy] [SciPy-user] FITS images with header-supplied axes? In-Reply-To: <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> References: <200804011041.02702.humufr@yahoo.fr> <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> Message-ID: <04DF97AA-4E62-408C-81BD-EEC155DBC7F7@nasa.gov> Here's my 'work python' list. Mostly codes for organizing image and volumetric datasets. I use PIL, PyFITS, scipy, numpy, VTK for 3-d data extraction and image generation. I have some cases where I need to retrieve the data in IDL to get a format that can be used on the python side. I have a number of custom modules which co-register the images and 3- d data into various geospatial coordinate systems. I also have modules that extract spacecraft orbit information from sites like the SSCweb (though the limitations in ZSI interfere with me migrated to their newer web interface). The co-registration works with time- dependent data and connects the data components at appropriate times (however, it does not interpolate between actual datasets). I use CGkit to map the images and objects created above onto 3-d objects and generate files which are rendered into images by the RenderMan computer graphics package. Tom On Apr 1, 2008, at 1:51 PM, Bridgman, William T. wrote: > I agree there is some lack of coordination on some of these > projects. Such is the nature of Open Source. > > I think there is also an issue where some of the projects are > overkill for what individuals might need for production-pipeline/ > research/educational uses. Hence many of us start from scratch to > keep the work compact. It's only later that we discover others > working in similar directions. > > I obtained a copy of pywcs many months ago, trying to get > heliographic coordinate systems installed. Other priorities > intervened and by the time I got back to it, migration to numpy and > changes in pyFITS seemed to break the previous version. I didn't get > a chance to make any revisions operational. > > Would there be any interest in members of the list publishing a short > description of what types of modules they are designing in their own > work? It might be worthwhile for coordination & possible > collaborations. My requirements for work projects are quite > different from my recreational & educational python projects. > > Tom > > On Apr 1, 2008, at 10:41 AM, humufr at yahoo.fr wrote: >> Hello, >> >> I just want to discuss about a problem that our python astronomer >> have and the >> answer from Perry is a very good example of it. There are too many >> projects >> which are doing exactly the same thing and essentially because >> nothing is >> centralized or worst, in the case of pywcs, not advertised. >> >> For example Perry told us about pywcs developped by STSCI (it was >> the first >> time I saw any reference to this project) but Adam spoke about >> astlib ( http://astlib.sourceforge.net/ ) which are different >> package with >> exactly the same goal and I did myself something similar (even if >> it was fast >> and dirtier). >> >> I think it's time to try to identified the most important task that >> the >> astronomer are needing and try to centralized all the effort at the >> same >> place. The astropy mail list is probably a good start as the >> astropy.scipy.org website. >> >> Perhaps we can start to identified the need and desiderata. We will >> have too >> many but that some most important or urgent must be identified. As >> example, >> we clearly needed the pywcs and thanks to STSCI we have it now. We >> also need >> a package to plot our data, images etc. Matplotlib is very good but >> does have >> a major problem (at least for my point of view), it's slow, very >> slow for big >> array or can't even produce the image if the image is too big. ( If I >> remember a precedent discussion, the problem is mainly due to Agg.) >> >> After we need to know, if possible, for which project if someone is >> doing >> something, who and how to contact him/her. So interested people can >> eventually help him/her to do it. It's seems that STSCI is doing >> most of the >> work but I'm pretty sure that other people are willingly to help >> them to >> extend python to be the ideal tool for astronomer. >> >> Just my 2 cents, >> >> Nicolas >> >> Le Sunday 30 March 2008 10:54:04 Perry Greenfield, vous avez ?crit : >>> On Mar 28, 2008, at 8:28 PM, Keflavich wrote: >>>> Is there any plotting routine in scipy / matplotlib that can plot a >>>> fits image with correct WCS coordinates on the axes? I know pyfits >>>> can load fits files, astLib has routines to interpret header >>> >>> coordinates, and I think you can make the axes different using >>> >>>> matplotlib transforms, but is there anything that puts all three >>>> together currently available? >>>> >>>> Thanks, >>>> Adam >>> >>> Well, we (STScI) recently wrapped WCSLIB to obtain a mapping >>> function >>> between pixel and sky coordinates for python (you can find it as >>> pywcs in >>> astrolib on scipy; that may have been what you were referring to). >>> >>> But I'm not sure you understand what you are asking for with >>> regard to >>> matplotlib. The new transforms stuff should make it much easier to >>> display >>> the sky coordinates in the interactive display. The axis labeling >>> is a >>> different matter. Suppose your image (let's say it's 1Kx1K for the >>> sake of >>> discussion) is rotated 45 degrees with regard to north (either >>> way, it >>> doesn't really matter). What would you expect to see for axis >>> labels? I >>> don't think it is at all obvious how people would want labeling to >>> be done >>> along the edges of the image. I can imagine someone wanting axes >>> or grids >>> superimposed on the image itself, but that's not quite the same >>> thing. Do >>> you want the image rotated so that it is resampled on to RA and >>> Dec and >>> displayed that way? >>> >>> In any event, no we haven't yet done anything to try to integrate >>> all three >>> things. Among other things we wanted to make sure that the api for >>> the wcs >>> info was suitable before doing a lot with it (and in the meantime, >>> Mike is >>> working on rewriting drizzle which is taking a lot of his time). >>> >>> Perry >>> _______________________________________________ >>> SciPy-user mailing list >>> SciPy-user at scipy.org >>> http://projects.scipy.org/mailman/listinfo/scipy-user >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://lists.astropy.scipy.org/mailman/listinfo/astropy > > **** Please Update my e-mail address to my *NEW* OneNASA address > below **** > -- > Dr. William T."Tom" Bridgman Scientific Visualization > Studio > Global Science & Technology, Inc. NASA/Goddard Space Flight > Center > Email: William.T.Bridgman at nasa.gov Code 610.3 > Phone: 301-286-1346 Greenbelt, MD 20771 > FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://lists.astropy.scipy.org/mailman/listinfo/astropy **** Please Update my e-mail address to my *NEW* OneNASA address below **** -- Dr. William T."Tom" Bridgman Scientific Visualization Studio Global Science & Technology, Inc. NASA/Goddard Space Flight Center Email: William.T.Bridgman at nasa.gov Code 610.3 Phone: 301-286-1346 Greenbelt, MD 20771 FAX: 301-286-1634 http://svs.gsfc.nasa.gov/ From jturner at gemini.edu Wed Apr 2 14:35:08 2008 From: jturner at gemini.edu (James Turner) Date: Wed, 02 Apr 2008 14:35:08 -0400 Subject: [AstroPy] [SciPy-user] FITS images with header-supplied axes? In-Reply-To: <04DF97AA-4E62-408C-81BD-EEC155DBC7F7@nasa.gov> References: <200804011041.02702.humufr@yahoo.fr> <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> <04DF97AA-4E62-408C-81BD-EEC155DBC7F7@nasa.gov> Message-ID: <47F3D1DC.5080008@gemini.edu> Hi everyone, I agree that it would be good for the community to get into the habit of exposing work on astronomical Python modules via SciPy (for example). It seems that the obvious place for this is "AstroLib", which has been part of the SciPy site for some time, but doesn't seem to have gathered a lot of momentum yet: http://scipy.org/AstroLib I'm also interested in WCS, as critical functionality that hasn't quite crystallized yet. Wrapping WCSLib did seem like an obvious direction when I first looked into this and talked to Perry, but it seemed that there may be licensing issues to investigate, with it being GPL (which I personally like, but I'm not sure about internal policy on relicensing everything). At Gemini, one thing we have been working on is a data access class that takes care of some logistics like figuring out the "type" (meaning instrument, for example, not numerical type) of data, accessing standard header information in an instrument-agnostic way, iterating over things like nod sets, FITS extensions and so on. In future, the class may also support array operations with automatic propagation of things like variance and header info. This is still really alpha code at the moment though and I'm not announcing it officially. Later in the year, we will be working on some generic spectroscopic data reduction tasks. These things are planned to operate both in a pipeline context and interactively. I also have a little script for mosaicing IFU data that might be of interest to someone, but it still needs converting to NumPy. That will probably work its way into the official Gemini DR package at some point. By the way, those of us at AURA sites will be meeting in June to discuss collaboration on new DR infrastructure (in general). Cheers, James. -- James E.H. Turner Gemini Observatory Southern Operations Centre, Casilla 603, Tel. (+56) 51 205609 La Serena, Chile. Fax. (+56) 51 205650 From perry at stsci.edu Wed Apr 2 16:00:33 2008 From: perry at stsci.edu (Perry Greenfield) Date: Wed, 2 Apr 2008 16:00:33 -0400 Subject: [AstroPy] [SciPy-user] FITS images with header-supplied axes? In-Reply-To: <47F3D1DC.5080008@gemini.edu> References: <200804011041.02702.humufr@yahoo.fr> <848B8309-7988-4CC7-9DB2-C6C851C273FD@nasa.gov> <04DF97AA-4E62-408C-81BD-EEC155DBC7F7@nasa.gov> <47F3D1DC.5080008@gemini.edu> Message-ID: <3A8958DB-498E-4134-9FFB-E3E13413F649@stsci.edu> On Apr 2, 2008, at 2:35 PM, James Turner wrote: > Hi everyone, > > I agree that it would be good for the community to get into the > habit of exposing work on astronomical Python modules via SciPy > (for example). It seems that the obvious place for this is "AstroLib", > which has been part of the SciPy site for some time, but doesn't seem > to have gathered a lot of momentum yet: > > http://scipy.org/AstroLib > > I'm also interested in WCS, as critical functionality that hasn't > quite crystallized yet. Wrapping WCSLib did seem like an obvious > direction when I first looked into this and talked to Perry, but it > seemed that there may be licensing issues to investigate, with it > being GPL (which I personally like, but I'm not sure about internal > policy on relicensing everything). > Turns out the license for WCSLIB was changed to LGPL so it's no longer a problem. So the pywcs that is on astrolib is still BSD (though I haven't checked to see if it has a license along with it, I'll have that done). Perry From rwagner at physics.ucsd.edu Thu Apr 10 22:21:39 2008 From: rwagner at physics.ucsd.edu (Rick Wagner) Date: Fri, 11 Apr 2008 04:21:39 +0200 Subject: [AstroPy] AstroGrid Python Message-ID: <8EB20EAF-BFDF-450D-A1C0-3D2C6E475C2D@physics.ucsd.edu> Hi, I wanted to point AstroGrid's Python package, which uses the Virtual Observatory for searching web services for data, querying catalogs, and handling data (amongst other things). Other than Peter Erwin's code, I haven't seen much talk about VO tools, so I wanted to get this out there. If there's interest in using the VO, I could write up a summary page on the wiki. And I'm betting some of the Space Telescope developers are up on the VO, too. There's an introduction on the AstroGrid site; at the bottom of the page you'll find more complete examples. http://www.astrogrid.org/wiki/Help/IntroScripting/AstrogridPython I recommend following Getting Started instructions closely, especially the statement "you need the AstroGrid VO Desktop running". VO Desktop is another client tool built by the AstroGrid folks, and they're designed to work together. It's pretty cool, unless you forget to start VO Desktop first. In fact, you may want to start by looking at the AstroGrid Desktop page. http://www.astrogrid.org/wiki/Home/AboutAGDesktop --Rick ------------------------------------------------------------------------ - Rick Wagner, Graduate Student Researcher UCSD Physics 9500 Gilman Drive La Jolla, CA 92093-0424 Email: rwagner at physics.ucsd.edu WWW: http://lca.ucsd.edu/projects/rpwagner (858) 822-4784 Phone ------------------------------------------------------------------------ - Measuring programming progress by lines of code is like measuring aircraft building progress by weight. --Bill Gates ------------------------------------------------------------------------ - From erwin at mpe.mpg.de Sun Apr 13 10:43:54 2008 From: erwin at mpe.mpg.de (Peter Erwin) Date: Sun, 13 Apr 2008 16:43:54 +0200 Subject: [AstroPy] New version of telarchive + fetchsdss Message-ID: Hi, This is an announcement of an updated version (1.4) of my quick-and- dirty, Python-based telescope archive search tool, available here: http://www.mpe.mpg.de/~erwin/code/ (A short summary of what it does is appended below.) New features (and fixes): - Searches of Gemini and CFHT archives now include a short summary of instruments and # exposures. - SDSS search now uses footprint server instead of main catalog; this ensure that image fields which have no sources in the catalog (e.g., due to the presence of very large galaxies) are still found. - Restored access to SMOKA (Japanese telescope archive), which had been disabled due to a change in the archive URL. Note that the package includes a separate script called "fetchsdss", which allows for command-line retrieval of FITS (or JPEG) images from the SDSS archive via object name, coordinates, or SDSS field specification. [previous summary, updated:] "Telarchive" is a command-line program which simplifies searching various public telescope archives to see if they might have data on a particular astronomical object or part of the sky. This includes the HST and general MAST archive, as well as the ESO, Gemini, UKIRT, CFHT, SMOKA (includes Subaru), and AAT ground-based archives, and imaging data from Data Release 6 (DR6) of the Sloan Digital Sky Survey. The NOAO Science Archive is also included, though it has very little publically available data at this point. The Isaac Newton Group Archive is also searched, but only for data from 2001 or earlier (newer data is handled by the new archive interface, which doesn't use standard HTTP forms and so is not accessible by this program). The program won't *get* the data for you, of course (but see "fetchsdss" for SDSS images), or even tell you very much about it -- for that, you still need to visit the individual archive web pages. But it will save you lots of clicking and typing in web-page forms if you just want to find out if there is *any* data available. An example (searching within a 2-arcminute box centered on the planetary nebula NGC 7027; searching on coordinates directly is also possible): $ telarchive "ngc 7027" 2.0 SIMBAD (Simbad 4, France): Found object coordinates: RA = 21 07 01.593, Dec = +42 14 10.18 Searching archives for ngc 7027 (RA = 21 07 01.593, dec = +42 14 10.18), with search box = 2.0 arcmin... ESO Archive: No data found. AAT Archive: No data found. NOAO Science Archive: No data found. HST Archive (at ESO): Data exists! (84 observations found) -- 12 WFPC2, 24 WF, 1 PC, 13 NICMOS, 3 FOC, 24 STIS, 13 FOS UKIRT Archive: Data exists! (943 observations found) Multimission Archive at STScI (MAST): Data exists! (66 observations found) COPERNICUS (1); FUSE (2); GALEX (29); IUE (34) ING Archive (old interface): Data exists! (577 observations found) 143 images, 391 spectra, 43 unclassified JKT -- RBS (1), FWHL (2), AGBX (102); INT -- WFC (9), PFCU (30), IDS (66), MES (17), FOS_1 (10); WHT -- TAURUS_2 (5), unknown (43), ISIS_BLUE_ARM (159), UES (58), ISIS_RED_ARM (75) Sloan Digital Sky Survey (DR6) Footprint Server: No data found. SMOKA (Subaru Mitaka Okayama Kiso Archive): Data exists! (921 observations found) 475 images and 446 spectra Subaru -- OHS/CISCO (9), Subaru -- IRCS (132), Subaru -- COMICS (331), Kiso -- 1k CCD (3), Subaru -- OHS/CISCO (29), Subaru -- IRCS (102), Subaru -- COMICS (281), Okayama -- SNG (3), Okayama -- HIDES (31) Gemini Science Archive: Data exists! (49 observations found) 4 imaging, 45 IFU NIRI (4), GMOS-N (45) CFHT Archive: Data exists! (213 observations found) BEAR (200), AOBIR (6), AOBVIS (3), GECKO (4) It's available as a gzipped tar file here: http://www.mpe.mpg.de/~erwin/code/ There are installation instructions/suggestions in the README file. (And feel free to email with questions or suggestions about it!) cheers, Peter ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)89 30000 3695 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From glenn.caltech at gmail.com Fri Apr 25 00:16:03 2008 From: glenn.caltech at gmail.com (G Jones) Date: Thu, 24 Apr 2008 21:16:03 -0700 Subject: [AstroPy] Streaming binary data format for spectra Message-ID: <38ebc70804242116t710cdc81v6e3103f6a88463bf@mail.gmail.com> Hello, I am working on radio astronomy instrumentation. In particular I have a spectrometer which dumps spectra very rapidly. I am using Python to capture this data and plot this data. In evaluating the equipment, I would like to make long running Allan variance measurements, and thus I would like to save every spectrum to disk for post processing. I realize I could dump the data to disk as just raw binary data, but if possible I would like to use an existing data format to avoid reinventing the wheel for reading and writing operations. I prefer to not aggregate everything into RAM and then dump it all do disk at once; it would be better to stream it to disk. I looked into using FITS, but it doesn't seem suited to dynamically increasing the size of the file. I tried using the pyfits streaming HDU construct, but it seemed to require specifying an initial dataset size first, then filling that dataset. NetCDF seemed like a good solution since it allows for an unlimited dimension to your data set, but I have not had any luck getting any of the python wrappers to work with this library. Does anyone have any suggestions of a data format that would fit my needs? Thanks, Glenn From rwagner at physics.ucsd.edu Fri Apr 25 00:25:03 2008 From: rwagner at physics.ucsd.edu (Rick Wagner) Date: Thu, 24 Apr 2008 21:25:03 -0700 Subject: [AstroPy] Streaming binary data format for spectra In-Reply-To: <38ebc70804242116t710cdc81v6e3103f6a88463bf@mail.gmail.com> References: <38ebc70804242116t710cdc81v6e3103f6a88463bf@mail.gmail.com> Message-ID: Hi, > I am working on radio astronomy instrumentation. In particular I have > a spectrometer which dumps spectra very rapidly. I am using Python to > capture this data and plot this data. In evaluating the equipment, I > would like to make long running Allan variance measurements, and thus > I would like to save every spectrum to disk for post processing. I > realize I could dump the data to disk as just raw binary data, but if > possible I would like to use an existing data format to avoid > reinventing the wheel for reading and writing operations. I prefer to > not aggregate everything into RAM and then dump it all do disk at > once; it would be better to stream it to disk. > I looked into using FITS, but it doesn't seem suited to dynamically > increasing the size of the file. I tried using the pyfits streaming > HDU construct, but it seemed to require specifying an initial dataset > size first, then filling that dataset. > NetCDF seemed like a good solution since it allows for an unlimited > dimension to your data set, but I have not had any luck getting any of > the python wrappers to work with this library. > Does anyone have any suggestions of a data format that would fit my > needs? I would recommend using the library that NetCDF is built on top of, HDF5 [1]. The Pytables [2] code provides a nice interface to the data, and you can get to your data from other languages, like C, C++, Fortran, etc. We use HDF5 in our group for massively (1000+ processor) simulations, and find it very fast, and the HDF5 is great to work with. If you have any questions, I'd be glad to help you get started. --Rick [1] http://hdfgroup.org/ [2] http://www.pytables.org/ ------------------------------------------------------------------------ - Rick Wagner, Graduate Student Researcher UCSD Physics 9500 Gilman Drive La Jolla, CA 92093-0424 Email: rwagner at physics.ucsd.edu WWW: http://lca.ucsd.edu/projects/rpwagner (858) 822-4784 Phone ------------------------------------------------------------------------ - Measuring programming progress by lines of code is like measuring aircraft building progress by weight. --Bill Gates ------------------------------------------------------------------------ -