From ghang.naoc at gmail.com Sun Jul 1 09:34:30 2012 From: ghang.naoc at gmail.com (gonghang.naoc) Date: Sun, 1 Jul 2012 21:34:30 +0800 Subject: [AstroPy] how to judge a point is in some rectangle or not? Message-ID: Suppose we know the coordinates of the rectangle and the point, is there a python script can handle this problem? What if the rectangle becomes a polygon? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mperrin at stsci.edu Sun Jul 1 09:36:45 2012 From: mperrin at stsci.edu (Marshall Perrin) Date: Sun, 1 Jul 2012 13:36:45 +0000 Subject: [AstroPy] how to judge a point is in some rectangle or not? In-Reply-To: References: Message-ID: <02C5A5A2-7BB0-4284-9C7F-FC64706F2211@stsci.edu> On Jul 1, 2012, at 3:34 PM, "gonghang.naoc" wrote: > Suppose we know the coordinates of the rectangle and the point, is there a python script can handle this problem? > What if the rectangle becomes a polygon? matplotlib.nxutils.pnpoly() See http://ask.scipy.org/en/topic/5-efficient-inside-polygon-test-for-an-array - Marshall > Thanks! > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From d.berry at jach.hawaii.edu Sun Jul 1 11:02:07 2012 From: d.berry at jach.hawaii.edu (David Berry) Date: Sun, 1 Jul 2012 16:02:07 +0100 Subject: [AstroPy] how to judge a point is in some rectangle or not? In-Reply-To: References: Message-ID: pyast (https://github.com/timj/starlink-pyast) has a wide range of region handling functions, including rectangles, polygons, circles, ellipses, etc. See for instance http://dsberry.github.com/starlink/node1.html#Polygon This is the python interface - see the C docs at http://www.starlink.ac.uk/docs/sun211.htx/node355.html#xref_astPolygon for more info. David On 1 July 2012 14:34, gonghang.naoc wrote: > Suppose we know the coordinates of the rectangle and the point, is there a > python script can handle this problem? > What if the rectangle becomes a polygon? > Thanks! > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > From erik.tollerud at gmail.com Mon Jul 2 20:06:39 2012 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 2 Jul 2012 17:06:39 -0700 Subject: [AstroPy] Gaussian with constant background In-Reply-To: References: Message-ID: Depending on how you think, you might also find pymodelfit (http://packages.python.org/PyModelFit/ - for full disclosure, I'm the author) useful - it has gaussian models for both 1D and 2D datasets that you can just fit like: gmod = pymodelfit.GaussianModel() gmod.fitData(x, y) And if you want a constant background as you're saying here, you could just add in an offset: gmod2 = pymodelfit.offset_model(pymodelfit.GaussianModel, 'bkgd') gmod2.bkgd = whatever your constant background should be gmod2.fitData(x, y, fixedpars='bkgd') This is all scipy.optimize functions under the hood, so it should perform pretty much the same - just rather easier to use. On Wed, Jun 27, 2012 at 1:10 PM, Vivienne Baldassare wrote: > Hello, > > I've been trying to figure out how to fit a gaussian with a constant > (non-zero) background. Aside from just subtracting off the background, I > haven't been able to do it, and googling around was not particularly > helpful. I'd appreciate any tips! > > Best, > Vivienne Baldassare > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Erik Tollerud From abhijithrajan at asu.edu Fri Jul 6 20:11:51 2012 From: abhijithrajan at asu.edu (Abhijith Rajan) Date: Fri, 6 Jul 2012 17:11:51 -0700 Subject: [AstroPy] querying Subaru database Message-ID: Hi, Does anyone on the astropy mailing list have code to request data from Subaru online database (Smoka). They do not appear to have any tools to upload lists (of star names) and locate datasets. Rather than go through ~800 stars by hand I was wondering if anyone has a convenient alternative. Thanks Abhi -------------- next part -------------- An HTML attachment was scrubbed... URL: From erwin at mpe.mpg.de Sat Jul 7 04:31:23 2012 From: erwin at mpe.mpg.de (Peter Erwin) Date: Sat, 7 Jul 2012 10:31:23 +0200 Subject: [AstroPy] querying Subaru database In-Reply-To: References: Message-ID: <5E5B1F0F-3B3C-4E1E-B201-00C9011FEE5E@mpe.mpg.de> Hi, My online archive-searching tool "telarchive", which can be found here http://www.mpe.mpg.de/~erwin/code/ includes a module which searches Smoka, though it only does this one object (or set of coordinates) at a time. Still, it shouldn't be too difficult to abstract the code and put a wrapper around it to loop through a list of targets. (Or you could run telarchive with the "--usearchive=smoka" flag, which restricts the search to that archive, and put a wrapper around that.) cheers, Peter On Jul 7, 2012, at 2:11 AM, Abhijith Rajan wrote: > Hi, > > Does anyone on the astropy mailing list have code to request data from Subaru online database (Smoka). They do not appear to have any tools to upload lists (of star names) and locate datasets. Rather than go through ~800 stars by hand I was wondering if anyone has a convenient alternative. > > Thanks > Abhi > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy ============================================================= Peter Erwin Max-Planck-Insitute for Extraterrestrial erwin at mpe.mpg.de Physics, Giessenbachstrasse tel. +49 (0)176 2481 7713 85748 Garching, Germany fax +49 (0)89 30000 3495 http://www.mpe.mpg.de/~erwin From astropy at liska.ath.cx Tue Jul 10 03:38:34 2012 From: astropy at liska.ath.cx (=?utf-8?Q?Ol=D0=B5?= Streicher) Date: Tue, 10 Jul 2012 09:38:34 +0200 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 References: Message-ID: Dear Astropy developers, astropy at liska.ath.cx (Ol? Streicher) writes: > * I would volunteer to do the packaging for Debian (and Ubuntu). I now had some time to look a bit deeper into the packaging of astropy; is the git repository. The package is not yet ready to use, however. I have now some questions and comment on the package: pyfits ====== * Will astropy.io.fits be an integral part of astropy, and pyfits remain just as a legacy (just like astropy.wcs resp. pywcs)? Or does astropy just include the latest version of pyfits? * astropy.io.fits also contains some third-party-library code (libz) that could be replaced by linking to the library. Many Linux distributions unbundle this -- see [1] for Fedora Linux (base of Scientific Linux) and [2] for Debian (and Ubuntu). I think that the astropy package is a good opportunity to make the code cleaner here and move this external code to cextern/ and having some options to use external libraries instead (like the wcslib/pywcs issue I already mentioned). Packages ======== * In the moment, I would just build astropy as one large package. I think that it does not make sense to split it up into smaller packages. The documentation would be separated, however (it is not built yet). * Since also legacy packages for pyfits, pywcs and vo are build (what about asciitable?), I would create for each of them a small legacy package that depends on the main package and provides the legacy namespace. However, the legacy namespace is only installed if the original is not found, which makes a package creation a bit instable. It would be nice if the legacy package creation could be forced even if the original package is installed. I could do this however by patching setup_helpers.py. Best regards Ole From embray at stsci.edu Tue Jul 10 10:43:38 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 10 Jul 2012 10:43:38 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: References: Message-ID: <4FFC3F9A.2060809@stsci.edu> Hi, thanks for your work on the Debian packaging. A few replies to your questions inline below: On 07/10/2012 03:38 AM, Ol? Streicher wrote: > Dear Astropy developers, > > astropy at liska.ath.cx (Ol? Streicher) writes: >> * I would volunteer to do the packaging for Debian (and Ubuntu). > > I now had some time to look a bit deeper into the packaging of astropy; > > is the git repository. The package is not yet ready to use, however. > > I have now some questions and comment on the package: > > pyfits > ====== > > * Will astropy.io.fits be an integral part of astropy, and pyfits remain > just as a legacy (just like astropy.wcs resp. pywcs)? Or does astropy > just include the latest version of pyfits? You can consider astropy.io.fits an eventual replacement for pyfits, though pyfits by itself will still be in development for a while. I am trying to time things so that astropy.io.fits is as close as possible to an actual release of pyfits. For that matter, it would be a good idea to include something in the documentation to this effect. To be specific: Astropy 0.1 can be considered a replacement for pyfits 3.1 (which is not yet released, but won't be much different). > * astropy.io.fits also contains some third-party-library code (libz) > that could be replaced by linking to the library. Many Linux > distributions unbundle this -- see [1] for Fedora Linux (base of > Scientific Linux) and [2] for Debian (and Ubuntu). I think that the > astropy package is a good opportunity to make the code cleaner here > and move this external code to cextern/ and having some options to use > external libraries instead (like the wcslib/pywcs issue I already > mentioned). Agreed. In addition to libz, io.fits bundles a few *pieces* of cfitsio, but not the entirety of it. Would you want to unbundle that too? In any case, moving it to cextern as you suggest. (I don't remember the conclusion of the wcslib dicussion--was it going to be moved into cextern as well?) The only problem I see is that in some of the zlib and cfitsio files, function names were renamed with a _pyfits_ prefix to prevent binding those symbols to the system versions of those libraries at runtime. I would need some way to resolve this--perhaps a define in a header file somewhere. Any ideas? > Packages > ======== > > * In the moment, I would just build astropy as one large package. I > think that it does not make sense to split it up into smaller > packages. The documentation would be separated, however (it is not > built yet). Agreed--breaking astropy into multiple packages I think would eliminate the point. Especially considering future goals to make all the sub-packages more cohesive with each other. > * Since also legacy packages for pyfits, pywcs and vo are build (what > about asciitable?), I would create for each of them a small legacy > package that depends on the main package and provides the legacy > namespace. However, the legacy namespace is only installed if the > original is not found, which makes a package creation a bit > instable. It would be nice if the legacy package creation could be > forced even if the original package is installed. I could do this > however by patching setup_helpers.py. I'm not sure I understand entirely--if I install the python-pyfits Debian package (or whatever it's called), and then install python-astropy, if it forces installation of the legacy shim for 'pyfits' that would break python-pyfits. Or are you suggesting that python-astropy would specify that it *replaces* python-pyfits and then have it force installation of the legacy shim? I would caution against that, since it might be replacing python-pyfits with a different, slightly incompatible version of pyfits. Are there any packages that depend on python-pyfits that this could break? Thanks, Erik From astropy at liska.ath.cx Tue Jul 10 11:33:43 2012 From: astropy at liska.ath.cx (=?utf-8?Q?Ol=D0=B5?= Streicher) Date: Tue, 10 Jul 2012 17:33:43 +0200 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 References: <4FFC3F9A.2060809@stsci.edu> Message-ID: Hi Erik, thank you for your fast and extensive answer! Erik Bray writes: >> * astropy.io.fits also contains some third-party-library code (libz) >> that could be replaced by linking to the library. Many Linux >> distributions unbundle this -- see [1] for Fedora Linux (base of >> Scientific Linux) and [2] for Debian (and Ubuntu). I think that the >> astropy package is a good opportunity to make the code cleaner here >> and move this external code to cextern/ and having some options to use >> external libraries instead (like the wcslib/pywcs issue I already >> mentioned). > > Agreed. In addition to libz, io.fits bundles a few *pieces* of cfitsio, > but not the entirety of it. Would you want to unbundle that too? Good question. I just had a look how pyfits was packaged in Debian and Fedora and found that they unbundle only zlib, so I concentrated on that. But since the part from fitsio seems not very small (a ~220k file), I would prefer to unbundle it as well -- at least if it is used mainly unchanged. > In any case, moving it to cextern as you suggest. (I don't remember > the conclusion of the wcslib dicussion--was it going to be moved into > cextern as well?) As far as I know, there is no (documented) decision on that yet. > The only problem I see is that in some of the zlib and cfitsio files, > function names were renamed with a _pyfits_ prefix to prevent binding > those symbols to the system versions of those libraries at runtime. I > would need some way to resolve this--perhaps a define in a header file > somewhere. Any ideas? Maybe this may be solved by the link order? Or re-#define these symbols in the cfitsio resp. zlib header files in io/fits/src? >> * In the moment, I would just build astropy as one large package. I >> think that it does not make sense to split it up into smaller >> packages. The documentation would be separated, however (it is not >> built yet). > > Agreed--breaking astropy into multiple packages I think would eliminate > the point. Especially considering future goals to make all the > sub-packages more cohesive with each other. OK. There may be a split of the architecture independent files (python scripts) from the architecture dependent ones (shared libs), but this is still in discussion yet. It will be also mainly invisible from the outside. >> * Since also legacy packages for pyfits, pywcs and vo are build (what >> about asciitable?), I would create for each of them a small legacy >> package that depends on the main package and provides the legacy >> namespace. However, the legacy namespace is only installed if the >> original is not found, which makes a package creation a bit >> instable. It would be nice if the legacy package creation could be >> forced even if the original package is installed. I could do this >> however by patching setup_helpers.py. > > I'm not sure I understand entirely--if I install the python-pyfits > Debian package (or whatever it's called), and then install > python-astropy, if it forces installation of the legacy shim for > 'pyfits' that would break python-pyfits. The following packages would be created: * python-astropy (without the directories dist-packages/pyfits/, dist-packages/pywcs/, dist-packages/vo/) * python-astropy-pyfits - contains the files in dist-packages/pyfits/ from astropy - depends on python-astropy - conflicts with the original "python-pyfits" package - provides a "pyfits" module (like python-pyfits) * python-astropy-pywcs, python-astropy-votable similarly Then one can just install python-astropy parallel to python-pyfits, and "import pyfits" would import the original (separate) package. Or one can remove the python-pyfits package and install the python-astropy-pyfits package; then "import pyfits" would use the legacy code from astropy. Both python-pyfits and python-astropy-pyfits cannot be installed at the same time; this is prevented by the "conflicts" attribute. I however need to check whether this fits into the Debian policy. But the machine that creates the packages has nothing to do with the machine where it is finally installed (Debian provides ready-to-use binary packages). So, I would like to be able to *create* the pyfits, pywcs and votable legacy packages even if the original ones are already installed on my build machine -- I will just put the files in the according package, and not use it here directly. The patch for this is quite simple, but a documented switch would be more robust. > Or are you suggesting that python-astropy would specify that it > *replaces* python-pyfits and then have it force installation of the > legacy shim? I would caution against that, since it might be replacing > python-pyfits with a different, slightly incompatible version of pyfits. > Are there any packages that depend on python-pyfits that this could break? I would vote for keeping the default as it is, since for most users the current way is optimal. It would just be nice to have some special interface for us packagers :-) Best regards Ole From tim.jenness at gmail.com Tue Jul 10 11:43:48 2012 From: tim.jenness at gmail.com (Tim Jenness) Date: Tue, 10 Jul 2012 08:43:48 -0700 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC3F9A.2060809@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> Message-ID: On Tue, Jul 10, 2012 at 7:43 AM, Erik Bray wrote: > Hi, thanks for your work on the Debian packaging. A few replies to your > questions inline below: > > Agreed. In addition to libz, io.fits bundles a few *pieces* of cfitsio, > but not the entirety of it. Would you want to unbundle that too? In > any case, moving it to cextern as you suggest. (I don't remember the > conclusion of the wcslib dicussion--was it going to be moved into > cextern as well?) > > The only problem I see is that in some of the zlib and cfitsio files, > function names were renamed with a _pyfits_ prefix to prevent binding > those symbols to the system versions of those libraries at runtime. I > would need some way to resolve this--perhaps a define in a header file > somewhere. Any ideas? When Ole put the AST library on debian AST initially came bundled with SOFA and PAL libraries. The way we dealt with that was with an include file that redefined the function names when we were using the private versions but could use the normal names when linking against the installed debian versions. There was a switch to the configure script for people wanting to link against the installed versions. -- Tim Jenness From sergiopr at fis.ucm.es Tue Jul 10 11:56:12 2012 From: sergiopr at fis.ucm.es (Sergio Pascual) Date: Tue, 10 Jul 2012 17:56:12 +0200 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC3F9A.2060809@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> Message-ID: Hi, I'm the Fedora packager of pyfits. I wasn't aware that pyfits bundles bits of cfitsio. What is funny, because cfitsio also bundles zlib... I would like to remove the cfisio bits and use the system cfitsio library , but I'm not sure if it is possible. If the current cfistio API is incompatible with pyfits it can be hard to accomplish. Is it documented somewhere which files were included from cfitsio, from which version and what changes, if any, were applied to them? Regards, Sergio 2012/7/10 Erik Bray : > Hi, thanks for your work on the Debian packaging. A few replies to your > questions inline below: > > On 07/10/2012 03:38 AM, Ol? Streicher wrote: >> Dear Astropy developers, >> >> astropy at liska.ath.cx (Ol? Streicher) writes: >>> * I would volunteer to do the packaging for Debian (and Ubuntu). >> >> I now had some time to look a bit deeper into the packaging of astropy; >> >> is the git repository. The package is not yet ready to use, however. >> >> I have now some questions and comment on the package: >> >> pyfits >> ====== >> >> * Will astropy.io.fits be an integral part of astropy, and pyfits remain >> just as a legacy (just like astropy.wcs resp. pywcs)? Or does astropy >> just include the latest version of pyfits? > > You can consider astropy.io.fits an eventual replacement for pyfits, > though pyfits by itself will still be in development for a while. I am > trying to time things so that astropy.io.fits is as close as possible to > an actual release of pyfits. For that matter, it would be a good idea > to include something in the documentation to this effect. > > To be specific: Astropy 0.1 can be considered a replacement for pyfits > 3.1 (which is not yet released, but won't be much different). > >> * astropy.io.fits also contains some third-party-library code (libz) >> that could be replaced by linking to the library. Many Linux >> distributions unbundle this -- see [1] for Fedora Linux (base of >> Scientific Linux) and [2] for Debian (and Ubuntu). I think that the >> astropy package is a good opportunity to make the code cleaner here >> and move this external code to cextern/ and having some options to use >> external libraries instead (like the wcslib/pywcs issue I already >> mentioned). > > Agreed. In addition to libz, io.fits bundles a few *pieces* of cfitsio, > but not the entirety of it. Would you want to unbundle that too? In > any case, moving it to cextern as you suggest. (I don't remember the > conclusion of the wcslib dicussion--was it going to be moved into > cextern as well?) > > The only problem I see is that in some of the zlib and cfitsio files, > function names were renamed with a _pyfits_ prefix to prevent binding > those symbols to the system versions of those libraries at runtime. I > would need some way to resolve this--perhaps a define in a header file > somewhere. Any ideas? > >> Packages >> ======== >> >> * In the moment, I would just build astropy as one large package. I >> think that it does not make sense to split it up into smaller >> packages. The documentation would be separated, however (it is not >> built yet). > > Agreed--breaking astropy into multiple packages I think would eliminate > the point. Especially considering future goals to make all the > sub-packages more cohesive with each other. > >> * Since also legacy packages for pyfits, pywcs and vo are build (what >> about asciitable?), I would create for each of them a small legacy >> package that depends on the main package and provides the legacy >> namespace. However, the legacy namespace is only installed if the >> original is not found, which makes a package creation a bit >> instable. It would be nice if the legacy package creation could be >> forced even if the original package is installed. I could do this >> however by patching setup_helpers.py. > > I'm not sure I understand entirely--if I install the python-pyfits > Debian package (or whatever it's called), and then install > python-astropy, if it forces installation of the legacy shim for > 'pyfits' that would break python-pyfits. > > Or are you suggesting that python-astropy would specify that it > *replaces* python-pyfits and then have it force installation of the > legacy shim? I would caution against that, since it might be replacing > python-pyfits with a different, slightly incompatible version of pyfits. > Are there any packages that depend on python-pyfits that this could break? > > Thanks, > > Erik > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- Sergio Pascual http://guaix.fis.ucm.es/~spr +34 91 394 5018 gpg fingerprint: 5203 B42D 86A0 5649 410A F4AC A35F D465 F263 BCCC Departamento de Astrof?sica -- Universidad Complutense de Madrid (Spain) From embray at stsci.edu Tue Jul 10 12:46:42 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 10 Jul 2012 12:46:42 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: References: <4FFC3F9A.2060809@stsci.edu> Message-ID: <4FFC5C72.7040205@stsci.edu> The issue with CFITSIO in pyfits is confusing. Pyfits only uses pieces of it to support tile-compression--everything it needs from CFITSIO to do that was cherry-picked into a single file called fitsio.c (the 220K file mentioned elsewhere in this thread). It sort of makes sense that it would do that, as CFITSIO in its entirety (including documentation) is about 13 MB uncompressed, and most of that is not needed in pyfits. fitsio.c does document which source files in the original CFITSIO that functions were cherry-picked from. However it also states that these functions were "copied and modified" without specifying how or why they were modified. Maybe someone else at STScI has a better handle on the history of that. There have been a couple big fixes to the code in fitsio.c that don't seem to have been upstreamed to CFITSIO. Finally, I don't know exactly what version of CFITSIO the code in pyfits was based on, but it hasn't been updated in 3 years. It might be worthwhile for me to update/refactor this a bit so that there's a better record of what CFITSIO version is used, and what has been changed. I think the cherry-picking approach still makes sense for pyfits/astropy since CFITSIO is a large-ish library. I think cherry-picking can still work while also supporting linking against a system version of CFITSIO. As for zlib, CFITSIO does include its own copy. I don't know how the Debian package for it is handling that. Seems like a bit of a mess worth trying to unravel. Erik On 07/10/2012 11:56 AM, Sergio Pascual wrote: > Hi, I'm the Fedora packager of pyfits. I wasn't aware that pyfits > bundles bits of cfitsio. What is funny, because cfitsio also bundles > zlib... > > I would like to remove the cfisio bits and use the system cfitsio > library , but I'm not sure if it is possible. If the current cfistio > API is incompatible with pyfits it can be hard to accomplish. > > Is it documented somewhere which files were included from cfitsio, > from which version and what changes, if > any, were applied to them? > > > Regards, Sergio > > 2012/7/10 Erik Bray: >> Hi, thanks for your work on the Debian packaging. A few replies to your >> questions inline below: >> >> On 07/10/2012 03:38 AM, Ol? Streicher wrote: >>> Dear Astropy developers, >>> >>> astropy at liska.ath.cx (Ol? Streicher) writes: >>>> * I would volunteer to do the packaging for Debian (and Ubuntu). >>> >>> I now had some time to look a bit deeper into the packaging of astropy; >>> >>> is the git repository. The package is not yet ready to use, however. >>> >>> I have now some questions and comment on the package: >>> >>> pyfits >>> ====== >>> >>> * Will astropy.io.fits be an integral part of astropy, and pyfits remain >>> just as a legacy (just like astropy.wcs resp. pywcs)? Or does astropy >>> just include the latest version of pyfits? >> >> You can consider astropy.io.fits an eventual replacement for pyfits, >> though pyfits by itself will still be in development for a while. I am >> trying to time things so that astropy.io.fits is as close as possible to >> an actual release of pyfits. For that matter, it would be a good idea >> to include something in the documentation to this effect. >> >> To be specific: Astropy 0.1 can be considered a replacement for pyfits >> 3.1 (which is not yet released, but won't be much different). >> >>> * astropy.io.fits also contains some third-party-library code (libz) >>> that could be replaced by linking to the library. Many Linux >>> distributions unbundle this -- see [1] for Fedora Linux (base of >>> Scientific Linux) and [2] for Debian (and Ubuntu). I think that the >>> astropy package is a good opportunity to make the code cleaner here >>> and move this external code to cextern/ and having some options to use >>> external libraries instead (like the wcslib/pywcs issue I already >>> mentioned). >> >> Agreed. In addition to libz, io.fits bundles a few *pieces* of cfitsio, >> but not the entirety of it. Would you want to unbundle that too? In >> any case, moving it to cextern as you suggest. (I don't remember the >> conclusion of the wcslib dicussion--was it going to be moved into >> cextern as well?) >> >> The only problem I see is that in some of the zlib and cfitsio files, >> function names were renamed with a _pyfits_ prefix to prevent binding >> those symbols to the system versions of those libraries at runtime. I >> would need some way to resolve this--perhaps a define in a header file >> somewhere. Any ideas? >> >>> Packages >>> ======== >>> >>> * In the moment, I would just build astropy as one large package. I >>> think that it does not make sense to split it up into smaller >>> packages. The documentation would be separated, however (it is not >>> built yet). >> >> Agreed--breaking astropy into multiple packages I think would eliminate >> the point. Especially considering future goals to make all the >> sub-packages more cohesive with each other. >> >>> * Since also legacy packages for pyfits, pywcs and vo are build (what >>> about asciitable?), I would create for each of them a small legacy >>> package that depends on the main package and provides the legacy >>> namespace. However, the legacy namespace is only installed if the >>> original is not found, which makes a package creation a bit >>> instable. It would be nice if the legacy package creation could be >>> forced even if the original package is installed. I could do this >>> however by patching setup_helpers.py. >> >> I'm not sure I understand entirely--if I install the python-pyfits >> Debian package (or whatever it's called), and then install >> python-astropy, if it forces installation of the legacy shim for >> 'pyfits' that would break python-pyfits. >> >> Or are you suggesting that python-astropy would specify that it >> *replaces* python-pyfits and then have it force installation of the >> legacy shim? I would caution against that, since it might be replacing >> python-pyfits with a different, slightly incompatible version of pyfits. >> Are there any packages that depend on python-pyfits that this could break? >> >> Thanks, >> >> Erik >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > > From embray at stsci.edu Tue Jul 10 12:49:44 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 10 Jul 2012 12:49:44 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: References: <4FFC3F9A.2060809@stsci.edu> Message-ID: <4FFC5D28.5000906@stsci.edu> On 07/10/2012 11:43 AM, Tim Jenness wrote: >> The only problem I see is that in some of the zlib and cfitsio files, >> function names were renamed with a _pyfits_ prefix to prevent binding >> those symbols to the system versions of those libraries at runtime. I >> would need some way to resolve this--perhaps a define in a header file >> somewhere. Any ideas? > > When Ole put the AST library on debian AST initially came bundled with > SOFA and PAL libraries. The way we dealt with that was with an include > file that redefined the function names when we were using the private > versions but could use the normal names when linking against the > installed debian versions. There was a switch to the configure script > for people wanting to link against the installed versions. That seems like the way to go. A define flag set at compile time could switch this on or off--a simple change to make for a Debian package. Erik From embray at stsci.edu Tue Jul 10 12:52:24 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 10 Jul 2012 12:52:24 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC5C72.7040205@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> Message-ID: <4FFC5DC8.3030605@stsci.edu> On 07/10/2012 12:46 PM, Erik Bray wrote: > fitsio.c does document which source files in the original CFITSIO that > functions were cherry-picked from. However it also states that these > functions were "copied and modified" without specifying how or why they > were modified. Maybe someone else at STScI has a better handle on the > history of that. There have been a couple big fixes to the code in > fitsio.c that don't seem to have been upstreamed to CFITSIO. I'm sorry, that previous sentence should read "a couple bug fixes". The few fixes I can find recorded in the SVN history are actually quite small. Erik From tim.jenness at gmail.com Tue Jul 10 13:02:24 2012 From: tim.jenness at gmail.com (Tim Jenness) Date: Tue, 10 Jul 2012 10:02:24 -0700 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC5C72.7040205@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> Message-ID: On Tue, Jul 10, 2012 at 9:46 AM, Erik Bray wrote: > Finally, I don't know exactly what version of CFITSIO the code in pyfits > was based on, but it hasn't been updated in 3 years. > > It might be worthwhile for me to update/refactor this a bit so that > there's a better record of what CFITSIO version is used, and what has > been changed. I think the cherry-picking approach still makes sense for > pyfits/astropy since CFITSIO is a large-ish library. I think > cherry-picking can still work while also supporting linking against a > system version of CFITSIO. > If it helps, I have an unofficial cfitsio git mirror at git://starlink.jach.hawaii.edu/cfitsio.git cfitsio-git - The vendor branch has every tar ball release since about v3.06 in it. I built it from the distribution tar files since I needed a git cfitsio for Starlink. Might help you work out which version your files originated from (once you've split them up again). You might want to consider shipping the cfitsio files you need in separate files to make it easier to keep upstream fixes available. You could also consider having a git submodule for cfitsio code where you have a pyfits branch that only retains the files you need. That would make it even easier to track upstream. I would be happy to push my cfitsio repository to github (I was trying to do that anyhow). -- Tim Jenness From jturner at gemini.edu Tue Jul 10 13:22:16 2012 From: jturner at gemini.edu (James Turner) Date: Tue, 10 Jul 2012 13:22:16 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC5DC8.3030605@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> <4FFC5DC8.3030605@stsci.edu> Message-ID: <4FFC64C8.1030205@gemini.edu> I'm just wondering about the motivation for bundling zlib. I looked into that issue fairly recently (not exhaustively but probably enough) and concluded that basically all Linux and Apple desktop installs come with zlib. In particular I believe the LSB requires it (though of course only certain distributions follow that). Is ABI stability a big concern even for something like zlib? Probably Erik, Ole et al. have a much better perspective on this than I have, but I thought it worth asking the question just to make sure there's a good answer ;-). Maybe it's just one less dependency to worry about whether it might break or not. Cheers, James. From embray at stsci.edu Tue Jul 10 13:38:14 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 10 Jul 2012 13:38:14 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC64C8.1030205@gemini.edu> References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> <4FFC5DC8.3030605@stsci.edu> <4FFC64C8.1030205@gemini.edu> Message-ID: <4FFC6886.8070804@stsci.edu> On 07/10/2012 01:22 PM, James Turner wrote: > I'm just wondering about the motivation for bundling zlib. I > looked into that issue fairly recently (not exhaustively but > probably enough) and concluded that basically all Linux and > Apple desktop installs come with zlib. In particular I believe > the LSB requires it (though of course only certain distributions > follow that). Is ABI stability a big concern even for something > like zlib? Probably Erik, Ole et al. have a much better > perspective on this than I have, but I thought it worth asking > the question just to make sure there's a good answer ;-). Maybe > it's just one less dependency to worry about whether it might > break or not. > > Cheers, > > James. I don't think ABI compatibility is as much a concern--I would think that's pretty stable (though I don't know). Also I don't think the availability of the library itself is of as much concern on *NIX systems. But on *Windows* it is very much necessary. Also, even if the library is installed, I'm not sure if the headers are always guaranteed to be installed. My guess would be that on most systems the headers for zlib are installed by default (they are on the OSX machines I have access to), but I don't know how guaranteed that is. But in any case, for the sake of Windows support (sans Cygwin, mingw, etc.) removing it outright is a non-starter I think. Erik From sergiopr at fis.ucm.es Tue Jul 10 13:52:38 2012 From: sergiopr at fis.ucm.es (Sergio Pascual) Date: Tue, 10 Jul 2012 19:52:38 +0200 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFC6886.8070804@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> <4FFC5DC8.3030605@stsci.edu> <4FFC64C8.1030205@gemini.edu> <4FFC6886.8070804@stsci.edu> Message-ID: In Linux, is common to have headers in a different precompilled package (zlib-devel.rpm vs zlib.rpm in Fedora, for example), which utterly confuses unaware users. 2012/7/10 Erik Bray : > On 07/10/2012 01:22 PM, James Turner wrote: >> I'm just wondering about the motivation for bundling zlib. I >> looked into that issue fairly recently (not exhaustively but >> probably enough) and concluded that basically all Linux and >> Apple desktop installs come with zlib. In particular I believe >> the LSB requires it (though of course only certain distributions >> follow that). Is ABI stability a big concern even for something >> like zlib? Probably Erik, Ole et al. have a much better >> perspective on this than I have, but I thought it worth asking >> the question just to make sure there's a good answer ;-). Maybe >> it's just one less dependency to worry about whether it might >> break or not. >> >> Cheers, >> >> James. > > I don't think ABI compatibility is as much a concern--I would think > that's pretty stable (though I don't know). Also I don't think the > availability of the library itself is of as much concern on *NIX > systems. But on *Windows* it is very much necessary. > > Also, even if the library is installed, I'm not sure if the headers are > always guaranteed to be installed. My guess would be that on most > systems the headers for zlib are installed by default (they are on the > OSX machines I have access to), but I don't know how guaranteed that is. > > But in any case, for the sake of Windows support (sans Cygwin, mingw, > etc.) removing it outright is a non-starter I think. > > Erik > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- Sergio Pascual http://guaix.fis.ucm.es/~spr +34 91 394 5018 gpg fingerprint: 5203 B42D 86A0 5649 410A F4AC A35F D465 F263 BCCC Departamento de Astrof?sica -- Universidad Complutense de Madrid (Spain) From embray at stsci.edu Tue Jul 10 17:05:53 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 10 Jul 2012 17:05:53 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: References: <4FFC3F9A.2060809@stsci.edu> Message-ID: <4FFC9931.4010305@stsci.edu> On 07/10/2012 11:33 AM, Ol? Streicher wrote: > Hi Erik, > > thank you for your fast and extensive answer! No problem :) >> Agreed. In addition to libz, io.fits bundles a few *pieces* of cfitsio, >> but not the entirety of it. Would you want to unbundle that too? > > Good question. I just had a look how pyfits was packaged in Debian and > Fedora and found that they unbundle only zlib, so I concentrated on > that. But since the part from fitsio seems not very small (a ~220k > file), I would prefer to unbundle it as well -- at least if it is used > mainly unchanged. I'm probably going to make some updates in the future to exactly how pyfits goes about including the bits of CFITSIO that it does, but in general I don't think there should be any problem with unbundling it. >> I'm not sure I understand entirely--if I install the python-pyfits >> Debian package (or whatever it's called), and then install >> python-astropy, if it forces installation of the legacy shim for >> 'pyfits' that would break python-pyfits. > > The following packages would be created: > > * python-astropy (without the directories dist-packages/pyfits/, > dist-packages/pywcs/, dist-packages/vo/) > * python-astropy-pyfits > - contains the files in dist-packages/pyfits/ from astropy > - depends on python-astropy > - conflicts with the original "python-pyfits" package > - provides a "pyfits" module (like python-pyfits) > * python-astropy-pywcs, python-astropy-votable similarly > > Then one can just install python-astropy parallel to python-pyfits, and > "import pyfits" would import the original (separate) package. Or one can > remove the python-pyfits package and install the python-astropy-pyfits > package; then "import pyfits" would use the legacy code from > astropy. Both python-pyfits and python-astropy-pyfits cannot be > installed at the same time; this is prevented by the "conflicts" > attribute. I however need to check whether this fits into the Debian > policy. > > But the machine that creates the packages has nothing to do with the > machine where it is finally installed (Debian provides ready-to-use > binary packages). So, I would like to be able to *create* the pyfits, > pywcs and votable legacy packages even if the original ones are already > installed on my build machine -- I will just put the files in the > according package, and not use it here directly. > > The patch for this is quite simple, but a documented switch would be > more robust. That all makes sense. I could see adding an install option for forcing installation of legacy packages. Feel free to add an issue report on GitHub for that; or I can add one. >> Or are you suggesting that python-astropy would specify that it >> *replaces* python-pyfits and then have it force installation of the >> legacy shim? I would caution against that, since it might be replacing >> python-pyfits with a different, slightly incompatible version of pyfits. >> Are there any packages that depend on python-pyfits that this could break? > > I would vote for keeping the default as it is, since for most users the > current way is optimal. It would just be nice to have some special > interface for us packagers :-) Agreed--this seems reasonable. Erik B. From mdroe at stsci.edu Wed Jul 11 12:13:22 2012 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 11 Jul 2012 12:13:22 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: References: <4FFC3F9A.2060809@stsci.edu> Message-ID: <4FFDA622.5070203@stsci.edu> On 07/10/2012 11:33 AM, Ol? Streicher wrote: >> In any case, moving it to cextern as you suggest. (I don't remember >> the conclusion of the wcslib dicussion--was it going to be moved into >> cextern as well?) > As far as I know, there is no (documented) decision on that yet. My plan is to (a) wait for a release of wcslib with the necessary patches, and then (b) move our local copy into cextern, and (c) provide a setup.cfg switch to use the system wcslib rather than the included one. I also plan to do (c) for expat, I just haven't got around to it yet. > > > But the machine that creates the packages has nothing to do with the > machine where it is finally installed (Debian provides ready-to-use > binary packages). So, I would like to be able to *create* the pyfits, > pywcs and votable legacy packages even if the original ones are already > installed on my build machine -- I will just put the files in the > according package, and not use it here directly. > > The patch for this is quite simple, but a documented switch would be > more robust. > PR #302 added an option to disable legacy shim building. You're suggesting here that it would also be nice to have something that *forces* legacy shim building? That should be easy enough to do following the pattern laid out in #302. Just wanted to confirm that adding a `--enable-legacy` commandline arg to setup.py that would always build the legacy shims would be helpful here. https://github.com/astropy/astropy/pull/302 Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Wed Jul 11 13:16:18 2012 From: mdroe at stsci.edu (Michael Droettboom) Date: Wed, 11 Jul 2012 13:16:18 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: <4FFDA622.5070203@stsci.edu> References: <4FFC3F9A.2060809@stsci.edu> <4FFDA622.5070203@stsci.edu> Message-ID: <4FFDB4E2.1070405@stsci.edu> Those interested in this discussion may want to look at PR #316, which makes creating options to link to system libraries easier to accomplish internally: https://github.com/astropy/astropy/pull/316 Mike On 07/11/2012 12:13 PM, Michael Droettboom wrote: > On 07/10/2012 11:33 AM, Ol? Streicher wrote: >>> In any case, moving it to cextern as you suggest. (I don't remember >>> the conclusion of the wcslib dicussion--was it going to be moved into >>> cextern as well?) >> As far as I know, there is no (documented) decision on that yet. > > My plan is to (a) wait for a release of wcslib with the necessary > patches, and then (b) move our local copy into cextern, and (c) > provide a setup.cfg switch to use the system wcslib rather than the > included one. > > I also plan to do (c) for expat, I just haven't got around to it yet. > >> >> But the machine that creates the packages has nothing to do with the >> machine where it is finally installed (Debian provides ready-to-use >> binary packages). So, I would like to be able to *create* the pyfits, >> pywcs and votable legacy packages even if the original ones are already >> installed on my build machine -- I will just put the files in the >> according package, and not use it here directly. >> >> The patch for this is quite simple, but a documented switch would be >> more robust. >> > > PR #302 added an option to disable legacy shim building. You're > suggesting here that it would also be nice to have something that > *forces* legacy shim building? That should be easy enough to do > following the pattern laid out in #302. Just wanted to confirm that > adding a `--enable-legacy` commandline arg to setup.py that would > always build the legacy shims would be helpful here. > > https://github.com/astropy/astropy/pull/302 > > Mike > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From astropy at liska.ath.cx Thu Jul 12 03:17:57 2012 From: astropy at liska.ath.cx (=?utf-8?Q?Ol=D0=B5?= Streicher) Date: Thu, 12 Jul 2012 09:17:57 +0200 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> <4FFC5DC8.3030605@stsci.edu> <4FFC64C8.1030205@gemini.edu> Message-ID: James Turner writes: > I'm just wondering about the motivation for bundling zlib. I > looked into that issue fairly recently (not exhaustively but > probably enough) and concluded that basically all Linux and > Apple desktop installs come with zlib. I would see the point in bundling zlib since at least one major system (MS Windows) does not have it on default. > In particular I believe the LSB requires it (though of course only > certain distributions follow that). LSB is very Linux specific, while one goal of astropy is being portable. > Is ABI stability a big concern even for something like zlib? I would (again) advertize the excellent "upstream-tracker" service here: This shows that the zlib ABI was stable in the last 14 years -- except for one removed function (gzflags()) in the latest version. Best Ole From jturner at gemini.edu Thu Jul 12 12:55:31 2012 From: jturner at gemini.edu (James Turner) Date: Thu, 12 Jul 2012 12:55:31 -0400 Subject: [AstroPy] [astropy-dev] ANN: Astropy v0.1 In-Reply-To: References: <4FFC3F9A.2060809@stsci.edu> <4FFC5C72.7040205@stsci.edu> <4FFC5DC8.3030605@stsci.edu> <4FFC64C8.1030205@gemini.edu> Message-ID: <4FFF0183.7000609@gemini.edu> > I would see the point in bundling zlib since at least one major system > (MS Windows) does not have it on default. Yep, I forget Windows. > I would (again) advertize the excellent "upstream-tracker" service here: > > This shows that the zlib ABI was stable in the last 14 years -- except > for one removed function (gzflags()) in the latest version. Oh, thanks for that -- looks very useful! Can even see what's coming in the next distros... James. From thomas.robitaille at gmail.com Thu Jul 26 11:54:32 2012 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Thu, 26 Jul 2012 17:54:32 +0200 Subject: [AstroPy] Montage in Python Message-ID: Hi everyone, I have been in touch with Bruce Berriman, one of the main developers of the Montage mosaicking toolkit (http://montage.ipac.caltech.edu/), who is interested in finding out how Python users are using Montage (as this may impact future directions for the package). [Note that if you use APLpy, you may be making use of Montage, so please read on!] If you indeed use or have used Montage before, please take a couple of minutes to email me at thomas.robitaille at gmail.com and let me know: - whether you use Montage via Python scripts, and if so whether you use it via the python-montage package, or via APLpy's make_rgb_cube and north=True functionality (which requires Montage), or whether you have developed your wrappers or just call the commands directly with subprocess or os.system (or other ways I haven't mentioned!) - what kind of projects you use Montage for (broadly) - whether you have developed any packages that use Montage or python-montage - whether you use Montage, but not from Python, and if so why not? Your replies, however short, would be very helpful! Thanks! Tom From ogk at astro.caltech.edu Thu Jul 26 12:54:52 2012 From: ogk at astro.caltech.edu (Oliver King) Date: Thu, 26 Jul 2012 09:54:52 -0700 Subject: [AstroPy] Pywcs not reading transformation matrix correctly Message-ID: Hi, I've discovered what I think is a bug in the pywcs module in astropy. I have a FITS file with a WCS that contains these parameters: CRVAL1 = 343.485065456 / RA of reference point CRVAL2 = 16.1466891384 / DEC of reference point CRPIX1 = 1094.26737976 / X reference pixel CRPIX2 = 1037.56068399 / Y reference pixel CUNIT1 = 'deg ' / X pixel scale units CUNIT2 = 'deg ' / Y pixel scale units CD1_1 = -7.75889930453E-05 / Transformation matrix CD1_2 = 4.5042903792E-07 / no comment CD2_1 = -3.9690360471E-07 / no comment CD2_2 = -7.76314691533E-05 / no comment Notice that the transformation matrix parameters are in exponential notation. However, when I read the WCS using astropy I get these parameters: In [14]: wcs.printwcs() WCS Keywords CD_11 CD_12: -7.7588993045299999 4.5042903790000004 CD_21 CD_22: -3.9690360400000002 -7.7631469099999997 CRVAL : 343.48506545599997 16.146689138399999 CRPIX : 1094.26737976 1037.5606839 NAXIS : 2048 2048 It has ignored the exponential notation. If I manually edit the transformation matrix parameters in the header to remove the exponential notation it reads in the WCS correctly. Is this a known bug? Is there a workaround or do I need to manually edit my FITS file headers? Cheers, Oliver ------ Oliver King Astronomy Department, Caltech +1-626-395-4197 From marquett at iap.fr Thu Jul 26 12:58:41 2012 From: marquett at iap.fr (Jean-Baptiste Marquette) Date: Thu, 26 Jul 2012 18:58:41 +0200 Subject: [AstroPy] Pywcs not reading transformation matrix correctly In-Reply-To: References: Message-ID: <76EE553F-1D74-4D8F-AB90-7EB0754B71A8@iap.fr> Hi Oliver, > Is this a known bug? Is there a workaround or do I need to manually edit my FITS file headers? What are your CTYPEn and RADECSYS ? Cheers JB -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogk at astro.caltech.edu Thu Jul 26 13:03:46 2012 From: ogk at astro.caltech.edu (Oliver King) Date: Thu, 26 Jul 2012 10:03:46 -0700 Subject: [AstroPy] Pywcs not reading transformation matrix correctly In-Reply-To: <76EE553F-1D74-4D8F-AB90-7EB0754B71A8@iap.fr> References: <76EE553F-1D74-4D8F-AB90-7EB0754B71A8@iap.fr> Message-ID: Hi JB, The WCS does not have a RADECSYS keyword; the CTYPE is TAN-SIP. Here are all the WCS parameters (excluding the polynomial coefficients). This is an image which has had a WCS attached by the astrometry.net software. CTYPE1 = 'RA---TAN-SIP' / TAN (gnomic) projection + SIP distortions CTYPE2 = 'DEC--TAN-SIP' / TAN (gnomic) projection + SIP distortions WCSAXES = 2 / no comment EQUINOX = 2000.0 / Equatorial coordinates definition (yr) LONPOLE = 180.0 / no comment LATPOLE = 0.0 / no comment CRVAL1 = 343.485065456 / RA of reference point CRVAL2 = 16.1466891384 / DEC of reference point CRPIX1 = 1094.26737976 / X reference pixel CRPIX2 = 1037.56068399 / Y reference pixel CUNIT1 = 'deg ' / X pixel scale units CUNIT2 = 'deg ' / Y pixel scale units CD1_1 = -7.75889930453E-05 / Transformation matrix CD1_2 = 4.5042903792E-07 / no comment CD2_1 = -3.9690360471E-07 / no comment CD2_2 = -7.76314691533E-05 / no comment IMAGEW = 2048 / Image width, in pixels. IMAGEH = 2048 / Image height, in pixels. Cheers, Oliver On Jul 26, 2012, at 9:58 AM, Jean-Baptiste Marquette wrote: > > Hi Oliver, > >> Is this a known bug? Is there a workaround or do I need to manually edit my FITS file headers? > > What are your CTYPEn and RADECSYS ? > > Cheers > JB ------ Oliver King Astronomy Department, Caltech +1-626-395-4197 From thomas.robitaille at gmail.com Thu Jul 26 13:10:07 2012 From: thomas.robitaille at gmail.com (Thomas Robitaille) Date: Thu, 26 Jul 2012 19:10:07 +0200 Subject: [AstroPy] Pywcs not reading transformation matrix correctly In-Reply-To: References: <76EE553F-1D74-4D8F-AB90-7EB0754B71A8@iap.fr> Message-ID: Which version of Astropy are you using? import astropy print astropy.__version__ I'm using the latest developer version, and it seems to be working: CD_11 CD_12: -7.7588993045300006e-05 4.5042903792000001e-07 CD_21 CD_22: -3.9690360471000002e-07 -7.7631469153300006e-05 CRVAL : 343.48506545599997 16.146689138399999 CRPIX : 1094.26737976 1037.5606839899999 NAXIS : 0.0 0.0 If you are using v0.1, could you try updating to the latest git version to see if it was fixed? Thanks! Tom On 26 July 2012 19:03, Oliver King wrote: > Hi JB, > > The WCS does not have a RADECSYS keyword; the CTYPE is TAN-SIP. Here are all the WCS parameters (excluding the polynomial coefficients). This is an image which has had a WCS attached by the astrometry.net software. > > CTYPE1 = 'RA---TAN-SIP' / TAN (gnomic) projection + SIP distortions > CTYPE2 = 'DEC--TAN-SIP' / TAN (gnomic) projection + SIP distortions > WCSAXES = 2 / no comment > EQUINOX = 2000.0 / Equatorial coordinates definition (yr) > LONPOLE = 180.0 / no comment > LATPOLE = 0.0 / no comment > CRVAL1 = 343.485065456 / RA of reference point > CRVAL2 = 16.1466891384 / DEC of reference point > CRPIX1 = 1094.26737976 / X reference pixel > CRPIX2 = 1037.56068399 / Y reference pixel > CUNIT1 = 'deg ' / X pixel scale units > CUNIT2 = 'deg ' / Y pixel scale units > CD1_1 = -7.75889930453E-05 / Transformation matrix > CD1_2 = 4.5042903792E-07 / no comment > CD2_1 = -3.9690360471E-07 / no comment > CD2_2 = -7.76314691533E-05 / no comment > IMAGEW = 2048 / Image width, in pixels. > IMAGEH = 2048 / Image height, in pixels. > > Cheers, > Oliver > > On Jul 26, 2012, at 9:58 AM, Jean-Baptiste Marquette wrote: > >> >> Hi Oliver, >> >>> Is this a known bug? Is there a workaround or do I need to manually edit my FITS file headers? >> >> What are your CTYPEn and RADECSYS ? >> >> Cheers >> JB > > ------ > Oliver King > Astronomy Department, Caltech > +1-626-395-4197 > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy From ogk at astro.caltech.edu Thu Jul 26 13:15:22 2012 From: ogk at astro.caltech.edu (Oliver King) Date: Thu, 26 Jul 2012 10:15:22 -0700 Subject: [AstroPy] Pywcs not reading transformation matrix correctly In-Reply-To: References: <76EE553F-1D74-4D8F-AB90-7EB0754B71A8@iap.fr> Message-ID: This might be the reason: 0.0.dev1119 I'll try updating to the latest version. On Jul 26, 2012, at 10:10 AM, Thomas Robitaille wrote: > Which version of Astropy are you using? > > import astropy > print astropy.__version__ > > I'm using the latest developer version, and it seems to be working: > > CD_11 CD_12: -7.7588993045300006e-05 4.5042903792000001e-07 > CD_21 CD_22: -3.9690360471000002e-07 -7.7631469153300006e-05 > CRVAL : 343.48506545599997 16.146689138399999 > CRPIX : 1094.26737976 1037.5606839899999 > NAXIS : 0.0 0.0 > > If you are using v0.1, could you try updating to the latest git > version to see if it was fixed? > > Thanks! > Tom > > On 26 July 2012 19:03, Oliver King wrote: >> Hi JB, >> >> The WCS does not have a RADECSYS keyword; the CTYPE is TAN-SIP. Here are all the WCS parameters (excluding the polynomial coefficients). This is an image which has had a WCS attached by the astrometry.net software. >> >> CTYPE1 = 'RA---TAN-SIP' / TAN (gnomic) projection + SIP distortions >> CTYPE2 = 'DEC--TAN-SIP' / TAN (gnomic) projection + SIP distortions >> WCSAXES = 2 / no comment >> EQUINOX = 2000.0 / Equatorial coordinates definition (yr) >> LONPOLE = 180.0 / no comment >> LATPOLE = 0.0 / no comment >> CRVAL1 = 343.485065456 / RA of reference point >> CRVAL2 = 16.1466891384 / DEC of reference point >> CRPIX1 = 1094.26737976 / X reference pixel >> CRPIX2 = 1037.56068399 / Y reference pixel >> CUNIT1 = 'deg ' / X pixel scale units >> CUNIT2 = 'deg ' / Y pixel scale units >> CD1_1 = -7.75889930453E-05 / Transformation matrix >> CD1_2 = 4.5042903792E-07 / no comment >> CD2_1 = -3.9690360471E-07 / no comment >> CD2_2 = -7.76314691533E-05 / no comment >> IMAGEW = 2048 / Image width, in pixels. >> IMAGEH = 2048 / Image height, in pixels. >> >> Cheers, >> Oliver >> >> On Jul 26, 2012, at 9:58 AM, Jean-Baptiste Marquette wrote: >> >>> >>> Hi Oliver, >>> >>>> Is this a known bug? Is there a workaround or do I need to manually edit my FITS file headers? >>> >>> What are your CTYPEn and RADECSYS ? >>> >>> Cheers >>> JB >> >> ------ >> Oliver King >> Astronomy Department, Caltech >> +1-626-395-4197 >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy ------ Oliver King Astronomy Department, Caltech +1-626-395-4197 From ogk at astro.caltech.edu Thu Jul 26 13:21:48 2012 From: ogk at astro.caltech.edu (Oliver King) Date: Thu, 26 Jul 2012 10:21:48 -0700 Subject: [AstroPy] Pywcs not reading transformation matrix correctly In-Reply-To: References: <76EE553F-1D74-4D8F-AB90-7EB0754B71A8@iap.fr> Message-ID: <56D6D1A8-4FC0-4596-8BBC-0ECCF50C4BF7@astro.caltech.edu> That fixed it - I guess that it was an already-squashed bug. Thanks for your help. Oliver On Jul 26, 2012, at 10:15 AM, Oliver King wrote: > This might be the reason: > 0.0.dev1119 > > I'll try updating to the latest version. > > On Jul 26, 2012, at 10:10 AM, Thomas Robitaille wrote: > >> Which version of Astropy are you using? >> >> import astropy >> print astropy.__version__ >> >> I'm using the latest developer version, and it seems to be working: >> >> CD_11 CD_12: -7.7588993045300006e-05 4.5042903792000001e-07 >> CD_21 CD_22: -3.9690360471000002e-07 -7.7631469153300006e-05 >> CRVAL : 343.48506545599997 16.146689138399999 >> CRPIX : 1094.26737976 1037.5606839899999 >> NAXIS : 0.0 0.0 >> >> If you are using v0.1, could you try updating to the latest git >> version to see if it was fixed? >> >> Thanks! >> Tom >> >> On 26 July 2012 19:03, Oliver King wrote: >>> Hi JB, >>> >>> The WCS does not have a RADECSYS keyword; the CTYPE is TAN-SIP. Here are all the WCS parameters (excluding the polynomial coefficients). This is an image which has had a WCS attached by the astrometry.net software. >>> >>> CTYPE1 = 'RA---TAN-SIP' / TAN (gnomic) projection + SIP distortions >>> CTYPE2 = 'DEC--TAN-SIP' / TAN (gnomic) projection + SIP distortions >>> WCSAXES = 2 / no comment >>> EQUINOX = 2000.0 / Equatorial coordinates definition (yr) >>> LONPOLE = 180.0 / no comment >>> LATPOLE = 0.0 / no comment >>> CRVAL1 = 343.485065456 / RA of reference point >>> CRVAL2 = 16.1466891384 / DEC of reference point >>> CRPIX1 = 1094.26737976 / X reference pixel >>> CRPIX2 = 1037.56068399 / Y reference pixel >>> CUNIT1 = 'deg ' / X pixel scale units >>> CUNIT2 = 'deg ' / Y pixel scale units >>> CD1_1 = -7.75889930453E-05 / Transformation matrix >>> CD1_2 = 4.5042903792E-07 / no comment >>> CD2_1 = -3.9690360471E-07 / no comment >>> CD2_2 = -7.76314691533E-05 / no comment >>> IMAGEW = 2048 / Image width, in pixels. >>> IMAGEH = 2048 / Image height, in pixels. >>> >>> Cheers, >>> Oliver >>> >>> On Jul 26, 2012, at 9:58 AM, Jean-Baptiste Marquette wrote: >>> >>>> >>>> Hi Oliver, >>>> >>>>> Is this a known bug? Is there a workaround or do I need to manually edit my FITS file headers? >>>> >>>> What are your CTYPEn and RADECSYS ? >>>> >>>> Cheers >>>> JB >>> >>> ------ >>> Oliver King >>> Astronomy Department, Caltech >>> +1-626-395-4197 >>> >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy > > ------ > Oliver King > Astronomy Department, Caltech > +1-626-395-4197 > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy ------ Oliver King Astronomy Department, Caltech +1-626-395-4197 From john.parejko at yale.edu Mon Jul 30 19:40:47 2012 From: john.parejko at yale.edu (John K. Parejko) Date: Mon, 30 Jul 2012 19:40:47 -0400 Subject: [AstroPy] reading one line from many small fits files Message-ID: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> Hey all, This is really more of a pyfits question, but I've upgraded to pyfits 3.1 (SVN), which is the version in astropy. I have data stored in thousands of ~few MB .fits files (photoObj files from SDSS) totaling a few TB of data, and I know the one single line I want to extract from some known subset of those files. But pyfits is taking more than a second per file to extract the fields I want, which seems very long, especially if it is using memmapped access, and thus should only have to read that single line (plus the header) from each file. I'm doing something like this: result = np.empty(len(data),dtype=dtype) for i,x in enumerate(data): getfilename(x[somefield]) photo = pyfits.open(photo,memmap=True) result[i] = photo[1].data[x[otherfield]-1] Is there a better way to go about this? Is pyfits known to be quite slow when reading a single row from a lot of different files? Anyone have suggestions on how to speed this up? Thanks, John From derek at astro.physik.uni-goettingen.de Mon Jul 30 20:57:12 2012 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 Jul 2012 02:57:12 +0200 Subject: [AstroPy] reading one line from many small fits files In-Reply-To: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> References: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> Message-ID: <91016FD8-9705-404D-8271-9B75DBEFD6D9@astro.physik.uni-goettingen.de> Hi John, On 31.07.2012, at 1:40AM, "John K. Parejko" wrote: > This is really more of a pyfits question, but I've upgraded to pyfits 3.1 (SVN), which is the version in astropy. > > I have data stored in thousands of ~few MB .fits files (photoObj files from SDSS) totaling a few TB of data, and I know the one single line I want to extract from some known subset of those files. But pyfits is taking more than a second per file to extract the fields I want, which seems very long, especially if it is using memmapped access, and thus should only have to read that single line (plus the header) from each file. > > I'm doing something like this: > > result = np.empty(len(data),dtype=dtype) > for i,x in enumerate(data): > getfilename(x[somefield]) > photo = pyfits.open(photo,memmap=True) > result[i] = photo[1].data[x[otherfield]-1] > > Is there a better way to go about this? Is pyfits known to be quite slow when reading a single row from a lot of different files? Anyone have suggestions on how to speed this up? that seems quite slow; it takes me about 50 ms to read a random line from the DR8 example file with pyfits 3.0.2. Unless the file access itself takes so long something appears to be odd. But the only thing coming to my mind now is that pyfits supports scaled column data (similar to BSCALE/BZERO in image HDUs, I assume), and if such keywords were present, they would probably cause a corresponding transformation for the entire bintable. They don't seem to exist in the standard SDSS files, though. Na?ve question: do you call photo.close() after each read? Cheers, Derek From embray at stsci.edu Tue Jul 31 11:52:07 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 31 Jul 2012 11:52:07 -0400 Subject: [AstroPy] reading one line from many small fits files In-Reply-To: <91016FD8-9705-404D-8271-9B75DBEFD6D9@astro.physik.uni-goettingen.de> References: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> <91016FD8-9705-404D-8271-9B75DBEFD6D9@astro.physik.uni-goettingen.de> Message-ID: <5017FF27.7080700@stsci.edu> On 07/30/2012 08:57 PM, Derek Homeier wrote: > Hi John, > > On 31.07.2012, at 1:40AM, "John K. Parejko" wrote: > >> This is really more of a pyfits question, but I've upgraded to pyfits 3.1 (SVN), which is the version in astropy. >> >> I have data stored in thousands of ~few MB .fits files (photoObj files from SDSS) totaling a few TB of data, and I know the one single line I want to extract from some known subset of those files. But pyfits is taking more than a second per file to extract the fields I want, which seems very long, especially if it is using memmapped access, and thus should only have to read that single line (plus the header) from each file. >> >> I'm doing something like this: >> >> result = np.empty(len(data),dtype=dtype) >> for i,x in enumerate(data): >> getfilename(x[somefield]) >> photo = pyfits.open(photo,memmap=True) >> result[i] = photo[1].data[x[otherfield]-1] >> >> Is there a better way to go about this? Is pyfits known to be quite slow when reading a single row from a lot of different files? Anyone have suggestions on how to speed this up? > > that seems quite slow; it takes me about 50 ms to read a random line from the DR8 example file > with pyfits 3.0.2. Unless the file access itself takes so long something appears to be odd. > But the only thing coming to my mind now is that pyfits supports scaled column data (similar to > BSCALE/BZERO in image HDUs, I assume), and if such keywords were present, they would probably > cause a corresponding transformation for the entire bintable. They don't seem to exist in the standard > SDSS files, though. > Na?ve question: do you call photo.close() after each read? > > Cheers, > Derek It probably shouldn't matter whether or not he's calling close(), but the question about BSCALE/BZERO is possibly relevant. Is the data you're reading from an image or a table? If it's an image, then as Derek wrote PyFITS is still pretty inefficient, in that it will transform the entire image, even if using mmap (which is the default now by the way). I have plans for overhauling this but it hasn't been a high priority for the most part. You can also turn off image scaling if you use do_not_scale_image_data=True when opening the file. That might speed things up. This is one area where using the .section feature on Image HDUs might still be useful. For example: result[i] = photo[1].section[x[otherfield] - 1] In PyFITS 3.1 that has improved support for scaling just sections of the file, which didn't work well before. That might also be faster. Of course this is all a moot point if this is not a scaled image. In any case, opening a file and reading a single row out of the data should not generally take as long as 1 second--especially if they're small files. Erik From aldcroft at head.cfa.harvard.edu Tue Jul 31 12:04:47 2012 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Tue, 31 Jul 2012 12:04:47 -0400 Subject: [AstroPy] reading one line from many small fits files In-Reply-To: <5017FF27.7080700@stsci.edu> References: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> <91016FD8-9705-404D-8271-9B75DBEFD6D9@astro.physik.uni-goettingen.de> <5017FF27.7080700@stsci.edu> Message-ID: On Tue, Jul 31, 2012 at 11:52 AM, Erik Bray wrote: > On 07/30/2012 08:57 PM, Derek Homeier wrote: >> Hi John, >> >> On 31.07.2012, at 1:40AM, "John K. Parejko" wrote: >> >>> This is really more of a pyfits question, but I've upgraded to pyfits 3.1 (SVN), which is the version in astropy. >>> >>> I have data stored in thousands of ~few MB .fits files (photoObj files from SDSS) totaling a few TB of data, and I know the one single line I want to extract from some known subset of those files. But pyfits is taking more than a second per file to extract the fields I want, which seems very long, especially if it is using memmapped access, and thus should only have to read that single line (plus the header) from each file. >>> >>> I'm doing something like this: >>> >>> result = np.empty(len(data),dtype=dtype) >>> for i,x in enumerate(data): >>> getfilename(x[somefield]) >>> photo = pyfits.open(photo,memmap=True) >>> result[i] = photo[1].data[x[otherfield]-1] >>> >>> Is there a better way to go about this? Is pyfits known to be quite slow when reading a single row from a lot of different files? Anyone have suggestions on how to speed this up? >> >> that seems quite slow; it takes me about 50 ms to read a random line from the DR8 example file >> with pyfits 3.0.2. Unless the file access itself takes so long something appears to be odd. >> But the only thing coming to my mind now is that pyfits supports scaled column data (similar to >> BSCALE/BZERO in image HDUs, I assume), and if such keywords were present, they would probably >> cause a corresponding transformation for the entire bintable. They don't seem to exist in the standard >> SDSS files, though. >> Na?ve question: do you call photo.close() after each read? >> >> Cheers, >> Derek > > It probably shouldn't matter whether or not he's calling close(), but > the question about BSCALE/BZERO is possibly relevant. Is the data > you're reading from an image or a table? If it's an image, then as > Derek wrote PyFITS is still pretty inefficient, in that it will > transform the entire image, even if using mmap (which is the default now > by the way). I have plans for overhauling this but it hasn't been a high > priority for the most part. You can also turn off image scaling if you > use do_not_scale_image_data=True when opening the file. That might > speed things up. > > This is one area where using the .section feature on Image HDUs might > still be useful. For example: > > result[i] = photo[1].section[x[otherfield] - 1] > > In PyFITS 3.1 that has improved support for scaling just sections of the > file, which didn't work well before. That might also be faster. > > Of course this is all a moot point if this is not a scaled image. In > any case, opening a file and reading a single row out of the data should > not generally take as long as 1 second--especially if they're small files. > > Erik > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > Hi Erik, Not to hijack the original question, but I have a very similar question. When reading a single column from a binary FITS table (with many other uninteresting columns) using PyFITS are there strategies that will improve performance? Is using memmap helpful? By default (without memmap) does PyFITS read the entire table into memory when a single column is requested? I could probably figure some of this out experimentally with a few tests, but as long as we're on the topic you probably just know the answer. :-) Thanks, Tom p.s. I think the original question was also referring to a table, not image. From derek at astro.physik.uni-goettingen.de Tue Jul 31 12:13:40 2012 From: derek at astro.physik.uni-goettingen.de (Derek Homeier) Date: Tue, 31 Jul 2012 18:13:40 +0200 Subject: [AstroPy] reading one line from many small fits files In-Reply-To: <5017FF27.7080700@stsci.edu> References: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> <91016FD8-9705-404D-8271-9B75DBEFD6D9@astro.physik.uni-goettingen.de> <5017FF27.7080700@stsci.edu> Message-ID: Hi Erik, On 31.07.2012, at 5:52PM, Erik Bray wrote: > It probably shouldn't matter whether or not he's calling close(), but > the question about BSCALE/BZERO is possibly relevant. Is the data > you're reading from an image or a table? If it's an image, then as > Derek wrote PyFITS is still pretty inefficient, in that it will > transform the entire image, even if using mmap (which is the default now > by the way). I have plans for overhauling this but it hasn't been a high > priority for the most part. You can also turn off image scaling if you > use do_not_scale_image_data=True when opening the file. That might > speed things up. > > This is one area where using the .section feature on Image HDUs might > still be useful. For example: > > result[i] = photo[1].section[x[otherfield] - 1] > > In PyFITS 3.1 that has improved support for scaling just sections of the > file, which didn't work well before. That might also be faster. > > Of course this is all a moot point if this is not a scaled image. In > any case, opening a file and reading a single row out of the data should > not generally take as long as 1 second--especially if they're small files. indeed the question was about tables, I just still had the discussion about automatic scaling and the potential type conversion coming with it in mind. And the docs mention `FITS_rec` is the data part of a table HDU's data part. This is a layer over the `recarray`, so we can deal with scaled columns. so I was wondering if some automatic scaling features might have been added (or be in testing) in 3.1 svn. Cheers, Derek From embray at stsci.edu Tue Jul 31 12:32:22 2012 From: embray at stsci.edu (Erik Bray) Date: Tue, 31 Jul 2012 12:32:22 -0400 Subject: [AstroPy] reading one line from many small fits files In-Reply-To: References: <853FBC70-C380-4813-98DA-18F5E883D83B@yale.edu> <91016FD8-9705-404D-8271-9B75DBEFD6D9@astro.physik.uni-goettingen.de> <5017FF27.7080700@stsci.edu> Message-ID: <50180896.3070901@stsci.edu> On 07/31/2012 12:13 PM, Derek Homeier wrote: > indeed the question was about tables, I just still had the discussion about automatic > scaling and the potential type conversion coming with it in mind. And the docs mention > > `FITS_rec` is the data part of a table HDU's data part. This is a > layer over the `recarray`, so we can deal with scaled columns. > > so I was wondering if some automatic scaling features might have been added > (or be in testing) in 3.1 svn. > > Cheers, > Derek PyFITS has had scaling of table columns with TSCAL and TZERO for a long time--to my knowledge nothing has changed there. (For what it's worth, the in-progress changelog for 3.1 can be viewed here: http://pyfits.readthedocs.org/en/latest/appendix/changelog.html#unreleased) Are you saying there is a noticeable performance degradation from a previous version of PyFITS? What version were you using before? There was at one time a severe performance issue with tables containing string columns, but that should be fixed. It's possible there is another? Though I will say that row-based access in PyFITS has always been rather inefficient--it was originally designed for more efficient column access. Though this is an artifact of using Numpy's recarray class in the background. There are plans to eventually replace this with the Astropy package's Table implementation. Erik