[AstroPy] AstroPy Digest, Vol 86, Issue 9
Sebastian
sebas0 at gmail.com
Mon Nov 11 14:52:04 EST 2013
Just wondering, will RICE compressed FITS files also be supported?
On Mon, Nov 11, 2013 at 2:50 PM, <astropy-request at scipy.org> wrote:
> Send AstroPy mailing list submissions to
> astropy at scipy.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.scipy.org/mailman/listinfo/astropy
> or, via email, send a message with subject or body 'help' to
> astropy-request at scipy.org
>
> You can reach the person managing the list at
> astropy-owner at scipy.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of AstroPy digest..."
>
>
> Today's Topics:
>
> 1. Re: FITS files and column descriptions (Perry Greenfield)
> 2. Re: FITS files and column descriptions (Michael Droettboom)
> 3. Re: FITS files and column descriptions (Aldcroft, Thomas)
> 4. Re: FITS files and column descriptions (Aldcroft, Thomas)
> 5. Re: FITS files and column descriptions (Yannick Roehlly)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 11 Nov 2013 08:51:05 -0500
> From: Perry Greenfield <stsci.perry at gmail.com>
> Subject: Re: [AstroPy] FITS files and column descriptions
> To: Paul Kuin <npkuin at gmail.com>
> Cc: "AstroPy at scipy.org" <astropy at scipy.org>
> Message-ID: <CCD766AE-2AF4-4EDE-B74C-5D46C347018B at gmail.com>
> Content-Type: text/plain; charset=windows-1252
>
> What TOPCAT is doing is in no way part of the FITS standard. As far as I
> can tell, it isn't even listed as a registered convention. It's exactly
> this sort of thing that drives us nuts regarding FITS. Somehow people
> expect us to read and support anything anyone happens to stuff into a FITS
> file and handle its semantics correctly.
>
> Sorry, it's not going to happen within the FITS module itself. If you want
> to use it, you need to layer something over FITS to handle it.
>
> Perry
>
> On Nov 7, 2013, at 4:59 PM, Paul Kuin wrote:
>
> > Some years ago, I was leading the NASA Astronomical Data Center, now
> gone. I worked with CDS to have a way to present a readable table along
> with the ability to automatically generate a FITS file. That lead to the
> "ReadMe" files that accompany the catalogs from CDS, which is storing the
> metadata.
> >
> > I think it is important to make sure no metadata are lost in the
> conversion process. Sure, the FITS standard says so-and-so, but in
> practice, one has to make amends so the standard can evolve.
> >
> > I have only used the astropy saving to tables a little bit because I got
> annoyed that there was loss of (meta) data.
> >
> > The reason I write is, that there needs to be some serious thought about
> this issue. as flagged by Yannick. My (biased) feeling is that you cannot
> go around having a second metadata file when going to ascii. Doing
> otherwise will lead to a plethora of ways to save the data file. Better to
> keep the data file clean, and put the jumk in a meta data file.
> >
> > Cheers,
> >
> > Paul
> >
> >
> > On Thu, Nov 7, 2013 at 9:20 PM, Yannick Roehlly <yannick.roehlly at lam.fr>
> wrote:
> > Hi all,
> >
> > I have some remarks / questions concerning the column descriptions in
> > FITS table files (and VO).
> >
> > 1. Long column description
> >
> > The FITS format requires that a column description must be 68
> > characters or less. Some tools like topcat / stills allows longer
> > descriptions because they use a kind of VO-table metada section in the
> > FITS header. If you open such file with astropy.io.fits and save it
> > again, you loose the long description. In fact, it's not an astropy
> > problem as these description as not in the "real FITS" column
> > descriptions in the header (TCOMxx), it is just because astropy does
> > not save the "VO-table" part of the topcat file.
> > Maybe it's worth raising a warning when opening such files, like "Hey,
> > you may loose metadata!" because one may be surprise not to have all
> > its column description.
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 11 Nov 2013 10:54:44 -0500
> From: Michael Droettboom <mdroe at stsci.edu>
> Subject: Re: [AstroPy] FITS files and column descriptions
> To: <astropy at scipy.org>
> Message-ID: <5280FDC4.6090300 at stsci.edu>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On 11/07/2013 04:20 PM, Yannick Roehlly wrote:
> > 3. Column descriptions in VO-Table
> >
> > When saving a table to VO-Table, astropy prettily indent the xml, e.g.
> >
> > <DESCRIPTION>
> > IAU Name
> > </DESCRIPTION>
> >
> > But when reading these tables with topcat, this one add the spaces to
> > the description (" IAU Name"). Maybe astropy should write the
> > description on one line.
> >
>
> I'll just address this part and leave the others to those more heavily
> involved with the FITS side of things.
>
> XML basically punts on whitespace handling, by saying that each
> "application of XML" (in this case, VOTable) should define how
> whitespace should be handled. Unfortunately, the VOTable spec is silent
> on this issue, so in effect, "we're both right". HTML, as an
> application of XML, would ignore the leading whitespace.
>
> http://www.w3.org/TR/REC-xml/#sec-white-space
>
> We should probably get some clarification from the VOTable committee on
> this point. I'll write an e-mail to votable at ivoa.net
>
> Mike
>
> --
>
> _
> |\/|o _|_ _. _ | | \.__ __|__|_|_ _ _ ._ _
> | ||(_| |(_|(/_| |_/|(_)(/_|_ |_|_)(_)(_)| | |
>
> http://www.droettboom.com
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/astropy/attachments/20131111/b86c7c56/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Mon, 11 Nov 2013 12:31:21 -0500
> From: "Aldcroft, Thomas" <aldcroft at head.cfa.harvard.edu>
> Subject: Re: [AstroPy] FITS files and column descriptions
> To: Yannick Roehlly <yannick.roehlly at lam.fr>
> Cc: "AstroPy at scipy.org" <astropy at scipy.org>
> Message-ID:
> <CAMtEP6yE=JnJcGZcr=bE9ifqE=
> 3cedz2cVd4ep_dUUtZ4ExEYw at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Thu, Nov 7, 2013 at 4:20 PM, Yannick Roehlly <yannick.roehlly at lam.fr
> >wrote:
>
> > Hi all,
> >
> > I have some remarks / questions concerning the column descriptions in
> > FITS table files (and VO).
> >
> > 1. Long column description
> >
> > The FITS format requires that a column description must be 68
> > characters or less. Some tools like topcat / stills allows longer
> > descriptions because they use a kind of VO-table metada section in the
> > FITS header. If you open such file with astropy.io.fits and save it
> > again, you loose the long description. In fact, it's not an astropy
> > problem as these description as not in the "real FITS" column
> > descriptions in the header (TCOMxx), it is just because astropy does
> > not save the "VO-table" part of the topcat file.
> > Maybe it's worth raising a warning when opening such files, like "Hey,
> > you may loose metadata!" because one may be surprise not to have all
> > its column description.
> >
> > 2. astropy.table and column descriptions in FITS files
> >
> > When opening a FITS file with astropy.tables(with last git checkout)
> > one does not have access to column descriptions. When saving an
> > astropy.io.table to FITS, the column description are lost. But when
> > opening a FITS file with description and saving it again to FITS, the
> > column descriptions are still here. I suppose that it's because
> > astropy.io.fits is under the hood.
> >
> > Is this because column description handling is not implemented or do I
> > miss something?
> >
>
> I believe this is not implemented but should be. I opened an issue
> https://github.com/astropy/astropy/issues/1768.
>
>
> >
> > 3. Column descriptions in VO-Table
> >
> > When saving a table to VO-Table, astropy prettily indent the xml, e.g.
> >
> > <DESCRIPTION>
> > IAU Name
> > </DESCRIPTION>
> >
> > But when reading these tables with topcat, this one add the spaces to
> > the description (" IAU Name"). Maybe astropy should write the
> > description on one line.
> >
> > Regards,
> >
> > Yannick
> >
> > --
> > Mystics always hope that science will some day overtake them.
> > -- Booth Tarkington
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/astropy/attachments/20131111/baf2b49d/attachment-0001.html
>
> ------------------------------
>
> Message: 4
> Date: Mon, 11 Nov 2013 12:45:35 -0500
> From: "Aldcroft, Thomas" <aldcroft at head.cfa.harvard.edu>
> Subject: Re: [AstroPy] FITS files and column descriptions
> To: Yannick Roehlly <yannick.roehlly at lam.fr>
> Cc: astropy <astropy at scipy.org>
> Message-ID:
> <CAMtEP6zBunfuZ2bN1KqFERD5LPFECYxTwHE-UPYqq5WqGMsn=
> g at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> On Fri, Nov 8, 2013 at 7:32 AM, Yannick Roehlly <yannick.roehlly at lam.fr
> >wrote:
>
> > Le Thu, 7 Nov 2013 21:59:46 +0000,
> > Paul Kuin <npkuin at gmail.com> a ?crit :
> >
> > > My (biased) feeling is that you cannot
> > > go around having a second metadata file when going to ascii.
> >
> >
> > Hi Paul,
> >
> > You are right, having all the metadata on a separate README file is
> > really important; I know people not even putting the column names in
> > the data files to be sure users read the README.
> > But I also think having the metadata inside the data file is useful as
> > a reminder when working with the data.
> >
>
> There is no technical reason you cannot store all reasonable metadata in a
> commented header that precedes the data in an ASCII table. The vast
> majority of table readers support a "comment" character to separate data
> from other information. Obviously this depends on having some standards
> for interpreting the commented header, but this is the same whether or not
> it resides in a separate file.
>
> Personally I think it is far preferable to have header metadata and the
> data in a single file, otherwise you have to maintain / distribute twice as
> many files and have a mechanism for establishing a correspondence between
> the meta (ReadMe) and the data file. This then makes it harder to rename
> table files since you need to update the contents of the ReadMe as well.
> Obviously there are some advantages, like being able to create a
> collection of data files (ala CDS) with a single ReadMe, but overall it
> seems simpler to have a single data table (with meta) per file.
>
> BTW, slightly OT but I did spend some time thinking about supporting
> arbitrary metadata in an ASCII format. The result is this pull request:
> https://github.com/astropy/astropy/pull/683
> It's basically just an exploratory idea that saves everything as JSON.
> This got stalled out and it may not be the best way to go (do we need yet
> another format?), but it's there if anyone is interested.
>
> - Tom
>
>
> >
> > Concerning the column descriptions in FITS files, pyfits /
> > astropy.io.fits does not handle them, but it is always possible to add
> > the TCOMM keywords manually to the HDU headers.
> >
> > Regards,
> >
> > Yannick
> >
> > --
> > What is mind? No matter. What is matter? Never mind.
> > -- Thomas Hewitt Key, 1799-1875
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mail.scipy.org/pipermail/astropy/attachments/20131111/da1e03bc/attachment-0001.html
>
> ------------------------------
>
> Message: 5
> Date: Mon, 11 Nov 2013 18:56:05 +0100
> From: Yannick Roehlly <yannick.roehlly at lam.fr>
> Subject: Re: [AstroPy] FITS files and column descriptions
> To: astropy at scipy.org
> Message-ID: <20131111185605.31916464 at lam.fr>
> Content-Type: text/plain; charset=UTF-8
>
> Le Mon, 11 Nov 2013 08:51:05 -0500,
> Perry Greenfield <stsci.perry at gmail.com> a ?crit :
>
> > What TOPCAT is doing is in no way part of the FITS standard.
>
> Hi Perry,
>
> I agree with you concerning the "VO metadata" embedded in the FITS
> headers. But we should definitively add the column descriptions (in the
> TCOMM keywords) when saving a table to FITS, maybe raising an
> exception / a warning when a description is longer than 68 characters.
>
> Regards,
>
> Yannick
>
> --
> It has just been discovered that research causes cancer in rats.
>
>
> ------------------------------
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
> End of AstroPy Digest, Vol 86, Issue 9
> **************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20131111/c8248dc8/attachment.html>
More information about the AstroPy
mailing list