From eric.emsellem at eso.org Tue Feb 12 07:32:03 2013 From: eric.emsellem at eso.org (Eric Emsellem) Date: Tue, 12 Feb 2013 13:32:03 +0100 Subject: [AstroPy] Log rebin in python? Message-ID: <511A3643.1040106@eso.org> Hi I was wondering if you would know of a simple package/module which would allow one to rebin/resample a 'spectrum' from linear to log(lambda) (and the other way around). It is not much to write it, but before I do so, I would like to know if someone else did it (I would guess the answer is yes). So a function which takes two 1D arrays: Lambda_lin, Flux and convert it in Lambda_log and Flux, conserving the flux. I see that Ian Crossfield has something like that on the web. But just wondering if something exists as a stand-alone module. thanks! Eric From wschoenell at gmail.com Tue Feb 12 09:36:05 2013 From: wschoenell at gmail.com (William Schoenell) Date: Tue, 12 Feb 2013 15:36:05 +0100 Subject: [AstroPy] Log rebin in python? In-Reply-To: <511A3643.1040106@eso.org> References: <511A3643.1040106@eso.org> Message-ID: Hi Eric, We did this here: https://bitbucket.org/astro_ufsc/pystarlight/src/20c5b3444bbed9ace92e32162bfd816b2e75da3b/src/pystarlight/util/StarlightUtils.py?at=default#cl-93 This function returns a transformation matrix which you should apply to your original fluxes. E.g.: [[ResampMat]] ? [F_orig] = [F_resampled] Cheers, William On Tue, Feb 12, 2013 at 1:32 PM, Eric Emsellem wrote: > Hi > > I was wondering if you would know of a simple package/module which would > allow > one to rebin/resample a 'spectrum' from linear to log(lambda) (and the > other way > around). It is not much to write it, but before I do so, I would like to > know if > someone else did it (I would guess the answer is yes). > > So a function which takes two 1D arrays: Lambda_lin, Flux and convert it in > Lambda_log and Flux, conserving the flux. > > I see that Ian Crossfield has something like that on the web. But just > wondering > if something exists as a stand-alone module. > > thanks! > > Eric > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- -------------------------------------------------------------------- William Schoenell Instituto de Astrof?sica de Andaluc?a - CSIC Glorieta de la Astronom?a, s/n. E-18008 Granada - Spain Work Phone: +34 958 230 612 -------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.parejko at yale.edu Tue Feb 12 10:46:17 2013 From: john.parejko at yale.edu (John K. Parejko) Date: Tue, 12 Feb 2013 10:46:17 -0500 Subject: [AstroPy] Log rebin in python? In-Reply-To: <511A3643.1040106@eso.org> References: <511A3643.1040106@eso.org> Message-ID: I ported the Michele Cappellari's logrebin.pro, which might not be exactly what you're looking for, but it does something very similar. http://fiddlernet.net/websvn/filedetails.php?repname=dev&path=%2Fpython%2Flogrebin.py There's a download link in the upper right. If someone wants to add this to astropy, feel free. John On 12Feb 2013, at 07:32, Eric Emsellem wrote: > Hi > > I was wondering if you would know of a simple package/module which would allow > one to rebin/resample a 'spectrum' from linear to log(lambda) (and the other way > around). It is not much to write it, but before I do so, I would like to know if > someone else did it (I would guess the answer is yes). > > So a function which takes two 1D arrays: Lambda_lin, Flux and convert it in > Lambda_log and Flux, conserving the flux. > > I see that Ian Crossfield has something like that on the web. But just wondering > if something exists as a stand-alone module. > > thanks! > > Eric > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- ************************* John Parejko john.parejko at yale.edu http://www.astro.yale.edu/~jp727/ 203 432-9759 JWG 465 Department of Physics Yale University New Haven, CT ************************** From jturner at gemini.edu Fri Feb 15 14:43:32 2013 From: jturner at gemini.edu (James Turner) Date: Fri, 15 Feb 2013 16:43:32 -0300 Subject: [AstroPy] Python Software Foundation News: Python trademark at risk in Europe: We need your help! Message-ID: <511E8FE4.5020002@gemini.edu> FYI: http://pyfound.blogspot.com/2013/02/python-trademark-at-risk-in-europe-we.html From erik.tollerud at gmail.com Wed Feb 20 09:56:20 2013 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Wed, 20 Feb 2013 09:56:20 -0500 Subject: [AstroPy] ANN: Astropy 0.2 released Message-ID: Dear colleagues, We are very happy to announce the first public release (v0.2) of the Astropy package, a core Python package for Astronomy: http://www.astropy.org Astropy is a community-driven package intended to contain much of the core functionality and common tools needed for performing astronomy and astrophysics with Python. Key functionality in this first release includes: * Manipulation of scalar and array quantities with physical units * Pre-defined physical and astronomical constants * Celestial coordinate representation and transformations * Reading/writing/editing of FITS files, VO tables, and many ASCII table formats * Class for grid-based data with meta-data and units * Powerful table creation/manipulation * Common cosmological calculations * World Coordinate System (WCS) transformations * Representation/transformation of Times and Dates to very high accuracy Additional major functionality is planned for the near future, including but not limited to model fitting, photometric analysis tools, and Virtual Observatory (VO) querying. Some of the functionality in Astropy was adapted from previously existing packages such as PyFITS, PyWCS, asciitable, and vo.table. Hence, Astropy works as a replacement for these separate packages. Instructions for installing Astropy are provided at the http://www.astropy.org website, and extensive documentation can be found at: http://docs.astropy.org Please report any issues, or request new features via our GitHub repository: https://github.com/astropy/astropy/issues Over 30 developers have contributed code for this release of Astropy, and you can find out more about the team behind Astropy here: http://www.astropy.org/team.html Please feel free to forward this announcement to anyone you think might be interested in this release. We hope that you enjoy using Astropy as much as we enjoyed developing it! Erik Tollerud, Thomas Robitaille, and Perry Greenfield on behalf of The Astropy Collaboration From sse at iaa.es Mon Feb 25 06:03:48 2013 From: sse at iaa.es (Susana Sanchez Exposito) Date: Mon, 25 Feb 2013 12:03:48 +0100 Subject: [AstroPy] extracting column names Message-ID: Hi all, Probably this is a newbie question, but how can I extract the names of the columns from the VOtable with the Astropy library? I want to show the VOTable data in a nice way, using the Qt library, so I need to extract the names of the columns and the data from a VOTable. I have tried it using the Numpy record array associated to the votable, see code below. But I have problems when the votable fields have 'ID' and also 'name'. I am wondering if there is a better way to find the column names. I would be very gratefully, If anyone can help me or give me any hint. Thanks, Susana. table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) data = table.array dtype_a=data.dtype column_names=[] for k,v in dtype_a.fields.iteritems(): column_names.append(k) -- Susana S?nchez Exp?sito Instituto de Astrof?sica de Andaluc?a IAA (CSIC) Camino Bajo de Hu?tor, 50. Granada E-18008 Tel:(+34) 958 121 311 / (+34) 958 230 618 Fax:(+34) 958 814 530 e-mail: sse at iaa.es From susanasanche at gmail.com Mon Feb 25 06:26:06 2013 From: susanasanche at gmail.com (Susana Sanchez) Date: Mon, 25 Feb 2013 12:26:06 +0100 Subject: [AstroPy] extracting column names from VOtable Message-ID: Hi all, Probably this is a newbie question, but how can I extract the names of the columns from the VOtable with the Astropy library? I want to show the VOTable data in a nice way, using the Qt library, so I need to extract the names of the columns and the data from a VOTable. I have tried it using the Numpy record array associated to the votable, see code below. But I have problems when the votable fields have 'ID' and also 'name'. I am wondering if there is a better way to find the column names. I would be very gratefully, If anyone can help me or give me any hint. Thanks, Susana. table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) data = table.array dtype_a=data.dtype column_names=[] for k,v in dtype_a.fields.iteritems(): column_names.append(k) From erik.tollerud at gmail.com Mon Feb 25 10:47:20 2013 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 25 Feb 2013 10:47:20 -0500 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: This is probably the easiest way: [f.name for f in table.fields] ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and those objects have all the information about the columns (including things like units). An alternative is to use `array` to get the numpy array from the votable, and the `dtype` has the column names. I.e., ``table.array.dtype.names`` should give the same thing. That won't include extra VO information like units and such, though. On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: > Hi all, > > Probably this is a newbie question, but how can I extract the names of > the columns from the VOtable with the Astropy library? > > I want to show the VOTable data in a nice way, using the Qt library, > so I need to extract the names of the columns and the data from a > VOTable. I have tried it using the Numpy record array associated to > the votable, see code below. But I have problems when the votable > fields have 'ID' and also 'name'. I am wondering if there is a better > way to find the column names. > > I would be very gratefully, If anyone can help me or give me any hint. > > Thanks, > Susana. > > > > table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) > data = table.array > dtype_a=data.dtype > column_names=[] > for k,v in dtype_a.fields.iteritems(): > column_names.append(k) > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -- Erik From susanasanche at gmail.com Mon Feb 25 11:18:32 2013 From: susanasanche at gmail.com (Susana Sanchez) Date: Mon, 25 Feb 2013 17:18:32 +0100 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: Thanks Erik, The first way you say ([f.name for f in table.fields]) it is just what I am looking for, but the alternative way, using the dtype array (table.array.dtype.names), does not give the same things. In those cases when the votable fields contain 'ID' and 'name', the table.array.dtype.names gives the values in 'ID' but not in 'name'. 2013/2/25 Erik Tollerud : > This is probably the easiest way: > > [f.name for f in table.fields] > > ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and > those objects have all the information about the columns (including > things like units). > > An alternative is to use `array` to get the numpy array from the > votable, and the `dtype` has the column names. I.e., > ``table.array.dtype.names`` should give the same thing. That won't > include extra VO information like units and such, though. > > > > On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: >> Hi all, >> >> Probably this is a newbie question, but how can I extract the names of >> the columns from the VOtable with the Astropy library? >> >> I want to show the VOTable data in a nice way, using the Qt library, >> so I need to extract the names of the columns and the data from a >> VOTable. I have tried it using the Numpy record array associated to >> the votable, see code below. But I have problems when the votable >> fields have 'ID' and also 'name'. I am wondering if there is a better >> way to find the column names. >> >> I would be very gratefully, If anyone can help me or give me any hint. >> >> Thanks, >> Susana. >> >> >> >> table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) >> data = table.array >> dtype_a=data.dtype >> column_names=[] >> for k,v in dtype_a.fields.iteritems(): >> column_names.append(k) >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > > > -- > Erik From susanasanche at gmail.com Mon Feb 25 11:45:03 2013 From: susanasanche at gmail.com (Susana Sanchez) Date: Mon, 25 Feb 2013 17:45:03 +0100 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: 2013/2/25 Erik Tollerud : > Ah, I see that - I didn't read to the bottom of your original post! > > Odd, I would have thought that would work. But as loks as the [f.name > for f in table.fields] method works, I guess it's fine. > > Is the VOTable that's doing this somewhere publicly available? It > might be useful if someone wants to check if this is a bug or > something... Yes, it is publicly available in the VO service "AMIGA catalogue", http://amiga.iaa.es/amigasearch. Attached you can find the votable that I was using. I got it from this VO service through TOPCAT. > On Mon, Feb 25, 2013 at 11:18 AM, Susana Sanchez wrote: >> Thanks Erik, >> >> The first way you say ([f.name for f in table.fields]) it is just what >> I am looking for, but the alternative way, using the dtype array >> (table.array.dtype.names), does not give the same things. In those >> cases when the votable fields contain 'ID' and 'name', the >> table.array.dtype.names gives the values in 'ID' but not in 'name'. >> >> >> >> 2013/2/25 Erik Tollerud : >>> This is probably the easiest way: >>> >>> [f.name for f in table.fields] >> >>> >>> ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and >>> those objects have all the information about the columns (including >>> things like units). >>> >>> An alternative is to use `array` to get the numpy array from the >>> votable, and the `dtype` has the column names. I.e., >>> ``table.array.dtype.names`` should give the same thing. That won't >>> include extra VO information like units and such, though. >>> >>> >>> >>> On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: >>>> Hi all, >>>> >>>> Probably this is a newbie question, but how can I extract the names of >>>> the columns from the VOtable with the Astropy library? >>>> >>>> I want to show the VOTable data in a nice way, using the Qt library, >>>> so I need to extract the names of the columns and the data from a >>>> VOTable. I have tried it using the Numpy record array associated to >>>> the votable, see code below. But I have problems when the votable >>>> fields have 'ID' and also 'name'. I am wondering if there is a better >>>> way to find the column names. >>>> >>>> I would be very gratefully, If anyone can help me or give me any hint. >>>> >>>> Thanks, >>>> Susana. >>>> >>>> >>>> >>>> table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) >>>> data = table.array >>>> dtype_a=data.dtype >>>> column_names=[] >>>> for k,v in dtype_a.fields.iteritems(): >>>> column_names.append(k) >>>> _______________________________________________ >>>> AstroPy mailing list >>>> AstroPy at scipy.org >>>> http://mail.scipy.org/mailman/listinfo/astropy >>> >>> >>> >>> -- >>> Erik > > > > -- > Erik -------------- next part -------------- A non-text attachment was scrubbed... Name: cig22.xml Type: text/xml Size: 11381 bytes Desc: not available URL: From erik.tollerud at gmail.com Mon Feb 25 12:23:46 2013 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 25 Feb 2013 12:23:46 -0500 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: Aha, I understand what's happening now. The fields that have both "ID" and "name" get *both* attached to the dtype, but `dtype.names` only gives you the ID. For example, if you do ``dt.descr[0]``, you get (('CIG Number', 'col1'), '|O8'), but ``dt.names[0]`` just gives 'col1'. So the way to get what you want out of the dtype is probably like this: [des[0][0] for des in dtype_a.descr] Although that's certainly rather awkward. I *think* this is easily fixable, and I agree with you that that's surprising behavior. This might be considered either a bug or a feature, request, but either way I'll put it in the issue tracker so it can hopefully get into the next version. Thanks! On Mon, Feb 25, 2013 at 11:45 AM, Susana Sanchez wrote: > 2013/2/25 Erik Tollerud : >> Ah, I see that - I didn't read to the bottom of your original post! >> >> Odd, I would have thought that would work. But as loks as the [f.name >> for f in table.fields] method works, I guess it's fine. >> >> Is the VOTable that's doing this somewhere publicly available? It >> might be useful if someone wants to check if this is a bug or >> something... > > > Yes, it is publicly available in the VO service "AMIGA catalogue", > http://amiga.iaa.es/amigasearch. Attached you can find the votable > that I was using. I got it from this VO service through TOPCAT. > > >> On Mon, Feb 25, 2013 at 11:18 AM, Susana Sanchez wrote: >>> Thanks Erik, >>> >>> The first way you say ([f.name for f in table.fields]) it is just what >>> I am looking for, but the alternative way, using the dtype array >>> (table.array.dtype.names), does not give the same things. In those >>> cases when the votable fields contain 'ID' and 'name', the >>> table.array.dtype.names gives the values in 'ID' but not in 'name'. >>> >>> >>> >>> 2013/2/25 Erik Tollerud : >>>> This is probably the easiest way: >>>> >>>> [f.name for f in table.fields] >>> >>>> >>>> ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and >>>> those objects have all the information about the columns (including >>>> things like units). >>>> >>>> An alternative is to use `array` to get the numpy array from the >>>> votable, and the `dtype` has the column names. I.e., >>>> ``table.array.dtype.names`` should give the same thing. That won't >>>> include extra VO information like units and such, though. >>>> >>>> >>>> >>>> On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: >>>>> Hi all, >>>>> >>>>> Probably this is a newbie question, but how can I extract the names of >>>>> the columns from the VOtable with the Astropy library? >>>>> >>>>> I want to show the VOTable data in a nice way, using the Qt library, >>>>> so I need to extract the names of the columns and the data from a >>>>> VOTable. I have tried it using the Numpy record array associated to >>>>> the votable, see code below. But I have problems when the votable >>>>> fields have 'ID' and also 'name'. I am wondering if there is a better >>>>> way to find the column names. >>>>> >>>>> I would be very gratefully, If anyone can help me or give me any hint. >>>>> >>>>> Thanks, >>>>> Susana. >>>>> >>>>> >>>>> >>>>> table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) >>>>> data = table.array >>>>> dtype_a=data.dtype >>>>> column_names=[] >>>>> for k,v in dtype_a.fields.iteritems(): >>>>> column_names.append(k) >>>>> _______________________________________________ >>>>> AstroPy mailing list >>>>> AstroPy at scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>> >>>> >>>> >>>> -- >>>> Erik >> >> >> >> -- >> Erik -- Erik From erik.tollerud at gmail.com Mon Feb 25 12:31:45 2013 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 25 Feb 2013 12:31:45 -0500 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: See https://github.com/astropy/astropy/issues/819 for the issue I just created to address this topic. On Mon, Feb 25, 2013 at 12:23 PM, Erik Tollerud wrote: > Aha, I understand what's happening now. The fields that have both > "ID" and "name" get *both* attached to the dtype, but `dtype.names` > only gives you the ID. For example, if you do ``dt.descr[0]``, you > get (('CIG Number', 'col1'), '|O8'), but ``dt.names[0]`` just gives > 'col1'. So the way to get what you want out of the dtype is probably > like this: > [des[0][0] for des in dtype_a.descr] > Although that's certainly rather awkward. > > I *think* this is easily fixable, and I agree with you that that's > surprising behavior. This might be considered either a bug or a > feature, request, but either way I'll put it in the issue tracker so > it can hopefully get into the next version. > > Thanks! > > On Mon, Feb 25, 2013 at 11:45 AM, Susana Sanchez wrote: >> 2013/2/25 Erik Tollerud : >>> Ah, I see that - I didn't read to the bottom of your original post! >>> >>> Odd, I would have thought that would work. But as loks as the [f.name >>> for f in table.fields] method works, I guess it's fine. >>> >>> Is the VOTable that's doing this somewhere publicly available? It >>> might be useful if someone wants to check if this is a bug or >>> something... >> >> >> Yes, it is publicly available in the VO service "AMIGA catalogue", >> http://amiga.iaa.es/amigasearch. Attached you can find the votable >> that I was using. I got it from this VO service through TOPCAT. >> >> >>> On Mon, Feb 25, 2013 at 11:18 AM, Susana Sanchez wrote: >>>> Thanks Erik, >>>> >>>> The first way you say ([f.name for f in table.fields]) it is just what >>>> I am looking for, but the alternative way, using the dtype array >>>> (table.array.dtype.names), does not give the same things. In those >>>> cases when the votable fields contain 'ID' and 'name', the >>>> table.array.dtype.names gives the values in 'ID' but not in 'name'. >>>> >>>> >>>> >>>> 2013/2/25 Erik Tollerud : >>>>> This is probably the easiest way: >>>>> >>>>> [f.name for f in table.fields] >>>> >>>>> >>>>> ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and >>>>> those objects have all the information about the columns (including >>>>> things like units). >>>>> >>>>> An alternative is to use `array` to get the numpy array from the >>>>> votable, and the `dtype` has the column names. I.e., >>>>> ``table.array.dtype.names`` should give the same thing. That won't >>>>> include extra VO information like units and such, though. >>>>> >>>>> >>>>> >>>>> On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: >>>>>> Hi all, >>>>>> >>>>>> Probably this is a newbie question, but how can I extract the names of >>>>>> the columns from the VOtable with the Astropy library? >>>>>> >>>>>> I want to show the VOTable data in a nice way, using the Qt library, >>>>>> so I need to extract the names of the columns and the data from a >>>>>> VOTable. I have tried it using the Numpy record array associated to >>>>>> the votable, see code below. But I have problems when the votable >>>>>> fields have 'ID' and also 'name'. I am wondering if there is a better >>>>>> way to find the column names. >>>>>> >>>>>> I would be very gratefully, If anyone can help me or give me any hint. >>>>>> >>>>>> Thanks, >>>>>> Susana. >>>>>> >>>>>> >>>>>> >>>>>> table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) >>>>>> data = table.array >>>>>> dtype_a=data.dtype >>>>>> column_names=[] >>>>>> for k,v in dtype_a.fields.iteritems(): >>>>>> column_names.append(k) >>>>>> _______________________________________________ >>>>>> AstroPy mailing list >>>>>> AstroPy at scipy.org >>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>> >>>>> >>>>> >>>>> -- >>>>> Erik >>> >>> >>> >>> -- >>> Erik > > > > -- > Erik -- Erik From erik.tollerud at gmail.com Mon Feb 25 11:37:53 2013 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 25 Feb 2013 11:37:53 -0500 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: Ah, I see that - I didn't read to the bottom of your original post! Odd, I would have thought that would work. But as loks as the [f.name for f in table.fields] method works, I guess it's fine. Is the VOTable that's doing this somewhere publicly available? It might be useful if someone wants to check if this is a bug or something... On Mon, Feb 25, 2013 at 11:18 AM, Susana Sanchez wrote: > Thanks Erik, > > The first way you say ([f.name for f in table.fields]) it is just what > I am looking for, but the alternative way, using the dtype array > (table.array.dtype.names), does not give the same things. In those > cases when the votable fields contain 'ID' and 'name', the > table.array.dtype.names gives the values in 'ID' but not in 'name'. > > > > 2013/2/25 Erik Tollerud : >> This is probably the easiest way: >> >> [f.name for f in table.fields] > >> >> ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and >> those objects have all the information about the columns (including >> things like units). >> >> An alternative is to use `array` to get the numpy array from the >> votable, and the `dtype` has the column names. I.e., >> ``table.array.dtype.names`` should give the same thing. That won't >> include extra VO information like units and such, though. >> >> >> >> On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: >>> Hi all, >>> >>> Probably this is a newbie question, but how can I extract the names of >>> the columns from the VOtable with the Astropy library? >>> >>> I want to show the VOTable data in a nice way, using the Qt library, >>> so I need to extract the names of the columns and the data from a >>> VOTable. I have tried it using the Numpy record array associated to >>> the votable, see code below. But I have problems when the votable >>> fields have 'ID' and also 'name'. I am wondering if there is a better >>> way to find the column names. >>> >>> I would be very gratefully, If anyone can help me or give me any hint. >>> >>> Thanks, >>> Susana. >>> >>> >>> >>> table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) >>> data = table.array >>> dtype_a=data.dtype >>> column_names=[] >>> for k,v in dtype_a.fields.iteritems(): >>> column_names.append(k) >>> _______________________________________________ >>> AstroPy mailing list >>> AstroPy at scipy.org >>> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> -- >> Erik -- Erik From susanasanche at gmail.com Mon Feb 25 12:46:57 2013 From: susanasanche at gmail.com (Susana Sanchez) Date: Mon, 25 Feb 2013 18:46:57 +0100 Subject: [AstroPy] saving a votable as a xml file using astropy Message-ID: Hi again, This time I have the doubt saving votables as xml. As I said in my previous mail, I want to show the VOTable in a nice way using Qt library, I also want to allow the users can modify this votable and then save the changes. Reading the documentation I saw that the astropy.io.votable.tree.Table has the method .to_xml, but probably I am not using it well because I get this error: In [159]: table = parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) In [160]: table.to_xml("./cig22_changed.xml") --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /home/susana/ in () /usr/local/lib/python2.6/dist-packages/astropy/io/votable/tree.pyc in to_xml(self, w, **kwargs) 2518 2519 def to_xml(self, w, **kwargs): -> 2520 with w.tag( 2521 u'TABLE', 2522 attrib=w.object_attrs( AttributeError: 'str' object has no attribute 'tag' Finally I got this way to save the xml: #Open the votable table = astropy.io.votable.parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) #Modify the table table.array['CIG Number'][0]="CIG SUSANA" #Create a VOTableFile instance of the table votable= VOTableFile.from_table(table.to_table()) #Save the xml votable.to_xml('cig22S.xml') I have two questions/comments: 1) Is this way correct? Or maybe there is a better way to save the changes made in a VOtable? 2) The VOTable created, 'cig22S.xml', is ok, but the fields have lost the 'name' value. Now, the 'ID' and the 'name' have the same value (the original value of ID). Thanks again, Susana. -------------- next part -------------- A non-text attachment was scrubbed... Name: cig22.xml Type: text/xml Size: 11381 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cig22S.xml Type: text/xml Size: 10865 bytes Desc: not available URL: From mdroe at stsci.edu Mon Feb 25 12:57:32 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Mon, 25 Feb 2013 12:57:32 -0500 Subject: [AstroPy] extracting column names from VOtable In-Reply-To: References: Message-ID: <512BA60C.3070505@stsci.edu> I admit, this is horribly confusing because the term "name" is used in VOTable and Numpy to represent different concepts. In VOTable, ID is guaranteed to be unique, but is not required. Names are not guaranteed to be unique, but are required. In numpy, names are required to be unique and are required. titles are not required, and are not required to be unique. So a name is not a name. The conceptual mapping is really that vo name == numpy title and vo ID == numpy name. Admittedly, this deserves better documentation in astropy, but I think this is actually the correct mapping of concepts. I have proposed to the VOTable committee to make IDs required, which would make this much more straightforward, but I have not heard a determination about this, and it appears that it will at least not make it into VOTable 1.3 -- we might have to wait for VOTable 1.4 for that if at all. Mike On 02/25/2013 12:31 PM, Erik Tollerud wrote: > See https://github.com/astropy/astropy/issues/819 for the issue I just > created to address this topic. > > On Mon, Feb 25, 2013 at 12:23 PM, Erik Tollerud wrote: >> Aha, I understand what's happening now. The fields that have both >> "ID" and "name" get *both* attached to the dtype, but `dtype.names` >> only gives you the ID. For example, if you do ``dt.descr[0]``, you >> get (('CIG Number', 'col1'), '|O8'), but ``dt.names[0]`` just gives >> 'col1'. So the way to get what you want out of the dtype is probably >> like this: >> [des[0][0] for des in dtype_a.descr] >> Although that's certainly rather awkward. >> >> I *think* this is easily fixable, and I agree with you that that's >> surprising behavior. This might be considered either a bug or a >> feature, request, but either way I'll put it in the issue tracker so >> it can hopefully get into the next version. >> >> Thanks! >> >> On Mon, Feb 25, 2013 at 11:45 AM, Susana Sanchez wrote: >>> 2013/2/25 Erik Tollerud : >>>> Ah, I see that - I didn't read to the bottom of your original post! >>>> >>>> Odd, I would have thought that would work. But as loks as the [f.name >>>> for f in table.fields] method works, I guess it's fine. >>>> >>>> Is the VOTable that's doing this somewhere publicly available? It >>>> might be useful if someone wants to check if this is a bug or >>>> something... >>> >>> Yes, it is publicly available in the VO service "AMIGA catalogue", >>> http://amiga.iaa.es/amigasearch. Attached you can find the votable >>> that I was using. I got it from this VO service through TOPCAT. >>> >>> >>>> On Mon, Feb 25, 2013 at 11:18 AM, Susana Sanchez wrote: >>>>> Thanks Erik, >>>>> >>>>> The first way you say ([f.name for f in table.fields]) it is just what >>>>> I am looking for, but the alternative way, using the dtype array >>>>> (table.array.dtype.names), does not give the same things. In those >>>>> cases when the votable fields contain 'ID' and 'name', the >>>>> table.array.dtype.names gives the values in 'ID' but not in 'name'. >>>>> >>>>> >>>>> >>>>> 2013/2/25 Erik Tollerud : >>>>>> This is probably the easiest way: >>>>>> >>>>>> [f.name for f in table.fields] >>>>>> ``table.fields`` is a list of `astropy.io.votable.tree.Field` objects, and >>>>>> those objects have all the information about the columns (including >>>>>> things like units). >>>>>> >>>>>> An alternative is to use `array` to get the numpy array from the >>>>>> votable, and the `dtype` has the column names. I.e., >>>>>> ``table.array.dtype.names`` should give the same thing. That won't >>>>>> include extra VO information like units and such, though. >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Feb 25, 2013 at 6:26 AM, Susana Sanchez wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> Probably this is a newbie question, but how can I extract the names of >>>>>>> the columns from the VOtable with the Astropy library? >>>>>>> >>>>>>> I want to show the VOTable data in a nice way, using the Qt library, >>>>>>> so I need to extract the names of the columns and the data from a >>>>>>> VOTable. I have tried it using the Numpy record array associated to >>>>>>> the votable, see code below. But I have problems when the votable >>>>>>> fields have 'ID' and also 'name'. I am wondering if there is a better >>>>>>> way to find the column names. >>>>>>> >>>>>>> I would be very gratefully, If anyone can help me or give me any hint. >>>>>>> >>>>>>> Thanks, >>>>>>> Susana. >>>>>>> >>>>>>> >>>>>>> >>>>>>> table = parse_single_table("/home/susana/Documents/examples/tables/cig22.xml",pedantic=False) >>>>>>> data = table.array >>>>>>> dtype_a=data.dtype >>>>>>> column_names=[] >>>>>>> for k,v in dtype_a.fields.iteritems(): >>>>>>> column_names.append(k) >>>>>>> _______________________________________________ >>>>>>> AstroPy mailing list >>>>>>> AstroPy at scipy.org >>>>>>> http://mail.scipy.org/mailman/listinfo/astropy >>>>>> >>>>>> >>>>>> -- >>>>>> Erik >>>> >>>> >>>> -- >>>> Erik >> >> >> -- >> Erik > > > -- > Erik > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Mon Feb 25 13:00:21 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Mon, 25 Feb 2013 13:00:21 -0500 Subject: [AstroPy] saving a votable as a xml file using astropy In-Reply-To: References: Message-ID: <512BA6B5.2070209@stsci.edu> VOTable is actually a container format for multiple tables, so you can only save that entire container to XML. The best thing to do to make modifications to save back out would be to retain a reference to the top-level VOTableFile object (not the votable.tree.Table object), and call `to_xml` on that. That approach should also maintain all of the metadata (what the original names and IDs were). Hope that helps, and feel free to ping me if you need more detail. Mike On 02/25/2013 12:46 PM, Susana Sanchez wrote: > Hi again, > > This time I have the doubt saving votables as xml. As I said in my > previous mail, I want to show the VOTable in a nice way using Qt > library, I also want to allow the users can modify this votable and > then save the changes. > > Reading the documentation I saw that the astropy.io.votable.tree.Table > has the method .to_xml, but probably I am not using it well because I > get this error: > > In [159]: table = > parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) > In [160]: table.to_xml("./cig22_changed.xml") > --------------------------------------------------------------------------- > AttributeError Traceback (most recent call last) > > /home/susana/ in () > > /usr/local/lib/python2.6/dist-packages/astropy/io/votable/tree.pyc in > to_xml(self, w, **kwargs) > 2518 > 2519 def to_xml(self, w, **kwargs): > -> 2520 with w.tag( > 2521 u'TABLE', > 2522 attrib=w.object_attrs( > > AttributeError: 'str' object has no attribute 'tag' > > > Finally I got this way to save the xml: > > #Open the votable > table = astropy.io.votable.parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) > > #Modify the table > table.array['CIG Number'][0]="CIG SUSANA" > > #Create a VOTableFile instance of the table > votable= VOTableFile.from_table(table.to_table()) > > #Save the xml > votable.to_xml('cig22S.xml') > > > I have two questions/comments: > > 1) Is this way correct? Or maybe there is a better way to save the > changes made in a VOtable? > > 2) The VOTable created, 'cig22S.xml', is ok, but the fields have lost > the 'name' value. Now, the 'ID' and the 'name' have the same value > (the original value of ID). > > > > Thanks again, > Susana. > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From sse at iaa.es Tue Feb 26 03:35:52 2013 From: sse at iaa.es (Susana Sanchez Exposito) Date: Tue, 26 Feb 2013 09:35:52 +0100 Subject: [AstroPy] saving a votable as a xml file using astropy In-Reply-To: <512BA6B5.2070209@stsci.edu> References: <512BA6B5.2070209@stsci.edu> Message-ID: Thanks Mike, At first, I was not sure what you mean, because I didn't find any attribute of VOTableFile for the data array, which can be modified. Then I realized the method VOTableFile.get_first_table() returns a *reference* of the first table (I thought it would return a *copy* of the table), so I guess the way you said is the way I code below, isn't it?: #Open the votable votable = parse("cig22.xml") #Get a reference of the table table=votable.get_first_table() #Modify the table table.array['CIG Number'][0]="CIG SUSANA" #The cig_s.xml table will contain the before modification votable.to_xml("cig_s.xml") Regards and thanks again for your help. Susana. 2013/2/25 Michael Droettboom : > VOTable is actually a container format for multiple tables, so you can only > save that entire container to XML. > > The best thing to do to make modifications to save back out would be to > retain a reference to the top-level VOTableFile object (not the > votable.tree.Table object), and call `to_xml` on that. That approach should > also maintain all of the metadata (what the original names and IDs were). > > Hope that helps, and feel free to ping me if you need more detail. > > Mike > > > On 02/25/2013 12:46 PM, Susana Sanchez wrote: > > Hi again, > > This time I have the doubt saving votables as xml. As I said in my > previous mail, I want to show the VOTable in a nice way using Qt > library, I also want to allow the users can modify this votable and > then save the changes. > > Reading the documentation I saw that the astropy.io.votable.tree.Table > has the method .to_xml, but probably I am not using it well because I > get this error: > > In [159]: table = > parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) > In [160]: table.to_xml("./cig22_changed.xml") > --------------------------------------------------------------------------- > AttributeError Traceback (most recent call last) > > /home/susana/ in () > > /usr/local/lib/python2.6/dist-packages/astropy/io/votable/tree.pyc in > to_xml(self, w, **kwargs) > 2518 > 2519 def to_xml(self, w, **kwargs): > -> 2520 with w.tag( > 2521 u'TABLE', > 2522 attrib=w.object_attrs( > > AttributeError: 'str' object has no attribute 'tag' > > > Finally I got this way to save the xml: > > #Open the votable > table = > astropy.io.votable.parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) > > #Modify the table > table.array['CIG Number'][0]="CIG SUSANA" > > #Create a VOTableFile instance of the table > votable= VOTableFile.from_table(table.to_table()) > > #Save the xml > votable.to_xml('cig22S.xml') > > > I have two questions/comments: > > 1) Is this way correct? Or maybe there is a better way to save the > changes made in a VOtable? > > 2) The VOTable created, 'cig22S.xml', is ok, but the fields have lost > the 'name' value. Now, the 'ID' and the 'name' have the same value > (the original value of ID). > > > > Thanks again, > Susana. > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy > -- Susana S?nchez Exp?sito Instituto de Astrof?sica de Andaluc?a IAA (CSIC) Camino Bajo de Hu?tor, 50. Granada E-18008 Tel:(+34) 958 121 311 / (+34) 958 230 618 Fax:(+34) 958 814 530 e-mail: sse at iaa.es From andrew.f.ptak at nasa.gov Tue Feb 26 09:51:01 2013 From: andrew.f.ptak at nasa.gov (Andy Ptak) Date: Tue, 26 Feb 2013 09:51:01 -0500 Subject: [AstroPy] Problem reading VOTable produced by HLA In-Reply-To: <512BA60C.3070505@stsci.edu> References: <512BA60C.3070505@stsci.edu> Message-ID: Hello, I am trying out reading VOTables using and am having trouble. Specifically reading tables returned by the Hubble Legacy Archive. The query was: http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389 which I attached. I am using astropy installed by pip on the enthought distribution on a mac, so its version 0.2 and numpy got upgraded to 1.7.0 (if that matters). First, is there a way to have a votable imported directly from query without saving it to a physical file first? Maybe its best practice to save VO queries to a file to trace errors, in case the votable is very large, etc. but when you know you are getting something back that is small and (as in this case) you mainly want a list of fields for further work, saving the full votable seams like an unnecessary extra step. The errors I am getting are below. I got similar errors when using a different votable call I found on the HLA website (basically the same except using acsSIAP.cgi?strict=1 Is this a problem with the Votable being returned by stsci, a bug or am I doing something wrong? Thanks, Andy Ptak In [4]: from astropy.io.votable import parse In [6]: votable = parse("test1.xml") WARNING: W42: test1.xml:3:0: W42: No XML namespace specified [astropy.io.votable.exceptions] WARNING: W03: test1.xml:7:4: W03: Implictly generating an ID from a name 'INPUT:POS' -> 'INPUT_POS' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:8:4: W03: Implictly generating an ID from a name 'INPUT:SIZE' -> 'INPUT_SIZE' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:9:4: W03: Implictly generating an ID from a name 'INPUT:FORMAT' -> 'INPUT_FORMAT' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:10:4: W03: Implictly generating an ID from a name 'INPUT:imagetype' -> 'INPUT_imagetype' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:11:4: W03: Implictly generating an ID from a name 'INPUT:inst' -> 'INPUT_inst' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:12:4: W03: Implictly generating an ID from a name 'INPUT:hrcmatch' -> 'INPUT_hrcmatch' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:13:4: W03: Implictly generating an ID from a name 'INPUT:zoom' -> 'INPUT_zoom' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:14:4: W03: Implictly generating an ID from a name 'INPUT:autoscale' -> 'INPUT_autoscale' [astropy.io.votable.exceptions] WARNING: W03: test1.xml:15:4: W03: Implictly generating an ID from a name 'INPUT:asinh' -> 'INPUT_asinh' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:16:4: W06: Invalid UCD 'VOX:STC_CoordRefFrame': UCD has invalid character ':' in 'VOX:STC_CoordRefFrame' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:17:4: W06: Invalid UCD 'VOX:WCS_CoordProjection': UCD has invalid character ':' in 'VOX:WCS_CoordProjection' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:19:6: W06: Invalid UCD 'VOX:Image_AccessReference': UCD has invalid character ':' in 'VOX:Image_AccessReference' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:82:6: W06: Invalid UCD 'VOX:Image_Format': UCD has invalid character ':' in 'VOX:Image_Format' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:91:6: W06: Invalid UCD 'VOX:Image_Title': UCD has invalid character ':' in 'VOX:Image_Title' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:94:6: W06: Invalid UCD 'VOX:Image_Naxis': UCD has invalid character ':' in 'VOX:Image_Naxis' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:97:6: W06: Invalid UCD 'VOX:Image_Scale': UCD has invalid character ':' in 'VOX:Image_Scale' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:100:6: W06: Invalid UCD 'VOX:Image_Naxes': UCD has invalid character ':' in 'VOX:Image_Naxes' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:103:6: W06: Invalid UCD 'VOX:WCS_CoordRefPixel': UCD has invalid character ':' in 'VOX:WCS_CoordRefPixel' [astropy.io.votable.exceptions] WARNING: W06: test1.xml:106:6: W06: Invalid UCD 'VOX:WCS_CoordRefValue': UCD has invalid character ':' in 'VOX:WCS_CoordRefValue' (suppressing further warnings of this type...) [astropy.io.votable.exceptions] ERROR: OverflowError: Python int too large to convert to C long [numpy.ma.core] --------------------------------------------------------------------------- OverflowError Traceback (most recent call last) /Volumes/Apps_and_Docs/aptak/Dropbox/HLA/ in () ----> 1 votable = parse("test1.xml") /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/table.pyc in parse(source, columns, invalid, pedantic, chunk_size, table_number, table_id, filename, _debug_python_based_parser) 114 _debug_python_based_parser=_debug_python_based_parser) as iterator: 115 return tree.VOTableFile( --> 116 config=config, pos=(1, 1)).parse(iterator, config) 117 118 /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc in parse(self, iterator, config) 3084 if start: 3085 tag_mapping.get(tag, self._add_unknown_tag)( -> 3086 iterator, tag, data, config, pos) 3087 elif tag == 'DESCRIPTION': 3088 if self.description is not None: /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc in _add_resource(self, iterator, tag, data, config, pos) 3015 resource = Resource(config=config, pos=pos, **data) 3016 self.resources.append(resource) -> 3017 resource.parse(self, iterator, config) 3018 3019 def _add_coosys(self, iterator, tag, data, config, pos): /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc in parse(self, votable, iterator, config) 2871 if start: 2872 tag_mapping.get(tag, self._add_unknown_tag)( -> 2873 iterator, tag, data, config, pos) 2874 elif tag == 'DESCRIPTION': 2875 if self.description is not None: /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc in _add_table(self, iterator, tag, data, config, pos) 2828 table = Table(self._votable, config=config, pos=pos, **data) 2829 self.tables.append(table) -> 2830 table.parse(iterator, config) 2831 2832 def _add_info(self, iterator, tag, data, config, pos): /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc in parse(self, iterator, config) 2212 'TABLEDATA', data.iterkeys(), config, pos) 2213 self.array = self._parse_tabledata( -> 2214 iterator, colnumbers, fields, config) 2215 break 2216 elif tag == 'BINARY': /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc in _parse_tabledata(self, iterator, colnumbers, fields, config) 2357 2358 array = _resize(array, alloc_rows) -> 2359 array[numrows:] = array_chunk 2360 if alloc_rows != 0: 2361 array.mask[numrows:] = mask_chunk /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in __setslice__(self, i, j, value) 3067 3068 """ -> 3069 self.__setitem__(slice(i, j), value) 3070 3071 /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc in __setitem__(self, indx, value) 3031 _mask = ndarray.__getattribute__(self, '_mask') 3032 # Set the data, then the mask -> 3033 ndarray.__setitem__(_data, indx, dval) 3034 ndarray.__setitem__(_mask, indx, mval) 3035 elif hasattr(indx, 'dtype') and (indx.dtype == MaskType): OverflowError: Python int too large to convert to C long -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test1.xml Type: application/xml Size: 17588 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Tue Feb 26 10:20:37 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 26 Feb 2013 10:20:37 -0500 Subject: [AstroPy] Problem reading VOTable produced by HLA In-Reply-To: References: <512BA60C.3070505@stsci.edu> Message-ID: <512CD2C5.2010403@stsci.edu> On 02/26/2013 09:51 AM, Andy Ptak wrote: > Hello, > I am trying out reading VOTables using and am having trouble. > Specifically reading tables returned by the Hubble Legacy Archive. > The query was: > http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389 > which I attached. > > I am using astropy installed by pip on the enthought distribution on a > mac, so its version 0.2 and numpy got upgraded to 1.7.0 (if that matters). > First, is there a way to have a votable imported directly from query > without saving it to a physical file first? Maybe its best practice > to save VO queries to a file to trace errors, in case the votable is > very large, etc. but when you know you are getting something back that > is small and (as in this case) you mainly want a list of fields for > further work, saving the full votable seams like an unnecessary extra > step. You can pass an http url to astropy.io.votable.parse, e.g.: In [2]: from astropy.io import votable In [3]: votable.parse("http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389") > > The errors I am getting are below. I got similar errors when using a > different votable call I found on the HLA website (basically the same > except using acsSIAP.cgi?strict=1 > Is this a problem with the Votable being returned by stsci, a bug or > am I doing something wrong? Interestingly, I'm not able to reproduce the exception here. (This is on Linux, with Python 2.7, astropy 0.2 and numpy 1.7.0). Would you mind tinkering around in the debugger for me? Turn on the IPython debugger with "%pdb", reproduce the crash, and then go up two levels and print out "array_chunk"? I'm curious if there's some messed up values getting in there. Thanks, Mike > > Thanks, > Andy Ptak > > > > In [4]: from astropy.io.votable import parse > In [6]: votable = parse("test1.xml") > WARNING: W42: test1.xml:3:0: W42: No XML namespace specified > [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:7:4: W03: Implictly generating an ID from a > name 'INPUT:POS' -> 'INPUT_POS' [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:8:4: W03: Implictly generating an ID from a > name 'INPUT:SIZE' -> 'INPUT_SIZE' [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:9:4: W03: Implictly generating an ID from a > name 'INPUT:FORMAT' -> 'INPUT_FORMAT' [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:10:4: W03: Implictly generating an ID from a > name 'INPUT:imagetype' -> 'INPUT_imagetype' > [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:11:4: W03: Implictly generating an ID from a > name 'INPUT:inst' -> 'INPUT_inst' [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:12:4: W03: Implictly generating an ID from a > name 'INPUT:hrcmatch' -> 'INPUT_hrcmatch' [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:13:4: W03: Implictly generating an ID from a > name 'INPUT:zoom' -> 'INPUT_zoom' [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:14:4: W03: Implictly generating an ID from a > name 'INPUT:autoscale' -> 'INPUT_autoscale' > [astropy.io.votable.exceptions] > WARNING: W03: test1.xml:15:4: W03: Implictly generating an ID from a > name 'INPUT:asinh' -> 'INPUT_asinh' [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:16:4: W06: Invalid UCD > 'VOX:STC_CoordRefFrame': UCD has invalid character ':' in > 'VOX:STC_CoordRefFrame' [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:17:4: W06: Invalid UCD > 'VOX:WCS_CoordProjection': UCD has invalid character ':' in > 'VOX:WCS_CoordProjection' [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:19:6: W06: Invalid UCD > 'VOX:Image_AccessReference': UCD has invalid character ':' in > 'VOX:Image_AccessReference' [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:82:6: W06: Invalid UCD 'VOX:Image_Format': UCD > has invalid character ':' in 'VOX:Image_Format' > [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:91:6: W06: Invalid UCD 'VOX:Image_Title': UCD > has invalid character ':' in 'VOX:Image_Title' > [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:94:6: W06: Invalid UCD 'VOX:Image_Naxis': UCD > has invalid character ':' in 'VOX:Image_Naxis' > [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:97:6: W06: Invalid UCD 'VOX:Image_Scale': UCD > has invalid character ':' in 'VOX:Image_Scale' > [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:100:6: W06: Invalid UCD 'VOX:Image_Naxes': UCD > has invalid character ':' in 'VOX:Image_Naxes' > [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:103:6: W06: Invalid UCD > 'VOX:WCS_CoordRefPixel': UCD has invalid character ':' in > 'VOX:WCS_CoordRefPixel' [astropy.io.votable.exceptions] > WARNING: W06: test1.xml:106:6: W06: Invalid UCD > 'VOX:WCS_CoordRefValue': UCD has invalid character ':' in > 'VOX:WCS_CoordRefValue' (suppressing further warnings of this type...) > [astropy.io.votable.exceptions] > ERROR: OverflowError: Python int too large to convert to C long > [numpy.ma.core] > --------------------------------------------------------------------------- > OverflowError Traceback (most recent call > last) > /Volumes/Apps_and_Docs/aptak/Dropbox/HLA/ in > () > ----> 1 votable = parse("test1.xml") > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/table.pyc > in parse(source, columns, invalid, pedantic, chunk_size, table_number, > table_id, filename, _debug_python_based_parser) > 114 _debug_python_based_parser=_debug_python_based_parser) as > iterator: > 115 return tree.VOTableFile( > --> 116 config=config, pos=(1, 1)).parse(iterator, config) > 117 > 118 > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc > in parse(self, iterator, config) > 3084 if start: > 3085 tag_mapping.get(tag, self._add_unknown_tag)( > -> 3086 iterator, tag, data, config, pos) > 3087 elif tag == 'DESCRIPTION': > 3088 if self.description is not None: > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc > in _add_resource(self, iterator, tag, data, config, pos) > 3015 resource = Resource(config=config, pos=pos, **data) > 3016 self.resources.append(resource) > -> 3017 resource.parse(self, iterator, config) > 3018 > 3019 def _add_coosys(self, iterator, tag, data, config, pos): > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc > in parse(self, votable, iterator, config) > 2871 if start: > 2872 tag_mapping.get(tag, self._add_unknown_tag)( > -> 2873 iterator, tag, data, config, pos) > 2874 elif tag == 'DESCRIPTION': > 2875 if self.description is not None: > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc > in _add_table(self, iterator, tag, data, config, pos) > 2828 table = Table(self._votable, config=config, pos=pos, > **data) > 2829 self.tables.append(table) > -> 2830 table.parse(iterator, config) > 2831 > 2832 def _add_info(self, iterator, tag, data, config, pos): > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc > in parse(self, iterator, config) > 2212 'TABLEDATA', data.iterkeys(), > config, pos) > 2213 self.array = self._parse_tabledata( > -> 2214 iterator, colnumbers, fields, config) > 2215 break > 2216 elif tag == 'BINARY': > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc > in _parse_tabledata(self, iterator, colnumbers, fields, config) > 2357 > 2358 array = _resize(array, alloc_rows) > -> 2359 array[numrows:] = array_chunk > 2360 if alloc_rows != 0: > 2361 array.mask[numrows:] = mask_chunk > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc > in __setslice__(self, i, j, value) > 3067 > 3068 """ > -> 3069 self.__setitem__(slice(i, j), value) > 3070 > 3071 > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc > in __setitem__(self, indx, value) > 3031 _mask = ndarray.__getattribute__(self, '_mask') > 3032 # Set the data, then the mask > > -> 3033 ndarray.__setitem__(_data, indx, dval) > 3034 ndarray.__setitem__(_mask, indx, mval) > 3035 elif hasattr(indx, 'dtype') and (indx.dtype == MaskType): > > OverflowError: Python int too large to convert to C long -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Tue Feb 26 11:08:47 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 26 Feb 2013 11:08:47 -0500 Subject: [AstroPy] Problem reading VOTable produced by HLA In-Reply-To: <8E84403D-9265-420E-8BD7-F2CA4768573B@nasa.gov> References: <512BA60C.3070505@stsci.edu> <512CD2C5.2010403@stsci.edu> <8E84403D-9265-420E-8BD7-F2CA4768573B@nasa.gov> Message-ID: <512CDE0F.1010001@stsci.edu> Column #24 is defined as follows: Here, a datatype of "int" means a 32-bit int, always, as per the VOTable spec. Yet the value in the first row of this column is -2208988800, which is too large to be represented in 32 bits. This field should really be updated to be of type "long". I'll go ahead and report this to the HLA folks. Unfortunately, it seems that Numpy complains about too long numbers going into a 32-bit field on a 32-bit platform, but not on a 64-platform, where it just silently truncates. It would be nice to at least have consistent behavior across platforms there. Again, I'll file a bug with Numpy about this to see what, if anything, that team may want to do. In any case, the real bug is in the file -- astropy is just correctly following directions here and only allocating as much memory as specified in the file. It would be nice to workaround this bug in the file, but it's a tricky one -- we would have to allocate the table in one way, come across the overflow later, and then reparse the whole file using a different table structure. None of our existing workarounds do that, so it's not the most straightforward thing to do. Mike On 02/26/2013 10:44 AM, Andy Ptak wrote: > Sure? > ipdb> print array_chunk > [('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref' > , > 11.9123, -25.2889, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', 'F475W', > 4, 1482.0, '2006-09-13 06:38:18', > 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', 10915, > '10915-ngc0253-wide5', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 > 13:59:02', 'HLSP', 4744.35009766, 'image/fits', '1900-01-01 00:00:00', > -2208988800L, 'angst_hst_acs-wfc_10915-ngc0253-wide5 ACS/WFC F475W > (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38889e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [2055.6 1148.67], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.9123 -25.2889], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.06388e-05 -8.9285e-06 -8.9285e-06 1.06388e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', 'Polygon > J2000 11.947768 -25.282758 11.905331 -25.237036 11.855680 > -25.274704 11.898112 -25.320439'), > ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz' > , > 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', 'F606W', > 4, 1508.0, '2006-09-13 09:42:19', > 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 10915, > 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 > 14:01:51', 'HLSP', 5917.68017578, 'image/fits', '1900-01-01 00:00:00', > -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC F606W (hlsp) > NGC0253-WIDE5', masked_array(data = [4220 4299], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38892e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [2110.0 2149.5], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.9017 -25.2787], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 1.06397e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 'Polygon J2000 > 11.947776 -25.282766 11.905328 -25.237033 11.855676 > -25.274701 11.898119 -25.320449'), > ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz' > , > 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', 'F814W', > 4, 1534.0, '2006-09-13 11:18:39', > 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 10915, > 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 > 14:01:51', 'HLSP', 8059.75976562, 'image/fits', '1900-01-01 00:00:00', > -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC F814W (hlsp) > NGC0253-WIDE5', masked_array(data = [4220 4298], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38892e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [2110.0 2149.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.9017 -25.2787], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 1.06397e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 'Polygon J2000 > 11.947772 -25.282761 11.905333 -25.237039 11.855682 > -25.274707 11.898114 -25.320444'), > ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz&blue=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz&size=ALL&format=fits' > , > 11.9017, -25.2787, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', > 'F814W/F606W', 8, 3042.0, '2006-09-13 09:42:19', > 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_f606w_v1_drz', 10915, > 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 > 14:01:51', 'HLSP', 6989.0, 'image/fits', '1900-01-01 00:00:00', > -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC F814W/F606W > (color) NGC0253-WIDE5', masked_array(data = [4220 4298], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38892e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [2110.0 2149.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.9017 -25.2787], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 1.06397e-05], > mask = [False False False False], > fill_value = 1e+20) > , > 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz,hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', > 'Polygon J2000 11.947772 -25.282761 11.905333 -25.237039 > 11.855682 -25.274707 11.898114 -25.320444'), > ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F475W' > , > 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', > 'F475W', 2, 1482.0, '2006-09-13 06:38:18', > 'HST_10915_98_ACS_WFC_F475W', 10915, '98', 'Dalcanton', '', 'Loaded', > 'ACCUM', '2008-01-17 03:05:31', 'HLA', 4744.36, 'image/fits', > '2006-09-14 00:47:14', 1158194834, '10915_98 ACS/WFC F475W (combined) > NGC0253-WIDE5', masked_array(data = [6100 6100], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38889e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [3050.0 3050.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.901736 -25.279734], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'HST_10915_98_ACS_WFC_F475W', 'Polygon J2000 11.87717390 > -25.29535800 11.89640629 -25.28200592 11.87697850 -25.29587050 > 11.87759056 -25.29648032 11.89825680 -25.31706610 11.89831934 > -25.31702147 11.89893180 -25.31763140 11.94817110 -25.28247540 > 11.92689000 -25.26128670 11.92682706 -25.26133165 11.92621490 > -25.26072200 11.92600258 -25.26087365 11.90560880 -25.24086630 > 11.90555076 -25.24090663 11.90493420 -25.24030160 11.85576560 > -25.27444230 11.85638210 -25.27504754 11.87649930 -25.29479280 > 11.87655706 -25.29475271'), > ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F606W' > , > 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', > 'F606W', 2, 1508.0, '2006-09-13 09:42:19', > 'HST_10915_98_ACS_WFC_F606W', 10915, '98', 'Dalcanton', '', 'Loaded', > 'ACCUM', '2008-01-17 03:05:31', 'HLA', 5917.68, 'image/fits', > '2006-09-14 10:03:41', 1158228221, '10915_98 ACS/WFC F606W (combined) > NGC0253-WIDE5', masked_array(data = [6100 6100], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38889e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [3050.0 3050.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.901736 -25.279734], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'HST_10915_98_ACS_WFC_F606W', 'Polygon J2000 11.92689010 > -25.26128690 11.92682692 -25.26133202 11.92621490 -25.26072250 > 11.92600228 -25.26087437 11.90560890 -25.24086820 11.90555051 > -25.24090876 11.90493430 -25.24030410 11.85576720 -25.27444250 > 11.85638324 -25.27504727 11.87650030 -25.29479150 11.87655775 > -25.29475163 11.87717450 -25.29535680 11.89645734 -25.28197019 > 11.87697940 -25.29587070 11.87759135 -25.29648042 11.89825680 > -25.31706560 11.89831905 -25.31702118 11.89893170 -25.31763130 > 11.94817060 -25.28247510'), > ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F814W' > , > 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', > 'F814W', 2, 1534.0, '2006-09-13 11:18:39', > 'HST_10915_98_ACS_WFC_F814W', 10915, '98', 'Dalcanton', '', 'Loaded', > 'ACCUM', '2008-01-17 03:05:31', 'HLA', 8059.76, 'image/fits', > '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC F814W (combined) > NGC0253-WIDE5', masked_array(data = [6100 6100], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38889e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [3050.0 3050.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.901736 -25.279734], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'HST_10915_98_ACS_WFC_F814W', 'Polygon J2000 11.90555223 > -25.24091185 11.90493480 -25.24030600 11.85576830 -25.27444290 > 11.87650110 -25.29479130 11.87655960 -25.29475070 11.87717680 > -25.29535630 11.89640118 -25.28201069 11.87698000 -25.29587030 > 11.89825710 -25.31706440 11.89832033 -25.31701928 11.89893250 > -25.31762890 11.94810647 -25.28252025 11.94816870 -25.28247580 > 11.92688930 -25.26128920 11.92682707 -25.26133365 11.92621460 > -25.26072370 11.92600121 -25.26087611 11.90560960 -25.24087200'), > ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_total' > , > 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', > 'detection', 6, 4524.0, '2006-09-13 06:38:18', > 'HST_10915_98_ACS_WFC_total', 10915, '98', 'Dalcanton', '', 'Loaded', > 'ACCUM', '2008-01-17 03:05:31', 'HLA', None, 'image/fits', '2006-09-14 > 16:27:26', 1158251246, '10915_98 ACS/WFC detection (combined) > NGC0253-WIDE5', masked_array(data = [6100 6100], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38889e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [3050.0 3050.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.901736 -25.279734], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], > mask = [False False False False], > fill_value = 1e+20) > , 'HST_10915_98_ACS_WFC_total', 'Polygon J2000 11.90560880 > -25.24086630 11.90555076 -25.24090663 11.90493420 -25.24030160 > 11.85576560 -25.27444230 11.85638210 -25.27504754 11.87649930 > -25.29479280 11.87655706 -25.29475271 11.87717390 -25.29535800 > 11.87717661 -25.29535612 11.89634395 -25.28205042 11.87697850 > -25.29587050 11.87759056 -25.29648032 11.89825680 -25.31706610 > 11.89831934 -25.31702147 11.89893180 -25.31763140 11.94817110 > -25.28247540 11.92689000 -25.26128670 11.92682706 -25.26133165 > 11.92621490 -25.26072200 11.92600258 -25.26087365'), > ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=HST_10915_98_ACS_WFC_F814W&green=HST_10915_98_ACS_WFC_F606W&blue=HST_10915_98_ACS_WFC_F475W&size=ALL&format=fits' > , > 11.901736, -25.279734, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', > 'F814W/F606W/F475W', 6, 4524.0, '2006-09-13 06:38:18', > 'HST_10915_98_ACS_WFC_F814W_F606W_F475W', 10915, '98', 'Dalcanton', > '', 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 6241.0, > 'image/fits', '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC > F814W/F606W/F475W (color) NGC0253-WIDE5', masked_array(data = [6100 6100], > mask = [False False], > fill_value = 999999) > , masked_array(data = [1.38889e-05], > mask = [False], > fill_value = 1e+20) > , 2, masked_array(data = [3050.0 3050.0], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [11.901736 -25.279734], > mask = [False False], > fill_value = 1e+20) > , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], > mask = [False False False False], > fill_value = 1e+20) > , > 'HST_10915_98_ACS_WFC_F814W,HST_10915_98_ACS_WFC_F606W,HST_10915_98_ACS_WFC_F475W', > 'Polygon J2000 11.90555223 -25.24091185 11.90493480 -25.24030600 > 11.85576830 -25.27444290 11.87650110 -25.29479130 11.87655960 > -25.29475070 11.87717680 -25.29535630 11.89640118 -25.28201069 > 11.87698000 -25.29587030 11.89825710 -25.31706440 11.89832033 > -25.31701928 11.89893250 -25.31762890 11.94810647 -25.28252025 > 11.94816870 -25.28247580 11.92688930 -25.26128920 11.92682707 > -25.26133365 11.92621460 -25.26072370 11.92600121 -25.26087611 > 11.90560960 -25.24087200')] > > On Feb 26, 2013, at 10:20 AM, Michael Droettboom > wrote: > >> On 02/26/2013 09:51 AM, Andy Ptak wrote: >>> Hello, >>> I am trying out reading VOTables using and am having trouble. >>> Specifically reading tables returned by the Hubble Legacy Archive. >>> The query was: >>> http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389 >>> which I attached. >>> >>> I am using astropy installed by pip on the enthought distribution on >>> a mac, so its version 0.2 and numpy got upgraded to 1.7.0 (if that >>> matters). >>> First, is there a way to have a votable imported directly from query >>> without saving it to a physical file first? Maybe its best >>> practice to save VO queries to a file to trace errors, in case the >>> votable is very large, etc. but when you know you are getting >>> something back that is small and (as in this case) you mainly want a >>> list of fields for further work, saving the full votable seams like >>> an unnecessary extra step. >> >> You can pass an http url to astropy.io.votable.parse, e.g.: >> >> In [2]: from astropy.io import votable >> >> In [3]: >> votable.parse("http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389") >> >>> >>> The errors I am getting are below. I got similar errors when using >>> a different votable call I found on the HLA website (basically the >>> same except using acsSIAP.cgi?strict=1 >>> Is this a problem with the Votable being returned by stsci, a bug or >>> am I doing something wrong? >> >> Interestingly, I'm not able to reproduce the exception here. (This >> is on Linux, with Python 2.7, astropy 0.2 and numpy 1.7.0). >> >> Would you mind tinkering around in the debugger for me? Turn on the >> IPython debugger with "%pdb", reproduce the crash, and then go up two >> levels and print out "array_chunk"? I'm curious if there's some >> messed up values getting in there. >> >> Thanks, >> Mike >> >>> >>> Thanks, >>> Andy Ptak >>> >>> >>> >>> In [4]: from astropy.io.votable import parse >>> In [6]: votable = parse("test1.xml") >>> WARNING: W42: test1.xml:3:0: W42: No XML namespace specified >>> [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:7:4: W03: Implictly generating an ID from a >>> name 'INPUT:POS' -> 'INPUT_POS' [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:8:4: W03: Implictly generating an ID from a >>> name 'INPUT:SIZE' -> 'INPUT_SIZE' [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:9:4: W03: Implictly generating an ID from a >>> name 'INPUT:FORMAT' -> 'INPUT_FORMAT' [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:10:4: W03: Implictly generating an ID from a >>> name 'INPUT:imagetype' -> 'INPUT_imagetype' >>> [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:11:4: W03: Implictly generating an ID from a >>> name 'INPUT:inst' -> 'INPUT_inst' [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:12:4: W03: Implictly generating an ID from a >>> name 'INPUT:hrcmatch' -> 'INPUT_hrcmatch' >>> [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:13:4: W03: Implictly generating an ID from a >>> name 'INPUT:zoom' -> 'INPUT_zoom' [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:14:4: W03: Implictly generating an ID from a >>> name 'INPUT:autoscale' -> 'INPUT_autoscale' >>> [astropy.io.votable.exceptions] >>> WARNING: W03: test1.xml:15:4: W03: Implictly generating an ID from a >>> name 'INPUT:asinh' -> 'INPUT_asinh' [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:16:4: W06: Invalid UCD >>> 'VOX:STC_CoordRefFrame': UCD has invalid character ':' in >>> 'VOX:STC_CoordRefFrame' [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:17:4: W06: Invalid UCD >>> 'VOX:WCS_CoordProjection': UCD has invalid character ':' in >>> 'VOX:WCS_CoordProjection' [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:19:6: W06: Invalid UCD >>> 'VOX:Image_AccessReference': UCD has invalid character ':' in >>> 'VOX:Image_AccessReference' [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:82:6: W06: Invalid UCD 'VOX:Image_Format': >>> UCD has invalid character ':' in 'VOX:Image_Format' >>> [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:91:6: W06: Invalid UCD 'VOX:Image_Title': >>> UCD has invalid character ':' in 'VOX:Image_Title' >>> [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:94:6: W06: Invalid UCD 'VOX:Image_Naxis': >>> UCD has invalid character ':' in 'VOX:Image_Naxis' >>> [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:97:6: W06: Invalid UCD 'VOX:Image_Scale': >>> UCD has invalid character ':' in 'VOX:Image_Scale' >>> [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:100:6: W06: Invalid UCD 'VOX:Image_Naxes': >>> UCD has invalid character ':' in 'VOX:Image_Naxes' >>> [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:103:6: W06: Invalid UCD >>> 'VOX:WCS_CoordRefPixel': UCD has invalid character ':' in >>> 'VOX:WCS_CoordRefPixel' [astropy.io.votable.exceptions] >>> WARNING: W06: test1.xml:106:6: W06: Invalid UCD >>> 'VOX:WCS_CoordRefValue': UCD has invalid character ':' in >>> 'VOX:WCS_CoordRefValue' (suppressing further warnings of this >>> type...) [astropy.io.votable.exceptions] >>> ERROR: OverflowError: Python int too large to convert to C long >>> [numpy.ma.core] >>> --------------------------------------------------------------------------- >>> OverflowError Traceback (most recent call last) >>> /Volumes/Apps_and_Docs/aptak/Dropbox/HLA/ >>> in () >>> ----> 1 votable = parse("test1.xml") >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/table.pyc >>> in parse(source, columns, invalid, pedantic, chunk_size, >>> table_number, table_id, filename, _debug_python_based_parser) >>> 114 _debug_python_based_parser=_debug_python_based_parser) as >>> iterator: >>> 115 return tree.VOTableFile( >>> --> 116 config=config, pos=(1, 1)).parse(iterator, config) >>> 117 >>> 118 >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>> in parse(self, iterator, config) >>> 3084 if start: >>> 3085 tag_mapping.get(tag, self._add_unknown_tag)( >>> -> 3086 iterator, tag, data, config, pos) >>> 3087 elif tag == 'DESCRIPTION': >>> 3088 if self.description is not None: >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>> in _add_resource(self, iterator, tag, data, config, pos) >>> 3015 resource = Resource(config=config, pos=pos, **data) >>> 3016 self.resources.append(resource) >>> -> 3017 resource.parse(self, iterator, config) >>> 3018 >>> 3019 def _add_coosys(self, iterator, tag, data, config, pos): >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>> in parse(self, votable, iterator, config) >>> 2871 if start: >>> 2872 tag_mapping.get(tag, self._add_unknown_tag)( >>> -> 2873 iterator, tag, data, config, pos) >>> 2874 elif tag == 'DESCRIPTION': >>> 2875 if self.description is not None: >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>> in _add_table(self, iterator, tag, data, config, pos) >>> 2828 table = Table(self._votable, config=config, pos=pos, >>> **data) >>> 2829 self.tables.append(table) >>> -> 2830 table.parse(iterator, config) >>> 2831 >>> 2832 def _add_info(self, iterator, tag, data, config, pos): >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>> in parse(self, iterator, config) >>> 2212 'TABLEDATA', data.iterkeys(), config, pos) >>> 2213 self.array = self._parse_tabledata( >>> -> 2214 iterator, colnumbers, fields, config) >>> 2215 break >>> 2216 elif tag == 'BINARY': >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>> in _parse_tabledata(self, iterator, colnumbers, fields, config) >>> 2357 >>> 2358 array = _resize(array, alloc_rows) >>> -> 2359 array[numrows:] = array_chunk >>> 2360 if alloc_rows != 0: >>> 2361 array.mask[numrows:] = mask_chunk >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>> in __setslice__(self, i, j, value) >>> 3067 >>> 3068 """ >>> -> 3069 self.__setitem__(slice(i, j), value) >>> 3070 >>> 3071 >>> >>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>> in __setitem__(self, indx, value) >>> 3031 _mask = ndarray.__getattribute__(self, '_mask') >>> 3032 # Set the data, then the mask >>> >>> -> 3033 ndarray.__setitem__(_data, indx, dval) >>> 3034 ndarray.__setitem__(_mask, indx, mval) >>> 3035 elif hasattr(indx, 'dtype') and (indx.dtype == >>> MaskType): >>> >>> OverflowError: Python int too large to convert to C long >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Tue Feb 26 12:20:41 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 26 Feb 2013 12:20:41 -0500 Subject: [AstroPy] saving a votable as a xml file using astropy In-Reply-To: References: <512BA6B5.2070209@stsci.edu> Message-ID: <512CEEE9.4000900@stsci.edu> Yes, exactly. Cheers, Mike On 02/26/2013 03:35 AM, Susana Sanchez Exposito wrote: > Thanks Mike, > > At first, I was not sure what you mean, because I didn't find any > attribute of VOTableFile for the data array, which can be modified. > Then I realized the method VOTableFile.get_first_table() returns a > *reference* of the first table (I thought it would return a *copy* of > the table), so I guess the way you said is the way I code below, isn't > it?: > > #Open the votable > votable = parse("cig22.xml") > > #Get a reference of the table > table=votable.get_first_table() > > #Modify the table > table.array['CIG Number'][0]="CIG SUSANA" > > #The cig_s.xml table will contain the before modification > votable.to_xml("cig_s.xml") > > > Regards and thanks again for your help. > Susana. > > > > > 2013/2/25 Michael Droettboom : >> VOTable is actually a container format for multiple tables, so you can only >> save that entire container to XML. >> >> The best thing to do to make modifications to save back out would be to >> retain a reference to the top-level VOTableFile object (not the >> votable.tree.Table object), and call `to_xml` on that. That approach should >> also maintain all of the metadata (what the original names and IDs were). >> >> Hope that helps, and feel free to ping me if you need more detail. >> >> Mike >> >> >> On 02/25/2013 12:46 PM, Susana Sanchez wrote: >> >> Hi again, >> >> This time I have the doubt saving votables as xml. As I said in my >> previous mail, I want to show the VOTable in a nice way using Qt >> library, I also want to allow the users can modify this votable and >> then save the changes. >> >> Reading the documentation I saw that the astropy.io.votable.tree.Table >> has the method .to_xml, but probably I am not using it well because I >> get this error: >> >> In [159]: table = >> parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) >> In [160]: table.to_xml("./cig22_changed.xml") >> --------------------------------------------------------------------------- >> AttributeError Traceback (most recent call last) >> >> /home/susana/ in () >> >> /usr/local/lib/python2.6/dist-packages/astropy/io/votable/tree.pyc in >> to_xml(self, w, **kwargs) >> 2518 >> 2519 def to_xml(self, w, **kwargs): >> -> 2520 with w.tag( >> 2521 u'TABLE', >> 2522 attrib=w.object_attrs( >> >> AttributeError: 'str' object has no attribute 'tag' >> >> >> Finally I got this way to save the xml: >> >> #Open the votable >> table = >> astropy.io.votable.parse_single_table("/home/susana/Documents/guipsy/examples/tables/cig22.xml",pedantic=False) >> >> #Modify the table >> table.array['CIG Number'][0]="CIG SUSANA" >> >> #Create a VOTableFile instance of the table >> votable= VOTableFile.from_table(table.to_table()) >> >> #Save the xml >> votable.to_xml('cig22S.xml') >> >> >> I have two questions/comments: >> >> 1) Is this way correct? Or maybe there is a better way to save the >> changes made in a VOtable? >> >> 2) The VOTable created, 'cig22S.xml', is ok, but the fields have lost >> the 'name' value. Now, the 'ID' and the 'name' have the same value >> (the original value of ID). >> >> >> >> Thanks again, >> Susana. >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy >> > > From mdroe at stsci.edu Tue Feb 26 12:51:57 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 26 Feb 2013 12:51:57 -0500 Subject: [AstroPy] Problem reading VOTable produced by HLA In-Reply-To: <512CDE0F.1010001@stsci.edu> References: <512BA60C.3070505@stsci.edu> <512CD2C5.2010403@stsci.edu> <8E84403D-9265-420E-8BD7-F2CA4768573B@nasa.gov> <512CDE0F.1010001@stsci.edu> Message-ID: <512CF63D.2050309@stsci.edu> Also see the following PR for astropy which will emit a warning and truncate out-of-range integer values: https://github.com/astropy/astropy/pull/825 Mike On 02/26/2013 11:08 AM, Michael Droettboom wrote: > Column #24 is defined as follows: > > ucd="TIME_EPOCH"> > > Here, a datatype of "int" means a 32-bit int, always, as per the > VOTable spec. > > Yet the value in the first row of this column is -2208988800, which is > too large to be represented in 32 bits. This field should really be > updated to be of type "long". I'll go ahead and report this to the > HLA folks. > > Unfortunately, it seems that Numpy complains about too long numbers > going into a 32-bit field on a 32-bit platform, but not on a > 64-platform, where it just silently truncates. It would be nice to at > least have consistent behavior across platforms there. Again, I'll > file a bug with Numpy about this to see what, if anything, that team > may want to do. > > In any case, the real bug is in the file -- astropy is just correctly > following directions here and only allocating as much memory as > specified in the file. It would be nice to workaround this bug in the > file, but it's a tricky one -- we would have to allocate the table in > one way, come across the overflow later, and then reparse the whole > file using a different table structure. None of our existing > workarounds do that, so it's not the most straightforward thing to do. > > Mike > > On 02/26/2013 10:44 AM, Andy Ptak wrote: >> Sure... >> ipdb> print array_chunk >> [('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref' >> , >> 11.9123, -25.2889, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F475W', 4, 1482.0, '2006-09-13 06:38:18', >> 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', 10915, >> '10915-ngc0253-wide5', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >> 13:59:02', 'HLSP', 4744.35009766, 'image/fits', '1900-01-01 >> 00:00:00', -2208988800L, 'angst_hst_acs-wfc_10915-ngc0253-wide5 >> ACS/WFC F475W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38889e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [2055.6 1148.67], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.9123 -25.2889], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.06388e-05 -8.9285e-06 -8.9285e-06 1.06388e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', 'Polygon >> J2000 11.947768 -25.282758 11.905331 -25.237036 11.855680 >> -25.274704 11.898112 -25.320439'), >> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz' >> , >> 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F606W', 4, 1508.0, '2006-09-13 09:42:19', >> 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 10915, >> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >> 14:01:51', 'HLSP', 5917.68017578, 'image/fits', '1900-01-01 >> 00:00:00', -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC >> F606W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4299], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38892e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [2110.0 2149.5], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.9017 -25.2787], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 1.06397e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 'Polygon J2000 >> 11.947776 -25.282766 11.905328 -25.237033 11.855676 >> -25.274701 11.898119 -25.320449'), >> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz' >> , >> 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F814W', 4, 1534.0, '2006-09-13 11:18:39', >> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 10915, >> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >> 14:01:51', 'HLSP', 8059.75976562, 'image/fits', '1900-01-01 >> 00:00:00', -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC >> F814W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38892e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [2110.0 2149.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.9017 -25.2787], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 1.06397e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 'Polygon J2000 >> 11.947772 -25.282761 11.905333 -25.237039 11.855682 >> -25.274707 11.898114 -25.320444'), >> ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz&blue=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz&size=ALL&format=fits' >> , >> 11.9017, -25.2787, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F814W/F606W', 8, 3042.0, '2006-09-13 09:42:19', >> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_f606w_v1_drz', 10915, >> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >> 14:01:51', 'HLSP', 6989.0, 'image/fits', '1900-01-01 00:00:00', >> -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC F814W/F606W >> (color) NGC0253-WIDE5', masked_array(data = [4220 4298], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38892e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [2110.0 2149.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.9017 -25.2787], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 1.06397e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , >> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz,hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', >> 'Polygon J2000 11.947772 -25.282761 11.905333 -25.237039 >> 11.855682 -25.274707 11.898114 -25.320444'), >> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F475W' >> , >> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F475W', 2, 1482.0, '2006-09-13 06:38:18', >> 'HST_10915_98_ACS_WFC_F475W', 10915, '98', 'Dalcanton', '', 'Loaded', >> 'ACCUM', '2008-01-17 03:05:31', 'HLA', 4744.36, 'image/fits', >> '2006-09-14 00:47:14', 1158194834, '10915_98 ACS/WFC F475W (combined) >> NGC0253-WIDE5', masked_array(data = [6100 6100], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38889e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [3050.0 3050.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.901736 -25.279734], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'HST_10915_98_ACS_WFC_F475W', 'Polygon J2000 11.87717390 >> -25.29535800 11.89640629 -25.28200592 11.87697850 -25.29587050 >> 11.87759056 -25.29648032 11.89825680 -25.31706610 11.89831934 >> -25.31702147 11.89893180 -25.31763140 11.94817110 -25.28247540 >> 11.92689000 -25.26128670 11.92682706 -25.26133165 11.92621490 >> -25.26072200 11.92600258 -25.26087365 11.90560880 -25.24086630 >> 11.90555076 -25.24090663 11.90493420 -25.24030160 11.85576560 >> -25.27444230 11.85638210 -25.27504754 11.87649930 -25.29479280 >> 11.87655706 -25.29475271'), >> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F606W' >> , >> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F606W', 2, 1508.0, '2006-09-13 09:42:19', >> 'HST_10915_98_ACS_WFC_F606W', 10915, '98', 'Dalcanton', '', 'Loaded', >> 'ACCUM', '2008-01-17 03:05:31', 'HLA', 5917.68, 'image/fits', >> '2006-09-14 10:03:41', 1158228221, '10915_98 ACS/WFC F606W (combined) >> NGC0253-WIDE5', masked_array(data = [6100 6100], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38889e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [3050.0 3050.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.901736 -25.279734], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'HST_10915_98_ACS_WFC_F606W', 'Polygon J2000 11.92689010 >> -25.26128690 11.92682692 -25.26133202 11.92621490 -25.26072250 >> 11.92600228 -25.26087437 11.90560890 -25.24086820 11.90555051 >> -25.24090876 11.90493430 -25.24030410 11.85576720 -25.27444250 >> 11.85638324 -25.27504727 11.87650030 -25.29479150 11.87655775 >> -25.29475163 11.87717450 -25.29535680 11.89645734 -25.28197019 >> 11.87697940 -25.29587070 11.87759135 -25.29648042 11.89825680 >> -25.31706560 11.89831905 -25.31702118 11.89893170 -25.31763130 >> 11.94817060 -25.28247510'), >> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F814W' >> , >> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F814W', 2, 1534.0, '2006-09-13 11:18:39', >> 'HST_10915_98_ACS_WFC_F814W', 10915, '98', 'Dalcanton', '', 'Loaded', >> 'ACCUM', '2008-01-17 03:05:31', 'HLA', 8059.76, 'image/fits', >> '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC F814W (combined) >> NGC0253-WIDE5', masked_array(data = [6100 6100], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38889e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [3050.0 3050.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.901736 -25.279734], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'HST_10915_98_ACS_WFC_F814W', 'Polygon J2000 11.90555223 >> -25.24091185 11.90493480 -25.24030600 11.85576830 -25.27444290 >> 11.87650110 -25.29479130 11.87655960 -25.29475070 11.87717680 >> -25.29535630 11.89640118 -25.28201069 11.87698000 -25.29587030 >> 11.89825710 -25.31706440 11.89832033 -25.31701928 11.89893250 >> -25.31762890 11.94810647 -25.28252025 11.94816870 -25.28247580 >> 11.92688930 -25.26128920 11.92682707 -25.26133365 11.92621460 >> -25.26072370 11.92600121 -25.26087611 11.90560960 -25.24087200'), >> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_total' >> , >> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'detection', 6, 4524.0, '2006-09-13 06:38:18', >> 'HST_10915_98_ACS_WFC_total', 10915, '98', 'Dalcanton', '', 'Loaded', >> 'ACCUM', '2008-01-17 03:05:31', 'HLA', None, 'image/fits', >> '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC detection >> (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38889e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [3050.0 3050.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.901736 -25.279734], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , 'HST_10915_98_ACS_WFC_total', 'Polygon J2000 11.90560880 >> -25.24086630 11.90555076 -25.24090663 11.90493420 -25.24030160 >> 11.85576560 -25.27444230 11.85638210 -25.27504754 11.87649930 >> -25.29479280 11.87655706 -25.29475271 11.87717390 -25.29535800 >> 11.87717661 -25.29535612 11.89634395 -25.28205042 11.87697850 >> -25.29587050 11.87759056 -25.29648032 11.89825680 -25.31706610 >> 11.89831934 -25.31702147 11.89893180 -25.31763140 11.94817110 >> -25.28247540 11.92689000 -25.26128670 11.92682706 -25.26133165 >> 11.92621490 -25.26072200 11.92600258 -25.26087365'), >> ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=HST_10915_98_ACS_WFC_F814W&green=HST_10915_98_ACS_WFC_F606W&blue=HST_10915_98_ACS_WFC_F475W&size=ALL&format=fits' >> , >> 11.901736, -25.279734, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >> 'F814W/F606W/F475W', 6, 4524.0, '2006-09-13 06:38:18', >> 'HST_10915_98_ACS_WFC_F814W_F606W_F475W', 10915, '98', 'Dalcanton', >> '', 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 6241.0, >> 'image/fits', '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC >> F814W/F606W/F475W (color) NGC0253-WIDE5', masked_array(data = [6100 >> 6100], >> mask = [False False], >> fill_value = 999999) >> , masked_array(data = [1.38889e-05], >> mask = [False], >> fill_value = 1e+20) >> , 2, masked_array(data = [3050.0 3050.0], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [11.901736 -25.279734], >> mask = [False False], >> fill_value = 1e+20) >> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >> mask = [False False False False], >> fill_value = 1e+20) >> , >> 'HST_10915_98_ACS_WFC_F814W,HST_10915_98_ACS_WFC_F606W,HST_10915_98_ACS_WFC_F475W', >> 'Polygon J2000 11.90555223 -25.24091185 11.90493480 -25.24030600 >> 11.85576830 -25.27444290 11.87650110 -25.29479130 11.87655960 >> -25.29475070 11.87717680 -25.29535630 11.89640118 -25.28201069 >> 11.87698000 -25.29587030 11.89825710 -25.31706440 11.89832033 >> -25.31701928 11.89893250 -25.31762890 11.94810647 -25.28252025 >> 11.94816870 -25.28247580 11.92688930 -25.26128920 11.92682707 >> -25.26133365 11.92621460 -25.26072370 11.92600121 -25.26087611 >> 11.90560960 -25.24087200')] >> >> On Feb 26, 2013, at 10:20 AM, Michael Droettboom > > wrote: >> >>> On 02/26/2013 09:51 AM, Andy Ptak wrote: >>>> Hello, >>>> I am trying out reading VOTables using and am having trouble. >>>> Specifically reading tables returned by the Hubble Legacy Archive. >>>> The query was: >>>> http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389 >>>> which I attached. >>>> >>>> I am using astropy installed by pip on the enthought distribution >>>> on a mac, so its version 0.2 and numpy got upgraded to 1.7.0 (if >>>> that matters). >>>> First, is there a way to have a votable imported directly from >>>> query without saving it to a physical file first? Maybe its best >>>> practice to save VO queries to a file to trace errors, in case the >>>> votable is very large, etc. but when you know you are getting >>>> something back that is small and (as in this case) you mainly want >>>> a list of fields for further work, saving the full votable seams >>>> like an unnecessary extra step. >>> >>> You can pass an http url to astropy.io.votable.parse, e.g.: >>> >>> In [2]: from astropy.io import votable >>> >>> In [3]: >>> votable.parse("http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389") >>> >>>> >>>> The errors I am getting are below. I got similar errors when using >>>> a different votable call I found on the HLA website (basically the >>>> same except using acsSIAP.cgi?strict=1 >>>> Is this a problem with the Votable being returned by stsci, a bug >>>> or am I doing something wrong? >>> >>> Interestingly, I'm not able to reproduce the exception here. (This >>> is on Linux, with Python 2.7, astropy 0.2 and numpy 1.7.0). >>> >>> Would you mind tinkering around in the debugger for me? Turn on the >>> IPython debugger with "%pdb", reproduce the crash, and then go up >>> two levels and print out "array_chunk"? I'm curious if there's some >>> messed up values getting in there. >>> >>> Thanks, >>> Mike >>> >>>> >>>> Thanks, >>>> Andy Ptak >>>> >>>> >>>> >>>> In [4]: from astropy.io.votable import parse >>>> In [6]: votable = parse("test1.xml") >>>> WARNING: W42: test1.xml:3:0: W42: No XML namespace specified >>>> [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:7:4: W03: Implictly generating an ID from a >>>> name 'INPUT:POS' -> 'INPUT_POS' [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:8:4: W03: Implictly generating an ID from a >>>> name 'INPUT:SIZE' -> 'INPUT_SIZE' [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:9:4: W03: Implictly generating an ID from a >>>> name 'INPUT:FORMAT' -> 'INPUT_FORMAT' [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:10:4: W03: Implictly generating an ID from >>>> a name 'INPUT:imagetype' -> 'INPUT_imagetype' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:11:4: W03: Implictly generating an ID from >>>> a name 'INPUT:inst' -> 'INPUT_inst' [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:12:4: W03: Implictly generating an ID from >>>> a name 'INPUT:hrcmatch' -> 'INPUT_hrcmatch' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:13:4: W03: Implictly generating an ID from >>>> a name 'INPUT:zoom' -> 'INPUT_zoom' [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:14:4: W03: Implictly generating an ID from >>>> a name 'INPUT:autoscale' -> 'INPUT_autoscale' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W03: test1.xml:15:4: W03: Implictly generating an ID from >>>> a name 'INPUT:asinh' -> 'INPUT_asinh' [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:16:4: W06: Invalid UCD >>>> 'VOX:STC_CoordRefFrame': UCD has invalid character ':' in >>>> 'VOX:STC_CoordRefFrame' [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:17:4: W06: Invalid UCD >>>> 'VOX:WCS_CoordProjection': UCD has invalid character ':' in >>>> 'VOX:WCS_CoordProjection' [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:19:6: W06: Invalid UCD >>>> 'VOX:Image_AccessReference': UCD has invalid character ':' in >>>> 'VOX:Image_AccessReference' [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:82:6: W06: Invalid UCD 'VOX:Image_Format': >>>> UCD has invalid character ':' in 'VOX:Image_Format' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:91:6: W06: Invalid UCD 'VOX:Image_Title': >>>> UCD has invalid character ':' in 'VOX:Image_Title' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:94:6: W06: Invalid UCD 'VOX:Image_Naxis': >>>> UCD has invalid character ':' in 'VOX:Image_Naxis' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:97:6: W06: Invalid UCD 'VOX:Image_Scale': >>>> UCD has invalid character ':' in 'VOX:Image_Scale' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:100:6: W06: Invalid UCD 'VOX:Image_Naxes': >>>> UCD has invalid character ':' in 'VOX:Image_Naxes' >>>> [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:103:6: W06: Invalid UCD >>>> 'VOX:WCS_CoordRefPixel': UCD has invalid character ':' in >>>> 'VOX:WCS_CoordRefPixel' [astropy.io.votable.exceptions] >>>> WARNING: W06: test1.xml:106:6: W06: Invalid UCD >>>> 'VOX:WCS_CoordRefValue': UCD has invalid character ':' in >>>> 'VOX:WCS_CoordRefValue' (suppressing further warnings of this >>>> type...) [astropy.io.votable.exceptions] >>>> ERROR: OverflowError: Python int too large to convert to C long >>>> [numpy.ma.core] >>>> --------------------------------------------------------------------------- >>>> OverflowError Traceback (most recent call last) >>>> /Volumes/Apps_and_Docs/aptak/Dropbox/HLA/ >>>> in () >>>> ----> 1 votable = parse("test1.xml") >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/table.pyc >>>> in parse(source, columns, invalid, pedantic, chunk_size, >>>> table_number, table_id, filename, _debug_python_based_parser) >>>> 114 _debug_python_based_parser=_debug_python_based_parser) as >>>> iterator: >>>> 115 return tree.VOTableFile( >>>> --> 116 config=config, pos=(1, 1)).parse(iterator, config) >>>> 117 >>>> 118 >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>> in parse(self, iterator, config) >>>> 3084 if start: >>>> 3085 tag_mapping.get(tag, self._add_unknown_tag)( >>>> -> 3086 iterator, tag, data, config, pos) >>>> 3087 elif tag == 'DESCRIPTION': >>>> 3088 if self.description is not None: >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>> in _add_resource(self, iterator, tag, data, config, pos) >>>> 3015 resource = Resource(config=config, pos=pos, **data) >>>> 3016 self.resources.append(resource) >>>> -> 3017 resource.parse(self, iterator, config) >>>> 3018 >>>> 3019 def _add_coosys(self, iterator, tag, data, config, pos): >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>> in parse(self, votable, iterator, config) >>>> 2871 if start: >>>> 2872 tag_mapping.get(tag, self._add_unknown_tag)( >>>> -> 2873 iterator, tag, data, config, pos) >>>> 2874 elif tag == 'DESCRIPTION': >>>> 2875 if self.description is not None: >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>> in _add_table(self, iterator, tag, data, config, pos) >>>> 2828 table = Table(self._votable, config=config, >>>> pos=pos, **data) >>>> 2829 self.tables.append(table) >>>> -> 2830 table.parse(iterator, config) >>>> 2831 >>>> 2832 def _add_info(self, iterator, tag, data, config, pos): >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>> in parse(self, iterator, config) >>>> 2212 'TABLEDATA', data.iterkeys(), config, pos) >>>> 2213 self.array = self._parse_tabledata( >>>> -> 2214 iterator, colnumbers, fields, config) >>>> 2215 break >>>> 2216 elif tag == 'BINARY': >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>> in _parse_tabledata(self, iterator, colnumbers, fields, config) >>>> 2357 >>>> 2358 array = _resize(array, alloc_rows) >>>> -> 2359 array[numrows:] = array_chunk >>>> 2360 if alloc_rows != 0: >>>> 2361 array.mask[numrows:] = mask_chunk >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>>> in __setslice__(self, i, j, value) >>>> 3067 >>>> 3068 """ >>>> -> 3069 self.__setitem__(slice(i, j), value) >>>> 3070 >>>> 3071 >>>> >>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>>> in __setitem__(self, indx, value) >>>> 3031 _mask = ndarray.__getattribute__(self, '_mask') >>>> 3032 # Set the data, then the mask >>>> >>>> -> 3033 ndarray.__setitem__(_data, indx, dval) >>>> 3034 ndarray.__setitem__(_mask, indx, mval) >>>> 3035 elif hasattr(indx, 'dtype') and (indx.dtype == >>>> MaskType): >>>> >>>> OverflowError: Python int too large to convert to C long >>> >> > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Tue Feb 26 15:21:04 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 26 Feb 2013 15:21:04 -0500 Subject: [AstroPy] Problem reading VOTable produced by HLA In-Reply-To: <512CF63D.2050309@stsci.edu> References: <512BA60C.3070505@stsci.edu> <512CD2C5.2010403@stsci.edu> <8E84403D-9265-420E-8BD7-F2CA4768573B@nasa.gov> <512CDE0F.1010001@stsci.edu> <512CF63D.2050309@stsci.edu> Message-ID: <512D1930.4080302@stsci.edu> I just got word from Rick White that this has been fixed on the HLA server as well. That was fast. Mike On 02/26/2013 12:51 PM, Michael Droettboom wrote: > Also see the following PR for astropy which will emit a warning and > truncate out-of-range integer values: > > https://github.com/astropy/astropy/pull/825 > > Mike > > On 02/26/2013 11:08 AM, Michael Droettboom wrote: >> Column #24 is defined as follows: >> >> > ucd="TIME_EPOCH"> >> >> Here, a datatype of "int" means a 32-bit int, always, as per the >> VOTable spec. >> >> Yet the value in the first row of this column is -2208988800, which >> is too large to be represented in 32 bits. This field should really >> be updated to be of type "long". I'll go ahead and report this to >> the HLA folks. >> >> Unfortunately, it seems that Numpy complains about too long numbers >> going into a 32-bit field on a 32-bit platform, but not on a >> 64-platform, where it just silently truncates. It would be nice to >> at least have consistent behavior across platforms there. Again, >> I'll file a bug with Numpy about this to see what, if anything, that >> team may want to do. >> >> In any case, the real bug is in the file -- astropy is just correctly >> following directions here and only allocating as much memory as >> specified in the file. It would be nice to workaround this bug in >> the file, but it's a tricky one -- we would have to allocate the >> table in one way, come across the overflow later, and then reparse >> the whole file using a different table structure. None of our >> existing workarounds do that, so it's not the most straightforward >> thing to do. >> >> Mike >> >> On 02/26/2013 10:44 AM, Andy Ptak wrote: >>> Sure... >>> ipdb> print array_chunk >>> [('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref' >>> , >>> 11.9123, -25.2889, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F475W', 4, 1482.0, '2006-09-13 06:38:18', >>> 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', 10915, >>> '10915-ngc0253-wide5', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 13:59:02', 'HLSP', 4744.35009766, 'image/fits', '1900-01-01 >>> 00:00:00', -2208988800L, 'angst_hst_acs-wfc_10915-ngc0253-wide5 >>> ACS/WFC F475W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2055.6 1148.67], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9123 -25.2889], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06388e-05 -8.9285e-06 -8.9285e-06 >>> 1.06388e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', >>> 'Polygon J2000 11.947768 -25.282758 11.905331 -25.237036 >>> 11.855680 -25.274704 11.898112 -25.320439'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz' >>> , >>> 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F606W', 4, 1508.0, '2006-09-13 09:42:19', >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 10915, >>> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 14:01:51', 'HLSP', 5917.68017578, 'image/fits', '1900-01-01 >>> 00:00:00', -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC >>> F606W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4299], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38892e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2110.0 2149.5], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9017 -25.2787], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 >>> 1.06397e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 'Polygon J2000 >>> 11.947776 -25.282766 11.905328 -25.237033 11.855676 >>> -25.274701 11.898119 -25.320449'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz' >>> , >>> 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W', 4, 1534.0, '2006-09-13 11:18:39', >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 10915, >>> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 14:01:51', 'HLSP', 8059.75976562, 'image/fits', '1900-01-01 >>> 00:00:00', -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC >>> F814W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38892e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2110.0 2149.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9017 -25.2787], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 >>> 1.06397e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 'Polygon J2000 >>> 11.947772 -25.282761 11.905333 -25.237039 11.855682 >>> -25.274707 11.898114 -25.320444'), >>> ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz&blue=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz&size=ALL&format=fits' >>> , >>> 11.9017, -25.2787, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W/F606W', 8, 3042.0, '2006-09-13 09:42:19', >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_f606w_v1_drz', 10915, >>> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 14:01:51', 'HLSP', 6989.0, 'image/fits', '1900-01-01 00:00:00', >>> -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC F814W/F606W >>> (color) NGC0253-WIDE5', masked_array(data = [4220 4298], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38892e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2110.0 2149.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9017 -25.2787], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 >>> 1.06397e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz,hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', >>> 'Polygon J2000 11.947772 -25.282761 11.905333 -25.237039 >>> 11.855682 -25.274707 11.898114 -25.320444'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F475W' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F475W', 2, 1482.0, '2006-09-13 06:38:18', >>> 'HST_10915_98_ACS_WFC_F475W', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 4744.36, >>> 'image/fits', '2006-09-14 00:47:14', 1158194834, '10915_98 ACS/WFC >>> F475W (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_F475W', 'Polygon J2000 11.87717390 >>> -25.29535800 11.89640629 -25.28200592 11.87697850 -25.29587050 >>> 11.87759056 -25.29648032 11.89825680 -25.31706610 11.89831934 >>> -25.31702147 11.89893180 -25.31763140 11.94817110 -25.28247540 >>> 11.92689000 -25.26128670 11.92682706 -25.26133165 11.92621490 >>> -25.26072200 11.92600258 -25.26087365 11.90560880 -25.24086630 >>> 11.90555076 -25.24090663 11.90493420 -25.24030160 11.85576560 >>> -25.27444230 11.85638210 -25.27504754 11.87649930 -25.29479280 >>> 11.87655706 -25.29475271'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F606W' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F606W', 2, 1508.0, '2006-09-13 09:42:19', >>> 'HST_10915_98_ACS_WFC_F606W', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 5917.68, >>> 'image/fits', '2006-09-14 10:03:41', 1158228221, '10915_98 ACS/WFC >>> F606W (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_F606W', 'Polygon J2000 11.92689010 >>> -25.26128690 11.92682692 -25.26133202 11.92621490 -25.26072250 >>> 11.92600228 -25.26087437 11.90560890 -25.24086820 11.90555051 >>> -25.24090876 11.90493430 -25.24030410 11.85576720 -25.27444250 >>> 11.85638324 -25.27504727 11.87650030 -25.29479150 11.87655775 >>> -25.29475163 11.87717450 -25.29535680 11.89645734 -25.28197019 >>> 11.87697940 -25.29587070 11.87759135 -25.29648042 11.89825680 >>> -25.31706560 11.89831905 -25.31702118 11.89893170 -25.31763130 >>> 11.94817060 -25.28247510'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F814W' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W', 2, 1534.0, '2006-09-13 11:18:39', >>> 'HST_10915_98_ACS_WFC_F814W', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 8059.76, >>> 'image/fits', '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC >>> F814W (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_F814W', 'Polygon J2000 11.90555223 >>> -25.24091185 11.90493480 -25.24030600 11.85576830 -25.27444290 >>> 11.87650110 -25.29479130 11.87655960 -25.29475070 11.87717680 >>> -25.29535630 11.89640118 -25.28201069 11.87698000 -25.29587030 >>> 11.89825710 -25.31706440 11.89832033 -25.31701928 11.89893250 >>> -25.31762890 11.94810647 -25.28252025 11.94816870 -25.28247580 >>> 11.92688930 -25.26128920 11.92682707 -25.26133365 11.92621460 >>> -25.26072370 11.92600121 -25.26087611 11.90560960 -25.24087200'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_total' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'detection', 6, 4524.0, '2006-09-13 06:38:18', >>> 'HST_10915_98_ACS_WFC_total', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', None, 'image/fits', >>> '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC detection >>> (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_total', 'Polygon J2000 11.90560880 >>> -25.24086630 11.90555076 -25.24090663 11.90493420 -25.24030160 >>> 11.85576560 -25.27444230 11.85638210 -25.27504754 11.87649930 >>> -25.29479280 11.87655706 -25.29475271 11.87717390 -25.29535800 >>> 11.87717661 -25.29535612 11.89634395 -25.28205042 11.87697850 >>> -25.29587050 11.87759056 -25.29648032 11.89825680 -25.31706610 >>> 11.89831934 -25.31702147 11.89893180 -25.31763140 11.94817110 >>> -25.28247540 11.92689000 -25.26128670 11.92682706 -25.26133165 >>> 11.92621490 -25.26072200 11.92600258 -25.26087365'), >>> ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=HST_10915_98_ACS_WFC_F814W&green=HST_10915_98_ACS_WFC_F606W&blue=HST_10915_98_ACS_WFC_F475W&size=ALL&format=fits' >>> , >>> 11.901736, -25.279734, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W/F606W/F475W', 6, 4524.0, '2006-09-13 06:38:18', >>> 'HST_10915_98_ACS_WFC_F814W_F606W_F475W', 10915, '98', 'Dalcanton', >>> '', 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 6241.0, >>> 'image/fits', '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC >>> F814W/F606W/F475W (color) NGC0253-WIDE5', masked_array(data = [6100 >>> 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , >>> 'HST_10915_98_ACS_WFC_F814W,HST_10915_98_ACS_WFC_F606W,HST_10915_98_ACS_WFC_F475W', >>> 'Polygon J2000 11.90555223 -25.24091185 11.90493480 -25.24030600 >>> 11.85576830 -25.27444290 11.87650110 -25.29479130 11.87655960 >>> -25.29475070 11.87717680 -25.29535630 11.89640118 -25.28201069 >>> 11.87698000 -25.29587030 11.89825710 -25.31706440 11.89832033 >>> -25.31701928 11.89893250 -25.31762890 11.94810647 -25.28252025 >>> 11.94816870 -25.28247580 11.92688930 -25.26128920 11.92682707 >>> -25.26133365 11.92621460 -25.26072370 11.92600121 -25.26087611 >>> 11.90560960 -25.24087200')] >>> >>> On Feb 26, 2013, at 10:20 AM, Michael Droettboom >> > wrote: >>> >>>> On 02/26/2013 09:51 AM, Andy Ptak wrote: >>>>> Hello, >>>>> I am trying out reading VOTables using and am having trouble. >>>>> Specifically reading tables returned by the Hubble Legacy >>>>> Archive. The query was: >>>>> http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389 >>>>> which I attached. >>>>> >>>>> I am using astropy installed by pip on the enthought distribution >>>>> on a mac, so its version 0.2 and numpy got upgraded to 1.7.0 (if >>>>> that matters). >>>>> First, is there a way to have a votable imported directly from >>>>> query without saving it to a physical file first? Maybe its best >>>>> practice to save VO queries to a file to trace errors, in case the >>>>> votable is very large, etc. but when you know you are getting >>>>> something back that is small and (as in this case) you mainly want >>>>> a list of fields for further work, saving the full votable seams >>>>> like an unnecessary extra step. >>>> >>>> You can pass an http url to astropy.io.votable.parse, e.g.: >>>> >>>> In [2]: from astropy.io import votable >>>> >>>> In [3]: >>>> votable.parse("http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389") >>>> >>>>> >>>>> The errors I am getting are below. I got similar errors when >>>>> using a different votable call I found on the HLA website >>>>> (basically the same except using acsSIAP.cgi?strict=1 >>>>> Is this a problem with the Votable being returned by stsci, a bug >>>>> or am I doing something wrong? >>>> >>>> Interestingly, I'm not able to reproduce the exception here. (This >>>> is on Linux, with Python 2.7, astropy 0.2 and numpy 1.7.0). >>>> >>>> Would you mind tinkering around in the debugger for me? Turn on the >>>> IPython debugger with "%pdb", reproduce the crash, and then go up >>>> two levels and print out "array_chunk"? I'm curious if there's >>>> some messed up values getting in there. >>>> >>>> Thanks, >>>> Mike >>>> >>>>> >>>>> Thanks, >>>>> Andy Ptak >>>>> >>>>> >>>>> >>>>> In [4]: from astropy.io.votable import parse >>>>> In [6]: votable = parse("test1.xml") >>>>> WARNING: W42: test1.xml:3:0: W42: No XML namespace specified >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:7:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:POS' -> 'INPUT_POS' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:8:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:SIZE' -> 'INPUT_SIZE' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:9:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:FORMAT' -> 'INPUT_FORMAT' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:10:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:imagetype' -> 'INPUT_imagetype' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:11:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:inst' -> 'INPUT_inst' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:12:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:hrcmatch' -> 'INPUT_hrcmatch' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:13:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:zoom' -> 'INPUT_zoom' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:14:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:autoscale' -> 'INPUT_autoscale' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:15:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:asinh' -> 'INPUT_asinh' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:16:4: W06: Invalid UCD >>>>> 'VOX:STC_CoordRefFrame': UCD has invalid character ':' in >>>>> 'VOX:STC_CoordRefFrame' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:17:4: W06: Invalid UCD >>>>> 'VOX:WCS_CoordProjection': UCD has invalid character ':' in >>>>> 'VOX:WCS_CoordProjection' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:19:6: W06: Invalid UCD >>>>> 'VOX:Image_AccessReference': UCD has invalid character ':' in >>>>> 'VOX:Image_AccessReference' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:82:6: W06: Invalid UCD 'VOX:Image_Format': >>>>> UCD has invalid character ':' in 'VOX:Image_Format' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:91:6: W06: Invalid UCD 'VOX:Image_Title': >>>>> UCD has invalid character ':' in 'VOX:Image_Title' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:94:6: W06: Invalid UCD 'VOX:Image_Naxis': >>>>> UCD has invalid character ':' in 'VOX:Image_Naxis' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:97:6: W06: Invalid UCD 'VOX:Image_Scale': >>>>> UCD has invalid character ':' in 'VOX:Image_Scale' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:100:6: W06: Invalid UCD 'VOX:Image_Naxes': >>>>> UCD has invalid character ':' in 'VOX:Image_Naxes' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:103:6: W06: Invalid UCD >>>>> 'VOX:WCS_CoordRefPixel': UCD has invalid character ':' in >>>>> 'VOX:WCS_CoordRefPixel' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:106:6: W06: Invalid UCD >>>>> 'VOX:WCS_CoordRefValue': UCD has invalid character ':' in >>>>> 'VOX:WCS_CoordRefValue' (suppressing further warnings of this >>>>> type...) [astropy.io.votable.exceptions] >>>>> ERROR: OverflowError: Python int too large to convert to C long >>>>> [numpy.ma.core] >>>>> --------------------------------------------------------------------------- >>>>> OverflowError Traceback (most recent call last) >>>>> /Volumes/Apps_and_Docs/aptak/Dropbox/HLA/ >>>>> in () >>>>> ----> 1 votable = parse("test1.xml") >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/table.pyc >>>>> in parse(source, columns, invalid, pedantic, chunk_size, >>>>> table_number, table_id, filename, _debug_python_based_parser) >>>>> 114 _debug_python_based_parser=_debug_python_based_parser) as >>>>> iterator: >>>>> 115 return tree.VOTableFile( >>>>> --> 116 config=config, pos=(1, 1)).parse(iterator, config) >>>>> 117 >>>>> 118 >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in parse(self, iterator, config) >>>>> 3084 if start: >>>>> 3085 tag_mapping.get(tag, self._add_unknown_tag)( >>>>> -> 3086 iterator, tag, data, config, pos) >>>>> 3087 elif tag == 'DESCRIPTION': >>>>> 3088 if self.description is not None: >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in _add_resource(self, iterator, tag, data, config, pos) >>>>> 3015 resource = Resource(config=config, pos=pos, **data) >>>>> 3016 self.resources.append(resource) >>>>> -> 3017 resource.parse(self, iterator, config) >>>>> 3018 >>>>> 3019 def _add_coosys(self, iterator, tag, data, config, pos): >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in parse(self, votable, iterator, config) >>>>> 2871 if start: >>>>> 2872 tag_mapping.get(tag, self._add_unknown_tag)( >>>>> -> 2873 iterator, tag, data, config, pos) >>>>> 2874 elif tag == 'DESCRIPTION': >>>>> 2875 if self.description is not None: >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in _add_table(self, iterator, tag, data, config, pos) >>>>> 2828 table = Table(self._votable, config=config, >>>>> pos=pos, **data) >>>>> 2829 self.tables.append(table) >>>>> -> 2830 table.parse(iterator, config) >>>>> 2831 >>>>> 2832 def _add_info(self, iterator, tag, data, config, pos): >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in parse(self, iterator, config) >>>>> 2212 'TABLEDATA', data.iterkeys(), config, pos) >>>>> 2213 self.array = self._parse_tabledata( >>>>> -> 2214 iterator, colnumbers, fields, config) >>>>> 2215 break >>>>> 2216 elif tag == 'BINARY': >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in _parse_tabledata(self, iterator, colnumbers, fields, config) >>>>> 2357 >>>>> 2358 array = _resize(array, alloc_rows) >>>>> -> 2359 array[numrows:] = array_chunk >>>>> 2360 if alloc_rows != 0: >>>>> 2361 array.mask[numrows:] = mask_chunk >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>>>> in __setslice__(self, i, j, value) >>>>> 3067 >>>>> 3068 """ >>>>> -> 3069 self.__setitem__(slice(i, j), value) >>>>> 3070 >>>>> 3071 >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>>>> in __setitem__(self, indx, value) >>>>> 3031 _mask = ndarray.__getattribute__(self, >>>>> '_mask') >>>>> 3032 # Set the data, then the mask >>>>> >>>>> -> 3033 ndarray.__setitem__(_data, indx, dval) >>>>> 3034 ndarray.__setitem__(_mask, indx, mval) >>>>> 3035 elif hasattr(indx, 'dtype') and (indx.dtype == >>>>> MaskType): >>>>> >>>>> OverflowError: Python int too large to convert to C long >>>> >>> >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdroe at stsci.edu Tue Feb 26 15:22:13 2013 From: mdroe at stsci.edu (Michael Droettboom) Date: Tue, 26 Feb 2013 15:22:13 -0500 Subject: [AstroPy] Problem reading VOTable produced by HLA In-Reply-To: <512CF63D.2050309@stsci.edu> References: <512BA60C.3070505@stsci.edu> <512CD2C5.2010403@stsci.edu> <8E84403D-9265-420E-8BD7-F2CA4768573B@nasa.gov> <512CDE0F.1010001@stsci.edu> <512CF63D.2050309@stsci.edu> Message-ID: <512D1975.2040001@stsci.edu> I just got word from Rick White that this has been fixed on the HLA server as well. That was fast. Mike On 02/26/2013 12:51 PM, Michael Droettboom wrote: > Also see the following PR for astropy which will emit a warning and > truncate out-of-range integer values: > > https://github.com/astropy/astropy/pull/825 > > Mike > > On 02/26/2013 11:08 AM, Michael Droettboom wrote: >> Column #24 is defined as follows: >> >> > ucd="TIME_EPOCH"> >> >> Here, a datatype of "int" means a 32-bit int, always, as per the >> VOTable spec. >> >> Yet the value in the first row of this column is -2208988800, which >> is too large to be represented in 32 bits. This field should really >> be updated to be of type "long". I'll go ahead and report this to >> the HLA folks. >> >> Unfortunately, it seems that Numpy complains about too long numbers >> going into a 32-bit field on a 32-bit platform, but not on a >> 64-platform, where it just silently truncates. It would be nice to >> at least have consistent behavior across platforms there. Again, >> I'll file a bug with Numpy about this to see what, if anything, that >> team may want to do. >> >> In any case, the real bug is in the file -- astropy is just correctly >> following directions here and only allocating as much memory as >> specified in the file. It would be nice to workaround this bug in >> the file, but it's a tricky one -- we would have to allocate the >> table in one way, come across the overflow later, and then reparse >> the whole file using a different table structure. None of our >> existing workarounds do that, so it's not the most straightforward >> thing to do. >> >> Mike >> >> On 02/26/2013 10:44 AM, Andy Ptak wrote: >>> Sure... >>> ipdb> print array_chunk >>> [('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref' >>> , >>> 11.9123, -25.2889, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F475W', 4, 1482.0, '2006-09-13 06:38:18', >>> 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', 10915, >>> '10915-ngc0253-wide5', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 13:59:02', 'HLSP', 4744.35009766, 'image/fits', '1900-01-01 >>> 00:00:00', -2208988800L, 'angst_hst_acs-wfc_10915-ngc0253-wide5 >>> ACS/WFC F475W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2055.6 1148.67], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9123 -25.2889], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06388e-05 -8.9285e-06 -8.9285e-06 >>> 1.06388e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'hlsp_angst_hst_acs-wfc_10915-ngc0253-wide5_f475w_v1_ref', >>> 'Polygon J2000 11.947768 -25.282758 11.905331 -25.237036 >>> 11.855680 -25.274704 11.898112 -25.320439'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz' >>> , >>> 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F606W', 4, 1508.0, '2006-09-13 09:42:19', >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 10915, >>> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 14:01:51', 'HLSP', 5917.68017578, 'image/fits', '1900-01-01 >>> 00:00:00', -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC >>> F606W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4299], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38892e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2110.0 2149.5], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9017 -25.2787], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 >>> 1.06397e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', 'Polygon J2000 >>> 11.947776 -25.282766 11.905328 -25.237033 11.855676 >>> -25.274701 11.898119 -25.320449'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz' >>> , >>> 11.9017, -25.2787, 5, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W', 4, 1534.0, '2006-09-13 11:18:39', >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 10915, >>> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 14:01:51', 'HLSP', 8059.75976562, 'image/fits', '1900-01-01 >>> 00:00:00', -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC >>> F814W (hlsp) NGC0253-WIDE5', masked_array(data = [4220 4298], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38892e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2110.0 2149.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9017 -25.2787], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 >>> 1.06397e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz', 'Polygon J2000 >>> 11.947772 -25.282761 11.905333 -25.237039 11.855682 >>> -25.274707 11.898114 -25.320444'), >>> ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz&blue=hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz&size=ALL&format=fits' >>> , >>> 11.9017, -25.2787, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W/F606W', 8, 3042.0, '2006-09-13 09:42:19', >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_f606w_v1_drz', 10915, >>> 'ngc0253-field01', 'Dalcanton', '', '', 'ACCUM', '2012-11-02 >>> 14:01:51', 'HLSP', 6989.0, 'image/fits', '1900-01-01 00:00:00', >>> -2208988800L, 'ghosts_hst_acs_ngc0253-field01 ACS/WFC F814W/F606W >>> (color) NGC0253-WIDE5', masked_array(data = [4220 4298], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38892e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [2110.0 2149.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.9017 -25.2787], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.06397e-05 -8.9279e-06 -8.9279e-06 >>> 1.06397e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , >>> 'hlsp_ghosts_hst_acs_ngc0253-field01_f814w_v1_drz,hlsp_ghosts_hst_acs_ngc0253-field01_f606w_v1_drz', >>> 'Polygon J2000 11.947772 -25.282761 11.905333 -25.237039 >>> 11.855682 -25.274707 11.898114 -25.320444'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F475W' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F475W', 2, 1482.0, '2006-09-13 06:38:18', >>> 'HST_10915_98_ACS_WFC_F475W', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 4744.36, >>> 'image/fits', '2006-09-14 00:47:14', 1158194834, '10915_98 ACS/WFC >>> F475W (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_F475W', 'Polygon J2000 11.87717390 >>> -25.29535800 11.89640629 -25.28200592 11.87697850 -25.29587050 >>> 11.87759056 -25.29648032 11.89825680 -25.31706610 11.89831934 >>> -25.31702147 11.89893180 -25.31763140 11.94817110 -25.28247540 >>> 11.92689000 -25.26128670 11.92682706 -25.26133165 11.92621490 >>> -25.26072200 11.92600258 -25.26087365 11.90560880 -25.24086630 >>> 11.90555076 -25.24090663 11.90493420 -25.24030160 11.85576560 >>> -25.27444230 11.85638210 -25.27504754 11.87649930 -25.29479280 >>> 11.87655706 -25.29475271'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F606W' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F606W', 2, 1508.0, '2006-09-13 09:42:19', >>> 'HST_10915_98_ACS_WFC_F606W', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 5917.68, >>> 'image/fits', '2006-09-14 10:03:41', 1158228221, '10915_98 ACS/WFC >>> F606W (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_F606W', 'Polygon J2000 11.92689010 >>> -25.26128690 11.92682692 -25.26133202 11.92621490 -25.26072250 >>> 11.92600228 -25.26087437 11.90560890 -25.24086820 11.90555051 >>> -25.24090876 11.90493430 -25.24030410 11.85576720 -25.27444250 >>> 11.85638324 -25.27504727 11.87650030 -25.29479150 11.87655775 >>> -25.29475163 11.87717450 -25.29535680 11.89645734 -25.28197019 >>> 11.87697940 -25.29587070 11.87759135 -25.29648042 11.89825680 >>> -25.31706560 11.89831905 -25.31702118 11.89893170 -25.31763130 >>> 11.94817060 -25.28247510'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_F814W' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W', 2, 1534.0, '2006-09-13 11:18:39', >>> 'HST_10915_98_ACS_WFC_F814W', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 8059.76, >>> 'image/fits', '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC >>> F814W (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_F814W', 'Polygon J2000 11.90555223 >>> -25.24091185 11.90493480 -25.24030600 11.85576830 -25.27444290 >>> 11.87650110 -25.29479130 11.87655960 -25.29475070 11.87717680 >>> -25.29535630 11.89640118 -25.28201069 11.87698000 -25.29587030 >>> 11.89825710 -25.31706440 11.89832033 -25.31701928 11.89893250 >>> -25.31762890 11.94810647 -25.28252025 11.94816870 -25.28247580 >>> 11.92688930 -25.26128920 11.92682707 -25.26133365 11.92621460 >>> -25.26072370 11.92600121 -25.26087611 11.90560960 -25.24087200'), >>> ('http://hla.stsci.edu/cgi-bin/getdata.cgi?dataset=HST_10915_98_ACS_WFC_total' >>> , >>> 11.901736, -25.279734, 2, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'detection', 6, 4524.0, '2006-09-13 06:38:18', >>> 'HST_10915_98_ACS_WFC_total', 10915, '98', 'Dalcanton', '', >>> 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', None, 'image/fits', >>> '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC detection >>> (combined) NGC0253-WIDE5', masked_array(data = [6100 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , 'HST_10915_98_ACS_WFC_total', 'Polygon J2000 11.90560880 >>> -25.24086630 11.90555076 -25.24090663 11.90493420 -25.24030160 >>> 11.85576560 -25.27444230 11.85638210 -25.27504754 11.87649930 >>> -25.29479280 11.87655706 -25.29475271 11.87717390 -25.29535800 >>> 11.87717661 -25.29535612 11.89634395 -25.28205042 11.87697850 >>> -25.29587050 11.87759056 -25.29648032 11.89825680 -25.31706610 >>> 11.89831934 -25.31702147 11.89893180 -25.31763140 11.94817110 >>> -25.28247540 11.92689000 -25.26128670 11.92682706 -25.26133165 >>> 11.92621490 -25.26072200 11.92600258 -25.26087365'), >>> ('http://hla.stsci.edu/cgi-bin/fitscut.cgi?red=HST_10915_98_ACS_WFC_F814W&green=HST_10915_98_ACS_WFC_F606W&blue=HST_10915_98_ACS_WFC_F475W&size=ALL&format=fits' >>> , >>> 11.901736, -25.279734, 4, 'NGC0253-WIDE5', 'ACS/WFC', 'WFCENTER', >>> 'F814W/F606W/F475W', 6, 4524.0, '2006-09-13 06:38:18', >>> 'HST_10915_98_ACS_WFC_F814W_F606W_F475W', 10915, '98', 'Dalcanton', >>> '', 'Loaded', 'ACCUM', '2008-01-17 03:05:31', 'HLA', 6241.0, >>> 'image/fits', '2006-09-14 16:27:26', 1158251246, '10915_98 ACS/WFC >>> F814W/F606W/F475W (color) NGC0253-WIDE5', masked_array(data = [6100 >>> 6100], >>> mask = [False False], >>> fill_value = 999999) >>> , masked_array(data = [1.38889e-05], >>> mask = [False], >>> fill_value = 1e+20) >>> , 2, masked_array(data = [3050.0 3050.0], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [11.901736 -25.279734], >>> mask = [False False], >>> fill_value = 1e+20) >>> , masked_array(data = [-1.38889e-05 0.0 0.0 1.38889e-05], >>> mask = [False False False False], >>> fill_value = 1e+20) >>> , >>> 'HST_10915_98_ACS_WFC_F814W,HST_10915_98_ACS_WFC_F606W,HST_10915_98_ACS_WFC_F475W', >>> 'Polygon J2000 11.90555223 -25.24091185 11.90493480 -25.24030600 >>> 11.85576830 -25.27444290 11.87650110 -25.29479130 11.87655960 >>> -25.29475070 11.87717680 -25.29535630 11.89640118 -25.28201069 >>> 11.87698000 -25.29587030 11.89825710 -25.31706440 11.89832033 >>> -25.31701928 11.89893250 -25.31762890 11.94810647 -25.28252025 >>> 11.94816870 -25.28247580 11.92688930 -25.26128920 11.92682707 >>> -25.26133365 11.92621460 -25.26072370 11.92600121 -25.26087611 >>> 11.90560960 -25.24087200')] >>> >>> On Feb 26, 2013, at 10:20 AM, Michael Droettboom >> > wrote: >>> >>>> On 02/26/2013 09:51 AM, Andy Ptak wrote: >>>>> Hello, >>>>> I am trying out reading VOTables using and am having trouble. >>>>> Specifically reading tables returned by the Hubble Legacy >>>>> Archive. The query was: >>>>> http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389 >>>>> which I attached. >>>>> >>>>> I am using astropy installed by pip on the enthought distribution >>>>> on a mac, so its version 0.2 and numpy got upgraded to 1.7.0 (if >>>>> that matters). >>>>> First, is there a way to have a votable imported directly from >>>>> query without saving it to a physical file first? Maybe its best >>>>> practice to save VO queries to a file to trace errors, in case the >>>>> votable is very large, etc. but when you know you are getting >>>>> something back that is small and (as in this case) you mainly want >>>>> a list of fields for further work, saving the full votable seams >>>>> like an unnecessary extra step. >>>> >>>> You can pass an http url to astropy.io.votable.parse, e.g.: >>>> >>>> In [2]: from astropy.io import votable >>>> >>>> In [3]: >>>> votable.parse("http://hla.stsci.edu/cgi-bin/hlaSIAP.cgi?inst=ACS&pos=11.888333,-25.288056&size=0.001389") >>>> >>>>> >>>>> The errors I am getting are below. I got similar errors when >>>>> using a different votable call I found on the HLA website >>>>> (basically the same except using acsSIAP.cgi?strict=1 >>>>> Is this a problem with the Votable being returned by stsci, a bug >>>>> or am I doing something wrong? >>>> >>>> Interestingly, I'm not able to reproduce the exception here. (This >>>> is on Linux, with Python 2.7, astropy 0.2 and numpy 1.7.0). >>>> >>>> Would you mind tinkering around in the debugger for me? Turn on the >>>> IPython debugger with "%pdb", reproduce the crash, and then go up >>>> two levels and print out "array_chunk"? I'm curious if there's >>>> some messed up values getting in there. >>>> >>>> Thanks, >>>> Mike >>>> >>>>> >>>>> Thanks, >>>>> Andy Ptak >>>>> >>>>> >>>>> >>>>> In [4]: from astropy.io.votable import parse >>>>> In [6]: votable = parse("test1.xml") >>>>> WARNING: W42: test1.xml:3:0: W42: No XML namespace specified >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:7:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:POS' -> 'INPUT_POS' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:8:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:SIZE' -> 'INPUT_SIZE' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:9:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:FORMAT' -> 'INPUT_FORMAT' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:10:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:imagetype' -> 'INPUT_imagetype' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:11:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:inst' -> 'INPUT_inst' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:12:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:hrcmatch' -> 'INPUT_hrcmatch' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:13:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:zoom' -> 'INPUT_zoom' [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:14:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:autoscale' -> 'INPUT_autoscale' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W03: test1.xml:15:4: W03: Implictly generating an ID from >>>>> a name 'INPUT:asinh' -> 'INPUT_asinh' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:16:4: W06: Invalid UCD >>>>> 'VOX:STC_CoordRefFrame': UCD has invalid character ':' in >>>>> 'VOX:STC_CoordRefFrame' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:17:4: W06: Invalid UCD >>>>> 'VOX:WCS_CoordProjection': UCD has invalid character ':' in >>>>> 'VOX:WCS_CoordProjection' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:19:6: W06: Invalid UCD >>>>> 'VOX:Image_AccessReference': UCD has invalid character ':' in >>>>> 'VOX:Image_AccessReference' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:82:6: W06: Invalid UCD 'VOX:Image_Format': >>>>> UCD has invalid character ':' in 'VOX:Image_Format' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:91:6: W06: Invalid UCD 'VOX:Image_Title': >>>>> UCD has invalid character ':' in 'VOX:Image_Title' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:94:6: W06: Invalid UCD 'VOX:Image_Naxis': >>>>> UCD has invalid character ':' in 'VOX:Image_Naxis' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:97:6: W06: Invalid UCD 'VOX:Image_Scale': >>>>> UCD has invalid character ':' in 'VOX:Image_Scale' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:100:6: W06: Invalid UCD 'VOX:Image_Naxes': >>>>> UCD has invalid character ':' in 'VOX:Image_Naxes' >>>>> [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:103:6: W06: Invalid UCD >>>>> 'VOX:WCS_CoordRefPixel': UCD has invalid character ':' in >>>>> 'VOX:WCS_CoordRefPixel' [astropy.io.votable.exceptions] >>>>> WARNING: W06: test1.xml:106:6: W06: Invalid UCD >>>>> 'VOX:WCS_CoordRefValue': UCD has invalid character ':' in >>>>> 'VOX:WCS_CoordRefValue' (suppressing further warnings of this >>>>> type...) [astropy.io.votable.exceptions] >>>>> ERROR: OverflowError: Python int too large to convert to C long >>>>> [numpy.ma.core] >>>>> --------------------------------------------------------------------------- >>>>> OverflowError Traceback (most recent call last) >>>>> /Volumes/Apps_and_Docs/aptak/Dropbox/HLA/ >>>>> in () >>>>> ----> 1 votable = parse("test1.xml") >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/table.pyc >>>>> in parse(source, columns, invalid, pedantic, chunk_size, >>>>> table_number, table_id, filename, _debug_python_based_parser) >>>>> 114 _debug_python_based_parser=_debug_python_based_parser) as >>>>> iterator: >>>>> 115 return tree.VOTableFile( >>>>> --> 116 config=config, pos=(1, 1)).parse(iterator, config) >>>>> 117 >>>>> 118 >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in parse(self, iterator, config) >>>>> 3084 if start: >>>>> 3085 tag_mapping.get(tag, self._add_unknown_tag)( >>>>> -> 3086 iterator, tag, data, config, pos) >>>>> 3087 elif tag == 'DESCRIPTION': >>>>> 3088 if self.description is not None: >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in _add_resource(self, iterator, tag, data, config, pos) >>>>> 3015 resource = Resource(config=config, pos=pos, **data) >>>>> 3016 self.resources.append(resource) >>>>> -> 3017 resource.parse(self, iterator, config) >>>>> 3018 >>>>> 3019 def _add_coosys(self, iterator, tag, data, config, pos): >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in parse(self, votable, iterator, config) >>>>> 2871 if start: >>>>> 2872 tag_mapping.get(tag, self._add_unknown_tag)( >>>>> -> 2873 iterator, tag, data, config, pos) >>>>> 2874 elif tag == 'DESCRIPTION': >>>>> 2875 if self.description is not None: >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in _add_table(self, iterator, tag, data, config, pos) >>>>> 2828 table = Table(self._votable, config=config, >>>>> pos=pos, **data) >>>>> 2829 self.tables.append(table) >>>>> -> 2830 table.parse(iterator, config) >>>>> 2831 >>>>> 2832 def _add_info(self, iterator, tag, data, config, pos): >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in parse(self, iterator, config) >>>>> 2212 'TABLEDATA', data.iterkeys(), config, pos) >>>>> 2213 self.array = self._parse_tabledata( >>>>> -> 2214 iterator, colnumbers, fields, config) >>>>> 2215 break >>>>> 2216 elif tag == 'BINARY': >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/astropy/io/votable/tree.pyc >>>>> in _parse_tabledata(self, iterator, colnumbers, fields, config) >>>>> 2357 >>>>> 2358 array = _resize(array, alloc_rows) >>>>> -> 2359 array[numrows:] = array_chunk >>>>> 2360 if alloc_rows != 0: >>>>> 2361 array.mask[numrows:] = mask_chunk >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>>>> in __setslice__(self, i, j, value) >>>>> 3067 >>>>> 3068 """ >>>>> -> 3069 self.__setitem__(slice(i, j), value) >>>>> 3070 >>>>> 3071 >>>>> >>>>> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >>>>> in __setitem__(self, indx, value) >>>>> 3031 _mask = ndarray.__getattribute__(self, >>>>> '_mask') >>>>> 3032 # Set the data, then the mask >>>>> >>>>> -> 3033 ndarray.__setitem__(_data, indx, dval) >>>>> 3034 ndarray.__setitem__(_mask, indx, mval) >>>>> 3035 elif hasattr(indx, 'dtype') and (indx.dtype == >>>>> MaskType): >>>>> >>>>> OverflowError: Python int too large to convert to C long >>>> >>> >> >> >> >> _______________________________________________ >> AstroPy mailing list >> AstroPy at scipy.org >> http://mail.scipy.org/mailman/listinfo/astropy > > > > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy -------------- next part -------------- An HTML attachment was scrubbed... URL: From aldcroft at head.cfa.harvard.edu Thu Feb 28 16:08:44 2013 From: aldcroft at head.cfa.harvard.edu (Tom Aldcroft) Date: Thu, 28 Feb 2013 16:08:44 -0500 Subject: [AstroPy] Astronomy & Astrophysics Mini-Symposium at SciPy2013 Message-ID: Dear colleagues, We are planning to have an Astronomy & Astrophysics Mini-Symposium at the SciPy2013 conference in Austin, TX in June 2013 [1]. The use of Python for astronomy research is growing rapidly, and we have seen the emergence of large and well-organized community efforts to develop analysis packages for astronomy. I'm excited to have the opportunity to organize and chair this mini-symposium. This is timely moment for the astronomy Python community to get together and present what's new and exciting, as well as to discuss and define future priorities. This will be an evening session taking 1.5 hours and consisting of 4 - 6 talks and some form of open discussion. I'd like to encourage everybody to attend SciPy2013 this year and participate in the overall discussion. We need viewpoints not only from people deeply involved in development, but also newcomers and those who are just trying to get their research done. I also want to specifically invite people to submit a talk or poster abstract so we can make the mini-symposium interesting. The abstract deadline is March 20. If you have questions or suggestions please contact me or post to this list as appropriate. Please forward this message to your colleagues who may not be on this list. I'm interested in ideas for discussion topics that would be productive. One possibility is examining some of the large community efforts (e.g. yt, SunPy, Astropy, etc) to look for opportunities for cross-project coordination. Best regards, Tom Aldcroft [1] http://conference.scipy.org/scipy2013/ From matthewturk at gmail.com Thu Feb 28 18:09:28 2013 From: matthewturk at gmail.com (Matthew Turk) Date: Thu, 28 Feb 2013 18:09:28 -0500 Subject: [AstroPy] Astronomy & Astrophysics Mini-Symposium at SciPy2013 In-Reply-To: References: Message-ID: Hi Tom, On Thu, Feb 28, 2013 at 4:08 PM, Tom Aldcroft wrote: > Dear colleagues, > > We are planning to have an Astronomy & Astrophysics Mini-Symposium at > the SciPy2013 conference in Austin, TX in June 2013 [1]. The use of > Python for astronomy research is growing rapidly, and we have seen the > emergence of large and well-organized community efforts to develop > analysis packages for astronomy. > > I'm excited to have the opportunity to organize and chair this > mini-symposium. This is timely moment for the astronomy Python > community to get together and present what's new and exciting, as well > as to discuss and define future priorities. This will be an evening > session taking 1.5 hours and consisting of 4 - 6 talks and some form > of open discussion. > > I'd like to encourage everybody to attend SciPy2013 this year and > participate in the overall discussion. We need viewpoints not only > from people deeply involved in development, but also newcomers and > those who are just trying to get their research done. I also want to > specifically invite people to submit a talk or poster abstract so we > can make the mini-symposium interesting. The abstract deadline is > March 20. > > If you have questions or suggestions please contact me or post to this > list as appropriate. Please forward this message to your colleagues > who may not be on this list. I'm interested in ideas for discussion > topics that would be productive. One possibility is examining some of > the large community efforts (e.g. yt, SunPy, Astropy, etc) to look for > opportunities for cross-project coordination. I'm co-organizing the BoFs this year, and cross-project coordination sounds like it could be a good fit if the discussions run long during the symposium. I'd also like to second Tom's call for participation -- if you haven't been, SciPy is a pretty amazing conference drawing together people from different domains of study. The talks are usually quite good and cover a wide range of interesting topics in many subjects. -Matt > > Best regards, > Tom Aldcroft > > [1] http://conference.scipy.org/scipy2013/ > _______________________________________________ > AstroPy mailing list > AstroPy at scipy.org > http://mail.scipy.org/mailman/listinfo/astropy