From burtscher at mpe.mpg.de  Wed Aug  5 10:31:29 2015
From: burtscher at mpe.mpg.de (Leonard Burtscher)
Date: Wed, 5 Aug 2015 16:31:29 +0200
Subject: [AstroPy] Bug in Table.view_in_browser(jsviewer=True) ?
Message-ID: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>

Hi,

The "show_in_browser" feature for Table is really great, especially when combined with jsviewer=True, e.g.

http://www.mpe.mpg.de/~burtscher/APEX/table.html

By clicking on the header row, one can sort by that column. However, numbers are not sorted correctly, with "9" being before "10" etc. (e.g. click on one of the "exptime" columns and look at the bottom of the table).

Is that something that could be solved within astropy or is it a Javascript issue?

-- Leo



From adam.g.ginsburg at gmail.com  Wed Aug  5 10:48:16 2015
From: adam.g.ginsburg at gmail.com (Adam Ginsburg)
Date: Wed, 5 Aug 2015 16:48:16 +0200
Subject: [AstroPy] Bug in Table.view_in_browser(jsviewer=True) ?
In-Reply-To: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>
References: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>
Message-ID: <CAEBNSwaod0gShvRuDxqQkJT5DxNcmfnM1mP9Z6+oseDva1MfnA@mail.gmail.com>

Hi Leonard,
     This is something that can be solved within astropy.  There is an
open issue that will probably solve this:
https://github.com/astropy/astropy/issues/2966
in case you want to try to tackle it based on Tom Robitaille's example.

On Wed, Aug 5, 2015 at 4:31 PM, Leonard Burtscher <burtscher at mpe.mpg.de> wrote:
> Hi,
>
> The "show_in_browser" feature for Table is really great, especially when combined with jsviewer=True, e.g.
>
> http://www.mpe.mpg.de/~burtscher/APEX/table.html
>
> By clicking on the header row, one can sort by that column. However, numbers are not sorted correctly, with "9" being before "10" etc. (e.g. click on one of the "exptime" columns and look at the bottom of the table).
>
> Is that something that could be solved within astropy or is it a Javascript issue?
>
> -- Leo
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy



-- 
Adam Ginsburg
Fellow, European Southern Observatory
http://www.adamgginsburg.com/


From embray at stsci.edu  Wed Aug  5 11:06:04 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 5 Aug 2015 11:06:04 -0400
Subject: [AstroPy] Bug in Table.view_in_browser(jsviewer=True) ?
In-Reply-To: <CAEBNSwaod0gShvRuDxqQkJT5DxNcmfnM1mP9Z6+oseDva1MfnA@mail.gmail.com>
References: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>
	<CAEBNSwaod0gShvRuDxqQkJT5DxNcmfnM1mP9Z6+oseDva1MfnA@mail.gmail.com>
Message-ID: <55C2265C.4050103@stsci.edu>

On 08/05/2015 10:48 AM, Adam Ginsburg wrote:
> Hi Leonard,
>       This is something that can be solved within astropy.  There is an
> open issue that will probably solve this:
> https://github.com/astropy/astropy/issues/2966
> in case you want to try to tackle it based on Tom Robitaille's example.

Indeed, some of the columns you'll notice do sort correctly numerically, like 
the "z" column in your table.  It's just when columns contain non-numbers like 
NaN.

Looks like a reasonably easy pull request since Tom already showed how to 
approach it (just need to add a custom sort method to dataTable in the 
javascript), if anyone has been looking for a contribution to make to Astropy...

Erik

> On Wed, Aug 5, 2015 at 4:31 PM, Leonard Burtscher <burtscher at mpe.mpg.de> wrote:
>> Hi,
>>
>> The "show_in_browser" feature for Table is really great, especially when combined with jsviewer=True, e.g.
>>
>> http://www.mpe.mpg.de/~burtscher/APEX/table.html
>>
>> By clicking on the header row, one can sort by that column. However, numbers are not sorted correctly, with "9" being before "10" etc. (e.g. click on one of the "exptime" columns and look at the bottom of the table).
>>
>> Is that something that could be solved within astropy or is it a Javascript issue?
>>
>> -- Leo
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>



From burtscher at mpe.mpg.de  Wed Aug  5 12:16:46 2015
From: burtscher at mpe.mpg.de (Leonard Burtscher)
Date: Wed, 5 Aug 2015 18:16:46 +0200
Subject: [AstroPy] Bug in Table.view_in_browser(jsviewer=True) ?
In-Reply-To: <55C2265C.4050103@stsci.edu>
References: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>
	<CAEBNSwaod0gShvRuDxqQkJT5DxNcmfnM1mP9Z6+oseDva1MfnA@mail.gmail.com>
	<55C2265C.4050103@stsci.edu>
Message-ID: <AA6E4C7E-6246-4D47-AAA6-33781C6B86C5@mpe.mpg.de>

Thanks, Adam and Erik, for your quick answers and sorry for not checking the bugtracker before. I won't get to this within the next week or so, but then I might try to fix it.

Cheers,
Leo


> Am 05.08.2015 um 17:06 schrieb Erik Bray <embray at stsci.edu>:
> 
> On 08/05/2015 10:48 AM, Adam Ginsburg wrote:
>> Hi Leonard,
>>      This is something that can be solved within astropy.  There is an
>> open issue that will probably solve this:
>> https://github.com/astropy/astropy/issues/2966
>> in case you want to try to tackle it based on Tom Robitaille's example.
> 
> Indeed, some of the columns you'll notice do sort correctly numerically, like 
> the "z" column in your table.  It's just when columns contain non-numbers like 
> NaN.
> 
> Looks like a reasonably easy pull request since Tom already showed how to 
> approach it (just need to add a custom sort method to dataTable in the 
> javascript), if anyone has been looking for a contribution to make to Astropy...
> 
> Erik
> 
>> On Wed, Aug 5, 2015 at 4:31 PM, Leonard Burtscher <burtscher at mpe.mpg.de> wrote:
>>> Hi,
>>> 
>>> The "show_in_browser" feature for Table is really great, especially when combined with jsviewer=True, e.g.
>>> 
>>> http://www.mpe.mpg.de/~burtscher/APEX/table.html
>>> 
>>> By clicking on the header row, one can sort by that column. However, numbers are not sorted correctly, with "9" being before "10" etc. (e.g. click on one of the "exptime" columns and look at the bottom of the table).
>>> 
>>> Is that something that could be solved within astropy or is it a Javascript issue?
>>> 
>>> -- Leo
>>> 
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>> 
>> 
>> 
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy



From embray at stsci.edu  Wed Aug  5 12:23:30 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 5 Aug 2015 12:23:30 -0400
Subject: [AstroPy] Bug in Table.view_in_browser(jsviewer=True) ?
In-Reply-To: <AA6E4C7E-6246-4D47-AAA6-33781C6B86C5@mpe.mpg.de>
References: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>
	<CAEBNSwaod0gShvRuDxqQkJT5DxNcmfnM1mP9Z6+oseDva1MfnA@mail.gmail.com>
	<55C2265C.4050103@stsci.edu>
	<AA6E4C7E-6246-4D47-AAA6-33781C6B86C5@mpe.mpg.de>
Message-ID: <55C23882.9060003@stsci.edu>

On 08/05/2015 12:16 PM, Leonard Burtscher wrote:
> Thanks, Adam and Erik, for your quick answers and sorry for not checking the
> bugtracker before. I won't get to this within the next week or so, but then I
> might try to fix it.

Much appreciated if you're able!

Thanks,
Erik

>> Am 05.08.2015 um 17:06 schrieb Erik Bray <embray at stsci.edu>:
>>
>> On 08/05/2015 10:48 AM, Adam Ginsburg wrote:
>>> Hi Leonard, This is something that can be solved within astropy.  There
>>> is an open issue that will probably solve this:
>>> https://github.com/astropy/astropy/issues/2966 in case you want to try to
>>> tackle it based on Tom Robitaille's example.
>>
>> Indeed, some of the columns you'll notice do sort correctly numerically,
>> like the "z" column in your table.  It's just when columns contain
>> non-numbers like NaN.
>>
>> Looks like a reasonably easy pull request since Tom already showed how to
>> approach it (just need to add a custom sort method to dataTable in the
>> javascript), if anyone has been looking for a contribution to make to
>> Astropy...
>>
>> Erik
>>
>>> On Wed, Aug 5, 2015 at 4:31 PM, Leonard Burtscher <burtscher at mpe.mpg.de>
>>> wrote:
>>>> Hi,
>>>>
>>>> The "show_in_browser" feature for Table is really great, especially
>>>> when combined with jsviewer=True, e.g.
>>>>
>>>> http://www.mpe.mpg.de/~burtscher/APEX/table.html
>>>>
>>>> By clicking on the header row, one can sort by that column. However,
>>>> numbers are not sorted correctly, with "9" being before "10" etc. (e.g.
>>>> click on one of the "exptime" columns and look at the bottom of the
>>>> table).
>>>>
>>>> Is that something that could be solved within astropy or is it a
>>>> Javascript issue?
>>>>
>>>> -- Leo
>>>>
>>>> _______________________________________________ AstroPy mailing list
>>>> AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>>
>>>
>>
>> _______________________________________________ AstroPy mailing list
>> AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________ AstroPy mailing list
> AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy
>



From ejensen1 at swarthmore.edu  Mon Aug 10 17:26:57 2015
From: ejensen1 at swarthmore.edu (Eric L. N. Jensen)
Date: Mon, 10 Aug 2015 17:26:57 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign string
	plus numbers for degrees, arcmin, arcsec
Message-ID: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>

Hi all,

I?m working with a table from CDS (i.e. from an ApJ paper) that specifies the declination coordinate in four separate columns:  a string for the sign of the declination, and three numerical columns for degrees, arcminutes, and arcseconds.  Is there a straightforward way to combine these into a single astropy.coordinates Angle object?  The standard Angle function (constructor?) only accepts a tuple with three entries.  

I guess what I?m looking for is the inverse method of ?signed_dms?. 

If it?s of use, the table I?m working with is here:  http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt 

Thanks in advance for your help with this,

Eric
 



From embray at stsci.edu  Mon Aug 10 17:49:54 2015
From: embray at stsci.edu (Erik Bray)
Date: Mon, 10 Aug 2015 17:49:54 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
Message-ID: <55C91C82.4070504@stsci.edu>

On 08/10/2015 05:26 PM, Eric L. N. Jensen wrote:
> Hi all,
>
> I?m working with a table from CDS (i.e. from an ApJ paper) that specifies the declination coordinate in four separate columns:  a string for the sign of the declination, and three numerical columns for degrees, arcminutes, and arcseconds.  Is there a straightforward way to combine these into a single astropy.coordinates Angle object?  The standard Angle function (constructor?) only accepts a tuple with three entries.
>
> I guess what I?m looking for is the inverse method of ?signed_dms?.
>
> If it?s of use, the table I?m working with is here:  http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt
>
> Thanks in advance for your help with this,

It seems a bit strange that the table format in this case indicates the sign of 
the declination as a completely separate field, rather than as just part of the 
numeric string.  But I'm not intimately familiar with the quirks of the CDS 
table format.  Is that normal?  I wonder if there's a way to get the CDS reader 
to format those fields as a single column.  Tom Aldcroft would know better.

You can instantiate an Angle array using a generator, so you could probably just 
slurp in the data using a generator expression, allowing you to transform the 
values one a time while creating the Angle rather than transforming the table 
all at once first (please bear with the LISPiness of this example):

dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
              for row in table['DE-', 'DEd', 'DEm', 'DEs'))


# (Note: originally I used row[2:] above, but there seems to be an unrelated bug 
involved in taking slices of rows...I'll report on that.)

Erik



From thomas.robitaille at gmail.com  Mon Aug 10 17:53:06 2015
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Mon, 10 Aug 2015 23:53:06 +0200
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <55C91C82.4070504@stsci.edu>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu>
Message-ID: <55C91D42.7000600@gmail.com>

Erik Bray wrote:
> On 08/10/2015 05:26 PM, Eric L. N. Jensen wrote:
>> Hi all,
>>
>> I?m working with a table from CDS (i.e. from an ApJ paper) that specifies the declination coordinate in four separate columns:  a string for the sign of the declination, and three numerical columns for degrees, arcminutes, and arcseconds.  Is there a straightforward way to combine these into a single astropy.coordinates Angle object?  The standard Angle function (constructor?) only accepts a tuple with three entries.
>>
>> I guess what I?m looking for is the inverse method of ?signed_dms?.
>>
>> If it?s of use, the table I?m working with is here:  http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt
>>
>> Thanks in advance for your help with this,
> 
> It seems a bit strange that the table format in this case indicates the sign of 
> the declination as a completely separate field, rather than as just part of the 
> numeric string.  But I'm not intimately familiar with the quirks of the CDS 
> table format.  Is that normal?  I wonder if there's a way to get the CDS reader 
> to format those fields as a single column.  Tom Aldcroft would know better.
> 
> You can instantiate an Angle array using a generator, so you could probably just 
> slurp in the data using a generator expression, allowing you to transform the 
> values one a time while creating the Angle rather than transforming the table 
> all at once first (please bear with the LISPiness of this example):
> 
> dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>               for row in table['DE-', 'DEd', 'DEm', 'DEs'))
> 
> 
> # (Note: originally I used row[2:] above, but there seems to be an unrelated bug 
> involved in taking slices of rows...I'll report on that.)

In case you want to do this without loops, np.char.add can be used to
concatenate string arrays element-wise. So for instance you could
concatenate the sign column and the degree column of the declination with

np.add(t['DE-'], t['DEd'])

then you would end up with three arrays instead of four.

Cheers,
Tom



> 
> Erik
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy


From embray at stsci.edu  Mon Aug 10 18:00:49 2015
From: embray at stsci.edu (Erik Bray)
Date: Mon, 10 Aug 2015 18:00:49 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <55C91D42.7000600@gmail.com>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu> <55C91D42.7000600@gmail.com>
Message-ID: <55C91F11.7080008@stsci.edu>

On 08/10/2015 05:53 PM, Thomas Robitaille wrote:
> Erik Bray wrote:
>> On 08/10/2015 05:26 PM, Eric L. N. Jensen wrote:
>>> Hi all,
>>>
>>> I?m working with a table from CDS (i.e. from an ApJ paper) that specifies the declination coordinate in four separate columns:  a string for the sign of the declination, and three numerical columns for degrees, arcminutes, and arcseconds.  Is there a straightforward way to combine these into a single astropy.coordinates Angle object?  The standard Angle function (constructor?) only accepts a tuple with three entries.
>>>
>>> I guess what I?m looking for is the inverse method of ?signed_dms?.
>>>
>>> If it?s of use, the table I?m working with is here:  http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt
>>>
>>> Thanks in advance for your help with this,
>>
>> It seems a bit strange that the table format in this case indicates the sign of
>> the declination as a completely separate field, rather than as just part of the
>> numeric string.  But I'm not intimately familiar with the quirks of the CDS
>> table format.  Is that normal?  I wonder if there's a way to get the CDS reader
>> to format those fields as a single column.  Tom Aldcroft would know better.
>>
>> You can instantiate an Angle array using a generator, so you could probably just
>> slurp in the data using a generator expression, allowing you to transform the
>> values one a time while creating the Angle rather than transforming the table
>> all at once first (please bear with the LISPiness of this example):
>>
>> dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>>                for row in table['DE-', 'DEd', 'DEm', 'DEs'))
>>
>>
>> # (Note: originally I used row[2:] above, but there seems to be an unrelated bug
>> involved in taking slices of rows...I'll report on that.)
>
> In case you want to do this without loops, np.char.add can be used to
> concatenate string arrays element-wise. So for instance you could
> concatenate the sign column and the degree column of the declination with
>
> np.add(t['DE-'], t['DEd'])
>
> then you would end up with three arrays instead of four.

This doesn't work because t['DE-'] is a string column, and t['DEd'] is a 
(masked) integer column.

In any case, doing a row-wise transformation allows doing this with one fewer loop.

Erik



From gizis at udel.edu  Mon Aug 10 20:19:07 2015
From: gizis at udel.edu (John Gizis)
Date: Mon, 10 Aug 2015 14:19:07 -1000
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <55C91C82.4070504@stsci.edu>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu>
Message-ID: <CAKd4CoO3Df_NSDnbd0D5jXKYnpqkvxqhTcnOxxSVYXDO1kxnUQ@mail.gmail.com>

"It seems a bit strange that the table format in this case indicates the
sign of
the declination as a completely separate field, rather than as just part of
the
numeric string.  But I'm not intimately familiar with the quirks of the CDS
table format. "

I don't think is unusual for CDS tables. I always thought it was because
Fortran77 didn't know -00 is different than +00.























On Mon, Aug 10, 2015 at 11:49 AM, Erik Bray <embray at stsci.edu> wrote:

> On 08/10/2015 05:26 PM, Eric L. N. Jensen wrote:
> > Hi all,
> >
> > I?m working with a table from CDS (i.e. from an ApJ paper) that
> specifies the declination coordinate in four separate columns:  a string
> for the sign of the declination, and three numerical columns for degrees,
> arcminutes, and arcseconds.  Is there a straightforward way to combine
> these into a single astropy.coordinates Angle object?  The standard Angle
> function (constructor?) only accepts a tuple with three entries.
> >
> > I guess what I?m looking for is the inverse method of ?signed_dms?.
> >
> > If it?s of use, the table I?m working with is here:
> http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt
> >
> > Thanks in advance for your help with this,
>
> It seems a bit strange that the table format in this case indicates the
> sign of
> the declination as a completely separate field, rather than as just part
> of the
> numeric string.  But I'm not intimately familiar with the quirks of the CDS
> table format.  Is that normal?  I wonder if there's a way to get the CDS
> reader
> to format those fields as a single column.  Tom Aldcroft would know better.
>
> You can instantiate an Angle array using a generator, so you could
> probably just
> slurp in the data using a generator expression, allowing you to transform
> the
> values one a time while creating the Angle rather than transforming the
> table
> all at once first (please bear with the LISPiness of this example):
>
> dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>               for row in table['DE-', 'DEd', 'DEm', 'DEs'))
>
>
> # (Note: originally I used row[2:] above, but there seems to be an
> unrelated bug
> involved in taking slices of rows...I'll report on that.)
>
> Erik
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



-- 
John E. Gizis
Professor
Department of Physics and Astronomy
University of Delaware

302-831-2668
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150810/5e0b5b9c/attachment.html>

From ejensen1 at swarthmore.edu  Mon Aug 10 23:13:26 2015
From: ejensen1 at swarthmore.edu (Eric Jensen)
Date: Mon, 10 Aug 2015 23:13:26 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
	string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <55C91F11.7080008@stsci.edu>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu> <55C91D42.7000600@gmail.com>
	<55C91F11.7080008@stsci.edu>
Message-ID: <87B7C4C5-44A5-4400-9739-E4D88867304C@swarthmore.edu>

Hi all,

Thanks for the suggestions.  These solutions both have the problem that I suspect the original table formatting was designed to avoid, which is the infamous ?minus zero? problem, affecting declinations between 0 and -1 degrees.  For those decs, the leading part of the coordinate is ?-00? which is fine as a string, but doesn?t preserve sign when evaluated as an integer:

In [1]: x = '-00'

In [2]: print int(x)
0

In [3]: print int(x) < 0
False


So I think what I probably need to do instead is construct a separate ?sign? variable with a +1 or -1, and then multiply it by the final coordinate.  I couldn?t see how to do it before (because of the masked integer vs. string column issue that Erik notes) but having seen these examples, it seems like I should be able to do something like this: 

for row in t['DE-', 'DEd', 'DEm', 'DEs']:
    dec = Angle(row[1], row[2], row[3]) * int(row[0] + '1')

but when trying to do that I get an error on units, whether or not I specify them explicitly.  Ultimately, I was able to make it work with this:


t['Dec'] = Angle((t['DEd'], t['DEm'], t['DEs']), unit=u.deg)
sign = np.array([])
# Construct an integer +1 or -1 sign variable from the string:
for sign_string in t['DE-?]:
    sign = np.append(sign,int(sign_string + '1'))
t['Dec'] *= sign

So my immediate problem is solved (thanks!) but the overall ?minus zero? problem is a real issue, I think - I hadn?t thought about it this generally at the outset, but any three-element angle constructor will have this problem if the angles are allowed to be negative.  So it still seems to me that a four-element angle constructor would be very useful (and perhaps a warning for the three-element version).  

Thanks,

Eric
 
P.S.  Just saw John Gizis? reply as I was about to send:

"I don't think is unusual for CDS tables. I always thought it was because Fortran77 didn't know -00 is different than +00. ?

It?s not just Fortran - Python too!   (And pretty much any other language you can name, unless you?re dealing with strings.) 

In [1]: +0 == -0
Out[1]: True


On Aug 10, 2015, at 6:00 PM, Erik Bray <embray at stsci.edu> wrote:

>>> You can instantiate an Angle array using a generator, so you could probably just
>>> slurp in the data using a generator expression, allowing you to transform the
>>> values one a time while creating the Angle rather than transforming the table
>>> all at once first (please bear with the LISPiness of this example):
>>> 
>>> dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>>>               for row in table['DE-', 'DEd', 'DEm', 'DEs'))
>>> 
>>> 
>>> # (Note: originally I used row[2:] above, but there seems to be an unrelated bug
>>> involved in taking slices of rows...I'll report on that.)
>> 
>> In case you want to do this without loops, np.char.add can be used to
>> concatenate string arrays element-wise. So for instance you could
>> concatenate the sign column and the degree column of the declination with
>> 
>> np.add(t['DE-'], t['DEd'])
>> 
>> then you would end up with three arrays instead of four.
> 
> This doesn't work because t['DE-'] is a string column, and t['DEd'] is a 
> (masked) integer column.
> 
> In any case, doing a row-wise transformation allows doing this with one fewer loop.



From aldcroft at head.cfa.harvard.edu  Mon Aug 10 23:18:27 2015
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Mon, 10 Aug 2015 23:18:27 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <CAKd4CoO3Df_NSDnbd0D5jXKYnpqkvxqhTcnOxxSVYXDO1kxnUQ@mail.gmail.com>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu>
	<CAKd4CoO3Df_NSDnbd0D5jXKYnpqkvxqhTcnOxxSVYXDO1kxnUQ@mail.gmail.com>
Message-ID: <CAMtEP6yX-92e1hzjq7qv70osFcr6NSr5DT1WkfzDNJ8=CmPwBA@mail.gmail.com>

I'll dispense with my standard rant on how moronic sexigesimal is, and
suggest this:

dec = Angle((np.where(t['DE-'] == '-', -1.0, 1.0) * t['DEd'], t['DEm'],
t['DEs']),
            unit='deg')

This relies on a tricky behavior that -0.0 has a sign of -1.  I'm not 100%
sure if that works on all platforms, but it works on 64-bit mac and linux.

- Tom


On Mon, Aug 10, 2015 at 8:19 PM, John Gizis <gizis at udel.edu> wrote:

> "It seems a bit strange that the table format in this case indicates the
> sign of
> the declination as a completely separate field, rather than as just part
> of the
> numeric string.  But I'm not intimately familiar with the quirks of the CDS
> table format. "
>
> I don't think is unusual for CDS tables. I always thought it was because
> Fortran77 didn't know -00 is different than +00.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Mon, Aug 10, 2015 at 11:49 AM, Erik Bray <embray at stsci.edu> wrote:
>
>> On 08/10/2015 05:26 PM, Eric L. N. Jensen wrote:
>> > Hi all,
>> >
>> > I?m working with a table from CDS (i.e. from an ApJ paper) that
>> specifies the declination coordinate in four separate columns:  a string
>> for the sign of the declination, and three numerical columns for degrees,
>> arcminutes, and arcseconds.  Is there a straightforward way to combine
>> these into a single astropy.coordinates Angle object?  The standard Angle
>> function (constructor?) only accepts a tuple with three entries.
>> >
>> > I guess what I?m looking for is the inverse method of ?signed_dms?.
>> >
>> > If it?s of use, the table I?m working with is here:
>> http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt
>> >
>> > Thanks in advance for your help with this,
>>
>> It seems a bit strange that the table format in this case indicates the
>> sign of
>> the declination as a completely separate field, rather than as just part
>> of the
>> numeric string.  But I'm not intimately familiar with the quirks of the
>> CDS
>> table format.  Is that normal?  I wonder if there's a way to get the CDS
>> reader
>> to format those fields as a single column.  Tom Aldcroft would know
>> better.
>>
>> You can instantiate an Angle array using a generator, so you could
>> probably just
>> slurp in the data using a generator expression, allowing you to transform
>> the
>> values one a time while creating the Angle rather than transforming the
>> table
>> all at once first (please bear with the LISPiness of this example):
>>
>> dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>>               for row in table['DE-', 'DEd', 'DEm', 'DEs'))
>>
>>
>> # (Note: originally I used row[2:] above, but there seems to be an
>> unrelated bug
>> involved in taking slices of rows...I'll report on that.)
>>
>> Erik
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>
>
>
> --
> John E. Gizis
> Professor
> Department of Physics and Astronomy
> University of Delaware
>
> 302-831-2668
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150810/9d6665b7/attachment.html>

From AP2015 at JohnMurrell.org.uk  Tue Aug 11 03:40:53 2015
From: AP2015 at JohnMurrell.org.uk (AP2015 at JohnMurrell.org.uk)
Date: Tue, 11 Aug 2015 08:40:53 +0100
Subject: [AstroPy] Constructing Angle from a tuple specified as sign - a
	possible work around
Message-ID: <004801d0d409$0d2cd320$27867960$@JohnMurrell.org.uk>

Eric,

 

One workaround would be to export the table from CDS Vizier in VOTable
format and then import it into TopCat. You can then save it as ASCII or CSV
or several other formats. As part of writing the table Vizier constructs new
RA & DEC columns in signed decimal degrees that may be easier to import and
work with ?

 

Depending what you are intending to do with the data you might be able to
manipulate and plot it in TopCat - it is quite a powerful tool for Catalogue
operations.

 

Regards

 

John Murrell

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150811/e5310ba3/attachment.html>

From AP2015 at JohnMurrell.org.uk  Tue Aug 11 05:08:40 2015
From: AP2015 at JohnMurrell.org.uk (AP2015 at JohnMurrell.org.uk)
Date: Tue, 11 Aug 2015 10:08:40 +0100
Subject: [AstroPy] Constructing Angle from a tuple specified as sign - a
	possible work around
Message-ID: <007e01d0d415$50e549b0$f2afdd10$@JohnMurrell.org.uk>

Eric,

 

One workaround would be to export the table from CDS Vizier in VOTable
format and then import it into TopCat. You can then save it as ASCII or CSV
or several other formats. As part of writing the table Vizier constructs new
RA & DEC columns in signed decimal degrees that may be easier to import and
work with ?

 

Depending what you are intending to do with the data you might be able to
manipulate and plot it in TopCat - it is quite a powerful tool for Catalogue
operations.

 

Regards

 

John Murrell

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150811/276f2203/attachment.html>

From embray at stsci.edu  Tue Aug 11 10:22:18 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 11 Aug 2015 10:22:18 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <87B7C4C5-44A5-4400-9739-E4D88867304C@swarthmore.edu>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu> <55C91D42.7000600@gmail.com>
	<55C91F11.7080008@stsci.edu>
	<87B7C4C5-44A5-4400-9739-E4D88867304C@swarthmore.edu>
Message-ID: <55CA051A.5060307@stsci.edu>

On 08/10/2015 11:13 PM, Eric Jensen wrote:
> Hi all,
>
> Thanks for the suggestions.  These solutions both have the problem that I
> suspect the original table formatting was designed to avoid, which is the
> infamous ?minus zero? problem, affecting declinations between 0 and -1
> degrees.  For those decs, the leading part of the coordinate is ?-00? which
> is fine as a string, but doesn?t preserve sign when evaluated as an integer:
>
> In [1]: x = '-00'
>
> In [2]: print int(x) 0
>
> In [3]: print int(x) < 0 False

Ah, right.  In my original suggestion I should have suggested float() instead of 
int().  IEEE floating point has distinct values for +0 and -0, and I believe the 
coordinates package can tell the difference between the two.

(Or so I thought, but after trying it out it appears not:

In [11]: Angle((-0.0, 0, 0), unit='deg').signed_dms
Out[11]: signed_dms_tuple(sign=0.0, d=0.0, m=0.0, s=0.0)

)

So maybe that should be fixed?  Or maybe this has already been discussed and 
rejected.  It does have the obvious problem that it only works if you explicitly 
use a floating point value for the degrees.

Erik


From embray at stsci.edu  Tue Aug 11 10:30:39 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 11 Aug 2015 10:30:39 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <87B7C4C5-44A5-4400-9739-E4D88867304C@swarthmore.edu>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu> <55C91D42.7000600@gmail.com>
	<55C91F11.7080008@stsci.edu>
	<87B7C4C5-44A5-4400-9739-E4D88867304C@swarthmore.edu>
Message-ID: <55CA070F.9060306@stsci.edu>

On 08/10/2015 11:13 PM, Eric Jensen wrote:
> "I don't think is unusual for CDS tables. I always thought it was because Fortran77 didn't know -00 is different than +00. ?
>
> It?s not just Fortran - Python too!   (And pretty much any other language you can name, unless you?re dealing with strings.)
>
> In [1]: +0 == -0
> Out[1]: True

This is the only logical possibility though.  However, as I mentioned in the 
last message it's important to note a difference between integers and floats in 
this regard.  For floats you still get:

In [17]: +0.0 == -0.0
Out[17]: True

However:

In [18]: +0.0 is -0.0
Out[18]: False

Unlike for integers where the aren't distinct representations for signed zero:

In [19]: +0 is -0
Out[19]: True

Although +0.0 == -0.0, signed float zeros still follow the usual arithmetic 
rules for signed zero:

In [19]: +0 is -0
Out[19]: True

In [20]: -0.0 * 1
Out[20]: -0.0

In [21]: -0.0 * -1
Out[21]: 0.0

In [22]: np.sqrt(-0.0)
Out[22]: -0.0

and so on.  One reasons it's good to get in the habit (and I admit I'm lazy 
about this myself) of appending a decimal to almost all numeric literals in 
scientific code so that it's stored as a float, and have the code convert to 
integers where it's appropriate to use integers.

Erik

> On Aug 10, 2015, at 6:00 PM, Erik Bray <embray at stsci.edu> wrote:
>
>>>> You can instantiate an Angle array using a generator, so you could probably just
>>>> slurp in the data using a generator expression, allowing you to transform the
>>>> values one a time while creating the Angle rather than transforming the table
>>>> all at once first (please bear with the LISPiness of this example):
>>>>
>>>> dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>>>>                for row in table['DE-', 'DEd', 'DEm', 'DEs'))
>>>>
>>>>
>>>> # (Note: originally I used row[2:] above, but there seems to be an unrelated bug
>>>> involved in taking slices of rows...I'll report on that.)
>>>
>>> In case you want to do this without loops, np.char.add can be used to
>>> concatenate string arrays element-wise. So for instance you could
>>> concatenate the sign column and the degree column of the declination with
>>>
>>> np.add(t['DE-'], t['DEd'])
>>>
>>> then you would end up with three arrays instead of four.
>>
>> This doesn't work because t['DE-'] is a string column, and t['DEd'] is a
>> (masked) integer column.
>>
>> In any case, doing a row-wise transformation allows doing this with one fewer loop.
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



From embray at stsci.edu  Tue Aug 11 10:32:02 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 11 Aug 2015 10:32:02 -0400
Subject: [AstroPy] Constructing Angle from a tuple specified as sign
 string plus numbers for degrees, arcmin, arcsec
In-Reply-To: <CAMtEP6yX-92e1hzjq7qv70osFcr6NSr5DT1WkfzDNJ8=CmPwBA@mail.gmail.com>
References: <406FD280-C848-45C8-B907-0CAD38C0A708@swarthmore.edu>
	<55C91C82.4070504@stsci.edu>
	<CAKd4CoO3Df_NSDnbd0D5jXKYnpqkvxqhTcnOxxSVYXDO1kxnUQ@mail.gmail.com>
	<CAMtEP6yX-92e1hzjq7qv70osFcr6NSr5DT1WkfzDNJ8=CmPwBA@mail.gmail.com>
Message-ID: <55CA0762.2040106@stsci.edu>

On 08/10/2015 11:18 PM, Aldcroft, Thomas wrote:
> I'll dispense with my standard rant on how moronic sexigesimal is, and suggest this:
>
> dec = Angle((np.where(t['DE-'] == '-', -1.0, 1.0) * t['DEd'], t['DEm'], t['DEs']),
>              unit='deg')
>
> This relies on a tricky behavior that -0.0 has a sign of -1.  I'm not 100% sure
> if that works on all platforms, but it works on 64-bit mac and linux.

+1 to Tom's version.  That seems the simplest of all.


> On Mon, Aug 10, 2015 at 8:19 PM, John Gizis <gizis at udel.edu
> <mailto:gizis at udel.edu>> wrote:
>
>     "It seems a bit strange that the table format in this case indicates the sign of
>     the declination as a completely separate field, rather than as just part of the
>     numeric string.  But I'm not intimately familiar with the quirks of the CDS
>     table format. "
>
>     I don't think is unusual for CDS tables. I always thought it was because
>     Fortran77 didn't know -00 is different than +00.
>
>     On Mon, Aug 10, 2015 at 11:49 AM, Erik Bray <embray at stsci.edu
>     <mailto:embray at stsci.edu>> wrote:
>
>         On 08/10/2015 05:26 PM, Eric L. N. Jensen wrote:
>         > Hi all,
>         >
>         > I?m working with a table from CDS (i.e. from an ApJ paper) that specifies the declination coordinate in four separate columns:  a string for the sign of the declination, and three numerical columns for degrees, arcminutes, and arcseconds.  Is there a straightforward way to combine these into a single astropy.coordinates Angle object?  The standard Angle function (constructor?) only accepts a tuple with three entries.
>         >
>         > I guess what I?m looking for is the inverse method of ?signed_dms?.
>         >
>         > If it?s of use, the table I?m working with is here:http://iopscience.iop.org/0067-0049/184/1/18/fulltext/apjs300673t4_mrt.txt
>         >
>         > Thanks in advance for your help with this,
>
>         It seems a bit strange that the table format in this case indicates the
>         sign of
>         the declination as a completely separate field, rather than as just part
>         of the
>         numeric string.  But I'm not intimately familiar with the quirks of the CDS
>         table format.  Is that normal?  I wonder if there's a way to get the CDS
>         reader
>         to format those fields as a single column.  Tom Aldcroft would know better.
>
>         You can instantiate an Angle array using a generator, so you could
>         probably just
>         slurp in the data using a generator expression, allowing you to
>         transform the
>         values one a time while creating the Angle rather than transforming the
>         table
>         all at once first (please bear with the LISPiness of this example):
>
>         dec = Angle(((int(row[0] + str(row[1])), row[2], row[3])
>                        for row in table['DE-', 'DEd', 'DEm', 'DEs'))
>
>
>         # (Note: originally I used row[2:] above, but there seems to be an
>         unrelated bug
>         involved in taking slices of rows...I'll report on that.)
>
>         Erik



From joe at neoturbine.net  Tue Aug 11 11:07:03 2015
From: joe at neoturbine.net (Joseph Booker)
Date: Tue, 11 Aug 2015 11:07:03 -0400
Subject: [AstroPy] Bug in Table.view_in_browser(jsviewer=True) ?
In-Reply-To: <55C23882.9060003@stsci.edu>
References: <4562EC6B-72D9-4FBD-9458-A815DF34510C@mpe.mpg.de>
	<CAEBNSwaod0gShvRuDxqQkJT5DxNcmfnM1mP9Z6+oseDva1MfnA@mail.gmail.com>
	<55C2265C.4050103@stsci.edu>
	<AA6E4C7E-6246-4D47-AAA6-33781C6B86C5@mpe.mpg.de>
	<55C23882.9060003@stsci.edu>
Message-ID: <CAJSjvtJ_kTWJcHHKttgPYWEfDCD-ET0Jn-K2FtmvkfPfzDXV9g@mail.gmail.com>

Just as a heads up, I implemented this in
https://github.com/astropy/astropy/pull/4052 (I've had to do something
similar so DataTables would sort numbers in scientific notation).

On Wed, Aug 5, 2015 at 12:23 PM, Erik Bray <embray at stsci.edu> wrote:

> On 08/05/2015 12:16 PM, Leonard Burtscher wrote:
> > Thanks, Adam and Erik, for your quick answers and sorry for not checking
> the
> > bugtracker before. I won't get to this within the next week or so, but
> then I
> > might try to fix it.
>
> Much appreciated if you're able!
>
> Thanks,
> Erik
>
> >> Am 05.08.2015 um 17:06 schrieb Erik Bray <embray at stsci.edu>:
> >>
> >> On 08/05/2015 10:48 AM, Adam Ginsburg wrote:
> >>> Hi Leonard, This is something that can be solved within astropy.  There
> >>> is an open issue that will probably solve this:
> >>> https://github.com/astropy/astropy/issues/2966 in case you want to
> try to
> >>> tackle it based on Tom Robitaille's example.
> >>
> >> Indeed, some of the columns you'll notice do sort correctly numerically,
> >> like the "z" column in your table.  It's just when columns contain
> >> non-numbers like NaN.
> >>
> >> Looks like a reasonably easy pull request since Tom already showed how
> to
> >> approach it (just need to add a custom sort method to dataTable in the
> >> javascript), if anyone has been looking for a contribution to make to
> >> Astropy...
> >>
> >> Erik
> >>
> >>> On Wed, Aug 5, 2015 at 4:31 PM, Leonard Burtscher <
> burtscher at mpe.mpg.de>
> >>> wrote:
> >>>> Hi,
> >>>>
> >>>> The "show_in_browser" feature for Table is really great, especially
> >>>> when combined with jsviewer=True, e.g.
> >>>>
> >>>> http://www.mpe.mpg.de/~burtscher/APEX/table.html
> >>>>
> >>>> By clicking on the header row, one can sort by that column. However,
> >>>> numbers are not sorted correctly, with "9" being before "10" etc.
> (e.g.
> >>>> click on one of the "exptime" columns and look at the bottom of the
> >>>> table).
> >>>>
> >>>> Is that something that could be solved within astropy or is it a
> >>>> Javascript issue?
> >>>>
> >>>> -- Leo
> >>>>
> >>>> _______________________________________________ AstroPy mailing list
> >>>> AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy
> >>>
> >>>
> >>>
> >>
> >> _______________________________________________ AstroPy mailing list
> >> AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy
> >
> > _______________________________________________ AstroPy mailing list
> > AstroPy at scipy.org http://mail.scipy.org/mailman/listinfo/astropy
> >
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150811/6d2da935/attachment.html>

From embray at stsci.edu  Tue Aug 11 16:43:16 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 11 Aug 2015 16:43:16 -0400
Subject: [AstroPy] [ANN] Astropy v1.0.4 released!
Message-ID: <55CA5E64.3080508@stsci.edu>

Hi all,

Astropy v1.0.4 has just been released.  This is a minor bug fix release--you can 
read the full list of fixes in the changelog:

https://github.com/astropy/astropy/blob/v1.0.4/CHANGES.rst#104-2015-08-11

There are a couple changes in particular worth calling out:

- The astropy.convolution routines now release the Python GIL during their
    main loops.  This allows convolution to be parallelized effectively.  For
    example an image can be broken into multiple chunks to be convolved in
    parallel in a multithreaded environment.

- Coordinate transformations on a scalar coordinate object now return a scalar
    (as opposed to a single-element array of coordinates).  This is a change in
    behavior from previous versions, but the old behavior was considered a bug,
    so it is fixed in this release.

The update package is available on PyPI: 
https://pypi.python.org/pypi/astropy/1.0.4 and can be upgraded with pip:

     $ pip install --upgrade astropy

Wheel builds have been provided for Windows for all currently supported versions 
of Python.

The update is also available as a conda package, so users of Anaconda can 
upgrade with:

     $ conda update astropy

Thanks to everyone who contributed code, documentation, and bug reports.

Erik


From embray at stsci.edu  Tue Aug 11 17:39:39 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 11 Aug 2015 17:39:39 -0400
Subject: [AstroPy] [ANN] Astropy v1.0.4 released!
In-Reply-To: <55CA5E64.3080508@stsci.edu>
References: <55CA5E64.3080508@stsci.edu>
Message-ID: <55CA6B9B.7010201@stsci.edu>

On 08/11/2015 04:43 PM, Erik Bray wrote:
> Hi all,
>
> Astropy v1.0.4 has just been released.  This is a minor bug fix release--you can
> read the full list of fixes in the changelog:
>
> https://github.com/astropy/astropy/blob/v1.0.4/CHANGES.rst#104-2015-08-11
>
> There are a couple changes in particular worth calling out:
>
> - The astropy.convolution routines now release the Python GIL during their
>      main loops.  This allows convolution to be parallelized effectively.  For
>      example an image can be broken into multiple chunks to be convolved in
>      parallel in a multithreaded environment.

Just a quick clarification--this change only affects the 
astropy.convolution.convolve [1] routine, the main loops of which are 
implemented in Cython and now release the GIL.

The convolve_fft [2] routine generally assumes that the FFT implementation in 
use (numpy.fft.fft [3] by default) is compiled from C or FORTRAN and already 
releases the GIL.

Erik


[1] 
http://docs.astropy.org/en/v1.0.4/api/astropy.convolution.convolve.html#astropy.convolution.convolve
[2] 
http://docs.astropy.org/en/v1.0.4/api/astropy.convolution.convolve_fft.html#astropy.convolution.convolve_fft
[3] 
http://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.fft.html#numpy.fft.fft


From tumlinson at stsci.edu  Wed Aug 19 10:00:36 2015
From: tumlinson at stsci.edu (Jason Tumlinson)
Date: Wed, 19 Aug 2015 14:00:36 +0000
Subject: [AstroPy] Convert FITS header into astropy Table entries?
Message-ID: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>

Hi,
I am trying to harvest the information from a large number of FITS headers, parse them and do statistics on values, in an efficient manner. I read in each header in the usual fashion:

h = fits.open('lbiy83i0q_x1d.fits?)
h0 = h[0].header

My goal then is to get all the header info into an astropy Table such that the table columns are given by the header keys and the values in the rows are given by the values in each of 100+ headers. I tried this:

In [38]: t = Table(rows=h0.values(), names=h0.keys())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-38-8a0ece2925d5> in <module>()
----> 1 t = Table(rows=h0.values(), names=h0.keys())

/Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pyc in __init__(self, data, masked, names, dtype, meta, copy, rows)
    240                 data = rows
    241             else:
--> 242                 rec_data = recarray_fromrecords(rows)
    243                 data = [rec_data[name] for name in rec_data.dtype.names]
    244

/Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pyc in recarray_fromrecords(rec_list)
    604     # to column arrays in the recarray module; it could be there is a better
    605     # way
--> 606     nfields = len(rec_list[0])
    607     obj = np.array(rec_list, dtype=object)
    608     array_list = [np.array(obj[..., i].tolist()) for i in range(nfields)]

TypeError: object of type 'bool' has no len()

to no avail. Is this possible, easy, or wrongheaded? Advise appreciated.

Thanks,
JT



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150819/a359b4c7/attachment.html>

From fred3public at gmail.com  Wed Aug 19 10:29:29 2015
From: fred3public at gmail.com (Fred Moolekamp)
Date: Wed, 19 Aug 2015 10:29:29 -0400
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
Message-ID: <CAGdJfBXZ8rhKXxNBVosr+XLd8NZv8bUy__PkmgzkP9zEtJNZbg@mail.gmail.com>

What you propose should work if you change

rows=h0.values()

to

rows=[tuple(h0.values)]

which will create a table with a single row. However, if your FITS file has
duplicate keys this will fail too and you will have to come up with a
scheme for renaming the duplicate keys for yourself.

On Wed, Aug 19, 2015 at 10:00 AM, Jason Tumlinson <tumlinson at stsci.edu>
wrote:

> Hi,
> I am trying to harvest the information from a large number of FITS
> headers, parse them and do statistics on values, in an efficient manner. I
> read in each header in the usual fashion:
>
> h = fits.open('lbiy83i0q_x1d.fits?)
> h0 = h[0].header
>
> My goal then is to get all the header info into an astropy Table such that
> the table columns are given by the header keys and the values in the rows
> are given by the values in each of 100+ headers. I tried this:
>
> In [38]: t = Table(rows=h0.values(), names=h0.keys())
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call
> last)
> <ipython-input-38-8a0ece2925d5> in <module>()
> ----> 1 t = Table(rows=h0.values(), names=h0.keys())
>
>
> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pyc
> in __init__(self, data, masked, names, dtype, meta, copy, rows)
>     240                 data = rows
>     241             else:
> --> 242                 rec_data = recarray_fromrecords(rows)
>     243                 data = [rec_data[name] for name in rec_data.dtype.
> names]
>     244
>
>
> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pyc
> in recarray_fromrecords(rec_list)
>     604     # to column arrays in the recarray module; it could be there
> is a better
>     605     # way
> --> 606     nfields = len(rec_list[0])
>     607     obj = np.array(rec_list, dtype=object)
>     608     array_list = [np.array(obj[..., i].tolist()) for i in range(
> nfields)]
>
> TypeError: object of type 'bool' has no len()
>
> to no avail. Is this possible, easy, or wrongheaded? Advise appreciated.
>
> Thanks,
> JT
>
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150819/a0d2c1ca/attachment.html>

From jens at astro.su.se  Wed Aug 19 10:36:32 2015
From: jens at astro.su.se (Jens Melinder)
Date: Wed, 19 Aug 2015 16:36:32 +0200
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
Message-ID: <059E3FAB-7B45-40E3-942B-3AC4A1A7CC75@astro.su.se>

Hi,
there are a couple of problems with doing it that way I think. 

1. Table will not like column names with the same name. For fits headers these are the HISTORY and COMMENT fields which will have multiple entries in the output list you get from h0.keys().
2. While not 100% certain I think also the empty keys (which are used to split the HST fist headers into different sections) will cause problems.
3. Finally, there is also a small syntax error in your call to Table. Instead of writing rows=h0.values() you should write rows = [h0.values()] (making it a list of lists)

If you don?t need the HISTORY and COMMENT fields a simple fix maybe something like this:
keys_cl=[]
vals_cl=[]
for ii in range(len(h0.keys())):
	if (h0.keys()[ii] != ??)&(h0.keys()[ii] != ?HISTORY?)&(h0.keys()[ii] != ?COMMENT?):
		keys_cl.append(h0.keys()[ii])
		vals_cl.append(h0.values()[ii])

tab = Table(rows=[vals_cl],names=keys_cl)

Good luck,
Jens 

19 aug 2015 kl. 16:00 skrev Jason Tumlinson <tumlinson at stsci.edu>:

> Hi,
> I am trying to harvest the information from a large number of FITS headers, parse them and do statistics on values, in an efficient manner. I read in each header in the usual fashion: 
> 
> h = fits.open('lbiy83i0q_x1d.fits?)
> h0 = h[0].header
> 
> My goal then is to get all the header info into an astropy Table such that the table columns are given by the header keys and the values in the rows are given by the values in each of 100+ headers. I tried this: 
> 
> In [38]: t = Table(rows=h0.values(), names=h0.keys())
> ---------------------------------------------------------------------------
> TypeError                                 Traceback (most recent call last)
> <ipython-input-38-8a0ece2925d5> in <module>()
> ----> 1 t = Table(rows=h0.values(), names=h0.keys())
> 
> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pyc in __init__(self, data, masked, names, dtype, meta, copy, rows)
>     240                 data = rows
>     241             else:
> --> 242                 rec_data = recarray_fromrecords(rows)
>     243                 data = [rec_data[name] for name in rec_data.dtype.names]
>     244 
> 
> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pyc in recarray_fromrecords(rec_list)
>     604     # to column arrays in the recarray module; it could be there is a better
>     605     # way
> --> 606     nfields = len(rec_list[0])
>     607     obj = np.array(rec_list, dtype=object)
>     608     array_list = [np.array(obj[..., i].tolist()) for i in range(nfields)]
> 
> TypeError: object of type 'bool' has no len()
> 
> to no avail. Is this possible, easy, or wrongheaded? Advise appreciated. 
> 
> Thanks, 
> JT
> 
> 
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-----------------------------------
Jens Melinder
Department of Astronomy
Stockholm University
jens at astro.su.se
+46 706471856




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150819/a0843ee4/attachment.html>

From embray at stsci.edu  Wed Aug 19 10:54:46 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 19 Aug 2015 10:54:46 -0400
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <059E3FAB-7B45-40E3-942B-3AC4A1A7CC75@astro.su.se>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
	<059E3FAB-7B45-40E3-942B-3AC4A1A7CC75@astro.su.se>
Message-ID: <55D498B6.9010000@stsci.edu>

On 08/19/2015 10:36 AM, Jens Melinder wrote:
> Hi,
> there are a couple of problems with doing it that way I think.
>
> 1. Table will not like column names with the same name. For fits headers these
> are the HISTORY and COMMENT fields which will have multiple entries in the
> output list you get from h0.keys().

Fortunately though when you do header['COMMENT'] or header['HISTORY'] it 
automatically combines the values of all COMMENT or HISTORY cards into a single, 
newline-separated value, so that makes things a little simpler)

What's more of a problem if you have duplicates of any non-commentary keyword 
(ex: a duplicate OBSERVER keyword).  This *should* never be the case, but 
technically FITS allows it, though it's strongly discouraged.  In my example 
below I ignore that case since it's fairly rare.

> 2. While not 100% certain I think also the empty keys (which are used to split
> the HST fist headers into different sections) will cause problems.

Right--they're treated the same as COMMENT and HISTORY cards (their keyword is 
just the empty string like in your example below).

> 3. Finally, there is also a small syntax error in your call to Table. Instead of
> writing rows=h0.values() you should write rows = [h0.values()] (making it a list
> of lists)
>
> If you don?t need the HISTORY and COMMENT fields a simple fix maybe something
> like this:
> keys_cl=[]
> vals_cl=[]
> for ii in range(len(h0.keys())):
> if (h0.keys()[ii] != ??)&(h0.keys()[ii] != ?HISTORY?)&(h0.keys()[ii] != ?COMMENT?):
> keys_cl.append(h0.keys()[ii])
> vals_cl.append(h0.values()[ii])
>
> tab = Table(rows=[vals_cl],names=keys_cl)

This example has a couple issues--for one header.keys() returns a generator so 
you can't just index it directly.  I think a simpler approach is:

 >>> unique_keys = sorted(set(header))
 >>> t = Table(rows=[[header[k] for k in unique_keys]], names=unique_keys)

If you wanted to exclude commentary keywords like '', 'COMMENT', and 'HISTORY' 
you might instead do:

 >>> unique_keys = sorted(key for key in header
...                      if key not in ('', 'COMMENT', 'HISTORY'))

This all also assumed that all the headers you're processing have the exact same 
set of keywords in them.  If that weren't the case you might have do take a 
subtly different approach to determining what columns your table should have 
(for example by looping over all headers first and adding their header keywords 
to a set of all known keywords).

Erik

> 19 aug 2015 kl. 16:00 skrev Jason Tumlinson <tumlinson at stsci.edu
> <mailto:tumlinson at stsci.edu>>:
>
>> Hi,
>> I am trying to harvest the information from a large number of FITS headers,
>> parse them and do statistics on values, in an efficient manner. I read in each
>> header in the usual fashion:
>>
>> h = fits.open('lbiy83i0q_x1d.fits?)
>> h0 = h[0].header
>>
>> My goal then is to get all the header info into an astropy Table such that the
>> table columns are given by the header keys and the values in the rows are
>> given by the values in each of 100+ headers. I tried this:
>>
>> In [38]: t = Table(rows=h0.values(), names=h0.keys())
>> ---------------------------------------------------------------------------
>> TypeError                                 Traceback (most recent call last)
>> <ipython-input-38-8a0ece2925d5>in <module>()
>> ----> 1t = Table(rows=h0.values(), names=h0.keys())
>>
>> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pycin
>> __init__(self, data, masked, names, dtype, meta, copy, rows)
>>     240                 data = rows
>>     241 else:
>> --> 242rec_data = recarray_fromrecords(rows)
>>     243                 data = [rec_data[name] for name in rec_data.dtype.names]
>>     244
>>
>> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pycin
>> recarray_fromrecords(rec_list)
>>     604# to column arrays in the recarray module; it could be there is a better
>>     605# way
>> --> 606nfields = len(rec_list[0])
>>     607     obj = np.array(rec_list, dtype=object)
>>     608     array_list = [np.array(obj[..., i].tolist()) for i in range(nfields)]
>>
>> TypeError: object of type 'bool' has no len()
>>
>> to no avail. Is this possible, easy, or wrongheaded? Advise appreciated.
>>
>> Thanks,
>> JT
>>
>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>> http://mail.scipy.org/mailman/listinfo/astropy
>
> -----------------------------------
> Jens Melinder
> Department of Astronomy
> Stockholm University
> jens at astro.su.se <mailto:jens at astro.su.se>
> +46 706471856
>
>
>
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



From jens at astro.su.se  Wed Aug 19 10:59:59 2015
From: jens at astro.su.se (Jens Melinder)
Date: Wed, 19 Aug 2015 16:59:59 +0200
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <55D498B6.9010000@stsci.edu>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
	<059E3FAB-7B45-40E3-942B-3AC4A1A7CC75@astro.su.se>
	<55D498B6.9010000@stsci.edu>
Message-ID: <673C4B09-8C56-4882-A6E0-98628E7B54EC@astro.su.se>


>> 
>> If you don?t need the HISTORY and COMMENT fields a simple fix maybe something
>> like this:
>> keys_cl=[]
>> vals_cl=[]
>> for ii in range(len(h0.keys())):
>> if (h0.keys()[ii] != ??)&(h0.keys()[ii] != ?HISTORY?)&(h0.keys()[ii] != ?COMMENT?):
>> keys_cl.append(h0.keys()[ii])
>> vals_cl.append(h0.values()[ii])
>> 
>> tab = Table(rows=[vals_cl],names=keys_cl)
> 
> This example has a couple issues--for one header.keys() returns a generator so 
> you can't just index it directly.  
 
Ah yes of course, I was trying it out with lists generated from the h0.keys() method. Thanks for pointing it out.

> I think a simpler approach is:
> 
>>>> unique_keys = sorted(set(header))
>>>> t = Table(rows=[[header[k] for k in unique_keys]], names=unique_keys)
> 
> If you wanted to exclude commentary keywords like '', 'COMMENT', and 'HISTORY' 
> you might instead do:
> 
>>>> unique_keys = sorted(key for key in header
> ...                      if key not in ('', 'COMMENT', 'HISTORY'))
> 
> This all also assumed that all the headers you're processing have the exact same 
> set of keywords in them.  If that weren't the case you might have do take a 
> subtly different approach to determining what columns your table should have 
> (for example by looping over all headers first and adding their header keywords 
> to a set of all known keywords).

Much simpler indeed.

Cheers,
Jens


-----------------------------------
Jens Melinder
Department of Astronomy
Stockholm University
jens at astro.su.se
+46 706471856




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150819/290a8536/attachment.html>

From jjk at uvic.ca  Wed Aug 19 11:14:08 2015
From: jjk at uvic.ca (JJ Kavelaars)
Date: Wed, 19 Aug 2015 08:14:08 -0700
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <059E3FAB-7B45-40E3-942B-3AC4A1A7CC75@astro.su.se>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
	<059E3FAB-7B45-40E3-942B-3AC4A1A7CC75@astro.su.se>
Message-ID: <9273EFDE-FA04-4AFC-A066-169A18F43325@uvic.ca>

Or something like:

hdulist=fits.open(filename)
h=hdulist[0].header
for key in ["HISTORY", "COMMENT", ""]:
   del h[key] 
names = sorted(h.keys())
rows = [h[k] for k in names]
t = Table(rows=[rows], names=names)

this only gets one value/key pair for each header key that may be duplicated.  Which one you get is not your choice here, so that might not be what is wanted.  Making a table with multiple rows is left as an exercise for the reader.

Doing those ?delete? commands without checking if the keyword exists outside a try/except is currently OK but won?t be in the future will raise an error if the keyword is missing.


JJ



From mcraig at mnstate.edu  Wed Aug 19 11:26:17 2015
From: mcraig at mnstate.edu (Matthew Craig)
Date: Wed, 19 Aug 2015 15:26:17 +0000
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
Message-ID: <6B503467-8E73-47EC-97A1-E5633C897372@mnstate.edu>

Hi Jason,

I?m late to the party, but ccdproc is about to add an ImageFileCollection obj that does (I think) what you are looking for?given a directory containing FITS files, make a table out of their headers (will not handle multiple extensions in a single file gracefully).

That functionality is available right now from another package, msumastro (http://msum-astro.readthedocs.org/en/latest/image_management.html or https://github.com/mwcraig/msumastro) so if you want it today I would get it from there.

If you decide to give it a try and find issues please report them, either at https://github.com/astropy/ccdproc/issues or at https://github.com/mwcraig/msumastro/issues or by email.

In a nutshell you would do:

>>> from msumastro import ImageFileCollection
>>> my_collection = ImageFileCollection(?path/to/dir/with/files?, keywords=?*?)
>>> my_table = my_collection.summary


Matt Craig

schedule:  http://physics.mnstate.edu/craig
??

Professor
Department of Physics and Astronomy
Minnesota State University Moorhead
1104 7th Ave S, Moorhead MN 56563

phone: (218) 477-2439
fax: (218) 477-2290

On Aug 19, 2015, at 9:00 AM, Jason Tumlinson <tumlinson at stsci.edu<mailto:tumlinson at stsci.edu>> wrote:

Hi,
I am trying to harvest the information from a large number of FITS headers, parse them and do statistics on values, in an efficient manner. I read in each header in the usual fashion:

h = fits.open('lbiy83i0q_x1d.fits?)
h0 = h[0].header

My goal then is to get all the header info into an astropy Table such that the table columns are given by the header keys and the values in the rows are given by the values in each of 100+ headers. I tried this:

In [38]: t = Table(rows=h0.values(), names=h0.keys())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-38-8a0ece2925d5> in <module>()
----> 1 t = Table(rows=h0.values(), names=h0.keys())

/Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pyc in __init__(self, data, masked, names, dtype, meta, copy, rows)
    240                 data = rows
    241             else:
--> 242                 rec_data = recarray_fromrecords(rows)
    243                 data = [rec_data[name] for name in rec_data.dtype.names]
    244

/Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pyc in recarray_fromrecords(rec_list)
    604     # to column arrays in the recarray module; it could be there is a better
    605     # way
--> 606     nfields = len(rec_list[0])
    607     obj = np.array(rec_list, dtype=object)
    608     array_list = [np.array(obj[..., i].tolist()) for i in range(nfields)]

TypeError: object of type 'bool' has no len()

to no avail. Is this possible, easy, or wrongheaded? Advise appreciated.

Thanks,
JT



_______________________________________________
AstroPy mailing list
AstroPy at scipy.org<mailto:AstroPy at scipy.org>
http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150819/23f50876/attachment.html>

From embray at stsci.edu  Wed Aug 19 13:27:11 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 19 Aug 2015 13:27:11 -0400
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <6B503467-8E73-47EC-97A1-E5633C897372@mnstate.edu>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
	<6B503467-8E73-47EC-97A1-E5633C897372@mnstate.edu>
Message-ID: <55D4BC6F.1020500@stsci.edu>

On 08/19/2015 11:26 AM, Matthew Craig wrote:
> Hi Jason,
>
> I?m late to the party, but ccdproc is about to add an ImageFileCollection obj
> that does (I think) what you are looking for?given a directory containing FITS
> files, make a table out of their headers (will not handle multiple extensions in
> a single file gracefully).

Couldn't it just include a column for the extension within that file?  Or is 
there something I'm missing?

Erik

> That functionality is available right now from another package, msumastro
> (http://msum-astro.readthedocs.org/en/latest/image_management.html or
> https://github.com/mwcraig/msumastro) so if you want it today I would get it
> from there.
>
> If you decide to give it a try and find issues please report them, either at
> https://github.com/astropy/ccdproc/issues or at
> https://github.com/mwcraig/msumastro/issues or by email.
>
> In a nutshell you would do:
>
>  >>> from msumastro import ImageFileCollection
>  >>> my_collection = ImageFileCollection(?path/to/dir/with/files?, keywords=?*?)
>>>>  my_table = my_collection.summary
>
>
> Matt Craig
>
> schedule: http://physics.mnstate.edu/craig
> ??
>
> Professor
> Department of Physics and Astronomy
> Minnesota State University Moorhead
> 1104 7th Ave S, Moorhead MN 56563
>
> phone: (218) 477-2439
> fax: (218) 477-2290
>
>> On Aug 19, 2015, at 9:00 AM, Jason Tumlinson <tumlinson at stsci.edu
>> <mailto:tumlinson at stsci.edu>> wrote:
>>
>> Hi,
>> I am trying to harvest the information from a large number of FITS headers,
>> parse them and do statistics on values, in an efficient manner. I read in each
>> header in the usual fashion:
>>
>> h = fits.open('lbiy83i0q_x1d.fits?)
>> h0 = h[0].header
>>
>> My goal then is to get all the header info into an astropy Table such that the
>> table columns are given by the header keys and the values in the rows are
>> given by the values in each of 100+ headers. I tried this:
>>
>> In [38]: t = Table(rows=h0.values(), names=h0.keys())
>> ---------------------------------------------------------------------------
>> TypeError                                 Traceback (most recent call last)
>> <ipython-input-38-8a0ece2925d5>in <module>()
>> ----> 1t = Table(rows=h0.values(), names=h0.keys())
>>
>> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pycin
>> __init__(self, data, masked, names, dtype, meta, copy, rows)
>>     240                 data = rows
>>     241 else:
>> --> 242rec_data = recarray_fromrecords(rows)
>>     243                 data = [rec_data[name] for name in rec_data.dtype.names]
>>     244
>>
>> /Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pycin
>> recarray_fromrecords(rec_list)
>>     604# to column arrays in the recarray module; it could be there is a better
>>     605# way
>> --> 606nfields = len(rec_list[0])
>>     607     obj = np.array(rec_list, dtype=object)
>>     608     array_list = [np.array(obj[..., i].tolist()) for i in range(nfields)]
>>
>> TypeError: object of type 'bool' has no len()
>>
>> to no avail. Is this possible, easy, or wrongheaded? Advise appreciated.
>>
>> Thanks,
>> JT
>>
>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



From mcraig at mnstate.edu  Wed Aug 19 13:34:29 2015
From: mcraig at mnstate.edu (Matthew Craig)
Date: Wed, 19 Aug 2015 17:34:29 +0000
Subject: [AstroPy] Convert FITS header into astropy Table entries?
In-Reply-To: <55D4BC6F.1020500@stsci.edu>
References: <BF0C9ABA-8AA5-4D23-A00D-712FA394CA9B@stsci.edu>
	<6B503467-8E73-47EC-97A1-E5633C897372@mnstate.edu>
	<55D4BC6F.1020500@stsci.edu>
Message-ID: <A4C3B785-2269-4FDC-9D51-C64D1EF0ED72@mnstate.edu>

I?m late to the party, but ccdproc is about to add an ImageFileCollection obj
that does (I think) what you are looking for?given a directory containing FITS
files, make a table out of their headers (will not handle multiple extensions in
a single file gracefully).

Couldn't it just include a column for the extension within that file?  Or is
there something I'm missing?

Yes, I think it *could*?but in its current state I don?t think it *would* :)

Matt Craig

schedule:  http://physics.mnstate.edu/craig
??

Professor
Department of Physics and Astronomy
Minnesota State University Moorhead
1104 7th Ave S, Moorhead MN 56563

phone: (218) 477-2439
fax: (218) 477-2290

On Aug 19, 2015, at 12:27 PM, Erik Bray <embray at stsci.edu<mailto:embray at stsci.edu>> wrote:

On 08/19/2015 11:26 AM, Matthew Craig wrote:
Hi Jason,

I?m late to the party, but ccdproc is about to add an ImageFileCollection obj
that does (I think) what you are looking for?given a directory containing FITS
files, make a table out of their headers (will not handle multiple extensions in
a single file gracefully).

Couldn't it just include a column for the extension within that file?  Or is
there something I'm missing?

Erik

That functionality is available right now from another package, msumastro
(http://msum-astro.readthedocs.org/en/latest/image_management.html or
https://github.com/mwcraig/msumastro) so if you want it today I would get it
from there.

If you decide to give it a try and find issues please report them, either at
https://github.com/astropy/ccdproc/issues or at
https://github.com/mwcraig/msumastro/issues or by email.

In a nutshell you would do:

from msumastro import ImageFileCollection
my_collection = ImageFileCollection(?path/to/dir/with/files?, keywords=?*?)
my_table = my_collection.summary


Matt Craig

schedule: http://physics.mnstate.edu/craig
??

Professor
Department of Physics and Astronomy
Minnesota State University Moorhead
1104 7th Ave S, Moorhead MN 56563

phone: (218) 477-2439
fax: (218) 477-2290

On Aug 19, 2015, at 9:00 AM, Jason Tumlinson <tumlinson at stsci.edu<mailto:tumlinson at stsci.edu>
<mailto:tumlinson at stsci.edu>> wrote:

Hi,
I am trying to harvest the information from a large number of FITS headers,
parse them and do statistics on values, in an efficient manner. I read in each
header in the usual fashion:

h = fits.open('lbiy83i0q_x1d.fits?)
h0 = h[0].header

My goal then is to get all the header info into an astropy Table such that the
table columns are given by the header keys and the values in the rows are
given by the values in each of 100+ headers. I tried this:

In [38]: t = Table(rows=h0.values(), names=h0.keys())
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-38-8a0ece2925d5>in <module>()
----> 1t = Table(rows=h0.values(), names=h0.keys())

/Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/table.pycin
__init__(self, data, masked, names, dtype, meta, copy, rows)
   240                 data = rows
   241 else:
--> 242rec_data = recarray_fromrecords(rows)
   243                 data = [rec_data[name] for name in rec_data.dtype.names]
   244

/Users/tumlinson/Ureka/python/lib/python2.7/site-packages/astropy/table/np_utils.pycin
recarray_fromrecords(rec_list)
   604# to column arrays in the recarray module; it could be there is a better
   605# way
--> 606nfields = len(rec_list[0])
   607     obj = np.array(rec_list, dtype=object)
   608     array_list = [np.array(obj[..., i].tolist()) for i in range(nfields)]

TypeError: object of type 'bool' has no len()

to no avail. Is this possible, easy, or wrongheaded? Advise appreciated.

Thanks,
JT



_______________________________________________
AstroPy mailing list
AstroPy at scipy.org<mailto:AstroPy at scipy.org> <mailto:AstroPy at scipy.org>
http://mail.scipy.org/mailman/listinfo/astropy



_______________________________________________
AstroPy mailing list
AstroPy at scipy.org<mailto:AstroPy at scipy.org>
http://mail.scipy.org/mailman/listinfo/astropy


_______________________________________________
AstroPy mailing list
AstroPy at scipy.org<mailto:AstroPy at scipy.org>
http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150819/82b9657c/attachment.html>

From eric.emsellem at eso.org  Fri Aug 21 06:32:11 2015
From: eric.emsellem at eso.org (Eric Emsellem)
Date: Fri, 21 Aug 2015 12:32:11 +0200
Subject: [AstroPy] Remove a model from a composite model in astropy.modeling
Message-ID: <55D6FE2B.1030605@eso.org>

Hi

I would like to know if something like this is possible with 
astropy.modeling:
Let's imagine I have a composite model made of 3 models, e.g.,

 > modelC = model1 + model2 + model3

where each model1, model2, model3 is a custom model or one of the 
integrated ones in astropy.modeling.models.

Can I do something which would be equivalent to:

 > modelC.remove(model2)

where now modelC would be  : model1 + model3

???

Thanks for any tips on this one!

Eric


From aldcroft at head.cfa.harvard.edu  Fri Aug 21 10:41:22 2015
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Fri, 21 Aug 2015 10:41:22 -0400
Subject: [AstroPy] Changing dtype when copying masked table throws
	TypeError
In-Reply-To: <55809E3E.6000602@buckeyemail.osu.edu>
References: <55809E3E.6000602@buckeyemail.osu.edu>
Message-ID: <CAMtEP6xsVZLCM75CfboVyCtrx6jdStaNnsoesGicNUYOyTQPFA@mail.gmail.com>

On Tue, Jun 16, 2015 at 6:07 PM, Gregory Simonian <
simonian.7 at buckeyemail.osu.edu> wrote:

> Hi all,
>
> I have a Table with several columns which were read in as strings because
> they had a combination of numeric and non-numeric characters. After
> cleaning out the non-numeric characters, I'd like to convert the dtype of
> the column from string to int.
>
> Looking at the astropy documentation for modifying tables [0], under the
> caveats section it states:
>
> "The data type for a column cannot be changed in place. In order to do
> this you must make a copy of the table with the column type changed
> appropriately."
>
> So I try making a new table (the table is small, so I don't mind the
> temporary memory hit), but I run into the error:
> "TypeError: Cannot set fill value of string with array of dtype int64"
> When checking the fill value of the column (which is masked from the
> reader), it's 'N/A', which makes sense that it wouldn't work. But when I
> change it to '0', I still get the same problem. I managed to find a minimal
> working example as:
>
> import numpy as np
> t = Table({'a': ['1']}, masked=True)
> t['a'].set_fill_value('0')
> u = Table(t, names=['a'], dtype=[np.int])
> > TypeError: Cannot set fill value of string with array of dtype int64
>
> When debugging the exception, it seems like the Table constructor doesn't
> convert the fill value of masked tables when generating a new table, which
> seems to defeat the ability to change the dtype of masked columns. I was
> wondering if other people have run by this, and if there's a more
> straightforward way of changing the dtype of a column. This is with Astropy
> 1.0.3 on Python 2.7.10 through Anaconda on CentOS 6.6.
>
> When checking to see if this was reported, I ran into issue #3809 [1],
> which might end up being the ultimate solution to my problem. It seems like
> a workaround would be:
>
> temp_column = np.ma.asanyarray(t['a'], np.int)
> del(t['a'])
> t['a'] = temp_column
>
>
Hi Gregory,

Sorry, your nice bug report fell through the cracks and didn't get a
response.  You've basically understood the issues correctly, and I just put
in a pull request that fixes this:

https://github.com/astropy/astropy/pull/4088/files

In your workaround you could have done something slightly simpler:

 temp_column = t['a'].astype(np.int)

Cheers,
Tom




> Thanks!
>
>                                             Gregory Simonian
>
> [0] http://docs.astropy.org/en/stable/table/modify_table.html#caveats
> [1] https://github.com/astropy/astropy/issues/3809
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150821/80cce2b0/attachment.html>

From embray at stsci.edu  Fri Aug 21 10:48:31 2015
From: embray at stsci.edu (Erik Bray)
Date: Fri, 21 Aug 2015 10:48:31 -0400
Subject: [AstroPy] Remove a model from a composite model in
 astropy.modeling
In-Reply-To: <55D6FE2B.1030605@eso.org>
References: <55D6FE2B.1030605@eso.org>
Message-ID: <55D73A3F.2040506@stsci.edu>

On 08/21/2015 06:32 AM, Eric Emsellem wrote:
> Hi
>
> I would like to know if something like this is possible with
> astropy.modeling:
> Let's imagine I have a composite model made of 3 models, e.g.,
>
>   > modelC = model1 + model2 + model3
>
> where each model1, model2, model3 is a custom model or one of the
> integrated ones in astropy.modeling.models.
>
> Can I do something which would be equivalent to:
>
>   > modelC.remove(model2)
>
> where now modelC would be  : model1 + model3
>
> ???
>
> Thanks for any tips on this one!

Hi Eric,

Glad to see you're trying out compound models.

right now the easiest way to do what you're asking would just be to take two 
slices that exclude model2 and combine them.  For example with:

 >>> modelC = model1 + model2 + model3

model2 has the index 1 (you can check this via the output of print(modelC)).

Then you can do:

 >>> modelC = modelC[:1] + modelC[2:]

This is even easier if you give names to the individual models so that you don't 
have to worry about their numeric index.  Although in this case it's a little 
awkward because when you use names in slices the end points are *inclusive* so 
you have to know which models come before and after 'B'

 >>> model = model1.rename('A') + model2.rename('B') + model3.rename('C')
 >>> model = model[:'A'] + model['C':]


I could see a case for a .remove() method, as well as .insert(), etc.  But it 
would have to take some care with how it handles the operators.  If you're just 
talking a linear sum it's straightforward enough.  But what if you have:

 >>> model = a + b * c

If you do model.remove('b') should it become  a + c or a * c?  I think in that 
case probably a + c makes sense due to the order of operations.  But what about

 >>> model = a + b ** c?

This is less likely to show up, but is possible.  Neither a + c or a ** c make 
much sense as a result.

So again I think this would be a good thing to have but there are subtleties 
that would have to be worked out.

Erik


From embray at stsci.edu  Fri Aug 21 10:57:54 2015
From: embray at stsci.edu (Erik Bray)
Date: Fri, 21 Aug 2015 10:57:54 -0400
Subject: [AstroPy] Remove a model from a composite model in
 astropy.modeling
In-Reply-To: <55D73A3F.2040506@stsci.edu>
References: <55D6FE2B.1030605@eso.org> <55D73A3F.2040506@stsci.edu>
Message-ID: <55D73C72.8000002@stsci.edu>

On 08/21/2015 10:48 AM, Erik Bray wrote:
> This is even easier if you give names to the individual models so that you don't
> have to worry about their numeric index.  Although in this case it's a little
> awkward because when you use names in slices the end points are *inclusive* so
> you have to know which models come before and after 'B'
>
>   >>> model = model1.rename('A') + model2.rename('B') + model3.rename('C')
>   >>> model = model[:'A'] + model['C':]

Another thought I just had is that it would be good to have a method to map a 
model's label (if it has one) to its index within the ordering of the 
submodules.  For example in the above case:

 >>> model.index('B')
1

Then you could do:

 >>> model = model[:model.index('B')] + model[model.index('B') + 1:]

Erik


From eric.emsellem at eso.org  Sun Aug 23 06:48:36 2015
From: eric.emsellem at eso.org (Eric Emsellem)
Date: Sun, 23 Aug 2015 12:48:36 +0200
Subject: [AstroPy] Remove a model from a composite model in
	astropy.modeling
Message-ID: <55D9A504.9090407@eso.org>

> right now the easiest way to do what you're asking would just be to take two
> slices that exclude model2 and combine them.  For example with:
>
>   >>> modelC = model1 + model2 + model3
>
> model2 has the index 1 (you can check this via the output of print(modelC)).
>
> Then you can do:
>
>   >>> modelC = modelC[:1] + modelC[2:]
>
> This is even easier if you give names to the individual models so that you don't
> have to worry about their numeric index.  Although in this case it's a little
> awkward because when you use names in slices the end points are*inclusive*  so
> you have to know which models come before and after 'B'
>
>   >>> model = model1.rename('A') + model2.rename('B') + model3.rename('C')
>   >>> model = model[:'A'] + model['C':]
>
>
> I could see a case for a .remove() method, as well as .insert(), etc.  But it
> would have to take some care with how it handles the operators.  If you're just
> talking a linear sum it's straightforward enough.  But what if you have:
>
>   >>> model = a + b * c
>
> If you do model.remove('b') should it become  a + c or a * c?  I think in that
> case probably a + c makes sense due to the order of operations.  But what about
>
>   >>> model = a + b ** c?
>
> This is less likely to show up, but is possible.  Neither a + c or a ** c make
> much sense as a result.
>
> So again I think this would be a good thing to have but there are subtleties
> that would have to be worked out.
>
> Erik

Thanks so much Erik. This makes a lot of sense although I had NOT 
realised you could just slice a composite model. I was a bit worried 
that adding was easy but "removing" was not due to the number of 
parameters you had to "clean".

I'll try the solutions you indicated.

As for a "remove" function: an idea would be to act on "blocks" instead 
and not on models directly. This would require some thinking indeed. In 
your example b**c would be a block, c would be a model. Only the second 
one could have a "remove" function. This would mean a hierarchy. Not 
sure you wish to pursue that path, which is quite a complex one and 
maybe not worth it.

In any cases, I am developing something which only uses additions, so I 
won't have that problem.

THANKS again!
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150823/6b6d101f/attachment.html>

From kevin.gullikson at gmail.com  Mon Aug 24 09:57:24 2015
From: kevin.gullikson at gmail.com (Kevin Gullikson)
Date: Mon, 24 Aug 2015 08:57:24 -0500
Subject: [AstroPy]  Using quantities is slow
Message-ID: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>

Hi,

I am writing an MCMC fitting code, and my current implementation uses the
(very useful) astropy quantities framework to take care of unit conversions
and such. However, the code is surprisingly slow so I profiled it with
%prun and it looks like the quantity utilities that wrap numpy functions
are taking a huge chunk of time. Is that just the price of convenience?

Here is an excerpt of the output from profiling my code.

         1827310507 function calls (1823056217 primitive calls) in
3659.725 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 25719008  400.735    0.000 1944.942    0.000 quantity.py:263(__array_prepare__)
159341911  157.611    0.000  182.754    0.000 {getattr}
32101370/31134490  148.248    0.000  180.318    0.000 {method 'format'
of 'unicode' objects}
 27459392  143.898    0.000  397.546    0.000 quantity.py:520(_new_view)
 25719008  141.664    0.000  566.152    0.000 quantity.py:421(__array_wrap__)
  6381408  137.927    0.000  733.341    0.000 core.py:2013(_expand_and_gather)
 63814347  136.370    0.000  373.226    0.000 {hasattr}
188542970  135.470    0.000  300.222    0.000 {isinstance}
 30360032  112.239    0.000  136.142    0.000 core.py:695(__hash__)
 44089728   99.326    0.000  164.752    0.000 abc.py:128(__instancecheck__)
 42350301   97.285    0.000  191.681    0.000 {method 'view' of
'numpy.ndarray' objects}
 30553408   95.704    0.000  236.855    0.000 quantity.py:695(__getattr__)
  6961536   90.321    0.000  675.250    0.000 quantity.py:756(__mul__)
   386752   87.576    0.000  900.927    0.002 Orbit.py:133(get_true_anomaly)
   193376   78.895    0.000 1178.998    0.006 Orbit.py:15(__init__)
 13729696   65.363    0.000  353.957    0.000 core.py:891(_to)
 58012800   59.299    0.000  128.197    0.000
quantity.py:260(__array_finalize__)
   386752   59.201    0.000   79.588    0.000
interpolate.py:1581(_evaluate_linear)
 73869632   55.839    0.000   55.839    0.000 _weakrefset.py:70(__contains__)
 15083328   53.805    0.000  266.936    0.000 core.py:2014(add_unit)
  2900640   42.100    0.000  535.141    0.000 quantity.py:780(__div__)


Kevin Gullikson
PhD Candidate
University of Texas Astronomy
RLM 15.310E
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150824/b9becbf6/attachment.html>

From andrew.hearin at yale.edu  Mon Aug 24 18:13:55 2015
From: andrew.hearin at yale.edu (Andrew Hearin)
Date: Mon, 24 Aug 2015 18:13:55 -0400
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
Message-ID: <024A4303-5925-41E0-9529-A39C8EBD3B5F@yale.edu>

Hi Kevin,

I?m really glad you asked this. I?m on the verge of implementing Quantities in my own work, and I will be very interested to hear what the workaround is.

Cheers,
Andrew


On Aug 24, 2015, at 9:57 AM, Kevin Gullikson <kevin.gullikson at gmail.com> wrote:

> Hi,
> 
> I am writing an MCMC fitting code, and my current implementation uses the (very useful) astropy quantities framework to take care of unit conversions and such. However, the code is surprisingly slow so I profiled it with %prun and it looks like the quantity utilities that wrap numpy functions are taking a huge chunk of time. Is that just the price of convenience?
> 
> Here is an excerpt of the output from profiling my code.
> 
>          1827310507 function calls (1823056217 primitive calls) in 3659.725 seconds
> 
>    Ordered by: internal time
> 
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>  25719008  400.735    0.000 1944.942    0.000 quantity.py:263(__array_prepare__)
> 159341911  157.611    0.000  182.754    0.000 {getattr}
> 32101370/31134490  148.248    0.000  180.318    0.000 {method 'format' of 'unicode' objects}
>  27459392  143.898    0.000  397.546    0.000 quantity.py:520(_new_view)
>  25719008  141.664    0.000  566.152    0.000 quantity.py:421(__array_wrap__)
>   6381408  137.927    0.000  733.341    0.000 core.py:2013(_expand_and_gather)
>  63814347  136.370    0.000  373.226    0.000 {hasattr}
> 188542970  135.470    0.000  300.222    0.000 {isinstance}
>  30360032  112.239    0.000  136.142    0.000 core.py:695(__hash__)
>  44089728   99.326    0.000  164.752    0.000 abc.py:128(__instancecheck__)
>  42350301   97.285    0.000  191.681    0.000 {method 'view' of 'numpy.ndarray' objects}
>  30553408   95.704    0.000  236.855    0.000 quantity.py:695(__getattr__)
>   6961536   90.321    0.000  675.250    0.000 quantity.py:756(__mul__)
>    386752   87.576    0.000  900.927    0.002 Orbit.py:133(get_true_anomaly)
>    193376   78.895    0.000 1178.998    0.006 Orbit.py:15(__init__)
>  13729696   65.363    0.000  353.957    0.000 core.py:891(_to)
>  58012800   59.299    0.000  128.197    0.000 quantity.py:260(__array_finalize__)
>    386752   59.201    0.000   79.588    0.000 interpolate.py:1581(_evaluate_linear)
>  73869632   55.839    0.000   55.839    0.000 _weakrefset.py:70(__contains__)
>  15083328   53.805    0.000  266.936    0.000 core.py:2014(add_unit)
>   2900640   42.100    0.000  535.141    0.000 quantity.py:780(__div__)
> 
> Kevin Gullikson
> PhD Candidate
> University of Texas Astronomy
> RLM 15.310E
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.scipy.org_mailman_listinfo_astropy&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=AHkQ8HPUDwzl0x62ybAnwN_OEebPRGDtcjUPBcnLYw4&m=x_Glih83ZTNtGOD126hO0h-K5DS2xM7tOX4tCZ6pKys&s=1ol12g0yGQ_OhCY6ZLEwL7aaDv2XF7YjekzlgDVjxxE&e= 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150824/94d71e5c/attachment.html>

From teuben at astro.umd.edu  Mon Aug 24 18:21:09 2015
From: teuben at astro.umd.edu (Peter Teuben)
Date: Mon, 24 Aug 2015 18:21:09 -0400
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <024A4303-5925-41E0-9529-A39C8EBD3B5F@yale.edu>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
	<024A4303-5925-41E0-9529-A39C8EBD3B5F@yale.edu>
Message-ID: <55DB98D5.7000806@astro.umd.edu>

Hi all
    The AMUSE project uses the comma operator to append units to
numbers, would be interesting
to hear of its performance compared to the astropy one. The concept was
discussed at Python in Astronomy
in Arjen's talk on AMUSE.

peter


On 08/24/2015 06:13 PM, Andrew Hearin wrote:
> Hi Kevin,
>
> I?m really glad you asked this. I?m on the verge of implementing
> Quantities in my own work, and I will be very interested to hear what
> the workaround is.
>
> Cheers,
> Andrew
>
>
> On Aug 24, 2015, at 9:57 AM, Kevin Gullikson
> <kevin.gullikson at gmail.com <mailto:kevin.gullikson at gmail.com>> wrote:
>
>> Hi,
>>
>> I am writing an MCMC fitting code, and my current implementation uses
>> the (very useful) astropy quantities framework to take care of unit
>> conversions and such. However, the code is surprisingly slow so I
>> profiled it with %prun and it looks like the quantity utilities that
>> wrap numpy functions are taking a huge chunk of time. Is that just
>> the price of convenience?
>>
>> Here is an excerpt of the output from profiling my code.
>>
>>          1827310507 function calls (1823056217 primitive calls) in 3659.725 seconds
>>
>>    Ordered by: internal time
>>
>>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>>  25719008  400.735    0.000 1944.942    0.000 quantity.py:263(__array_prepare__)
>> 159341911  157.611    0.000  182.754    0.000 {getattr}
>> 32101370/31134490  148.248    0.000  180.318    0.000 {method 'format' of 'unicode' objects}
>>  27459392  143.898    0.000  397.546    0.000 quantity.py:520(_new_view)
>>  25719008  141.664    0.000  566.152    0.000 quantity.py:421(__array_wrap__)
>>   6381408  137.927    0.000  733.341    0.000 core.py:2013(_expand_and_gather)
>>  63814347  136.370    0.000  373.226    0.000 {hasattr}
>> 188542970  135.470    0.000  300.222    0.000 {isinstance}
>>  30360032  112.239    0.000  136.142    0.000 core.py:695(__hash__)
>>  44089728   99.326    0.000  164.752    0.000 abc.py:128(__instancecheck__)
>>  42350301   97.285    0.000  191.681    0.000 {method 'view' of 'numpy.ndarray' objects}
>>  30553408   95.704    0.000  236.855    0.000 quantity.py:695(__getattr__)
>>   6961536   90.321    0.000  675.250    0.000 quantity.py:756(__mul__)
>>    386752   87.576    0.000  900.927    0.002 Orbit.py:133(get_true_anomaly)
>>    193376   78.895    0.000 1178.998    0.006 Orbit.py:15(__init__)
>>  13729696   65.363    0.000  353.957    0.000 core.py:891(_to)
>>  58012800   59.299    0.000  128.197    0.000 quantity.py:260(__array_finalize__)
>>    386752   59.201    0.000   79.588    0.000 interpolate.py:1581(_evaluate_linear)
>>  73869632   55.839    0.000   55.839    0.000 _weakrefset.py:70(__contains__)
>>  15083328   53.805    0.000  266.936    0.000 core.py:2014(add_unit)
>>   2900640   42.100    0.000  535.141    0.000 quantity.py:780(__div__)
>>
>> Kevin Gullikson
>> PhD Candidate
>> University of Texas Astronomy
>> RLM 15.310E
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__mail.scipy.org_mailman_listinfo_astropy&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=AHkQ8HPUDwzl0x62ybAnwN_OEebPRGDtcjUPBcnLYw4&m=x_Glih83ZTNtGOD126hO0h-K5DS2xM7tOX4tCZ6pKys&s=1ol12g0yGQ_OhCY6ZLEwL7aaDv2XF7YjekzlgDVjxxE&e=
>>
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150824/7143d0b5/attachment.html>

From embray at stsci.edu  Mon Aug 24 18:26:40 2015
From: embray at stsci.edu (Erik Bray)
Date: Mon, 24 Aug 2015 18:26:40 -0400
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
Message-ID: <55DB9A20.5080006@stsci.edu>

On 08/24/2015 09:57 AM, Kevin Gullikson wrote:
> Hi,
>
> I am writing an MCMC fitting code, and my current implementation uses the (very
> useful) astropy quantities framework to take care of unit conversions and such.
> However, the code is surprisingly slow so I profiled it with %prun and it looks
> like the quantity utilities that wrap numpy functions are taking a huge chunk of
> time. Is that just the price of convenience?

Hi Kevin,

It's hard to say exactly where your sticking point is without seeing some code. 
  But it sounds like you're probably evaluating some function that performs a 
computation using quantities in a loop for interpolation.  If the unit 
conversion needs to be computed every time it goes through that loop that's 
going to always add an enormous amount of overhead (though I think this will be 
improved when Numpy 1.10 is out and we can take advantage of __numpy_ufunc__ 
support).

Better to convert the input arrays so that they're all expressed in the same 
base units.  The exact heuristics to use for this could be very simple, or could 
be quite complex (perhaps to minimize the amount of error).  Then perform the 
fit with unadorned arrays, and restore the units later.

I'm working on similar code to do exactly this for astropy.modeling, so that 
models which have units attached the parameters and/or inputs can be fitted 
efficiently.  And in any case the fitting algorithms in SciPy don't natively 
support units, so it's necessary to make some conversions first.

Perhaps the routine for choosing the best set of conversions to do given a list 
of quantities could be made general enough to be useful outside of 
astropy.modeling (though once that unit support is fully working you might be 
interested to try implementing your MCMC code within the astropy.modeling 
framework).

Erik

> Here is an excerpt of the output from profiling my code.
>
>           1827310507 function calls (1823056217 primitive calls) in 3659.725 seconds
>
>     Ordered by: internal time
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>   25719008  400.735    0.000 1944.942    0.000 quantity.py:263(__array_prepare__)
> 159341911  157.611    0.000  182.754    0.000 {getattr}
> 32101370/31134490  148.248    0.000  180.318    0.000 {method 'format' of 'unicode' objects}
>   27459392  143.898    0.000  397.546    0.000 quantity.py:520(_new_view)
>   25719008  141.664    0.000  566.152    0.000 quantity.py:421(__array_wrap__)
>    6381408  137.927    0.000  733.341    0.000 core.py:2013(_expand_and_gather)
>   63814347  136.370    0.000  373.226    0.000 {hasattr}
> 188542970  135.470    0.000  300.222    0.000 {isinstance}
>   30360032  112.239    0.000  136.142    0.000 core.py:695(__hash__)
>   44089728   99.326    0.000  164.752    0.000 abc.py:128(__instancecheck__)
>   42350301   97.285    0.000  191.681    0.000 {method 'view' of 'numpy.ndarray' objects}
>   30553408   95.704    0.000  236.855    0.000 quantity.py:695(__getattr__)
>    6961536   90.321    0.000  675.250    0.000 quantity.py:756(__mul__)
>     386752   87.576    0.000  900.927    0.002 Orbit.py:133(get_true_anomaly)
>     193376   78.895    0.000 1178.998    0.006 Orbit.py:15(__init__)
>   13729696   65.363    0.000  353.957    0.000 core.py:891(_to)
>   58012800   59.299    0.000  128.197    0.000 quantity.py:260(__array_finalize__)
>     386752   59.201    0.000   79.588    0.000 interpolate.py:1581(_evaluate_linear)
>   73869632   55.839    0.000   55.839    0.000 _weakrefset.py:70(__contains__)
>   15083328   53.805    0.000  266.936    0.000 core.py:2014(add_unit)
>    2900640   42.100    0.000  535.141    0.000 quantity.py:780(__div__)



From derek at astro.physik.uni-goettingen.de  Mon Aug 24 18:32:29 2015
From: derek at astro.physik.uni-goettingen.de (Derek Homeier)
Date: Tue, 25 Aug 2015 00:32:29 +0200
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
Message-ID: <69F6DBCE-C479-40B5-88C4-DE24AE9B6410@astro.physik.uni-goettingen.de>

On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com> wrote:
> 
> I am writing an MCMC fitting code, and my current implementation uses the (very useful) astropy quantities framework to take care of unit conversions and such. However, the code is surprisingly slow so I profiled it with %prun and it looks like the quantity utilities that wrap numpy functions are taking a huge chunk of time. Is that just the price of convenience?
> 
I am not an expert on the quantities/units implementation, but I would imagine that they could indeed
introduce a significant overhead, as the class adds a number of validations and checks for a function
calling a quantity object. Especially with rather numerous operations on comparatively small arrays
this could become problematic.
Did you experiment with using decorators as described in
http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities

- iiuc this would allow bypassing some of the validations?

Another option might be to only pass plain numpy arrays to the innermost loops of your code with .value,
of course at the expense of some of the convenience gained in the first place...

HTH
					Derek



From embray at stsci.edu  Mon Aug 24 18:41:26 2015
From: embray at stsci.edu (Erik Bray)
Date: Mon, 24 Aug 2015 18:41:26 -0400
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <69F6DBCE-C479-40B5-88C4-DE24AE9B6410@astro.physik.uni-goettingen.de>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
	<69F6DBCE-C479-40B5-88C4-DE24AE9B6410@astro.physik.uni-goettingen.de>
Message-ID: <55DB9D96.6050008@stsci.edu>

On 08/24/2015 06:32 PM, Derek Homeier wrote:
> On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com> wrote:
>>
>> I am writing an MCMC fitting code, and my current implementation uses the (very useful) astropy quantities framework to take care of unit conversions and such. However, the code is surprisingly slow so I profiled it with %prun and it looks like the quantity utilities that wrap numpy functions are taking a huge chunk of time. Is that just the price of convenience?
>>
> I am not an expert on the quantities/units implementation, but I would imagine that they could indeed
> introduce a significant overhead, as the class adds a number of validations and checks for a function
> calling a quantity object. Especially with rather numerous operations on comparatively small arrays
> this could become problematic.
> Did you experiment with using decorators as described in
> http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
>
> - iiuc this would allow bypassing some of the validations?

I don't think this is really relevant to the OP's question (though always good 
to know about).  This decorator just makes it easy to bypass writing a lot of 
boilerplate code for checking the units of its inputs--it doesn't remove any 
overhead (in fact it adds it--but this was used for cases where that overhead 
already existed--the code was just highly repetitive).

> Another option might be to only pass plain numpy arrays to the innermost loops of your code with .value,
> of course at the expense of some of the convenience gained in the first place...

Right, that's the big problem.  The thing Quantity supports (which most similar 
libraries in Python don't) is that it *is* a Numpy ndarray, and with few 
exceptions [1] works transparently with all of Numpy's universal functions. 
However, Numpy currently doesn't make it easy to wrap calls to ufuncs, so 
there's some overhead associated with redoing unit conversions on every function 
call (and even if Numpy did make this easier, as it will in 1.10, performing 
unit conversions repeatedly in a loop will add significant overhead to any 
fitting operation, so better to do the unit conversions first so that all 
quantities are expressed in the same base units and orders of magnitude, then as 
Derek suggested pass in the raw arrays (from .value) to high performance code.

Having a mechanism to automate such conversions will be useful I think.

Erik


[1] http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues


From npkuin at gmail.com  Mon Aug 24 20:58:19 2015
From: npkuin at gmail.com (Paul Kuin)
Date: Tue, 25 Aug 2015 01:58:19 +0100
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <55DB9D96.6050008@stsci.edu>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
	<69F6DBCE-C479-40B5-88C4-DE24AE9B6410@astro.physik.uni-goettingen.de>
	<55DB9D96.6050008@stsci.edu>
Message-ID: <CANoQ6N2U=UBi-hQti44wyzrrZqcuBBJeDZ-ufq2HWK2L7Du8oQ@mail.gmail.com>

I'm converting my code to use astropy.units, and decided to use it mainly
at the interface level, and use my own set of fixed units inside the
software units.

That said, it seems to me that it would be useful if you could set some
flag to tell the code that the units are not going to change until you
unset the flag. Then the "y = xx.value" steps are not needed. Or perhaps
this is a really stupid idea because I don't know the ins and outs. .. ;-)

Paul

On Mon, Aug 24, 2015 at 11:41 PM, Erik Bray <embray at stsci.edu> wrote:

> On 08/24/2015 06:32 PM, Derek Homeier wrote:
> > On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com>
> wrote:
> >>
> >> I am writing an MCMC fitting code, and my current implementation uses
> the (very useful) astropy quantities framework to take care of unit
> conversions and such. However, the code is surprisingly slow so I profiled
> it with %prun and it looks like the quantity utilities that wrap numpy
> functions are taking a huge chunk of time. Is that just the price of
> convenience?
> >>
> > I am not an expert on the quantities/units implementation, but I would
> imagine that they could indeed
> > introduce a significant overhead, as the class adds a number of
> validations and checks for a function
> > calling a quantity object. Especially with rather numerous operations on
> comparatively small arrays
> > this could become problematic.
> > Did you experiment with using decorators as described in
> >
> http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
> >
> > - iiuc this would allow bypassing some of the validations?
>
> I don't think this is really relevant to the OP's question (though always
> good
> to know about).  This decorator just makes it easy to bypass writing a lot
> of
> boilerplate code for checking the units of its inputs--it doesn't remove
> any
> overhead (in fact it adds it--but this was used for cases where that
> overhead
> already existed--the code was just highly repetitive).
>
> > Another option might be to only pass plain numpy arrays to the innermost
> loops of your code with .value,
> > of course at the expense of some of the convenience gained in the first
> place...
>
> Right, that's the big problem.  The thing Quantity supports (which most
> similar
> libraries in Python don't) is that it *is* a Numpy ndarray, and with few
> exceptions [1] works transparently with all of Numpy's universal functions.
> However, Numpy currently doesn't make it easy to wrap calls to ufuncs, so
> there's some overhead associated with redoing unit conversions on every
> function
> call (and even if Numpy did make this easier, as it will in 1.10,
> performing
> unit conversions repeatedly in a loop will add significant overhead to any
> fitting operation, so better to do the unit conversions first so that all
> quantities are expressed in the same base units and orders of magnitude,
> then as
> Derek suggested pass in the raw arrays (from .value) to high performance
> code.
>
> Having a mechanism to automate such conversions will be useful I think.
>
> Erik
>
>
> [1]
> http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



-- 

* * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
phone +44-(0)1483 (prefix) -204927 (work)
mobile +44(0)7806985366  skype ID: npkuin
Mullard Space Science Laboratory  ? University College London  ?
Holmbury St Mary ? Dorking ? Surrey RH5 6NT?  U.K.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150825/3195e5cf/attachment.html>

From adam.g.ginsburg at gmail.com  Tue Aug 25 01:25:05 2015
From: adam.g.ginsburg at gmail.com (Adam Ginsburg)
Date: Tue, 25 Aug 2015 07:25:05 +0200
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <CANoQ6N2U=UBi-hQti44wyzrrZqcuBBJeDZ-ufq2HWK2L7Du8oQ@mail.gmail.com>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
	<69F6DBCE-C479-40B5-88C4-DE24AE9B6410@astro.physik.uni-goettingen.de>
	<55DB9D96.6050008@stsci.edu>
	<CANoQ6N2U=UBi-hQti44wyzrrZqcuBBJeDZ-ufq2HWK2L7Du8oQ@mail.gmail.com>
Message-ID: <CAEBNSwaRTPtaxUQsYffTunPQDSr6RGXBJwNY2UVPL-uF06=bgQ@mail.gmail.com>

I've encountered performance problems with units before, and one issue
is that creating new composite units (e.g., u.km/u.s) comes with some
overhead.  If such operations are a significant overhead, you can set
them up in advance, e.g.:

_kms = u.km/u.s

and use that elsewhere in the code.  You can also get a small boost
from using u.Quantity(x, unit) instead of x*unit.

In [1]: %timeit 5*u.km/u.s
The slowest run took 4.81 times longer than the fastest. This could
mean that an intermediate result is being cached
10000 loops, best of 3: 40.6 ?s per loop

In [2]: %timeit u.Quantity(5, u.km/u.s)
The slowest run took 4.38 times longer than the fastest. This could
mean that an intermediate result is being cached
10000 loops, best of 3: 30.1 ?s per loop

In [3]: _kms = u.km/u.s

In [4]: %timeit u.Quantity(5, _kms)
The slowest run took 6.71 times longer than the fastest. This could
mean that an intermediate result is being cached
100000 loops, best of 3: 7.89 ?s per loop

(thanks to Tom Robitaille who pointed all this out to me)

On Tue, Aug 25, 2015 at 2:58 AM, Paul Kuin <npkuin at gmail.com> wrote:
> I'm converting my code to use astropy.units, and decided to use it mainly at
> the interface level, and use my own set of fixed units inside the software
> units.
>
> That said, it seems to me that it would be useful if you could set some flag
> to tell the code that the units are not going to change until you unset the
> flag. Then the "y = xx.value" steps are not needed. Or perhaps this is a
> really stupid idea because I don't know the ins and outs. .. ;-)
>
> Paul
>
> On Mon, Aug 24, 2015 at 11:41 PM, Erik Bray <embray at stsci.edu> wrote:
>>
>> On 08/24/2015 06:32 PM, Derek Homeier wrote:
>> > On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <kevin.gullikson at gmail.com>
>> > wrote:
>> >>
>> >> I am writing an MCMC fitting code, and my current implementation uses
>> >> the (very useful) astropy quantities framework to take care of unit
>> >> conversions and such. However, the code is surprisingly slow so I profiled
>> >> it with %prun and it looks like the quantity utilities that wrap numpy
>> >> functions are taking a huge chunk of time. Is that just the price of
>> >> convenience?
>> >>
>> > I am not an expert on the quantities/units implementation, but I would
>> > imagine that they could indeed
>> > introduce a significant overhead, as the class adds a number of
>> > validations and checks for a function
>> > calling a quantity object. Especially with rather numerous operations on
>> > comparatively small arrays
>> > this could become problematic.
>> > Did you experiment with using decorators as described in
>> >
>> > http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
>> >
>> > - iiuc this would allow bypassing some of the validations?
>>
>> I don't think this is really relevant to the OP's question (though always
>> good
>> to know about).  This decorator just makes it easy to bypass writing a lot
>> of
>> boilerplate code for checking the units of its inputs--it doesn't remove
>> any
>> overhead (in fact it adds it--but this was used for cases where that
>> overhead
>> already existed--the code was just highly repetitive).
>>
>> > Another option might be to only pass plain numpy arrays to the innermost
>> > loops of your code with .value,
>> > of course at the expense of some of the convenience gained in the first
>> > place...
>>
>> Right, that's the big problem.  The thing Quantity supports (which most
>> similar
>> libraries in Python don't) is that it *is* a Numpy ndarray, and with few
>> exceptions [1] works transparently with all of Numpy's universal
>> functions.
>> However, Numpy currently doesn't make it easy to wrap calls to ufuncs, so
>> there's some overhead associated with redoing unit conversions on every
>> function
>> call (and even if Numpy did make this easier, as it will in 1.10,
>> performing
>> unit conversions repeatedly in a loop will add significant overhead to any
>> fitting operation, so better to do the unit conversions first so that all
>> quantities are expressed in the same base units and orders of magnitude,
>> then as
>> Derek suggested pass in the raw arrays (from .value) to high performance
>> code.
>>
>> Having a mechanism to automate such conversions will be useful I think.
>>
>> Erik
>>
>>
>> [1]
>> http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
>
> --
>
> * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
> Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
> phone +44-(0)1483 (prefix) -204927 (work)
> mobile +44(0)7806985366  skype ID: npkuin
> Mullard Space Science Laboratory  ? University College London  ?
> Holmbury St Mary ? Dorking ? Surrey RH5 6NT?  U.K.
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



-- 
Adam Ginsburg
Fellow, European Southern Observatory
http://www.adamgginsburg.com/


From eric.emsellem at eso.org  Tue Aug 25 07:43:14 2015
From: eric.emsellem at eso.org (Eric Emsellem)
Date: Tue, 25 Aug 2015 13:43:14 +0200
Subject: [AstroPy] Composite models or input arrays
Message-ID: <55DC54D2.3020208@eso.org>

Hi

I just realised something odd (but nice!) about astropy.modeling.

If I have a model, e.g., with 3 parameters (p1, p2, p3), I can do two 
things to get some sort of composite model :

Option 1
 >>> M1 = mymodel(p1=1,p2=1,p3=0)
 >>> M2 = mymodel(p1=2,p2=2,p3=0)

 >>> Mtotal1 = M1 + M2

or

Option 2
 >>> Mtotal2 = mymodel(p1=[1,2],p2=[1,2], p3=[0,0])

In the first case, I'll have parameters set up one block after the other 
(parameters = [1, 1, 0, 2, 2, 0]) while in the second example, I'll have 
them as I did write them: parameters = [1, 2, 1, 2, 0, 0].
In the first case, if I evaluate the model it gives the sum as written 
in Mtotal1(x) = (M1+M2)(x), while in the second case, I get [M1(x), 
M2(x)], so the broadcasting is different indeed.

Both ways are useful. On one hand, Option 1 is the natural way to work 
out a composite model. But Option 2 is using an internal broadcasting I 
guess, hence it is just a matter of providing the right input arrays of 
parameters, and you can always do the sum after the fact by doing e.g., 
Mtotal(x).sum() for example.

However, Mtotal2 does not seem to act on arrays (again broadcasting 
issues?), so I cannot do Mtotal2(x) where x is an array, while it seems 
I can do that with Mtotal1 and it will return the right output.

Is that correct, or am I on the wrong track here?

thanks for any tips :-)

Eric


From kevin.gullikson at gmail.com  Tue Aug 25 10:57:36 2015
From: kevin.gullikson at gmail.com (Kevin Gullikson)
Date: Tue, 25 Aug 2015 09:57:36 -0500
Subject: [AstroPy] Using quantities is slow
In-Reply-To: <CAEBNSwaRTPtaxUQsYffTunPQDSr6RGXBJwNY2UVPL-uF06=bgQ@mail.gmail.com>
References: <CAEUX0Xe6tsbScqucQC14Omr+1ObkNYYowxsvXYJng6tw+h4cEQ@mail.gmail.com>
	<69F6DBCE-C479-40B5-88C4-DE24AE9B6410@astro.physik.uni-goettingen.de>
	<55DB9D96.6050008@stsci.edu>
	<CANoQ6N2U=UBi-hQti44wyzrrZqcuBBJeDZ-ufq2HWK2L7Du8oQ@mail.gmail.com>
	<CAEBNSwaRTPtaxUQsYffTunPQDSr6RGXBJwNY2UVPL-uF06=bgQ@mail.gmail.com>
Message-ID: <CAEUX0Xd4C=m3LxXZDQN6-ZmNiuE6hLaiOXEev6jS-8OVrJaNEA@mail.gmail.com>

Thanks for the suggestions everyone! I ended up using units for the
interface code that gets called once, and using plain numpy arrays for the
likelihood calculations. That gave me a ~4x speedup!


Kevin Gullikson
PhD Candidate
University of Texas Astronomy
RLM 15.310E

On Tue, Aug 25, 2015 at 12:25 AM, Adam Ginsburg <adam.g.ginsburg at gmail.com>
wrote:

> I've encountered performance problems with units before, and one issue
> is that creating new composite units (e.g., u.km/u.s) comes with some
> overhead.  If such operations are a significant overhead, you can set
> them up in advance, e.g.:
>
> _kms = u.km/u.s
>
> and use that elsewhere in the code.  You can also get a small boost
> from using u.Quantity(x, unit) instead of x*unit.
>
> In [1]: %timeit 5*u.km/u.s
> The slowest run took 4.81 times longer than the fastest. This could
> mean that an intermediate result is being cached
> 10000 loops, best of 3: 40.6 ?s per loop
>
> In [2]: %timeit u.Quantity(5, u.km/u.s)
> The slowest run took 4.38 times longer than the fastest. This could
> mean that an intermediate result is being cached
> 10000 loops, best of 3: 30.1 ?s per loop
>
> In [3]: _kms = u.km/u.s
>
> In [4]: %timeit u.Quantity(5, _kms)
> The slowest run took 6.71 times longer than the fastest. This could
> mean that an intermediate result is being cached
> 100000 loops, best of 3: 7.89 ?s per loop
>
> (thanks to Tom Robitaille who pointed all this out to me)
>
> On Tue, Aug 25, 2015 at 2:58 AM, Paul Kuin <npkuin at gmail.com> wrote:
> > I'm converting my code to use astropy.units, and decided to use it
> mainly at
> > the interface level, and use my own set of fixed units inside the
> software
> > units.
> >
> > That said, it seems to me that it would be useful if you could set some
> flag
> > to tell the code that the units are not going to change until you unset
> the
> > flag. Then the "y = xx.value" steps are not needed. Or perhaps this is a
> > really stupid idea because I don't know the ins and outs. .. ;-)
> >
> > Paul
> >
> > On Mon, Aug 24, 2015 at 11:41 PM, Erik Bray <embray at stsci.edu> wrote:
> >>
> >> On 08/24/2015 06:32 PM, Derek Homeier wrote:
> >> > On 24 Aug 2015, at 3:57 pm, Kevin Gullikson <
> kevin.gullikson at gmail.com>
> >> > wrote:
> >> >>
> >> >> I am writing an MCMC fitting code, and my current implementation uses
> >> >> the (very useful) astropy quantities framework to take care of unit
> >> >> conversions and such. However, the code is surprisingly slow so I
> profiled
> >> >> it with %prun and it looks like the quantity utilities that wrap
> numpy
> >> >> functions are taking a huge chunk of time. Is that just the price of
> >> >> convenience?
> >> >>
> >> > I am not an expert on the quantities/units implementation, but I would
> >> > imagine that they could indeed
> >> > introduce a significant overhead, as the class adds a number of
> >> > validations and checks for a function
> >> > calling a quantity object. Especially with rather numerous operations
> on
> >> > comparatively small arrays
> >> > this could become problematic.
> >> > Did you experiment with using decorators as described in
> >> >
> >> >
> http://astropy.readthedocs.org/en/stable/units/quantity.html#functions-accepting-quantities
> >> >
> >> > - iiuc this would allow bypassing some of the validations?
> >>
> >> I don't think this is really relevant to the OP's question (though
> always
> >> good
> >> to know about).  This decorator just makes it easy to bypass writing a
> lot
> >> of
> >> boilerplate code for checking the units of its inputs--it doesn't remove
> >> any
> >> overhead (in fact it adds it--but this was used for cases where that
> >> overhead
> >> already existed--the code was just highly repetitive).
> >>
> >> > Another option might be to only pass plain numpy arrays to the
> innermost
> >> > loops of your code with .value,
> >> > of course at the expense of some of the convenience gained in the
> first
> >> > place...
> >>
> >> Right, that's the big problem.  The thing Quantity supports (which most
> >> similar
> >> libraries in Python don't) is that it *is* a Numpy ndarray, and with few
> >> exceptions [1] works transparently with all of Numpy's universal
> >> functions.
> >> However, Numpy currently doesn't make it easy to wrap calls to ufuncs,
> so
> >> there's some overhead associated with redoing unit conversions on every
> >> function
> >> call (and even if Numpy did make this easier, as it will in 1.10,
> >> performing
> >> unit conversions repeatedly in a loop will add significant overhead to
> any
> >> fitting operation, so better to do the unit conversions first so that
> all
> >> quantities are expressed in the same base units and orders of magnitude,
> >> then as
> >> Derek suggested pass in the raw arrays (from .value) to high performance
> >> code.
> >>
> >> Having a mechanism to automate such conversions will be useful I think.
> >>
> >> Erik
> >>
> >>
> >> [1]
> >>
> http://astropy.readthedocs.org/en/stable/known_issues.html#quantity-issues
> >> _______________________________________________
> >> AstroPy mailing list
> >> AstroPy at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/astropy
> >
> >
> >
> >
> > --
> >
> > * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
> > Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
> > phone +44-(0)1483 (prefix) -204927 (work)
> > mobile +44(0)7806985366  skype ID: npkuin
> > Mullard Space Science Laboratory  ? University College London  ?
> > Holmbury St Mary ? Dorking ? Surrey RH5 6NT?  U.K.
> >
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
> >
>
>
>
> --
> Adam Ginsburg
> Fellow, European Southern Observatory
> http://www.adamgginsburg.com/
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150825/cbeb0a89/attachment.html>

From embray at stsci.edu  Tue Aug 25 11:17:21 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 25 Aug 2015 11:17:21 -0400
Subject: [AstroPy] Composite models or input arrays
In-Reply-To: <55DC54D2.3020208@eso.org>
References: <55DC54D2.3020208@eso.org>
Message-ID: <55DC8701.4040109@stsci.edu>

On 08/25/2015 07:43 AM, Eric Emsellem wrote:
> Hi
>
> I just realised something odd (but nice!) about astropy.modeling.
>
> If I have a model, e.g., with 3 parameters (p1, p2, p3), I can do two
> things to get some sort of composite model :
>
> Option 1
>   >>> M1 = mymodel(p1=1,p2=1,p3=0)
>   >>> M2 = mymodel(p1=2,p2=2,p3=0)
>
>   >>> Mtotal1 = M1 + M2
>
> or
>
> Option 2
>   >>> Mtotal2 = mymodel(p1=[1,2],p2=[1,2], p3=[0,0])

Or Option 3:

MySummedModel = MyModel + MyModel

model_instance = MySummedModel(1, 1, 0, 2, 2, 0)

> In the first case, I'll have parameters set up one block after the other
> (parameters = [1, 1, 0, 2, 2, 0]) while in the second example, I'll have
> them as I did write them: parameters = [1, 2, 1, 2, 0, 0].
> In the first case, if I evaluate the model it gives the sum as written
> in Mtotal1(x) = (M1+M2)(x), while in the second case, I get [M1(x),
> M2(x)], so the broadcasting is different indeed.

That's not really the same as what I would call "broadcasting".  In the latter 
case you're treating the parameters as vectors.  This "works", but how this is 
defined, if at all, is rather dependent on the model.

> Both ways are useful. On one hand, Option 1 is the natural way to work
> out a composite model. But Option 2 is using an internal broadcasting I
> guess, hence it is just a matter of providing the right input arrays of
> parameters, and you can always do the sum after the fact by doing e.g.,
> Mtotal(x).sum() for example.

Indeed you could do something like that.  But if you want to treat a single 
model instance as a *collection* of models (of which you would then sum the 
results) there's a subtly different thing you should do.  Read on....

> However, Mtotal2 does not seem to act on arrays (again broadcasting
> issues?), so I cannot do Mtotal2(x) where x is an array, while it seems
> I can do that with Mtotal1 and it will return the right output.
>
> Is that correct, or am I on the wrong track here?

What you want is to use a model set.  This is a still poorly understood feature 
and I'm open to suggestions for how to make this more "obvious" (part of the 
difficulty in that has been supporting a multitude of different use cases 
simultaneously).  You can read about that here:

http://docs.astropy.org/en/v1.0.4/modeling/models.html

In short, a *model set* is what you want if you want to represent a collection 
of independent instances of the same model with different parameters.

When you do:

     >>> Mtotal2 = mymodel(p1=[1,2],p2=[1,2], p3=[0,0])

What you're doing is making a model where the parameters have 2D vector values. 
  Which is fine, and will work in many cases.  But the components of these 
parameters are not necessarily linearly independent, depending on how the 
particular model is defined.

On the other hand, if you specify the n_models=2 argument like:

     >>> Mtotal2 = mymodel(p1=[1, 2], p2=[1, 2], p3=[0, 0], n_models=2)

This is like an array of models of whatever type "mymodel" is, with parameters 
(1, 1, 0) and (2, 2, 0).  In other words it's more like having a list of models 
like [mymodel(1, 1, 0), mymodel(2, 2, 0)].  When these are evaluated on an input 
the broadcasting is such that they are evaluated independently (but still 
generally more efficiently than if you made a list of models and evaluated them 
one by one--especially when you're scaling up to hundreds or thousands).

One important distinction to make though, is that if you have a model set, when 
you pass in an array-like argument the default assumption is that the first axis 
of that array corresponds to the number of models in the set, as though you are 
passing in a separate input to each model.  That is, the size of the first array 
axis is assumed to be the same as the number of models in the set.  So for 
example if I have:

 >>> m = Gaussian1D([1, 2], [0, 0], [0.1, 0.1], n_models=2)

and try to evaluate it on an array of shape (3,):

 >>> m([1, 2, 3])
ERROR: ValueError: Input argument 'x' does not have the correct dimensions in 
model_set_axis=0 for a model set with n_models=2. [astropy.modeling.core]

However, as explained in the docs, the way to say "no, I really just want to 
evaluate each model in the set on the same 3 element array" is:

 >>> m([.1, .2, .3], model_set_axis=False)
array([[ 0.60653066,  0.13533528,  0.011109  ],
        [ 1.21306132,  0.27067057,  0.02221799]])

There you see that you get a result of shape (2, 3).  The rows (2) correspond to 
each model in the set, while the columns correspond to the evaluation of that 
model on each of the inputs 0.1, 0.2, and 0.3.

If it helps clarify the difference, try also using a shape (2,) array.  This 
will work with or without model_set_axis=False, but with different results in 
each case.  Without you get:

 >>> m([.1, .2])
array([ 0.60653066,  0.27067057])

This is equivalent to passing 0.1 to Gaussian1D(1, 0, 0.1) and 0.2 to 
Gaussian1D(2, 0, 0.1).

On the other hand:

 >>> m([.1, .2], model_set_axis=False)
array([[ 0.60653066,  0.13533528],
        [ 1.21306132,  0.27067057]])

This passes the input [0.1, 0.2] to Gaussian1D(1, 0, 0.1), and the same to 
Gaussian1D(2, 0, 0.1).

For your summation example this latter usecase is probably what you want. 
Although I do hope to implement some tricks to speed up compound models where 
only a single operator is used (i.e. a reduction operation like sum or product), 
for now you'll get the best performance out of a model set, and then performing 
a reduction along the desired axis:

 >>> mymodel(p1=p1s, p2=p2s, p3=p3s, n_models=n_sources)
 >>> the_summed_sources = mymodel(input, 
model_set_axis=False).sum(axis=mymodel.model_set_axis)

(where above mymodel.model_set_axis is just 0 by default)

Erik


From aldcroft at head.cfa.harvard.edu  Tue Aug 25 13:07:42 2015
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Tue, 25 Aug 2015 13:07:42 -0400
Subject: [AstroPy] Changing dtype when copying masked table throws
	TypeError
In-Reply-To: <CAMtEP6xsVZLCM75CfboVyCtrx6jdStaNnsoesGicNUYOyTQPFA@mail.gmail.com>
References: <55809E3E.6000602@buckeyemail.osu.edu>
	<CAMtEP6xsVZLCM75CfboVyCtrx6jdStaNnsoesGicNUYOyTQPFA@mail.gmail.com>
Message-ID: <CAMtEP6zAiA3Rusj6zPHND-fu-1MJUD76W0h9WttUCgpo8PcLrA@mail.gmail.com>

On Fri, Aug 21, 2015 at 10:41 AM, Aldcroft, Thomas <
aldcroft at head.cfa.harvard.edu> wrote:

>
>
> On Tue, Jun 16, 2015 at 6:07 PM, Gregory Simonian <
> simonian.7 at buckeyemail.osu.edu> wrote:
>
>> Hi all,
>>
>> I have a Table with several columns which were read in as strings because
>> they had a combination of numeric and non-numeric characters. After
>> cleaning out the non-numeric characters, I'd like to convert the dtype of
>> the column from string to int.
>>
>> Looking at the astropy documentation for modifying tables [0], under the
>> caveats section it states:
>>
>> "The data type for a column cannot be changed in place. In order to do
>> this you must make a copy of the table with the column type changed
>> appropriately."
>>
>> So I try making a new table (the table is small, so I don't mind the
>> temporary memory hit), but I run into the error:
>> "TypeError: Cannot set fill value of string with array of dtype int64"
>> When checking the fill value of the column (which is masked from the
>> reader), it's 'N/A', which makes sense that it wouldn't work. But when I
>> change it to '0', I still get the same problem. I managed to find a minimal
>> working example as:
>>
>> import numpy as np
>> t = Table({'a': ['1']}, masked=True)
>> t['a'].set_fill_value('0')
>> u = Table(t, names=['a'], dtype=[np.int])
>> > TypeError: Cannot set fill value of string with array of dtype int64
>>
>> When debugging the exception, it seems like the Table constructor doesn't
>> convert the fill value of masked tables when generating a new table, which
>> seems to defeat the ability to change the dtype of masked columns. I was
>> wondering if other people have run by this, and if there's a more
>> straightforward way of changing the dtype of a column. This is with Astropy
>> 1.0.3 on Python 2.7.10 through Anaconda on CentOS 6.6.
>>
>> When checking to see if this was reported, I ran into issue #3809 [1],
>> which might end up being the ultimate solution to my problem. It seems like
>> a workaround would be:
>>
>> temp_column = np.ma.asanyarray(t['a'], np.int)
>> del(t['a'])
>> t['a'] = temp_column
>>
>>
> Hi Gregory,
>
> Sorry, your nice bug report fell through the cracks and didn't get a
> response.  You've basically understood the issues correctly, and I just put
> in a pull request that fixes this:
>
> https://github.com/astropy/astropy/pull/4088/files
>


Just to close out this thread, #4088 has been merged.


>
>
> In your workaround you could have done something slightly simpler:
>
>  temp_column = t['a'].astype(np.int)
>
> Cheers,
> Tom
>
>
>
>
>> Thanks!
>>
>>                                             Gregory Simonian
>>
>> [0] http://docs.astropy.org/en/stable/table/modify_table.html#caveats
>> [1] https://github.com/astropy/astropy/issues/3809
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150825/8031ec75/attachment.html>

From eric.emsellem at eso.org  Wed Aug 26 03:19:48 2015
From: eric.emsellem at eso.org (Eric Emsellem)
Date: Wed, 26 Aug 2015 09:19:48 +0200
Subject: [AstroPy] Composite models or input arrays
Message-ID: <55DD6894.2090603@eso.org>

> What you want is to use a model set.  This is a still poorly understood feature
> and I'm open to suggestions for how to make this more "obvious" (part of the
> difficulty in that has been supporting a multitude of different use cases
> simultaneously).  You can read about that here:
>
> http://docs.astropy.org/en/v1.0.4/modeling/models.html
>
> In short, a*model set*  is what you want if you want to represent a collection
> of independent instances of the same model with different parameters.
wahh... quite powerful. Sorry to have missed that part of the 
documentation. This is really great and solves quite a number of issues 
I had in mind.

At the end, I need to choose between (the examples below would need 
extra lines/variables, but this is I hope explicit enough):

1- for i in range(N) : ModelT += MyModel(par1= , par2= , par3= )

(each Model being the same MyModel but with different instances provided 
as going through the for loop)

(I could also use model_instance = ModelT(....) to set the parameters 
after the fact as you suggested)

or

2- ModelT = MyModel(par1=[...], par2=[...], par3=[...], n_models=N)

The second option (model set) seems quite natural. It also has the 
advantages to have both the "parameters" as an array including all 
parameters (2d array), and each individual parameter (par1, par2, par3) 
as an individual array (I checked that 'parameters' and e.g., 'par1' are 
views of each others so when I modify one the other one is also 
modified, via a setter/view I guess -> great!).

=> However, I don't know how to slice the model set then (e.g., compute 
ModelT(x) just using e.g., the 1st and 3rd model of that set). By the 
way, with a compound model, I guess I cannot do ModelT[[0,2]] to access 
only model 1 and 3.

=> More generally, and as mentioned in a previous email, I would also 
like to be able to add more instances of MyModel, or remove some of them.

With Option 1, I can just do ModelT = ModelT + MyModel(par1=, par2=, 
par3=) to add one

or

ModelT = ModelT[0] + ModelT[2] to remove one (as you suggested)

Could you do that easily with a model set? (1- compute the output on a 
slice of the models, a selected subset, and later on add or even remove 
completely one of the models in the model set?).

Anyways, this is quite amazing. I was a bit dubious when I started to 
examine the possibility to use the astropy.modeling. I am more than 
convinced now and this is going well beyond what I had imagined (and I 
haven't seen everything yet by far). Thanks so much for these developments!

Eric

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150826/67529627/attachment.html>

From embray at stsci.edu  Wed Aug 26 11:48:52 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 26 Aug 2015 11:48:52 -0400
Subject: [AstroPy] Composite models or input arrays
In-Reply-To: <55DD6894.2090603@eso.org>
References: <55DD6894.2090603@eso.org>
Message-ID: <55DDDFE4.8080605@stsci.edu>

On 08/26/2015 03:19 AM, Eric Emsellem wrote:
> => However, I don't know how to slice the model set then (e.g., compute
> ModelT(x) just using e.g., the 1st and 3rd model of that set). By the way, with
> a compound model, I guess I cannot do ModelT[[0,2]] to access only model 1 and 3.

Currently not possible--you'd have to slice the parameter arrays themselves and 
just create a new model instance with the sliced parameters.

You've hit on exactly what I *want* to be able to do though.  Just as the 
Astropy Time object [1] can represent an array of times (of the same format and 
scale) and SkyCoord can represent an array of coordinates in the same frame, a 
Model object (with n_models > 1) can be thought of an array of models of the 
same type just with different parameters.

This feature just wasn't conceived of in exact analogy to those (though in 
retrospect it's a good way to think of it), so it isn't quite possible yet to 
fully manipulate a model set like an array of models.  But I would very much 
*like* to have that working sooner or later.

The main trickiness is that indexing of model objects is already used by 
compound models (and only by compound models) to index submodels / subexpressions.

What I might do is just have multiple indices--so that when using a model set 
the first index indexes models in the set, and the second index (if it's a 
compound model) indexes submodels.  For single "scalar(?)" models, the first 
index would always just index submodels of compound models.  That's what I'm 
thinking anyways, but I haven't looked into what the impact is on existing code.

> => More generally, and as mentioned in a previous email, I would also like to be
> able to add more instances of MyModel, or remove some of them.
>
> With Option 1, I can just do ModelT = ModelT + MyModel(par1=, par2=, par3=) to
> add one
>
> or
>
> ModelT = ModelT[0] + ModelT[2] to remove one (as you suggested)
>
> Could you do that easily with a model set? (1- compute the output on a slice of
> the models, a selected subset, and later on add or even remove completely one of
> the models in the model set?).

Ah, I see.  That would be a good use case.  If I can get array-like operations 
working better on model sets then you'd be able to slice/stack them.  You should 
open a ticket for this on GitHub if you think of it.

> Anyways, this is quite amazing. I was a bit dubious when I started to examine
> the possibility to use the astropy.modeling. I am more than convinced now and
> this is going well beyond what I had imagined (and I haven't seen everything yet
> by far). Thanks so much for these developments!

Thanks!  I hope it will be useful to you.

Erik

[1] http://docs.astropy.org/en/stable/time/index.html#getting-started
[2] http://docs.astropy.org/en/stable/coordinates/index.html#getting-started




From mcbeth at broggs.org  Fri Aug 28 08:13:48 2015
From: mcbeth at broggs.org (Jeffrey Brent McBeth)
Date: Fri, 28 Aug 2015 08:13:48 -0400
Subject: [AstroPy] Confused by coordinate transforms ICRS/ITRS/GCRS
Message-ID: <20150828121348.GH5880@broggs.org>

I've been working on a personal project figuring out where stars are from the point of view of the ISS (after reading about the difficulty of the transform in a paper on the Hubble).

So, I have a catalog in ICRS, and location of the ISS in ITRS.  All fine and good, now I want calculate orientations, and take into account some of the weirdness like parallax and velocity aberration.

I have some code below that shows some of the things I've tried and the various ways I see things going "wrong".  I'm perfectly willing to admit the wrong is more likely to be me, but in that case I would like to be educated.

Thanks for your time and patience,
 Jeff

#+name: confusingTransforms
#+begin_src: python
import astropy
import astropy.coordinates
import astropy.time
import astropy.units as u

# For now, lets just talk about the center of the earth
# You may complain "this isn't in the sky", but I get similar results
# even when I'm well away from the earth, this keeps things simple, and since
# I'm ultimately trying for orientation transforms, doesn't matter
earth = astropy.coordinates.SkyCoord(x=0,y=0,z=0,frame='itrs',unit=u.m)
# And a spot one meter away
# This goes wrong with larger distances too, so it isn't simply precision
almostEarth = astropy.coordinates.SkyCoord(x=0,y=0,z=1,frame='itrs',unit=u.m)

# I would expect _most_ prints below to be 1 meter
print 'ITRS/ITRS:',earth.separation_3d(almostEarth)
print 'ICRS/ITRS:',earth.icrs.separation_3d(almostEarth)
print 'ICRS/ICRS:',earth.icrs.separation_3d(almostEarth.icrs)

print 'GCRS/ITRS:',earth.gcrs.separation_3d(almostEarth)
print 'GCRS/GCRS:',earth.gcrs.separation_3d(almostEarth.gcrs)

now = astropy.time.Time.now()
later = now+1*u.second

# A notional ISS, I have the real numbers, but this keeps it simple
platform = astropy.coordinates.GCRS(obsgeoloc=[0,0,400]*u.km,obsgeovel=[0,0,7.5]*u.km/u.s,obstime=now)

platformLater = astropy.coordinates.GCRS(obsgeoloc=[0,0,400]*u.km,obsgeovel=[0,0,7.5]*u.km/u.s,obstime=later)

print 'GCRS(Now  )/GCRS(Now  ):',earth.transform_to(platform).separation_3d(almostEarth.transform_to(platform))
print 'GCRS(Later)/GCRS(Later):',earth.transform_to(platformLater).separation_3d(almostEarth.transform_to(platformLater))

# These two, due to earth motion should probably be around 30km
print 'GCRS(Now  )/GCRS(Later):',earth.transform_to(platform).separation_3d(almostEarth.transform_to(platformLater))
print 'GCRS(Later)/GCRS(Now  ):',earth.transform_to(platformLater).separation_3d(almostEarth.transform_to(platform))
#+end_src
-- 
"The man who does not read good books has no advantage over 
 the man who cannot read them."
 -- Mark Twain


From brinkmann at mpia.de  Mon Aug 31 09:36:09 2015
From: brinkmann at mpia.de (Steffen Brinkmann)
Date: Mon, 31 Aug 2015 15:36:09 +0200
Subject: [AstroPy] Find record(s) with specific key in fits table
Message-ID: <55E45849.4090805@mpia.de>

Hello,

Currently, after a tedious hour of trying different things, I do the
following to extract records with a specific key from a fits table. In
this case the key name is 'OBJECT' and, say, I am looking for '18_Sco':

$ from astropy.io import fits
$ fits_data = fits.getdata("file.fits")
$ fits_data[list(fits_data['OBJECT']).index('18_Sco')]

The disadvantage is that only the first record with that 'OBJECT' key is
found.

This:
$ fits_data[list(fits_data['OBJECT']) == '18_Sco']

does not work. The output is a different record with another 'OBJECT'
name. No idea why...

I can use a list comprehension of the python built-in filter() but that
lasts minutes. The file has >10? records.

There must be an elegant way to do this right! Right?

Thank you, cheers,

Steffen


-- 
Steffen Brinkmann
Max-Planck-Institut f?r Astronomie        tel: +49(0)6221-528-458
K?nigstuhl 17                             fax: +49(0)6221-528-246
69117 Heidelberg                          email: brinkmann at mpia.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0xD969459A.asc
Type: application/pgp-keys
Size: 1727 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/astropy/attachments/20150831/a47c8cb5/attachment.key>

From hodge at stsci.edu  Mon Aug 31 09:43:11 2015
From: hodge at stsci.edu (Phil Hodge)
Date: Mon, 31 Aug 2015 09:43:11 -0400
Subject: [AstroPy] Find record(s) with specific key in fits table
In-Reply-To: <55E45849.4090805@mpia.de>
References: <55E45849.4090805@mpia.de>
Message-ID: <55E459EF.6020606@stsci.edu>

On 08/31/2015 09:36 AM, Steffen Brinkmann wrote:
> Currently, after a tedious hour of trying different things, I do the
> following to extract records with a specific key from a fits table. In
> this case the key name is 'OBJECT' and, say, I am looking for '18_Sco':
>
> $ from astropy.io import fits
> $ fits_data = fits.getdata("file.fits")
> $ fits_data[list(fits_data['OBJECT']).index('18_Sco')]

You could try this:

column = fits_data.field("OBJECT")
selected = (column == '18_Sco')
selected_data = fits_data[selected]

Phil


From brinkmann at mpia.de  Mon Aug 31 09:52:20 2015
From: brinkmann at mpia.de (Steffen Brinkmann)
Date: Mon, 31 Aug 2015 15:52:20 +0200
Subject: [AstroPy] Find record(s) with specific key in fits table
In-Reply-To: <55E459EF.6020606@stsci.edu>
References: <55E45849.4090805@mpia.de> <55E459EF.6020606@stsci.edu>
Message-ID: <55E45C14.7080007@mpia.de>

Hello Phil,

that did the trick. Thank you.

I wrote a function to encapsulate this task:

def select_by_key(fits_data, key, val):
    column = fits_data.field(key)
    selected = (column == val)
    return fits_data[selected]

selected_data = select_by_key(fits_data, 'OBJECT', 18_Sco')


Cheers, Steffen


On 31.08.2015 15:43, Phil Hodge wrote:
> On 08/31/2015 09:36 AM, Steffen Brinkmann wrote:
>> Currently, after a tedious hour of trying different things, I do the
>> following to extract records with a specific key from a fits table. In
>> this case the key name is 'OBJECT' and, say, I am looking for '18_Sco':
>>
>> $ from astropy.io import fits
>> $ fits_data = fits.getdata("file.fits")
>> $ fits_data[list(fits_data['OBJECT']).index('18_Sco')]
> 
> You could try this:
> 
> column = fits_data.field("OBJECT")
> selected = (column == '18_Sco')
> selected_data = fits_data[selected]
> 
> Phil
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
> 


-- 
Steffen Brinkmann
Max-Planck-Institut f?r Astronomie        tel: +49(0)6221-528-458
K?nigstuhl 17                             fax: +49(0)6221-528-246
69117 Heidelberg                          email: brinkmann at mpia.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0xD969459A.asc
Type: application/pgp-keys
Size: 1727 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/astropy/attachments/20150831/99a3aa70/attachment.key>

From embray at stsci.edu  Mon Aug 31 10:28:31 2015
From: embray at stsci.edu (Erik Bray)
Date: Mon, 31 Aug 2015 10:28:31 -0400
Subject: [AstroPy] Find record(s) with specific key in fits table
In-Reply-To: <55E45849.4090805@mpia.de>
References: <55E45849.4090805@mpia.de>
Message-ID: <55E4648F.8050406@stsci.edu>

On 08/31/2015 09:36 AM, Steffen Brinkmann wrote:
> Hello,
>
> Currently, after a tedious hour of trying different things, I do the
> following to extract records with a specific key from a fits table. In
> this case the key name is 'OBJECT' and, say, I am looking for '18_Sco':
>
> $ from astropy.io import fits
> $ fits_data = fits.getdata("file.fits")
> $ fits_data[list(fits_data['OBJECT']).index('18_Sco')]
>
> The disadvantage is that only the first record with that 'OBJECT' key is
> found.
>
> This:
> $ fits_data[list(fits_data['OBJECT']) == '18_Sco']
>
> does not work. The output is a different record with another 'OBJECT'
> name. No idea why...

Because you're comparing a list to a string.  This is the same as if you tried:

 >>> ['a', 'b', 'c'] == 'd'
False

Python lists compare to other objects to by comparing element-wise.  They'll 
only compare true to something if that something is another list containing the 
same values in the same order.

In Python True and False are equivalent to the integers 1 and 0, so the line of 
code you quoted is always equivalent to:

 >>> fits_data[0]

Since the comparison will always return False.

Phil's response is the right approach.

FITS data returned by astropy.io.fits is in a Numpy array (for the most part, 
with some caveats, though none that apply here).  If you'll be working with 
Python extensively you'll save yourself a lot of trouble in the future by 
getting comfortable with Numpy (much of which is inspired by matlab, if that helps).

The Numpy documentation is found here: 
http://docs.scipy.org/doc/numpy/user/index.html

I have an introductory tutorial to Numpy that I teach, which can be read here:
https://github.com/embray/notebooks/blob/master/numpy.ipynb

You should be able to find other people at MPIA who are familiar with Numpy as well.

Best,
Erik