From long at stsci.edu  Fri Jan  2 13:48:52 2015
From: long at stsci.edu (Knox Long)
Date: Fri, 2 Jan 2015 18:48:52 +0000
Subject: [AstroPy] Removing rows from tables
Message-ID: <D0CC442F.16552%long@stsci.edu>

Is there a simple way to remove rows from an ascii table based on some condition without first converting table to a grouped_table.

I guess if if could determine how to create a slice based on a condition this would be possible.

Thanks,
Knox
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150102/7c8cf393/attachment.html>

From aldcroft at head.cfa.harvard.edu  Fri Jan  2 15:03:01 2015
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Fri, 2 Jan 2015 15:03:01 -0500
Subject: [AstroPy] Removing rows from tables
In-Reply-To: <D0CC442F.16552%long@stsci.edu>
References: <D0CC442F.16552%long@stsci.edu>
Message-ID: <CAMtEP6yDJ_TuV6Ye6fGu+dM3WfGdENVdOfg3VdBSbnSv3spKgg@mail.gmail.com>

On Fri, Jan 2, 2015 at 1:48 PM, Knox Long <long at stsci.edu> wrote:

>  Is there a simple way to remove rows from an ascii table based on some
> condition without first converting table to a grouped_table.
>
>  I guess if if could determine how to create a slice based on a condition
> this would be possible.
>

You can use the `Table.remove_rows()` method (
http://astropy.readthedocs.org/en/stable/api/astropy.table.Table.html#astropy.table.Table.remove_rows
).

This will accept a slice or a list of row indexes.  If you have a boolean
mask then you can simply use that, e.g.:

>>> t = Table(....)
>>> remove_mask = t['a'] > 10  # for example
>>> t_new = t[~remove_mask]

Cheers,
Tom


>
>  Thanks,
> Knox
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150102/12db8ecb/attachment.html>

From simonian.7 at buckeyemail.osu.edu  Tue Jan  6 22:35:54 2015
From: simonian.7 at buckeyemail.osu.edu (Gregory Simonian)
Date: Tue, 6 Jan 2015 22:35:54 -0500
Subject: [AstroPy] Abbreviated data types in IPAC tables
Message-ID: <54ACA99A.8030107@buckeyemail.osu.edu>

Hi all,

I'm trying to read in an IPAC table from the WISE IRSA catalog using
Astropy v4.1, which for some columns abbreviate "double" as "doubl". At
first I thought this was a bug in the database, but I was informed that
abbreviations off the end are actually a legal part of the standard.[0]

However, whenever I try to read it using astropy, I get the error:

In [2]: Table.read("test_output.tbl", format="ascii.ipac", guess=False)
<snip>
ValueError: Unknown data type ""doubl"" for column "w1ba"

Is this a part of the standard that hasn't been implemented yet in
astropy? Or is there some setting that I haven't found that will enable
this to work? Or am I just doing something wrong? I've attached a small
output file that reproduces this error for anyone that's curious. Thanks!

                                   Gregory Simonian

[0] http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html
-------------- next part --------------
\
\fixlen = T
\primary    = 0
\RowsRetrieved =                 1
\QueryTime     =  00:00:00.00199
\ORIGIN  = 'IPAC Infrared Science Archive (IRSA), Caltech/JPL'
\DATETIME= '2015-01-06 19:15:44'
\DATABASE= 'AllWISE Source Catalog (wise_allwise_p3as_psd)'
\EQUINOX = 'J2000'
\NOTE    = 'Column names with suffix _01 are from the input file Utest_upload.tbl.'
\NOTE    = 'Column cntr_01 is a sequential counter added to ID each row in the user input file Utest_upload.tbl.'
\NOTE    = 'dist_x is the distance in arcsec between each input source and its potential match in the AllWISE Source Catalog.'
\NOTE    = 'pang_x is position angle, measured eastward from north, between the input source and potential match in the AllWISE Source Catalog.
\NOTE    = 'Output rows are ordered by increasing cntr_01, and for each cntr_01 by increasing dist_x.'
\StatusFile = '/workspace/TMP_I0e6Zj_13982/Gator/irsa/13982/log.13982.html'
\SQL     = 'WHERE  A.cntr_01 = X.cntr1_x and B.cntr=X.cntr2_x '
\SQL     = 'SELECT (10 column names follow in next row.)'
\Upload   = 'file=Utest_upload.tbl radius=2 unit=arcsec'
\
\
|             cntr_01|             dist_x|             pang_x|       objstr_01|              ra_01|             dec_01| w1ba| w2ba| w3ba| w4ba|
|                long|             double|             double|            char|             double|             double|doubl|doubl|doubl|doubl|
|                    |                   |                   |                |                   |                   |     |     |     |     |
|                null|               null|               null|            null|               null|               null| null| null| null| null|
                    1            0.111821          -69.259471             test           24.777210           -7.512610  0.85  0.85  0.85  0.87


From thomas.robitaille at gmail.com  Wed Jan  7 05:20:52 2015
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Wed, 07 Jan 2015 11:20:52 +0100
Subject: [AstroPy] Abbreviated data types in IPAC tables
In-Reply-To: <54ACA99A.8030107@buckeyemail.osu.edu>
References: <54ACA99A.8030107@buckeyemail.osu.edu>
Message-ID: <54AD0884.7090106@gmail.com>

Hi Gregory,

I think that this is indeed a bug in Astropy and we should fix it to
recognize abbreviated data types. Can you open an issue on the astropy
bug tracker on Github?

Thanks!
Tom

Gregory Simonian wrote:
> Hi all,
> 
> I'm trying to read in an IPAC table from the WISE IRSA catalog using
> Astropy v4.1, which for some columns abbreviate "double" as "doubl". At
> first I thought this was a bug in the database, but I was informed that
> abbreviations off the end are actually a legal part of the standard.[0]
> 
> However, whenever I try to read it using astropy, I get the error:
> 
> In [2]: Table.read("test_output.tbl", format="ascii.ipac", guess=False)
> <snip>
> ValueError: Unknown data type ""doubl"" for column "w1ba"
> 
> Is this a part of the standard that hasn't been implemented yet in
> astropy? Or is there some setting that I haven't found that will enable
> this to work? Or am I just doing something wrong? I've attached a small
> output file that reproduces this error for anyone that's curious. Thanks!
> 
>                                    Gregory Simonian
> 
> [0] http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy


From npkuin at gmail.com  Wed Jan  7 07:44:55 2015
From: npkuin at gmail.com (Paul Kuin)
Date: Wed, 7 Jan 2015 12:44:55 +0000
Subject: [AstroPy] Abbreviated data types in IPAC tables
In-Reply-To: <54ACA99A.8030107@buckeyemail.osu.edu>
References: <54ACA99A.8030107@buckeyemail.osu.edu>
Message-ID: <CANoQ6N3DmqkWXptH0EzCkJ3ciA+wLRHUB4b_Rz68z8yi0TyzEA@mail.gmail.com>

I was under the impression that astropy reads FITS, and it looks like the
IPAC ascii table standard is incompatable. Perhaps IPAC can elaborate?

On Wed, Jan 7, 2015 at 3:35 AM, Gregory Simonian <
simonian.7 at buckeyemail.osu.edu> wrote:

> Hi all,
>
> I'm trying to read in an IPAC table from the WISE IRSA catalog using
> Astropy v4.1, which for some columns abbreviate "double" as "doubl". At
> first I thought this was a bug in the database, but I was informed that
> abbreviations off the end are actually a legal part of the standard.[0]
>
> However, whenever I try to read it using astropy, I get the error:
>
> In [2]: Table.read("test_output.tbl", format="ascii.ipac", guess=False)
> <snip>
> ValueError: Unknown data type ""doubl"" for column "w1ba"
>
> Is this a part of the standard that hasn't been implemented yet in
> astropy? Or is there some setting that I haven't found that will enable
> this to work? Or am I just doing something wrong? I've attached a small
> output file that reproduces this error for anyone that's curious. Thanks!
>
>                                    Gregory Simonian
>
> [0] http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>


-- 

* * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
phone +44-(0)1483 (prefix) -204927 (work)
mobile +44(0)7806985366  skype ID: npkuin
Mullard Space Science Laboratory  ? University College London  ?
Holmbury St Mary ? Dorking ? Surrey RH5 6NT?  U.K.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150107/635e2f8e/attachment.html>

From thomas.robitaille at gmail.com  Wed Jan  7 08:00:07 2015
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Wed, 07 Jan 2015 14:00:07 +0100
Subject: [AstroPy] Abbreviated data types in IPAC tables
In-Reply-To: <CANoQ6N3DmqkWXptH0EzCkJ3ciA+wLRHUB4b_Rz68z8yi0TyzEA@mail.gmail.com>
References: <54ACA99A.8030107@buckeyemail.osu.edu>
	<CANoQ6N3DmqkWXptH0EzCkJ3ciA+wLRHUB4b_Rz68z8yi0TyzEA@mail.gmail.com>
Message-ID: <54AD2DD7.8000507@gmail.com>

Astropy can actually read many ASCII formats (including IPAC) in
addition to FITS, VO Tables, etc. This is just an issue related to
abbreviated data types (i.e. it works fine when the data types are
complete).

Cheers,
Tom

Paul Kuin wrote:
> I was under the impression that astropy reads FITS, and it looks like
> the IPAC ascii table standard is incompatable. Perhaps IPAC can elaborate?
> 
> On Wed, Jan 7, 2015 at 3:35 AM, Gregory Simonian
> <simonian.7 at buckeyemail.osu.edu <mailto:simonian.7 at buckeyemail.osu.edu>>
> wrote:
> 
>     Hi all,
> 
>     I'm trying to read in an IPAC table from the WISE IRSA catalog using
>     Astropy v4.1, which for some columns abbreviate "double" as "doubl". At
>     first I thought this was a bug in the database, but I was informed that
>     abbreviations off the end are actually a legal part of the standard.[0]
> 
>     However, whenever I try to read it using astropy, I get the error:
> 
>     In [2]: Table.read("test_output.tbl", format="ascii.ipac", guess=False)
>     <snip>
>     ValueError: Unknown data type ""doubl"" for column "w1ba"
> 
>     Is this a part of the standard that hasn't been implemented yet in
>     astropy? Or is there some setting that I haven't found that will enable
>     this to work? Or am I just doing something wrong? I've attached a small
>     output file that reproduces this error for anyone that's curious.
>     Thanks!
> 
>                                        Gregory Simonian
> 
>     [0] http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html
> 
>     _______________________________________________
>     AstroPy mailing list
>     AstroPy at scipy.org <mailto:AstroPy at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/astropy
> 
> 
> 
> 
> -- 
> 
> * * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
> Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk <mailto:n.kuin at ucl.ac.uk>)      
> phone +44-(0)1483 (prefix) -204927 (work)
> mobile +44(0)7806985366  skype ID: npkuin
> Mullard Space Science Laboratory  ? University College London  ?
> Holmbury St Mary ? Dorking ? Surrey RH5 6NT?  U.K.
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy


From simonian.7 at buckeyemail.osu.edu  Wed Jan  7 19:00:14 2015
From: simonian.7 at buckeyemail.osu.edu (Gregory Simonian)
Date: Wed, 7 Jan 2015 19:00:14 -0500
Subject: [AstroPy] Abbreviated data types in IPAC tables
In-Reply-To: <54AD0884.7090106@gmail.com>
References: <54ACA99A.8030107@buckeyemail.osu.edu> <54AD0884.7090106@gmail.com>
Message-ID: <54ADC88E.7040304@buckeyemail.osu.edu>

Hi Tom,

I opened the bug right here:

https://github.com/astropy/astropy/issues/3279

Let me know if there's anything I left out. Thank you and the rest of
the astropy team for working on these tools; they save a lot of work!

                            Gregory

On 01/07/2015 05:20 AM, Thomas Robitaille wrote:
> Hi Gregory,
> 
> I think that this is indeed a bug in Astropy and we should fix it to
> recognize abbreviated data types. Can you open an issue on the astropy
> bug tracker on Github?
> 
> Thanks!
> Tom
> 
> Gregory Simonian wrote:
>> Hi all,
>>
>> I'm trying to read in an IPAC table from the WISE IRSA catalog using
>> Astropy v4.1, which for some columns abbreviate "double" as "doubl". At
>> first I thought this was a bug in the database, but I was informed that
>> abbreviations off the end are actually a legal part of the standard.[0]
>>
>> However, whenever I try to read it using astropy, I get the error:
>>
>> In [2]: Table.read("test_output.tbl", format="ascii.ipac", guess=False)
>> <snip>
>> ValueError: Unknown data type ""doubl"" for column "w1ba"
>>
>> Is this a part of the standard that hasn't been implemented yet in
>> astropy? Or is there some setting that I haven't found that will enable
>> this to work? Or am I just doing something wrong? I've attached a small
>> output file that reproduces this error for anyone that's curious. Thanks!
>>
>>                                    Gregory Simonian
>>
>> [0] http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
> 



From aldcroft at head.cfa.harvard.edu  Thu Jan  8 09:41:53 2015
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Thu, 8 Jan 2015 09:41:53 -0500
Subject: [AstroPy] Abbreviated data types in IPAC tables
In-Reply-To: <54ADC88E.7040304@buckeyemail.osu.edu>
References: <54ACA99A.8030107@buckeyemail.osu.edu> <54AD0884.7090106@gmail.com>
	<54ADC88E.7040304@buckeyemail.osu.edu>
Message-ID: <CAMtEP6z=axhxCRMDtMFShL51B+C=SGh+B4YtYu+kMpWsSiFSXg@mail.gmail.com>

For the record this has been closed now with a PR that fixes this:

  https://github.com/astropy/astropy/pull/3279

- Tom

On Wed, Jan 7, 2015 at 7:00 PM, Gregory Simonian <
simonian.7 at buckeyemail.osu.edu> wrote:

> Hi Tom,
>
> I opened the bug right here:
>
> https://github.com/astropy/astropy/issues/3279
>
> Let me know if there's anything I left out. Thank you and the rest of
> the astropy team for working on these tools; they save a lot of work!
>
>                             Gregory
>
> On 01/07/2015 05:20 AM, Thomas Robitaille wrote:
> > Hi Gregory,
> >
> > I think that this is indeed a bug in Astropy and we should fix it to
> > recognize abbreviated data types. Can you open an issue on the astropy
> > bug tracker on Github?
> >
> > Thanks!
> > Tom
> >
> > Gregory Simonian wrote:
> >> Hi all,
> >>
> >> I'm trying to read in an IPAC table from the WISE IRSA catalog using
> >> Astropy v4.1, which for some columns abbreviate "double" as "doubl". At
> >> first I thought this was a bug in the database, but I was informed that
> >> abbreviations off the end are actually a legal part of the standard.[0]
> >>
> >> However, whenever I try to read it using astropy, I get the error:
> >>
> >> In [2]: Table.read("test_output.tbl", format="ascii.ipac", guess=False)
> >> <snip>
> >> ValueError: Unknown data type ""doubl"" for column "w1ba"
> >>
> >> Is this a part of the standard that hasn't been implemented yet in
> >> astropy? Or is there some setting that I haven't found that will enable
> >> this to work? Or am I just doing something wrong? I've attached a small
> >> output file that reproduces this error for anyone that's curious.
> Thanks!
> >>
> >>                                    Gregory Simonian
> >>
> >> [0] http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html
> >>
> >> _______________________________________________
> >> AstroPy mailing list
> >> AstroPy at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/astropy
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
> >
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150108/c7e3d6b9/attachment.html>

From thomas.robitaille at gmail.com  Thu Jan  8 09:54:50 2015
From: thomas.robitaille at gmail.com (Thomas Robitaille)
Date: Thu, 08 Jan 2015 14:54:50 +0000
Subject: [AstroPy] reading tables causes seg fault
References: <AEB22144-0E71-476B-8CBD-3379BD5DBD8D@gmail.com>
Message-ID: <CAGMHX_11==nxBdOqne2+5UwpOz0aez8=VMj=Qcj18KGgTyTGvA@mail.gmail.com>

Dear Semyeong,

There is no easy way to search the mailing list as far as I know, though it
should be indexed by Google so maybe you can find old threads that way.

Regarding the issue you have run into, if you can reproduce it
consistently, could you open an issue in the tracker:

https://github.com/astropy/astropy/issues

Please include the full error message.

Thanks!
Tom


On Wed Dec 17 2014 at 9:10:21 PM Semyeong Oh <semyeong.oh at gmail.com> wrote:

> Hello astropy,
>
> I?m having trouble reading ascii tables using astropy.table.Table.
>
> d = {}
> d[1] = Table.read(table1, format='ascii')
> d[2] = Table.read(table2, format='ascii?)
> ?
>
> Each table is less than 700Kbytes. If I just read one table, it?s all
> good, but python shell would
> exit with segmentation fault upon reading the second.
> Actually, I?ve had a similar experience with reading other ascii tables
> recently, but then the files were quite huge, so I thought maybe it?s
> related to that.
>
> I?m using astropy 0.4.2, anaconda python distribution and OSX 10.10.1.
> $ conda info
> Current conda install:
>
>              platform : osx-64
>         conda version : 3.7.3
>   conda-build version : 1.3.1
>        python version : 2.7.8.final.0
>      requests version : 2.5.0
>
> Also, is there a way to search astropy mailing list? I couldn?t find one
> on archive, and I think it?d be really helpful.
>
> Best,
> Semyeong
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150108/62e4f6e4/attachment.html>

From ejensen1 at swarthmore.edu  Thu Jan  8 10:27:20 2015
From: ejensen1 at swarthmore.edu (Eric L. N. Jensen)
Date: Thu, 8 Jan 2015 10:27:20 -0500
Subject: [AstroPy] reading tables causes seg fault
In-Reply-To: <CAGMHX_11==nxBdOqne2+5UwpOz0aez8=VMj=Qcj18KGgTyTGvA@mail.gmail.com>
References: <AEB22144-0E71-476B-8CBD-3379BD5DBD8D@gmail.com>
	<CAGMHX_11==nxBdOqne2+5UwpOz0aez8=VMj=Qcj18KGgTyTGvA@mail.gmail.com>
Message-ID: <55D97EA5-981E-49D1-AD36-F111F3D2F153@swarthmore.edu>

>> On Wed Dec 17 2014 at 9:10:21 PM Semyeong Oh <semyeong.oh at gmail.com> wrote:
>> Also, is there a way to search astropy mailing list? I couldn?t find one on archive, and I think it?d be really helpful.
> 
> On Jan 8, 2015, at 9:54 AM, Thomas Robitaille <thomas.robitaille at gmail.com> wrote
> There is no easy way to search the mailing list as far as I know, though it should be indexed by Google so maybe you can find old threads that way.

More specifically, if you add "site:mail.scipy.org/pipermail/astropy/? to your Google search, it will only search that part of that site, which contains the astropy list archive.  So for example

https://www.google.com/search?q=site%3Amail.scipy.org%2Fpipermail%2Fastropy%2F+fault&ie=utf-8&oe=utf-8

will search the list for the word ?fault?. 

If you want to restrict your search term to appearing only in the title of the page (which should be the original Subject line of the message), then prepend it with ?allintitle:?, for example

https://www.google.com/webhp?hl=en#hl=en&q=allintitle:ascii+site:mail.scipy.org%2Fpipermail%2Fastropy

will find pages with ?ascii? in the title - about 14 pages, whereas searching the whole list for ?ascii? has 191 results. 

Hope this helps - 

Eric
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150108/ceea50e2/attachment.html>

From skwok at keck.hawaii.edu  Fri Jan  9 20:39:56 2015
From: skwok at keck.hawaii.edu (Shui Kwok)
Date: Sat, 10 Jan 2015 01:39:56 +0000
Subject: [AstroPy] Jobs at W. M. Keck Observatory
Message-ID: <E1D5BAA761375F4F9E27DF9F7A314138435B764B@wsmail1.keck.hawaii.edu>

Dear all,

The W. M. Keck Observatory has two openings for Software Engineers, one of which is specifically for astronomical data reduction.
Please see http://keckobservatory.iapplicants.com/searchjobs.php for more information.

Aloha,

Shui H. Kwok
Senior Software Engineer
W. M. Keck Observatory

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150110/43724c3e/attachment.html>

From tribeiro at ufs.br  Sun Jan 11 09:41:02 2015
From: tribeiro at ufs.br (Tiago Ribeiro de Souza)
Date: Sun, 11 Jan 2015 11:41:02 -0300
Subject: [AstroPy] Fellowship available to develop pipeline
Message-ID: <1DBBE1C0-851A-49C8-BF64-1B6FEF69D880@ufs.br>

Dear colleagues,

We have a postdoctoral position available for developing data reduction pipeline. The position will be based on IAG/USP in S?o Paulo, one of the leading institutions in Brasil. Up to 50% of the time can be dedicated to science. It is a 1-year contract and we need to fill it right away. If you are interested or know someone that might be, please send me an email with CV highlighting computational experience and scientific interest. Further enquiries regarding this position can also be redirected to me. 

Cheers, 

T.

?
Tiago Ribeiro de Souza
Universidade Federal de Sergipe

From tribeiro at ufs.br  Sun Jan 11 09:41:18 2015
From: tribeiro at ufs.br (Tiago Ribeiro de Souza)
Date: Sun, 11 Jan 2015 11:41:18 -0300
Subject: [AstroPy] Fellowship available to develop pipeline
Message-ID: <97C51916-8BF4-4D60-9DFD-3FC72B42D8FC@ufs.br>

Dear colleagues,

We have a postdoctoral position available for developing data reduction pipeline. The position will be based on IAG/USP in S?o Paulo, one of the leading institutions in Brasil. Up to 50% of the time can be dedicated to science. It is a 1-year contract and we need to fill it right away. If you are interested or know someone that might be, please send me an email with CV highlighting computational experience and scientific interest. Further enquiries regarding this position can also be redirected to me. 

Cheers, 

T.

?
Tiago Ribeiro de Souza
Universidade Federal de Sergipe

From jzuhone at gmail.com  Wed Jan 14 18:05:45 2015
From: jzuhone at gmail.com (John ZuHone)
Date: Wed, 14 Jan 2015 18:05:45 -0500
Subject: [AstroPy] ANN: yt-3.1 released
Message-ID: <B0179BAE-2EFE-412D-BF12-DA56103CEB6C@gmail.com>

The yt community is proud to announce the release of yt 3.1.

yt (http://yt-project.org) is an open source, community-developed
toolkit for analysis and visualization of volumetric data of all
types, with a particular emphasis on astrophysical simulations and
nuclear engineering simulations.

This is a scheduled feature release. Highlighted changes in yt 3.1:

Major changes:
++++++++++++++

* The RADMC-3D export analysis module has been updated.
* Performance improvements for grid frontends.
* Added a frontend for Dark Matter-only NMSU Art simulations.
* The absorption spectrum generator has been updated.
* The PerspectiveCamera has been updated and a new SphericalCamera has been added.
* The unit system now supports unit equivalencies and has improved support for MKS units.
* Data object selection can now be chained, allowing selecting based on multiple constraints.
* Added the ability to manually override the simulation unit system.
* The documentation has been reorganized and has seen substantial improvements. 

Minor or bugfix changes:
++++++++++++++++++++++++

* The Gadget InternalEnergy and StarFormationRate fields are now read in with the correct units.
* Substantial improvements for the PPVCube analysis module and support for FITS datasets.
* The center of a PlotWindow plot can now be set to the maximum or minimum of any field.
* Projections are now performed using an explicit path length field for all coordinate systems.
* Fix for the camera.draw_line function.
* Minor fixes and improvements for yt plots.
* Significant documentation reorganization and improvement.
* Miscellaneous code cleanup.
* yt now hooks up to the python logging infrastructure in a more standard fashion, avoiding issues 
  with yt logging showing up with using other libraries.
* Improvements for the yt-rockstar interface.
* It is now possible to supply a default value for get_field_parameter.
* A bug in the interpretation of the units of RAMSES simulations has been fixed.
* Improvements and bugfixes for the halo analysis framework.
* Fix issues with the default setting for the "center" field parameter.
* yt can now be run in parallel on a subset of available processors using an MPI subcommunicator.
* Fix for incorrect units when loading an Athena simulation as a time series.
* Improved support for Enzo 3.0 simulations that have not produced any active particles.
* Fix for periodic radius vector calculation.
* Improvements for the Maestro and Castro frontends.
* Clump finding is now supported for more generic types of data.
* Fix unit consistency issue when mixing dimensionless unit symbols.
* Improved memory footprint in the photon_simulator.
* Large grids in Athena datasets produced by the join_vtk script can now be optionally split, 
  improving parallel performance. 
* Slice plots now accept a ?data_source" keyword argument.
* Nearest neighbor distance field added.
* Improvements for the ORION2 frontend.
* Enzo 3.0 frontend can now read active particle attributes that are arrays of any shape.
* Fixes for accessing deposit fields for FLASH data. 
* Added wrapper functions for numpy array manipulation functions.
* Added support for packed HDF5 Enzo datasets.

A more comprehensive list of the changes in this release, with links to the corresponding pull requests, 
can be found at http://yt-project.org/docs/3.1/reference/changelog.html. 

Standard Installation Methods
-----------------------------

As with previous releases, you can install yt from source using one of 
the following methods. 

1) From the install script (http://yt-project.org/#getyt):

# Installation

$ wget http://hg.yt-project.org/yt/raw/stable/doc/install_script.sh
$ bash install_script.sh

# Update

$ yt update

2) From pip (source or binary wheel, see below for more details):

# Installation

$ pip install yt

# Update

$ pip install -U yt

3) From the Anaconda Python Distribution (https://store.continuum.io/cshop/anaconda/): 

# Installation

$ conda install yt

# Update 

$ conda update yt

Note that it might take a day or two for the conda package to be updated.

If you are on the ?stable? branch, updating will bring you from yt 3.0.2 to 3.1, incorporating all 
changes since 3.0.2, whereas if you are on the ?dev? or ?yt? branch, only the changes since 
your last update should be incorporated. 

NEW: Installing Binary Packages via pip
---------------------------------------

New to this release is the ability to install binary packages (?wheels?) using
pip on Windows and Mac OS X (64-bit only for both). This has the advantage 
of not needing to install yt from source using a proper compiler setup, which
has caused occasional problems on both of these platforms and prevented us
from installing yt easily on other Python distributions. 

We have so far been able to install and run the binary distribution via pip on the 
following platforms and Python stacks:

Windows x86_64: 

* Enthought Canopy Python (https://www.enthought.com/products/canopy/)
* WinPython (http://winpython.sourceforge.net/) 

Mac OS X x86_64:

* Enthought Canopy Python (https://www.enthought.com/products/canopy/)
* Homebrew Python (http://brew.sh/)
* Python.org Python
* Mac OS X?s system Python
* MacPorts Python (https://www.macports.org/)

This is somewhat experimental, so other distributions may work (or not), 
please submit bug reports or successes to the mailing list or to the Bitbucket
issues page (http://bitbucket.org/yt_analysis/yt/issues). 

All distributions must be Python v. 2.7. The requirements for installing yt 
via this method are the same as from source:

* NumPy
* h5py
* HDF5
* SymPy
* Matplotlib
* IPython (not required, but strongly recommended)

To install a new version of yt on one of these platforms, simply do 

$ pip install yt

and you should get the binary distribution automatically. Also, if your 
python installation is system-wide (e.g., the Mac system Python) 
you might need to run pip with administrator privileges. 

For more information, including more installation instructions, links to
community resources, and information on contributing to yt?s
development, please see the yt homepage at http://yt-project.org and
the documentation for yt-3.1 at http://yt-project.org/docs/3.1.

yt is the product of a large community of developers and users and we
are extraordinarily grateful for and proud of their contributions. Please 
forward this announcement on to any interested parties.

As always, if you have any questions, concerns, or run into any trouble 
updating please don't hesitate to send a message to the mailing list or 
stop by our IRC channel.

Thank you,

The yt development team

From aldcroft at head.cfa.harvard.edu  Thu Jan 15 13:46:24 2015
From: aldcroft at head.cfa.harvard.edu (Aldcroft, Thomas)
Date: Thu, 15 Jan 2015 13:46:24 -0500
Subject: [AstroPy] Enhanced Character Separated Values table format for
	discussion
Message-ID: <CAMtEP6wnGVjc2D=L9d38HoPBJA9gYngYw_G8oPGha-QT0er3QQ@mail.gmail.com>

There is an Astropy Proposed Enhancement that could benefit from discussion
with an audience outside the usual astropy-dev crowd.  Please see the pull
request link and abstract below.

To avoid confusion, *please* comment on the github PR page if possible, not
on this mailing list.

Cheers,
Tom

APE-6 : https://github.com/astropy/astropy-APEs/pull/7

Abstract
--------

Data tables in astronomical analysis frequently contain additional metadata
beyond just the column names and data values.  Common attributes include
the numerical data type, the physical unit, and a longer textual
description of the column content.  These attributes can be represented in
binary formats such as FITS, but the available options for a text-only
format are inadequate.

APE6 proposes to fill this void by specification of a standard for the
interchange of tabular data in a text-only format.  The format handles the
key issue of serializing column specifications and table metadata by using
a YAML-encoded data structure.  The actual tabular data are stored in a
standard character separated values (CSV) format, giving compatibility with
a wide variety of non-specialized CSV table readers.

Using YAML makes it extremely easy for applications *and humans* to read
both the standardized data format elements (e.g. column name, type,
description) as well as complex metadata structures.  YAML also lends
itself to simple table modifications by humans in a plain text editor.

The reference Python implementation in ``astropy.io.ascii`` is relatively
straightforward and will provide a significant benefit of allowing text
serialization of most astropy Table objects, persistent storage, and
subsequent interchange with other users.

Although developed in the context of Astropy, there is nothing
astronomy-specific in this format and it could be useful for other domains
or languages.  However, promoting that is beyond the scope of this APE.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150115/4c48e45b/attachment.html>

From dkirkby at uci.edu  Sat Jan 17 20:29:11 2015
From: dkirkby at uci.edu (David Kirkby)
Date: Sun, 18 Jan 2015 01:29:11 +0000
Subject: [AstroPy] Advice on reading FITS file with many HDUs
Message-ID: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>

I am reading a ~700 Mb file containing ~45K HDUs and looking for advice on
how to speed things up. I typically only want to read the first 2 HDUs
right after opening the file, and then a small number of randomly selected
HDUs while my program runs. The first 2 HDUs are the largest, but still
only represent about10% of the total file size.

The following command takes about 42 seconds:

  fits.open('LSST_i.fits',mode='readonly',memmap=False)

Changing to memap=True makes no difference but leads to "Too many open
files" if I try to read too many HDUs after opening the file.

Is this what I should expect? Any suggestions for opening the file and
reading a small number of HDUs faster? If necessary, I can change the
format of the file I am reading.

In case it helps, the file I am testing with can be downloaded from:


http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-DESC&folder=12915647

thanks,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150118/8a314d10/attachment.html>

From stsci.perry at gmail.com  Sat Jan 17 21:32:00 2015
From: stsci.perry at gmail.com (Perry Greenfield)
Date: Sat, 17 Jan 2015 21:32:00 -0500
Subject: [AstroPy] Advice on reading FITS file with many HDUs
In-Reply-To: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
References: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
Message-ID: <498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>

My first reaction is that you really are asking a lot of the fits module (45K headers!). If it was only to read the first two, we could consider some option to that module not to locate all headers in the file, but just the first N, but asking for random access basically requires reading most of them anyway.

My second reaction is this is a poor use of FITS. Isn't there something more efficient you could do with storing the information in a FITS file? I know, I know, someone else decided this (usually) and you don't  have any control over that. Still, it makes me wonder seeing things like FITS taken to this extreme level.

Really, this seems a better job for CFITSIO (I'd be curious to see how much faster it is though since it does have to run through most of the file to find the header locations as well, but will avoid the overhead of creating 45K objects).

Cheers, Perry 

On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:

> I am reading a ~700 Mb file containing ~45K HDUs and looking for advice on how to speed things up. I typically only want to read the first 2 HDUs right after opening the file, and then a small number of randomly selected HDUs while my program runs. The first 2 HDUs are the largest, but still only represent about10% of the total file size.
> 
> The following command takes about 42 seconds:
> 
>   fits.open('LSST_i.fits',mode='readonly',memmap=False)
> 
> Changing to memap=True makes no difference but leads to "Too many open files" if I try to read too many HDUs after opening the file.
> 
> Is this what I should expect? Any suggestions for opening the file and reading a small number of HDUs faster? If necessary, I can change the format of the file I am reading.
> 
> In case it helps, the file I am testing with can be downloaded from: 
> 
>   http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-DESC&folder=12915647
> 
> thanks,
> David
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy



From dkirkby at uci.edu  Sat Jan 17 22:45:44 2015
From: dkirkby at uci.edu (David Kirkby)
Date: Sun, 18 Jan 2015 03:45:44 +0000
Subject: [AstroPy] Advice on reading FITS file with many HDUs
References: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
	<498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>
Message-ID: <CAF4SvuVgczVNoVyccWRUnYTL=7x2favW43ecO9ATRDcKqPZwvA@mail.gmail.com>

Thanks for the feedback, Perry. Yes, point taken about asking too much from
FITS.  I was hoping there might be a simple fix, but my plan B is to save
the 45K-2 HDUs in an HDF5 file instead. That's a bit more hassle but
probably worth it.

David

On Sat Jan 17 2015 at 6:33:32 PM Perry Greenfield <stsci.perry at gmail.com>
wrote:

> My first reaction is that you really are asking a lot of the fits module
> (45K headers!). If it was only to read the first two, we could consider
> some option to that module not to locate all headers in the file, but just
> the first N, but asking for random access basically requires reading most
> of them anyway.
>
> My second reaction is this is a poor use of FITS. Isn't there something
> more efficient you could do with storing the information in a FITS file? I
> know, I know, someone else decided this (usually) and you don't  have any
> control over that. Still, it makes me wonder seeing things like FITS taken
> to this extreme level.
>
> Really, this seems a better job for CFITSIO (I'd be curious to see how
> much faster it is though since it does have to run through most of the file
> to find the header locations as well, but will avoid the overhead of
> creating 45K objects).
>
> Cheers, Perry
>
> On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:
>
> > I am reading a ~700 Mb file containing ~45K HDUs and looking for advice
> on how to speed things up. I typically only want to read the first 2 HDUs
> right after opening the file, and then a small number of randomly selected
> HDUs while my program runs. The first 2 HDUs are the largest, but still
> only represent about10% of the total file size.
> >
> > The following command takes about 42 seconds:
> >
> >   fits.open('LSST_i.fits',mode='readonly',memmap=False)
> >
> > Changing to memap=True makes no difference but leads to "Too many open
> files" if I try to read too many HDUs after opening the file.
> >
> > Is this what I should expect? Any suggestions for opening the file and
> reading a small number of HDUs faster? If necessary, I can change the
> format of the file I am reading.
> >
> > In case it helps, the file I am testing with can be downloaded from:
> >
> >   http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-
> DESC&folder=12915647
> >
> > thanks,
> > David
> > _______________________________________________
> > AstroPy mailing list
> > AstroPy at scipy.org
> > http://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150118/50f0947c/attachment.html>

From kylebarbary at gmail.com  Sat Jan 17 23:21:30 2015
From: kylebarbary at gmail.com (Kyle Barbary)
Date: Sat, 17 Jan 2015 20:21:30 -0800
Subject: [AstroPy] Advice on reading FITS file with many HDUs
In-Reply-To: <CAF4SvuVgczVNoVyccWRUnYTL=7x2favW43ecO9ATRDcKqPZwvA@mail.gmail.com>
References: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
	<498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>
	<CAF4SvuVgczVNoVyccWRUnYTL=7x2favW43ecO9ATRDcKqPZwvA@mail.gmail.com>
Message-ID: <CANO7-x8E58s065t2+657pcXoOcvrBdODpvFrszfQRW1AE6cTgg@mail.gmail.com>

Hi David,

Erin Sheldon?s fitsio <https://github.com/esheldon/fitsio/> seems to do a
bit better:

In [1]: import fitsio

In [2]: f = fitsio.FITS("LSST_i_descwl.fits");

In [3]: f[0]
Out[3]:

  file: LSST_i_descwl.fits
  extension: 0
  type: IMAGE_HDU
  image info:
    data type: f4
    dims: [4096,4096]

Line 2 (opening the file) is nearly instantaneous, Line 3 (getting the info
for the first HDU) takes maybe a second. Other HDU access after that (even
near the end of the file) seems nearly instantaneous as well.

Best,
Kyle
?

On Sat, Jan 17, 2015 at 7:45 PM, David Kirkby <dkirkby at uci.edu> wrote:

> Thanks for the feedback, Perry. Yes, point taken about asking too much
> from FITS.  I was hoping there might be a simple fix, but my plan B is to
> save the 45K-2 HDUs in an HDF5 file instead. That's a bit more hassle but
> probably worth it.
>
> David
>
> On Sat Jan 17 2015 at 6:33:32 PM Perry Greenfield <stsci.perry at gmail.com>
> wrote:
>
>> My first reaction is that you really are asking a lot of the fits module
>> (45K headers!). If it was only to read the first two, we could consider
>> some option to that module not to locate all headers in the file, but just
>> the first N, but asking for random access basically requires reading most
>> of them anyway.
>>
>> My second reaction is this is a poor use of FITS. Isn't there something
>> more efficient you could do with storing the information in a FITS file? I
>> know, I know, someone else decided this (usually) and you don't  have any
>> control over that. Still, it makes me wonder seeing things like FITS taken
>> to this extreme level.
>>
>> Really, this seems a better job for CFITSIO (I'd be curious to see how
>> much faster it is though since it does have to run through most of the file
>> to find the header locations as well, but will avoid the overhead of
>> creating 45K objects).
>>
>> Cheers, Perry
>>
>> On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:
>>
>> > I am reading a ~700 Mb file containing ~45K HDUs and looking for advice
>> on how to speed things up. I typically only want to read the first 2 HDUs
>> right after opening the file, and then a small number of randomly selected
>> HDUs while my program runs. The first 2 HDUs are the largest, but still
>> only represent about10% of the total file size.
>> >
>> > The following command takes about 42 seconds:
>> >
>> >   fits.open('LSST_i.fits',mode='readonly',memmap=False)
>> >
>> > Changing to memap=True makes no difference but leads to "Too many open
>> files" if I try to read too many HDUs after opening the file.
>> >
>> > Is this what I should expect? Any suggestions for opening the file and
>> reading a small number of HDUs faster? If necessary, I can change the
>> format of the file I am reading.
>> >
>> > In case it helps, the file I am testing with can be downloaded from:
>> >
>> >   http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-
>> DESC&folder=12915647
>> >
>> > thanks,
>> > David
>> > _______________________________________________
>> > AstroPy mailing list
>> > AstroPy at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/astropy
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150117/83eff618/attachment.html>

From dkirkby at uci.edu  Mon Jan 19 13:56:34 2015
From: dkirkby at uci.edu (David Kirkby)
Date: Mon, 19 Jan 2015 18:56:34 +0000
Subject: [AstroPy] Advice on reading FITS file with many HDUs
References: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
	<498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>
	<CAF4SvuVgczVNoVyccWRUnYTL=7x2favW43ecO9ATRDcKqPZwvA@mail.gmail.com>
	<CANO7-x8E58s065t2+657pcXoOcvrBdODpvFrszfQRW1AE6cTgg@mail.gmail.com>
Message-ID: <CAF4SvuUPBJHsAdittvb+Bm9bf-=n9BxUmgR8mtYuoTuP3opnzg@mail.gmail.com>

Hi Kyle,

Thanks for the suggestion.  It turns out that fitsio is significantly
faster at reading my files with many small HDUs, but is very slow at
writing them (details at https://github.com/esheldon/fitsio/issues/32).
Fortunately, I can have the best of both worlds and won't have to abandon
FITS files for this application.

David

On Sat Jan 17 2015 at 8:22:00 PM Kyle Barbary <kylebarbary at gmail.com> wrote:

> Hi David,
>
> Erin Sheldon?s fitsio <https://github.com/esheldon/fitsio/> seems to do a
> bit better:
>
> In [1]: import fitsio
>
> In [2]: f = fitsio.FITS("LSST_i_descwl.fits");
>
> In [3]: f[0]
> Out[3]:
>
>   file: LSST_i_descwl.fits
>   extension: 0
>   type: IMAGE_HDU
>   image info:
>     data type: f4
>     dims: [4096,4096]
>
> Line 2 (opening the file) is nearly instantaneous, Line 3 (getting the
> info for the first HDU) takes maybe a second. Other HDU access after that
> (even near the end of the file) seems nearly instantaneous as well.
>
> Best,
> Kyle
> ?
>
> On Sat, Jan 17, 2015 at 7:45 PM, David Kirkby <dkirkby at uci.edu> wrote:
>
>> Thanks for the feedback, Perry. Yes, point taken about asking too much
>> from FITS.  I was hoping there might be a simple fix, but my plan B is to
>> save the 45K-2 HDUs in an HDF5 file instead. That's a bit more hassle but
>> probably worth it.
>>
>> David
>>
>> On Sat Jan 17 2015 at 6:33:32 PM Perry Greenfield <stsci.perry at gmail.com>
>> wrote:
>>
>>> My first reaction is that you really are asking a lot of the fits module
>>> (45K headers!). If it was only to read the first two, we could consider
>>> some option to that module not to locate all headers in the file, but just
>>> the first N, but asking for random access basically requires reading most
>>> of them anyway.
>>>
>>> My second reaction is this is a poor use of FITS. Isn't there something
>>> more efficient you could do with storing the information in a FITS file? I
>>> know, I know, someone else decided this (usually) and you don't  have any
>>> control over that. Still, it makes me wonder seeing things like FITS taken
>>> to this extreme level.
>>>
>>> Really, this seems a better job for CFITSIO (I'd be curious to see how
>>> much faster it is though since it does have to run through most of the file
>>> to find the header locations as well, but will avoid the overhead of
>>> creating 45K objects).
>>>
>>> Cheers, Perry
>>>
>>> On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:
>>>
>>> > I am reading a ~700 Mb file containing ~45K HDUs and looking for
>>> advice on how to speed things up. I typically only want to read the first 2
>>> HDUs right after opening the file, and then a small number of randomly
>>> selected HDUs while my program runs. The first 2 HDUs are the largest, but
>>> still only represent about10% of the total file size.
>>> >
>>> > The following command takes about 42 seconds:
>>> >
>>> >   fits.open('LSST_i.fits',mode='readonly',memmap=False)
>>> >
>>> > Changing to memap=True makes no difference but leads to "Too many open
>>> files" if I try to read too many HDUs after opening the file.
>>> >
>>> > Is this what I should expect? Any suggestions for opening the file and
>>> reading a small number of HDUs faster? If necessary, I can change the
>>> format of the file I am reading.
>>> >
>>> > In case it helps, the file I am testing with can be downloaded from:
>>> >
>>> >   http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-D
>>> ESC&folder=12915647
>>> >
>>> > thanks,
>>> > David
>>> > _______________________________________________
>>> > AstroPy mailing list
>>> > AstroPy at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/astropy
>>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150119/3a43b510/attachment.html>

From embray at stsci.edu  Tue Jan 20 13:22:40 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 20 Jan 2015 13:22:40 -0500
Subject: [AstroPy] Advice on reading FITS file with many HDUs
In-Reply-To: <498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>
References: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
	<498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>
Message-ID: <54BE9CF0.7000901@stsci.edu>

On 01/17/2015 09:32 PM, Perry Greenfield wrote:
> My first reaction is that you really are asking a lot of the fits module (45K headers!). If it was only to read the first two, we could consider some option to that module not to locate all headers in the file, but just the first N, but asking for random access basically requires reading most of them anyway.
>
> My second reaction is this is a poor use of FITS. Isn't there something more efficient you could do with storing the information in a FITS file? I know, I know, someone else decided this (usually) and you don't  have any control over that. Still, it makes me wonder seeing things like FITS taken to this extreme level.
>
> Really, this seems a better job for CFITSIO (I'd be curious to see how much faster it is though since it does have to run through most of the file to find the header locations as well, but will avoid the overhead of creating 45K objects).

For what it's worth, I have plans drawn up for a complete redesign of the I/O 
for PyFITS that would speed up things like this quite a bit (and would enable 
reading only the required HDUs without having to access the others).  But in the 
meantime I agree.

 > On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:
 >> Changing to memap=True makes no difference but leads to "Too many open 
files" if I try to read too many HDUs after opening the file.

That's surprising though, since memmap=True doesn't do anything unless you're 
also reading in the data of each HDU.

Erik

> On Jan 17, 2015, at 8:29 PM, David Kirkby wrote:
>
>> I am reading a ~700 Mb file containing ~45K HDUs and looking for advice on how to speed things up. I typically only want to read the first 2 HDUs right after opening the file, and then a small number of randomly selected HDUs while my program runs. The first 2 HDUs are the largest, but still only represent about10% of the total file size.
>>
>> The following command takes about 42 seconds:
>>
>>    fits.open('LSST_i.fits',mode='readonly',memmap=False)
>>
>> Changing to memap=True makes no difference but leads to "Too many open files" if I try to read too many HDUs after opening the file.
>>
>> Is this what I should expect? Any suggestions for opening the file and reading a small number of HDUs faster? If necessary, I can change the format of the file I am reading.
>>
>> In case it helps, the file I am testing with can be downloaded from:
>>
>>    http://srs.slac.stanford.edu/DataCatalog/?experiment=LSST-DESC&folder=12915647
>>
>> thanks,
>> David
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



From dkirkby at uci.edu  Tue Jan 20 13:47:22 2015
From: dkirkby at uci.edu (David Kirkby)
Date: Tue, 20 Jan 2015 18:47:22 +0000
Subject: [AstroPy] Advice on reading FITS file with many HDUs
References: <CAF4SvuX23S4jveBiFbP+KVPOMyTGohVDZRomJShOOEkVmU8U0g@mail.gmail.com>
	<498EE049-DF04-4B67-A59C-8A6C728E095D@gmail.com>
	<54BE9CF0.7000901@stsci.edu>
Message-ID: <CAF4SvuWvoywqiXR-MLLB3kgSOv5RHavNa3Uoc=+boz48cMPWAQ@mail.gmail.com>

>
>
> That's surprising though, since memmap=True doesn't do anything unless
> you're
> also reading in the data of each HDU.
>
>
The "Too many open files" problem does not occur when I open the file, but
when I start using the data from too many HDUs data.  I think this is
expected, since you open a file per HDU with memmap=True, and the problem
is just a relatively low OS limit on the max open files.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150120/11f2d9e7/attachment.html>

From eric at depagne.org  Wed Jan 21 06:51:45 2015
From: eric at depagne.org (=?ISO-8859-1?Q?=C9ric?= Depagne)
Date: Wed, 21 Jan 2015 13:51:45 +0200
Subject: [AstroPy] Astroquery question.
Message-ID: <2795874.fIBDUAeORS@notebook>

Hi all.

I'm not sure it's the right place to ask. If not, let me know.

I'm querying the Simbad database, and I'd like to know how if the query 
retrieves only the data shown when one prints the result of the query, or if 
there is more data retrieved, but it needs additional code to get it.

Specifically, I'd like to retrieve the magnitudes of various objects. When one 
queries simbad directly, this information is present, but when using 
Simbad.query_object, it is not there.

Since query_objectids provide other info, I suspect it must be possible to get 
the magnitudes from a query.

Thanks.

?ric.
-- 
Un clavier azerty en vaut deux
----------------------------------------------------------
?ric Depagne                            eric at depagne.org



From avillaum at ucsc.edu  Wed Jan 21 07:45:50 2015
From: avillaum at ucsc.edu (Alexa Villaume)
Date: Wed, 21 Jan 2015 04:45:50 -0800
Subject: [AstroPy] Astroquery question.
In-Reply-To: <2795874.fIBDUAeORS@notebook>
References: <2795874.fIBDUAeORS@notebook>
Message-ID: <9D640CBB-25FF-4671-BD70-ACBE8C16AFD3@ucsc.edu>

Hi ?ric,

You may be able to retrieve the magnitudes by customizing the VOTable to include that field.

Look at the section ?Specifying which VOTable fields to include in the result? in the documentation (https://astroquery.readthedocs.org/en/latest/simbad/simbad.html).

Alexa

On Jan 21, 2015, at 3:51 AM, ?ric Depagne <eric at depagne.org> wrote:

> Hi all.
> 
> I'm not sure it's the right place to ask. If not, let me know.
> 
> I'm querying the Simbad database, and I'd like to know how if the query 
> retrieves only the data shown when one prints the result of the query, or if 
> there is more data retrieved, but it needs additional code to get it.
> 
> Specifically, I'd like to retrieve the magnitudes of various objects. When one 
> queries simbad directly, this information is present, but when using 
> Simbad.query_object, it is not there.
> 
> Since query_objectids provide other info, I suspect it must be possible to get 
> the magnitudes from a query.
> 
> Thanks.
> 
> ?ric.
> -- 
> Un clavier azerty en vaut deux
> ----------------------------------------------------------
> ?ric Depagne                            eric at depagne.org
> 
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy



From eric at depagne.org  Wed Jan 21 08:14:31 2015
From: eric at depagne.org (=?ISO-8859-1?Q?=C9ric?= Depagne)
Date: Wed, 21 Jan 2015 15:14:31 +0200
Subject: [AstroPy] Astroquery question.
In-Reply-To: <9D640CBB-25FF-4671-BD70-ACBE8C16AFD3@ucsc.edu>
References: <2795874.fIBDUAeORS@notebook>
	<9D640CBB-25FF-4671-BD70-ACBE8C16AFD3@ucsc.edu>
Message-ID: <1477418.RqkHcWNVdz@notebook>

On Wednesday 21 January 2015 04:45:50 Alexa Villaume wrote:
Hi Alexa, 


Many thanks for your answer, it indeed did the trick!


?ric.
[1]
/> Hi ?ric,/
/> /
/> You may be able to retrieve the magnitudes by customizing the VOTable to/
/> include that field./
/> /
/> Look at the section ?Specifying which VOTable fields to include in the/
/> result? in the documentation/
/> (https://astroquery.readthedocs.org/en/latest/simbad/simbad.html[2])./
/> /
/> Alexa/


-- 
Un clavier azerty en vaut deux
----------------------------------------------------------
?ric Depagne


--------
[1] 
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAkAAAAJCAMAAADXT/YiAA
ACiFBMVEUAAAAAADMAAGYAAJkAAMwAAP8AMwAAMzMAM2YAM5kAM8wAM/8AZ
gAAZjMAZmYAZpkAZswAZv8AmQAAmTMAmWYAmZkAmcwAmf8AzAAAzDMAzGYAzJ
kAzMwAzP8A/wAA/zMA/2YA/5kA/8wA//8zAAAzADMzAGYzAJkzAMwzAP8zMwAzMzMz
M2YzM5kzM8wzM/8zZgAzZjMzZmYzZpkzZswzZv8zmQAzmTMzmWYzmZkzmcwzmf8zzAA
zzDMzzGYzzJkzzMwzzP8z/wAz/zMz/2Yz/5kz/8wz//9mAABmADNmAGZmAJlmAMxmAP9m
MwBmMzNmM2ZmM5lmM8xmM/9mZgBmZjNmZmZmZplmZsxmZv9mmQBmmTNmmWZ
mmZlmmcxmmf9mzABmzDNmzGZmzJlmzMxmzP9m/wBm/zNm/2Zm/5lm/8xm//+ZAACZA
DOZAGaZAJmZAMyZAP+ZMwCZMzOZM2aZM5mZM8yZM/+ZZgCZZjOZZmaZZpmZZs
yZZv+ZmQCZmTOZmWaZmZmZmcyZmf+ZzACZzDOZzGaZzJmZzMyZzP+Z/wCZ/zOZ/2
aZ/5mZ/8yZ///MAADMADPMAGbMAJnMAMzMAP/MMwDMMzPMM2bMM5nMM8zMM//
MZgDMZjPMZmbMZpnMZszMZv/MmQDMmTPMmWbMmZnMmczMmf/MzADMzDPMzG
bMzJnMzMzMzP/M/wDM/zPM/2bM/5nM/8zM////AAD/ADP/AGb/AJn/AMz/AP//MwD/MzP
/M2b/M5n/M8z/M///ZgD/ZjP/Zmb/Zpn/Zsz/Zv//mQD/mTP/mWb/mZn/mcz/mf//zAD/zDP/
zGb/zJn/zMz/zP///wD//zP//2b//5n//8z///9jsLtRAAAAJUlEQVR4AWNoDINChurrEBDGEANl
oYoxgMB1AurA5sVUh4U1AgDi1TJf4feK3QAAAABJRU5ErkJggg==
[2] https://astroquery.readthedocs.org/en/latest/simbad/simbad.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150121/b52129f8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data:image/png;	base64,iVBORw0KGgoAAAANSUhEUgAAAAkAAAAJCAMAAADXT/YiAAACiFBMVEUAAAAAADMAAGYAAJkAAMwAAP8AMwAAMzMAM2YAM5kAM8wAM/8AZgAAZjMAZmYAZpkAZswAZv8AmQAAmTMAmWYAmZkAmcwAmf8AzAAAzDMAzGYAzJkAzMwAzP8A/wAA/zMA/2YA/5kA/8wA//8zAAAzADMzAGYzAJkzAMwzAP8zMwAzMzMzM2YzM5kzM8wzM/8zZgAzZjMzZmYzZpkzZswzZv8zmQAzmTMzmWYzmZkzmcwzmf8zzAAzzDMzzGYzzJkzzMwzzP8z/wAz/zMz/2Yz/5kz/8wz//9mAABmADNmAGZmAJlmAMxmAP9mMwBmMzNmM2ZmM5lmM8xmM/9mZgBmZjNmZmZmZplmZsxmZv9mmQBmmTNmmWZmmZlmmcxmmf9mzABmzDNmzGZmzJlmzMxmzP9m/wBm/zNm/2Zm/5lm/8xm//+ZAACZADOZAGaZAJmZAMyZAP+ZMwCZMzOZM2aZM5mZM8yZM/+ZZgCZZjOZZmaZZpmZZsyZZv+ZmQCZmTOZmWaZmZmZmcyZmf+ZzACZzDOZzGaZzJmZzMyZzP+Z/wCZ/zOZ/2aZ/5mZ/8yZ///MAADMADPMAGbMAJnMAMzMAP/MMwDMMzPMM2bMM5nMM8zMM//MZgDMZjPMZmbMZpnMZszMZv/MmQDMmTPMmWbMmZnMmczMmf/MzADMzDPMzGbMzJnMzMzMzP/M/wDM/zPM/2bM/5nM/8zM////AAD/ADP/AGb/AJn/AMz/AP//MwD/MzP/M2b/M5n/M8z/M///ZgD/ZjP/Zmb/Zpn/Zsz/Zv//mQD/mTP/mWb/mZn/mcz/mf//zAD/zDP/zGb/zJn/zMz/zP///wD//zP//2b//5n//8z///9jsLtRAAAAJUlEQVR4AWNoDINChurrEBDGEANloYoxgMB1AurA5sVUh4U1AgDi1TJf4feK3QAAAABJRU5ErkJggg==
Type: image/png
Size: 170 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/astropy/attachments/20150121/b52129f8/attachment.png>

From embray at stsci.edu  Wed Jan 21 16:12:37 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 21 Jan 2015 16:12:37 -0500
Subject: [AstroPy] [ANN] Astropy v0.4.4 released
Message-ID: <54C01645.9000708@stsci.edu>

Hi all,

Not a week after v0.4.3 was released, here is a surprise v0.4.4  The release is
listed on PyPI here:

https://pypi.python.org/pypi/astropy/0.4.4

The git tag is here:

https://github.com/astropy/astropy/tree/v0.4.4

This release is primarily to address a security update in Python 2.7.9
and 3.4 [1] that rendered the astropy.vo.samp module unusable.  This
is also a security enhancement, albeit one only likely to affect a
small number of users (anyone using SAMP over HTTPS across a network).

Thanks to Ole Streicher for reporting the issue, and to Mike
Droettboom for providing a quick fix.


This also included a fix for an issue [2] that was reported fixed in
Astropy v0.4.3, but the fix for which was actually excluded from that
release by mistake.

See the full changelog [3] for details on the fixes that were included.

Erik B.

[1] https://github.com/astropy/astropy/issues/3306
[2] https://github.com/astropy/astropy/pull/2909
[3] https://github.com/astropy/astropy/blob/v0.4.4/CHANGES.rst#044-2015-01-21


From stephenbailey at lbl.gov  Mon Jan 26 20:50:16 2015
From: stephenbailey at lbl.gov (Stephen Bailey)
Date: Mon, 26 Jan 2015 17:50:16 -0800
Subject: [AstroPy] writing 2D string column to fits binary table
Message-ID: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>

astropy.io.fits and fitsio aficionados,

I'm trying to write a FITS binary table that includes a column that is a 2D
array of strings.  Curiously, only files written by astropy.io.fits and
read by fitsio pass the test of actually reconstructing the input numpy
array.

astropy.io.fits loses the 2D information when reading back in and ends up
with a 1D array of null separated characters per row.

fitsio.fits pads its file with spaces when reading back in.

But the file written by astropy.io.fits and then read by fitsio gets me
back what I put in.

Details:

import numpy as np

from astropy.io import fits

import fitsio


n = 10

dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])

data = np.zeros(n, dtype=dtype)

data['X'] = np.arange(n)

data['Y'] = np.arange(n)

data['ABC'][:, 0] = 'a'

data['ABC'][:, 1] = 'b'

data['ABC'][:, 2] = 'c'

data['ABC'][0] = ['x', 'y', 'z']


fits.writeto('apio.fits', data)

fitsio.write('fio.fits', data)

astropy.io.fits complains:

/Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
UserWarning: TDIM1 value (5,3) does not fit with the size of the array
items (5).  TDIM1 will be ignored.

  actual_nitems, indx + 1))

fitsio seems happy.  Reading it back in:

In [*15*]: np.array(fits.getdata('apio.fits'))

Out[*15*]:

array([('x\x00\x00\x00\x00y\x00\x00\x00\x00z', 0, 0.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 1, 1.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 2, 2.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 3, 3.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 4, 4.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 5, 5.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 6, 6.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 7, 7.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 8, 8.0),

       ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 9, 9.0)],

      dtype=[('ABC', 'S15'), ('X', '>i8'), ('Y', '>f8')])


In [*16*]: fitsio.read('fio.fits')

Out[*16*]:

array([(['x    ', 'y    ', 'z    '], 0, 0.0),

       (['a    ', 'b    ', 'c    '], 1, 1.0),

       (['a    ', 'b    ', 'c    '], 2, 2.0),

       (['a    ', 'b    ', 'c    '], 3, 3.0),

       (['a    ', 'b    ', 'c    '], 4, 4.0),

       (['a    ', 'b    ', 'c    '], 5, 5.0),

       (['a    ', 'b    ', 'c    '], 6, 6.0),

       (['a    ', 'b    ', 'c    '], 7, 7.0),

       (['a    ', 'b    ', 'c    '], 8, 8.0),

       (['a    ', 'b    ', 'c    '], 9, 9.0)],

      dtype=[('ABC', 'S5', (3,)), ('X', '>i8'), ('Y', '>f8')])


In [*17*]: data

Out[*17*]:

array([(['x', 'y', 'z'], 0, 0.0), (['a', 'b', 'c'], 1, 1.0),

       (['a', 'b', 'c'], 2, 2.0), (['a', 'b', 'c'], 3, 3.0),

       (['a', 'b', 'c'], 4, 4.0), (['a', 'b', 'c'], 5, 5.0),

       (['a', 'b', 'c'], 6, 6.0), (['a', 'b', 'c'], 7, 7.0),

       (['a', 'b', 'c'], 8, 8.0), (['a', 'b', 'c'], 9, 9.0)],

      dtype=[('ABC', 'S5', (3,)), ('X', '<i8'), ('Y', '<f8')])

But fitsio reading the file that astropy.io.fits wrote is good (despite the
warning that astropy.io.fits gave when writing the file):

In [*18*]: fitsio.read('apio.fits')

Out[*18*]:

array([(['x', 'y', 'z'], 0, 0.0), (['a', 'b', 'c'], 1, 1.0),

       (['a', 'b', 'c'], 2, 2.0), (['a', 'b', 'c'], 3, 3.0),

       (['a', 'b', 'c'], 4, 4.0), (['a', 'b', 'c'], 5, 5.0),

       (['a', 'b', 'c'], 6, 6.0), (['a', 'b', 'c'], 7, 7.0),

       (['a', 'b', 'c'], 8, 8.0), (['a', 'b', 'c'], 9, 9.0)],

      dtype=[('ABC', 'S5', (3,)), ('X', '>i8'), ('Y', '>f8')])

Final check of combinations:

In [*19*]: np.all(data == np.array(fits.getdata('apio.fits')))

Out[*19*]: False


In [*20*]: np.all(data == np.array(fits.getdata('fio.fits')))

Out[*20*]: False


In [*21*]: np.all(data == fitsio.read('apio.fits'))

Out[*21*]: True


In [*22*]: np.all(data == fitsio.read('fio.fits'))

Out[*22*]: False

Feature?  Bug?  User error?  Other?

Thanks for the help,

Stephen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150126/93dbd272/attachment.html>

From embray at stsci.edu  Tue Jan 27 11:13:37 2015
From: embray at stsci.edu (Erik Bray)
Date: Tue, 27 Jan 2015 11:13:37 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
Message-ID: <54C7B931.2080109@stsci.edu>

On 01/26/2015 08:50 PM, Stephen Bailey wrote:
> astropy.io.fits and fitsio aficionados,
>
> I'm trying to write a FITS binary table that includes a column that is a 2D
> array of strings.  Curiously, only files written by astropy.io.fits and read by
> fitsio pass the test of actually reconstructing the input numpy array.
>
> astropy.io.fits loses the 2D information when reading back in and ends up with a
> 1D array of null separated characters per row.
>
> fitsio.fits pads its file with spaces when reading back in.
>
> But the file written by astropy.io.fits and then read by fitsio gets me back
> what I put in.
>
> Details:
>
> import numpy as np
>
> from astropy.io <http://astropy.io> import fits
>
> import fitsio
>
>
> n = 10
>
> dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])
>
> data = np.zeros(n, dtype=dtype)
>
> data['X'] = np.arange(n)
>
> data['Y'] = np.arange(n)
>
> data['ABC'][:, 0] = 'a'
>
> data['ABC'][:, 1] = 'b'
>
> data['ABC'][:, 2] = 'c'
>
> data['ABC'][0] = ['x', 'y', 'z']
>
>
> fits.writeto('apio.fits', data)
>
> fitsio.write('fio.fits', data)
>
>
> astropy.io.fits complains:
>
> /Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
> UserWarning: TDIM1 value (5,3) does not fit with the size of the array items
> (5).  TDIM1 will be ignored.
>
>    actual_nitems, indx + 1))

... snip ...

>
> Feature?  Bug?  User error?  Other?

Maybe a little of each, but mostly a bug, I think :)  The data is being written 
correctly--the code that generates that warning is just getting confused about 
axis order when it compares the generated TDIM1 keyword to the shape of the 
data.  In other words, cross talk between different parts of the code.

This this: 
https://github.com/astropy/astropy/blob/master/astropy/io/fits/tests/test_table.py#L1704

shows an example where that won't happen because the Column objects are all 
hand-written with proper values given for the TDIMn keywords.  But in principle 
non of that should be necessary and your example should just work (and it does 
in fact write the file correctly; it just shouldn't display that warning.

 > In [*15*]: np.array(fits.getdata('apio.fits'))

If you just write fits.getdata('apio.fits'), you see that the FITS_rec structure 
returned has the correct format.  I think the fact that converting it to a 
normal ndarray drops that formatting has to do with historical cruft, but I 
think the answer there, for now, is "don't do that" :)

Erik


From erin.sheldon at gmail.com  Tue Jan 27 13:28:14 2015
From: erin.sheldon at gmail.com (Erin Sheldon)
Date: Tue, 27 Jan 2015 13:28:14 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
Message-ID: <CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>

Hi Stephen -

This is a difficult problem I think.

In FITS, the strings must get padded with spaces when written.  That is my
understanding of the format.

There is no way to know the intention of the user, whether spaces in the
original data are "significant" or not.  This information is not available in
a FITS file, only the size of the field in bytes.

The convention in many codes is to strip all trailing whitespace on reading.
If I recall correctly, IDL mrdfits does not strip.

If a user thinks the whitespace is significant they will be surprised when
reading that the spaces are not there.   If the user thinks the whitespace is
not significant they will may be surprised if it is there.

I took what I think is probably a controversial stance:  I will not lose user
data.  So I always read the full field and retain spaces.  It is up to the
user to strip them if that matters.

I am willing to reconsider based on good arguments.

An idea comes to mind: maybe support a strip_strings= keyword for the FITS
object and the reader routines.

thanks for the discussion,
-e

On 1/26/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
> astropy.io.fits and fitsio aficionados,
>
> I'm trying to write a FITS binary table that includes a column that is a 2D
> array of strings.  Curiously, only files written by astropy.io.fits and
> read by fitsio pass the test of actually reconstructing the input numpy
> array.
>
> astropy.io.fits loses the 2D information when reading back in and ends up
> with a 1D array of null separated characters per row.
>
> fitsio.fits pads its file with spaces when reading back in.
>
> But the file written by astropy.io.fits and then read by fitsio gets me
> back what I put in.
>
> Details:
>
> import numpy as np
>
> from astropy.io import fits
>
> import fitsio
>
>
> n = 10
>
> dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])
>
> data = np.zeros(n, dtype=dtype)
>
> data['X'] = np.arange(n)
>
> data['Y'] = np.arange(n)
>
> data['ABC'][:, 0] = 'a'
>
> data['ABC'][:, 1] = 'b'
>
> data['ABC'][:, 2] = 'c'
>
> data['ABC'][0] = ['x', 'y', 'z']
>
>
> fits.writeto('apio.fits', data)
>
> fitsio.write('fio.fits', data)
>
> astropy.io.fits complains:
>
> /Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
> UserWarning: TDIM1 value (5,3) does not fit with the size of the array
> items (5).  TDIM1 will be ignored.
>
>   actual_nitems, indx + 1))
>
> fitsio seems happy.  Reading it back in:
>
> In [*15*]: np.array(fits.getdata('apio.fits'))
>
> Out[*15*]:
>
> array([('x\x00\x00\x00\x00y\x00\x00\x00\x00z', 0, 0.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 1, 1.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 2, 2.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 3, 3.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 4, 4.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 5, 5.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 6, 6.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 7, 7.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 8, 8.0),
>
>        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 9, 9.0)],
>
>       dtype=[('ABC', 'S15'), ('X', '>i8'), ('Y', '>f8')])
>
>
> In [*16*]: fitsio.read('fio.fits')
>
> Out[*16*]:
>
> array([(['x    ', 'y    ', 'z    '], 0, 0.0),
>
>        (['a    ', 'b    ', 'c    '], 1, 1.0),
>
>        (['a    ', 'b    ', 'c    '], 2, 2.0),
>
>        (['a    ', 'b    ', 'c    '], 3, 3.0),
>
>        (['a    ', 'b    ', 'c    '], 4, 4.0),
>
>        (['a    ', 'b    ', 'c    '], 5, 5.0),
>
>        (['a    ', 'b    ', 'c    '], 6, 6.0),
>
>        (['a    ', 'b    ', 'c    '], 7, 7.0),
>
>        (['a    ', 'b    ', 'c    '], 8, 8.0),
>
>        (['a    ', 'b    ', 'c    '], 9, 9.0)],
>
>       dtype=[('ABC', 'S5', (3,)), ('X', '>i8'), ('Y', '>f8')])
>
>
> In [*17*]: data
>
> Out[*17*]:
>
> array([(['x', 'y', 'z'], 0, 0.0), (['a', 'b', 'c'], 1, 1.0),
>
>        (['a', 'b', 'c'], 2, 2.0), (['a', 'b', 'c'], 3, 3.0),
>
>        (['a', 'b', 'c'], 4, 4.0), (['a', 'b', 'c'], 5, 5.0),
>
>        (['a', 'b', 'c'], 6, 6.0), (['a', 'b', 'c'], 7, 7.0),
>
>        (['a', 'b', 'c'], 8, 8.0), (['a', 'b', 'c'], 9, 9.0)],
>
>       dtype=[('ABC', 'S5', (3,)), ('X', '<i8'), ('Y', '<f8')])
>
> But fitsio reading the file that astropy.io.fits wrote is good (despite the
> warning that astropy.io.fits gave when writing the file):
>
> In [*18*]: fitsio.read('apio.fits')
>
> Out[*18*]:
>
> array([(['x', 'y', 'z'], 0, 0.0), (['a', 'b', 'c'], 1, 1.0),
>
>        (['a', 'b', 'c'], 2, 2.0), (['a', 'b', 'c'], 3, 3.0),
>
>        (['a', 'b', 'c'], 4, 4.0), (['a', 'b', 'c'], 5, 5.0),
>
>        (['a', 'b', 'c'], 6, 6.0), (['a', 'b', 'c'], 7, 7.0),
>
>        (['a', 'b', 'c'], 8, 8.0), (['a', 'b', 'c'], 9, 9.0)],
>
>       dtype=[('ABC', 'S5', (3,)), ('X', '>i8'), ('Y', '>f8')])
>
> Final check of combinations:
>
> In [*19*]: np.all(data == np.array(fits.getdata('apio.fits')))
>
> Out[*19*]: False
>
>
> In [*20*]: np.all(data == np.array(fits.getdata('fio.fits')))
>
> Out[*20*]: False
>
>
> In [*21*]: np.all(data == fitsio.read('apio.fits'))
>
> Out[*21*]: True
>
>
> In [*22*]: np.all(data == fitsio.read('fio.fits'))
>
> Out[*22*]: False
>
> Feature?  Bug?  User error?  Other?
>
> Thanks for the help,
>
> Stephen
>


-- 
Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com


From stephenbailey at lbl.gov  Tue Jan 27 18:22:39 2015
From: stephenbailey at lbl.gov (Stephen Bailey)
Date: Tue, 27 Jan 2015 15:22:39 -0800
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
	<CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>
Message-ID: <CAMCHcTq8RqK_OyxdyrqiH814ikNwdJnRAQ779P_=YoDTCzOz4Q@mail.gmail.com>

On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon <erin.sheldon at gmail.com>
wrote:

> Hi Stephen -
>
> This is a difficult problem I think.
>
> In FITS, the strings must get padded with spaces when written.  That is my
> understanding of the format.
>

From
https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
:

8.3.3 Data Sequence -> 8.3.3.1 Main Data Table  -> Character

If the value of the TFORMn keyword specifies data type A, field n shall
contain a character string of  zero or more members, composed of  ASCII
text. This character string may be terminated before the length specified
by the repeat count  by  an ASCII NULL (hexadecimal code 00). Characters
after the first ASCII NULL are not defined. A string with the number of
characters specified by the repeat count is not NULL terminated. Null
strings are defined by the presence of an ASCII NULL as the first character.


This indicates that padding with NULLs is the correct thing to do when the
string is shorter than the field.  This is also how the data are in the
ndarray memory and it is distinguishable from the case of strings that end
with trailing spaces when reading it back in.

There is no way to know the intention of the user, whether spaces in the
> original data are "significant" or not.  This information is not available
> in
> a FITS file, only the size of the field in bytes.


> The convention in many codes is to strip all trailing whitespace on
> reading.
> If I recall correctly, IDL mrdfits does not strip.
>
> If a user thinks the whitespace is significant they will be surprised when
> reading that the spaces are not there.   If the user thinks the whitespace
> is
> not significant they will may be surprised if it is there.
>
> I took what I think is probably a controversial stance:  I will not lose
> user
> data.  So I always read the full field and retain spaces.  It is up to the
> user to strip them if that matters.
>

I completely agree with the philosophy of not losing user data, even when
it is a pain for the user.  But in this case, I think the standard allows
(even specifies!) to pad with NULLs instead of spaces if the input strings
are genuinely shorter than the field.

Agree?

Stephen



>
> I am willing to reconsider based on good arguments.
>
> An idea comes to mind: maybe support a strip_strings= keyword for the FITS
> object and the reader routines.
>
> thanks for the discussion,
> -e
>
> On 1/26/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
> > astropy.io.fits and fitsio aficionados,
> >
> > I'm trying to write a FITS binary table that includes a column that is a
> 2D
> > array of strings.  Curiously, only files written by astropy.io.fits and
> > read by fitsio pass the test of actually reconstructing the input numpy
> > array.
> >
> > astropy.io.fits loses the 2D information when reading back in and ends up
> > with a 1D array of null separated characters per row.
> >
> > fitsio.fits pads its file with spaces when reading back in.
> >
> > But the file written by astropy.io.fits and then read by fitsio gets me
> > back what I put in.
> >
> > Details:
> >
> > import numpy as np
> >
> > from astropy.io import fits
> >
> > import fitsio
> >
> >
> > n = 10
> >
> > dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])
> >
> > data = np.zeros(n, dtype=dtype)
> >
> > data['X'] = np.arange(n)
> >
> > data['Y'] = np.arange(n)
> >
> > data['ABC'][:, 0] = 'a'
> >
> > data['ABC'][:, 1] = 'b'
> >
> > data['ABC'][:, 2] = 'c'
> >
> > data['ABC'][0] = ['x', 'y', 'z']
> >
> >
> > fits.writeto('apio.fits', data)
> >
> > fitsio.write('fio.fits', data)
> >
> > astropy.io.fits complains:
> >
> >
> /Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
> > UserWarning: TDIM1 value (5,3) does not fit with the size of the array
> > items (5).  TDIM1 will be ignored.
> >
> >   actual_nitems, indx + 1))
> >
> > fitsio seems happy.  Reading it back in:
> >
> > In [*15*]: np.array(fits.getdata('apio.fits'))
> >
> > Out[*15*]:
> >
> > array([('x\x00\x00\x00\x00y\x00\x00\x00\x00z', 0, 0.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 1, 1.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 2, 2.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 3, 3.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 4, 4.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 5, 5.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 6, 6.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 7, 7.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 8, 8.0),
> >
> >        ('a\x00\x00\x00\x00b\x00\x00\x00\x00c', 9, 9.0)],
> >
> >       dtype=[('ABC', 'S15'), ('X', '>i8'), ('Y', '>f8')])
> >
> >
> > In [*16*]: fitsio.read('fio.fits')
> >
> > Out[*16*]:
> >
> > array([(['x    ', 'y    ', 'z    '], 0, 0.0),
> >
> >        (['a    ', 'b    ', 'c    '], 1, 1.0),
> >
> >        (['a    ', 'b    ', 'c    '], 2, 2.0),
> >
> >        (['a    ', 'b    ', 'c    '], 3, 3.0),
> >
> >        (['a    ', 'b    ', 'c    '], 4, 4.0),
> >
> >        (['a    ', 'b    ', 'c    '], 5, 5.0),
> >
> >        (['a    ', 'b    ', 'c    '], 6, 6.0),
> >
> >        (['a    ', 'b    ', 'c    '], 7, 7.0),
> >
> >        (['a    ', 'b    ', 'c    '], 8, 8.0),
> >
> >        (['a    ', 'b    ', 'c    '], 9, 9.0)],
> >
> >       dtype=[('ABC', 'S5', (3,)), ('X', '>i8'), ('Y', '>f8')])
> >
> >
> > In [*17*]: data
> >
> > Out[*17*]:
> >
> > array([(['x', 'y', 'z'], 0, 0.0), (['a', 'b', 'c'], 1, 1.0),
> >
> >        (['a', 'b', 'c'], 2, 2.0), (['a', 'b', 'c'], 3, 3.0),
> >
> >        (['a', 'b', 'c'], 4, 4.0), (['a', 'b', 'c'], 5, 5.0),
> >
> >        (['a', 'b', 'c'], 6, 6.0), (['a', 'b', 'c'], 7, 7.0),
> >
> >        (['a', 'b', 'c'], 8, 8.0), (['a', 'b', 'c'], 9, 9.0)],
> >
> >       dtype=[('ABC', 'S5', (3,)), ('X', '<i8'), ('Y', '<f8')])
> >
> > But fitsio reading the file that astropy.io.fits wrote is good (despite
> the
> > warning that astropy.io.fits gave when writing the file):
> >
> > In [*18*]: fitsio.read('apio.fits')
> >
> > Out[*18*]:
> >
> > array([(['x', 'y', 'z'], 0, 0.0), (['a', 'b', 'c'], 1, 1.0),
> >
> >        (['a', 'b', 'c'], 2, 2.0), (['a', 'b', 'c'], 3, 3.0),
> >
> >        (['a', 'b', 'c'], 4, 4.0), (['a', 'b', 'c'], 5, 5.0),
> >
> >        (['a', 'b', 'c'], 6, 6.0), (['a', 'b', 'c'], 7, 7.0),
> >
> >        (['a', 'b', 'c'], 8, 8.0), (['a', 'b', 'c'], 9, 9.0)],
> >
> >       dtype=[('ABC', 'S5', (3,)), ('X', '>i8'), ('Y', '>f8')])
> >
> > Final check of combinations:
> >
> > In [*19*]: np.all(data == np.array(fits.getdata('apio.fits')))
> >
> > Out[*19*]: False
> >
> >
> > In [*20*]: np.all(data == np.array(fits.getdata('fio.fits')))
> >
> > Out[*20*]: False
> >
> >
> > In [*21*]: np.all(data == fitsio.read('apio.fits'))
> >
> > Out[*21*]: True
> >
> >
> > In [*22*]: np.all(data == fitsio.read('fio.fits'))
> >
> > Out[*22*]: False
> >
> > Feature?  Bug?  User error?  Other?
> >
> > Thanks for the help,
> >
> > Stephen
> >
>
>
> --
> Erin Scott Sheldon
> Brookhaven National Laboratory erin dot sheldon at gmail dot com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150127/e3425001/attachment.html>

From stephenbailey at lbl.gov  Tue Jan 27 18:27:45 2015
From: stephenbailey at lbl.gov (Stephen Bailey)
Date: Tue, 27 Jan 2015 15:27:45 -0800
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <54C7B931.2080109@stsci.edu>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
	<54C7B931.2080109@stsci.edu>
Message-ID: <CAMCHcTog0CL4GzNGpmKmcyokp6D7FWH7-D+=hAk3QKXuad6unw@mail.gmail.com>

On Tue, Jan 27, 2015 at 8:13 AM, Erik Bray <embray at stsci.edu> wrote:

> On 01/26/2015 08:50 PM, Stephen Bailey wrote:
> > astropy.io.fits and fitsio aficionados,
> >
> > I'm trying to write a FITS binary table that includes a column that is a
> 2D
> > array of strings.  Curiously, only files written by astropy.io.fits and
> read by
> > fitsio pass the test of actually reconstructing the input numpy array.
> >
> > astropy.io.fits loses the 2D information when reading back in and ends
> up with a
> > 1D array of null separated characters per row.
> >
> > fitsio.fits pads its file with spaces when reading back in.
> >
> > But the file written by astropy.io.fits and then read by fitsio gets me
> back
> > what I put in.
> >
> > Details:
> >
> > import numpy as np
> >
> > from astropy.io <http://astropy.io> import fits
> >
> > import fitsio
> >
> >
> > n = 10
> >
> > dtype = ([('ABC', 'S5', (3,)), ('X', int), ('Y', float)])
> >
> > data = np.zeros(n, dtype=dtype)
> >
> > data['X'] = np.arange(n)
> >
> > data['Y'] = np.arange(n)
> >
> > data['ABC'][:, 0] = 'a'
> >
> > data['ABC'][:, 1] = 'b'
> >
> > data['ABC'][:, 2] = 'c'
> >
> > data['ABC'][0] = ['x', 'y', 'z']
> >
> >
> > fits.writeto('apio.fits', data)
> >
> > fitsio.write('fio.fits', data)
> >
> >
> > astropy.io.fits complains:
> >
> >
> /Users/sbailey/anaconda/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py:782:
> > UserWarning: TDIM1 value (5,3) does not fit with the size of the array
> items
> > (5).  TDIM1 will be ignored.
> >
> >    actual_nitems, indx + 1))
>
> ... snip ...
>
> >
> > Feature?  Bug?  User error?  Other?
>
> Maybe a little of each, but mostly a bug, I think :)  The data is being
> written
> correctly--the code that generates that warning is just getting confused
> about
> axis order when it compares the generated TDIM1 keyword to the shape of the
> data.  In other words, cross talk between different parts of the code.
>
> This this:
>
> https://github.com/astropy/astropy/blob/master/astropy/io/fits/tests/test_table.py#L1704
>
> shows an example where that won't happen because the Column objects are all
> hand-written with proper values given for the TDIMn keywords.  But in
> principle
> non of that should be necessary and your example should just work (and it
> does
> in fact write the file correctly; it just shouldn't display that warning.
>
>  > In [*15*]: np.array(fits.getdata('apio.fits'))
>
> If you just write fits.getdata('apio.fits'), you see that the FITS_rec
> structure
> returned has the correct format.  I think the fact that converting it to a
> normal ndarray drops that formatting has to do with historical cruft, but I
> think the answer there, for now, is "don't do that" :)
>

Alas, I always prefer to get back to vanilla ndarray objects as quickly as
possible when doing I/O so that downstream algorithm code can rely on
ndarray only, not on the particular I/O format from whence it came.

>From your description, I think there are two separate astropy bug reports
brewing here:
1. cosmetic: astropy.io.fits.writeto() shouldn't issue that warning since
it actually is writing the file correctly
2. important: the conversion from the FITS_rec structure to the np.array is
not converting the chararray() portion correctly to maintain the 2D nature
of the string array.

Regards,

Stephen



>
> Erik
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150127/dbf7c774/attachment.html>

From erin.sheldon at gmail.com  Tue Jan 27 18:45:09 2015
From: erin.sheldon at gmail.com (Erin Sheldon)
Date: Tue, 27 Jan 2015 18:45:09 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAMCHcTq8RqK_OyxdyrqiH814ikNwdJnRAQ779P_=YoDTCzOz4Q@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
	<CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>
	<CAMCHcTq8RqK_OyxdyrqiH814ikNwdJnRAQ779P_=YoDTCzOz4Q@mail.gmail.com>
Message-ID: <CABOnLw1SSjnv0zqckXAk_S2P9yCwV6BJan-4fAz0RPcy7XA8Aw@mail.gmail.com>

On 1/27/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
> On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon <erin.sheldon at gmail.com>
> wrote:
>
>> Hi Stephen -
>>
>> This is a difficult problem I think.
>>
>> In FITS, the strings must get padded with spaces when written.  That is
>> my
>> understanding of the format.
>>
>
> From
> https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
> :
>
> 8.3.3 Data Sequence -> 8.3.3.1 Main Data Table  -> Character
>
> If the value of the TFORMn keyword specifies data type A, field n shall
> contain a character string of  zero or more members, composed of  ASCII
> text. This character string may be terminated before the length specified
> by the repeat count  by  an ASCII NULL (hexadecimal code 00). Characters
> after the first ASCII NULL are not defined. A string with the number of
> characters specified by the repeat count is not NULL terminated. Null
> strings are defined by the presence of an ASCII NULL as the first
> character.
>
>
> This indicates that padding with NULLs is the correct thing to do when the
> string is shorter than the field.  This is also how the data are in the
> ndarray memory and it is distinguishable from the case of strings that end
> with trailing spaces when reading it back in.

Hi Stephen -

Interesting, compare this to the cfitsio manual

http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html

So this may be cfitsio specific, and perhaps something we should work around
in fitsio.

-e

-- 
Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com


From stephenbailey at lbl.gov  Tue Jan 27 19:29:20 2015
From: stephenbailey at lbl.gov (Stephen Bailey)
Date: Tue, 27 Jan 2015 16:29:20 -0800
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CABOnLw1SSjnv0zqckXAk_S2P9yCwV6BJan-4fAz0RPcy7XA8Aw@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
	<CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>
	<CAMCHcTq8RqK_OyxdyrqiH814ikNwdJnRAQ779P_=YoDTCzOz4Q@mail.gmail.com>
	<CABOnLw1SSjnv0zqckXAk_S2P9yCwV6BJan-4fAz0RPcy7XA8Aw@mail.gmail.com>
Message-ID: <CAMCHcTogdZWU3e_SJgBuuZ6Cjs6EVXk2JSCyt3jQTFB0ADUKcg@mail.gmail.com>

On Tue, Jan 27, 2015 at 3:45 PM, Erin Sheldon <erin.sheldon at gmail.com>
wrote:

> On 1/27/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
> > On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon <erin.sheldon at gmail.com>
> > wrote:
> >
> >> Hi Stephen -
> >>
> >> This is a difficult problem I think.
> >>
> >> In FITS, the strings must get padded with spaces when written.  That is
> >> my
> >> understanding of the format.
> >>
> >
> > From
> >
> https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
> > :
> >
> > 8.3.3 Data Sequence -> 8.3.3.1 Main Data Table  -> Character
> >
> > If the value of the TFORMn keyword specifies data type A, field n shall
> > contain a character string of  zero or more members, composed of  ASCII
> > text. This character string may be terminated before the length specified
> > by the repeat count  by  an ASCII NULL (hexadecimal code 00). Characters
> > after the first ASCII NULL are not defined. A string with the number of
> > characters specified by the repeat count is not NULL terminated. Null
> > strings are defined by the presence of an ASCII NULL as the first
> > character.
> >
> >
> > This indicates that padding with NULLs is the correct thing to do when
> the
> > string is shorter than the field.  This is also how the data are in the
> > ndarray memory and it is distinguishable from the case of strings that
> end
> > with trailing spaces when reading it back in.
>
> Hi Stephen -
>
> Interesting, compare this to the cfitsio manual
>
> http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html
>
> So this may be cfitsio specific, and perhaps something we should work
> around
> in fitsio.
>

Gack.  My interpretation of that is that cfitsio is violating the FITS
standard by doing something crazy with padding and stripping spaces.  If
you have some way of working around that in fitsio, that would be great.

Stephen


>
> -e
>
> --
> Erin Scott Sheldon
> Brookhaven National Laboratory erin dot sheldon at gmail dot com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150127/3faac60c/attachment.html>

From benjamin.weaver at nyu.edu  Tue Jan 27 19:32:50 2015
From: benjamin.weaver at nyu.edu (Benjamin Alan Weaver)
Date: Tue, 27 Jan 2015 19:32:50 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
Message-ID: <CAK8F6Hrwb-LW2TOsQyBJVo5kBWkeiuJ5D4jeVz_fUqZm2ud-4A@mail.gmail.com>

Hello y'all,

> https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000

The date at the bottom says 1999.  The FITS 3.0 standard from
2008-2010 appears to be more consistent with how cfitsio does it (see
http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf):

Character fields. The value of a character-formatted (Aw) field is a
character string of width w containing the characters in columns
TBCOLn through TBCOLn+w ? 1. The character string shall be composed of
the restricted set of ASCII text characters with decimal values in the
range 32 through 126 (hexadecimal 20 through 7E).

Note the restriction to decimal values > 0.  This would appear to rule
out ASCII NULL characters.

Kia ora koutou,
Benjamin Alan Weaver

On Tue, Jan 27, 2015 at 7:29 PM, Stephen Bailey <stephenbailey at lbl.gov> wrote:
>
> On Tue, Jan 27, 2015 at 3:45 PM, Erin Sheldon <erin.sheldon at gmail.com>
> wrote:
>>
>> On 1/27/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
>> > On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon <erin.sheldon at gmail.com>
>> > wrote:
>> >
>> >> Hi Stephen -
>> >>
>> >> This is a difficult problem I think.
>> >>
>> >> In FITS, the strings must get padded with spaces when written.  That is
>> >> my
>> >> understanding of the format.
>> >>
>> >
>> > From
>> >
>> > https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
>> > :
>> >
>> > 8.3.3 Data Sequence -> 8.3.3.1 Main Data Table  -> Character
>> >
>> > If the value of the TFORMn keyword specifies data type A, field n shall
>> > contain a character string of  zero or more members, composed of  ASCII
>> > text. This character string may be terminated before the length
>> > specified
>> > by the repeat count  by  an ASCII NULL (hexadecimal code 00). Characters
>> > after the first ASCII NULL are not defined. A string with the number of
>> > characters specified by the repeat count is not NULL terminated. Null
>> > strings are defined by the presence of an ASCII NULL as the first
>> > character.
>> >
>> >
>> > This indicates that padding with NULLs is the correct thing to do when
>> > the
>> > string is shorter than the field.  This is also how the data are in the
>> > ndarray memory and it is distinguishable from the case of strings that
>> > end
>> > with trailing spaces when reading it back in.
>>
>> Hi Stephen -
>>
>> Interesting, compare this to the cfitsio manual
>>
>> http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html
>>
>> So this may be cfitsio specific, and perhaps something we should work
>> around
>> in fitsio.
>
>
> Gack.  My interpretation of that is that cfitsio is violating the FITS
> standard by doing something crazy with padding and stripping spaces.  If you
> have some way of working around that in fitsio, that would be great.
>
> Stephen
>
>>
>>
>> -e
>>
>> --
>> Erin Scott Sheldon
>> Brookhaven National Laboratory erin dot sheldon at gmail dot com
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



-- 
a.k.a. The Dream Weaver

Outside of a dog, a book is man's best friend. Inside of a dog it's
too dark to read.
  --Groucho Marx


From benjamin.weaver at nyu.edu  Tue Jan 27 19:37:23 2015
From: benjamin.weaver at nyu.edu (Benjamin Alan Weaver)
Date: Tue, 27 Jan 2015 19:37:23 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAK8F6Hrwb-LW2TOsQyBJVo5kBWkeiuJ5D4jeVz_fUqZm2ud-4A@mail.gmail.com>
References: <CAK8F6Hrwb-LW2TOsQyBJVo5kBWkeiuJ5D4jeVz_fUqZm2ud-4A@mail.gmail.com>
Message-ID: <CAK8F6Hq_AWgEggRSid+aJJE4fBC6EKH6WsiUO7vfao+=guaAsw@mail.gmail.com>

Hello y'all,

Please disregard the previous email.  That applies to ASCII tables not
Binary tables.

Ugh, why do they list ASCII tables first?

Kia ora koutou,
Benjamin Alan Weaver


On Tue, Jan 27, 2015 at 7:32 PM, Benjamin Alan Weaver
<benjamin.weaver at nyu.edu> wrote:
> Hello y'all,
>
>> https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
>
> The date at the bottom says 1999.  The FITS 3.0 standard from
> 2008-2010 appears to be more consistent with how cfitsio does it (see
> http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf):
>
> Character fields. The value of a character-formatted (Aw) field is a
> character string of width w containing the characters in columns
> TBCOLn through TBCOLn+w ? 1. The character string shall be composed of
> the restricted set of ASCII text characters with decimal values in the
> range 32 through 126 (hexadecimal 20 through 7E).
>
> Note the restriction to decimal values > 0.  This would appear to rule
> out ASCII NULL characters.
>
> Kia ora koutou,
> Benjamin Alan Weaver
>
> On Tue, Jan 27, 2015 at 7:29 PM, Stephen Bailey <stephenbailey at lbl.gov> wrote:
>>
>> On Tue, Jan 27, 2015 at 3:45 PM, Erin Sheldon <erin.sheldon at gmail.com>
>> wrote:
>>>
>>> On 1/27/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
>>> > On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon <erin.sheldon at gmail.com>
>>> > wrote:
>>> >
>>> >> Hi Stephen -
>>> >>
>>> >> This is a difficult problem I think.
>>> >>
>>> >> In FITS, the strings must get padded with spaces when written.  That is
>>> >> my
>>> >> understanding of the format.
>>> >>
>>> >
>>> > From
>>> >
>>> > https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
>>> > :
>>> >
>>> > 8.3.3 Data Sequence -> 8.3.3.1 Main Data Table  -> Character
>>> >
>>> > If the value of the TFORMn keyword specifies data type A, field n shall
>>> > contain a character string of  zero or more members, composed of  ASCII
>>> > text. This character string may be terminated before the length
>>> > specified
>>> > by the repeat count  by  an ASCII NULL (hexadecimal code 00). Characters
>>> > after the first ASCII NULL are not defined. A string with the number of
>>> > characters specified by the repeat count is not NULL terminated. Null
>>> > strings are defined by the presence of an ASCII NULL as the first
>>> > character.
>>> >
>>> >
>>> > This indicates that padding with NULLs is the correct thing to do when
>>> > the
>>> > string is shorter than the field.  This is also how the data are in the
>>> > ndarray memory and it is distinguishable from the case of strings that
>>> > end
>>> > with trailing spaces when reading it back in.
>>>
>>> Hi Stephen -
>>>
>>> Interesting, compare this to the cfitsio manual
>>>
>>> http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html
>>>
>>> So this may be cfitsio specific, and perhaps something we should work
>>> around
>>> in fitsio.
>>
>>
>> Gack.  My interpretation of that is that cfitsio is violating the FITS
>> standard by doing something crazy with padding and stripping spaces.  If you
>> have some way of working around that in fitsio, that would be great.
>>
>> Stephen
>>
>>>
>>>
>>> -e
>>>
>>> --
>>> Erin Scott Sheldon
>>> Brookhaven National Laboratory erin dot sheldon at gmail dot com
>>
>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at scipy.org
>> http://mail.scipy.org/mailman/listinfo/astropy
>>
>
>
>
> --
> a.k.a. The Dream Weaver
>
> Outside of a dog, a book is man's best friend. Inside of a dog it's
> too dark to read.
>   --Groucho Marx



-- 
a.k.a. The Dream Weaver

Outside of a dog, a book is man's best friend. Inside of a dog it's
too dark to read.
  --Groucho Marx


From npkuin at gmail.com  Tue Jan 27 21:59:21 2015
From: npkuin at gmail.com (Paul Kuin)
Date: Wed, 28 Jan 2015 02:59:21 +0000
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAK8F6Hq_AWgEggRSid+aJJE4fBC6EKH6WsiUO7vfao+=guaAsw@mail.gmail.com>
References: <CAK8F6Hrwb-LW2TOsQyBJVo5kBWkeiuJ5D4jeVz_fUqZm2ud-4A@mail.gmail.com>
	<CAK8F6Hq_AWgEggRSid+aJJE4fBC6EKH6WsiUO7vfao+=guaAsw@mail.gmail.com>
Message-ID: <CANoQ6N2Uu31T1yM3oP7NQdy=So_FDm=Ns-Bm4rtcm9C0DHV=2A@mail.gmail.com>

Hi Stephen, all,

You can ask Bill Pence, who's member of the IAU FITS working group.

fits @ fits.gsfc.nasa.gov

There might be a need somewhere to preserve the original ascii record,
(spaces and all) and to do that padding was to be done with a special
character. I think it would be smart to ask Bill Pence for clarification on
this issue.

- Paul

On Wed, Jan 28, 2015 at 12:37 AM, Benjamin Alan Weaver <
benjamin.weaver at nyu.edu> wrote:

> Hello y'all,
>
> Please disregard the previous email.  That applies to ASCII tables not
> Binary tables.
>
> Ugh, why do they list ASCII tables first?
>
> Kia ora koutou,
> Benjamin Alan Weaver
>
>
> On Tue, Jan 27, 2015 at 7:32 PM, Benjamin Alan Weaver
> <benjamin.weaver at nyu.edu> wrote:
> > Hello y'all,
> >
> >>
> https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
> >
> > The date at the bottom says 1999.  The FITS 3.0 standard from
> > 2008-2010 appears to be more consistent with how cfitsio does it (see
> > http://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf):
> >
> > Character fields. The value of a character-formatted (Aw) field is a
> > character string of width w containing the characters in columns
> > TBCOLn through TBCOLn+w ? 1. The character string shall be composed of
> > the restricted set of ASCII text characters with decimal values in the
> > range 32 through 126 (hexadecimal 20 through 7E).
> >
> > Note the restriction to decimal values > 0.  This would appear to rule
> > out ASCII NULL characters.
> >
> > Kia ora koutou,
> > Benjamin Alan Weaver
> >
> > On Tue, Jan 27, 2015 at 7:29 PM, Stephen Bailey <stephenbailey at lbl.gov>
> wrote:
> >>
> >> On Tue, Jan 27, 2015 at 3:45 PM, Erin Sheldon <erin.sheldon at gmail.com>
> >> wrote:
> >>>
> >>> On 1/27/15, Stephen Bailey <stephenbailey at lbl.gov> wrote:
> >>> > On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon <
> erin.sheldon at gmail.com>
> >>> > wrote:
> >>> >
> >>> >> Hi Stephen -
> >>> >>
> >>> >> This is a difficult problem I think.
> >>> >>
> >>> >> In FITS, the strings must get padded with spaces when written.
> That is
> >>> >> my
> >>> >> understanding of the format.
> >>> >>
> >>> >
> >>> > From
> >>> >
> >>> >
> https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
> >>> > :
> >>> >
> >>> > 8.3.3 Data Sequence -> 8.3.3.1 Main Data Table  -> Character
> >>> >
> >>> > If the value of the TFORMn keyword specifies data type A, field n
> shall
> >>> > contain a character string of  zero or more members, composed of
> ASCII
> >>> > text. This character string may be terminated before the length
> >>> > specified
> >>> > by the repeat count  by  an ASCII NULL (hexadecimal code 00).
> Characters
> >>> > after the first ASCII NULL are not defined. A string with the number
> of
> >>> > characters specified by the repeat count is not NULL terminated. Null
> >>> > strings are defined by the presence of an ASCII NULL as the first
> >>> > character.
> >>> >
> >>> >
> >>> > This indicates that padding with NULLs is the correct thing to do
> when
> >>> > the
> >>> > string is shorter than the field.  This is also how the data are in
> the
> >>> > ndarray memory and it is distinguishable from the case of strings
> that
> >>> > end
> >>> > with trailing spaces when reading it back in.
> >>>
> >>> Hi Stephen -
> >>>
> >>> Interesting, compare this to the cfitsio manual
> >>>
> >>> http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html
> >>>
> >>> So this may be cfitsio specific, and perhaps something we should work
> >>> around
> >>> in fitsio.
> >>
> >>
> >> Gack.  My interpretation of that is that cfitsio is violating the FITS
> >> standard by doing something crazy with padding and stripping spaces.
> If you
> >> have some way of working around that in fitsio, that would be great.
> >>
> >> Stephen
> >>
> >>>
> >>>
> >>> -e
> >>>
> >>> --
> >>> Erin Scott Sheldon
> >>> Brookhaven National Laboratory erin dot sheldon at gmail dot com
> >>
> >>
> >>
> >> _______________________________________________
> >> AstroPy mailing list
> >> AstroPy at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/astropy
> >>
> >
> >
> >
> > --
> > a.k.a. The Dream Weaver
> >
> > Outside of a dog, a book is man's best friend. Inside of a dog it's
> > too dark to read.
> >   --Groucho Marx
>
>
>
> --
> a.k.a. The Dream Weaver
>
> Outside of a dog, a book is man's best friend. Inside of a dog it's
> too dark to read.
>   --Groucho Marx
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>



-- 

* * * * * * * * http://www.mssl.ucl.ac.uk/~npmk/ * * * *
Dr. N.P.M. Kuin      (n.kuin at ucl.ac.uk)
phone +44-(0)1483 (prefix) -204927 (work)
mobile +44(0)7806985366  skype ID: npkuin
Mullard Space Science Laboratory  ? University College London  ?
Holmbury St Mary ? Dorking ? Surrey RH5 6NT?  U.K.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150128/ef9dfb1b/attachment.html>

From bgarwood at nrao.edu  Tue Jan 27 22:27:34 2015
From: bgarwood at nrao.edu (Bob Garwood)
Date: Tue, 27 Jan 2015 22:27:34 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAMCHcTogdZWU3e_SJgBuuZ6Cjs6EVXk2JSCyt3jQTFB0ADUKcg@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>	<CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>	<CAMCHcTq8RqK_OyxdyrqiH814ikNwdJnRAQ779P_=YoDTCzOz4Q@mail.gmail.com>	<CABOnLw1SSjnv0zqckXAk_S2P9yCwV6BJan-4fAz0RPcy7XA8Aw@mail.gmail.com>
	<CAMCHcTogdZWU3e_SJgBuuZ6Cjs6EVXk2JSCyt3jQTFB0ADUKcg@mail.gmail.com>
Message-ID: <54C85726.8010600@nrao.edu>

On 1/27/2015 7:29 PM, Stephen Bailey wrote:
>
> On Tue, Jan 27, 2015 at 3:45 PM, Erin Sheldon <erin.sheldon at gmail.com 
> <mailto:erin.sheldon at gmail.com>> wrote:
>
>     On 1/27/15, Stephen Bailey <stephenbailey at lbl.gov
>     <mailto:stephenbailey at lbl.gov>> wrote:
>     > On Tue, Jan 27, 2015 at 10:28 AM, Erin Sheldon
>     <erin.sheldon at gmail.com <mailto:erin.sheldon at gmail.com>>
>     > wrote:
>     >
>     >> Hi Stephen -
>     >>
>     >> This is a difficult problem I think.
>     >>
>     >> In FITS, the strings must get padded with spaces when written. 
>     That is
>     >> my
>     >> understanding of the format.
>     >>
>     >
>     > From
>     >
>     https://archive.stsci.edu/fits/fits_standard/node70.html#SECTION001233130000000000000
>     > :
>     >
>     > 8.3.3 Data Sequence -> 8.3.3.1 Main Data Table -> Character
>     >
>     > If the value of the TFORMn keyword specifies data type A, field
>     n shall
>     > contain a character string of  zero or more members, composed
>     of  ASCII
>     > text. This character string may be terminated before the length
>     specified
>     > by the repeat count  by  an ASCII NULL (hexadecimal code 00).
>     Characters
>     > after the first ASCII NULL are not defined. A string with the
>     number of
>     > characters specified by the repeat count is not NULL terminated.
>     Null
>     > strings are defined by the presence of an ASCII NULL as the first
>     > character.
>     >
>     >
>     > This indicates that padding with NULLs is the correct thing to
>     do when the
>     > string is shorter than the field.  This is also how the data are
>     in the
>     > ndarray memory and it is distinguishable from the case of
>     strings that end
>     > with trailing spaces when reading it back in.
>
>     Hi Stephen -
>
>     Interesting, compare this to the cfitsio manual
>
>     http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html
>
>     So this may be cfitsio specific, and perhaps something we should
>     work around
>     in fitsio.
>
>
> Gack.  My interpretation of that is that cfitsio is violating the FITS 
> standard by doing something crazy with padding and stripping spaces.  
> If you have some way of working around that in fitsio, that would be 
> great.

My interpretation of this section of the cfitsio manual is that this 
only applies to header values and ascii tables.  I think there are no 
such limits for binary tables (but the documentation isn't quite 
explicit in that regards and I haven't tested it to see).

Bob

>
> Stephen
>
>
>     -e
>
>     --
>     Erin Scott Sheldon
>     Brookhaven National Laboratory erin dot sheldon at gmail dot com
>
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150127/b1cbe4a5/attachment.html>

From erin.sheldon at gmail.com  Tue Jan 27 22:38:42 2015
From: erin.sheldon at gmail.com (Erin Sheldon)
Date: Tue, 27 Jan 2015 22:38:42 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <54C85726.8010600@nrao.edu>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>
	<CABOnLw1o+D0tpu=pzVkD+TqDMPD=Hs94b8Xtpyey=K2Crx=n2g@mail.gmail.com>
	<CAMCHcTq8RqK_OyxdyrqiH814ikNwdJnRAQ779P_=YoDTCzOz4Q@mail.gmail.com>
	<CABOnLw1SSjnv0zqckXAk_S2P9yCwV6BJan-4fAz0RPcy7XA8Aw@mail.gmail.com>
	<CAMCHcTogdZWU3e_SJgBuuZ6Cjs6EVXk2JSCyt3jQTFB0ADUKcg@mail.gmail.com>
	<54C85726.8010600@nrao.edu>
Message-ID: <CABOnLw1nWzMUsR7kMoDMbqxqt4xHs3Bhnw+S=RfLjJ+MW18pHg@mail.gmail.com>

On 1/27/15, Bob Garwood <bgarwood at nrao.edu> wrote:
>>     Hi Stephen -
>>
>>     Interesting, compare this to the cfitsio manual
>>
>>     http://heasarc.nasa.gov/docs/software/fitsio/c/c_user/node24.html
>>
>>     So this may be cfitsio specific, and perhaps something we should
>>     work around
>>     in fitsio.
>>
>>
>> Gack.  My interpretation of that is that cfitsio is violating the FITS
>> standard by doing something crazy with padding and stripping spaces.
>> If you have some way of working around that in fitsio, that would be
>> great.
>
> My interpretation of this section of the cfitsio manual is that this
> only applies to header values and ascii tables.  I think there are no
> such limits for binary tables (but the documentation isn't quite
> explicit in that regards and I haven't tested it to see).

It certainly applies to binary tables too; that section is for all tables.  It
does throw the word ASCII  around a lot, but that is referring to the data.

I agree discussion with Bill Pence is a good idea.


It is a simple change to the CFITSIO code to get the behavior we want:  pad
with nulls instead of spaces.  I just did this in the fitsio codebase (which
embeds cfitsio) in a one line change and it works as expected.

-e

-- 
Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com


From embray at stsci.edu  Wed Jan 28 11:14:59 2015
From: embray at stsci.edu (Erik Bray)
Date: Wed, 28 Jan 2015 11:14:59 -0500
Subject: [AstroPy] writing 2D string column to fits binary table
In-Reply-To: <CAMCHcTog0CL4GzNGpmKmcyokp6D7FWH7-D+=hAk3QKXuad6unw@mail.gmail.com>
References: <CAMCHcToojt_UgzzyS=8xJA0sRf1sAVVk37-yTt_8Q9CgWSXt-w@mail.gmail.com>	<54C7B931.2080109@stsci.edu>
	<CAMCHcTog0CL4GzNGpmKmcyokp6D7FWH7-D+=hAk3QKXuad6unw@mail.gmail.com>
Message-ID: <54C90B03.8060203@stsci.edu>

On 01/27/2015 06:27 PM, Stephen Bailey wrote:
> Alas, I always prefer to get back to vanilla ndarray objects as quickly as
> possible when doing I/O so that downstream algorithm code can rely on ndarray
> only, not on the particular I/O format from whence it came.

Understandable.

>  From your description, I think there are two separate astropy bug reports
> brewing here:
> 1. cosmetic: astropy.io.fits.writeto() shouldn't issue that warning since it
> actually is writing the file correctly

Easy to fix.

> 2. important: the conversion from the FITS_rec structure to the np.array is not
> converting the chararray() portion correctly to maintain the 2D nature of the
> string array.

Less easy, believe it or not.  There's not a lot of control over this without 
changing the data format.  The FITS_rec structure is extremely long in the tooth 
though and has long been planned for replacement, likely with the Astropy Table 
class.  I've been tasked with that work but it has been on the backburner behind 
other priorities.

Erik



From erik.tollerud at gmail.com  Thu Jan 29 15:49:03 2015
From: erik.tollerud at gmail.com (Erik Tollerud)
Date: Thu, 29 Jan 2015 15:49:03 -0500
Subject: [AstroPy] Testing release candidate for astropy v1.0
Message-ID: <CAARXUwVJLwF_efOxm=FQ0EZ5RRZ5WPh8mY8OMESfWuKBvUPLpQ@mail.gmail.com>

Hello all,

The astropy v1.0 release candidate is available for testing on pypi:
https://pypi.python.org/pypi/astropy/1.0rc1

Please help us test if you can! You can install it with pip by doing "pip
install --pre astropy".  If you're using an anaconda distribution, you'll
either need to "conda remove astropy" or make a new environment before
installing the rc from pip.


Documentation is available at:
http://astropy.readthedocs.org/en/v1.0rc1/

Of particular note is the what's new section for 1.0:
http://astropy.readthedocs.org/en/v1.0rc1/whatsnew/1.0.html

Assuming no major blocking bugs are found in the RC, we will aim to do the
final 1.0 release in about a week.


-- 
Erik T
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150129/c1e07131/attachment.html>

From trive at astro.su.se  Thu Jan 29 17:23:03 2015
From: trive at astro.su.se (=?UTF-8?B?VGjDuGdlciBFbWlsICBSaXZlcmEtVGhvcnNlbg==?=)
Date: Thu, 29 Jan 2015 23:23:03 +0100
Subject: [AstroPy] Testing release candidate for astropy v1.0
In-Reply-To: <CAARXUwVJLwF_efOxm=FQ0EZ5RRZ5WPh8mY8OMESfWuKBvUPLpQ@mail.gmail.com>
References: <CAARXUwVJLwF_efOxm=FQ0EZ5RRZ5WPh8mY8OMESfWuKBvUPLpQ@mail.gmail.com>
Message-ID: <54CAB2C7.4010302@astro.su.se>

Nice and impressive work!

On 01/29/2015 09:49 PM, Erik Tollerud wrote:
> Hello all,
>
> The astropy v1.0 release candidate is available for testing on pypi: 
> https://pypi.python.org/pypi/astropy/1.0rc1
>
> Please help us test if you can! You can install it with pip by doing
> "pip install --pre astropy".  If you're using an anaconda
> distribution, you'll either need to "conda remove astropy" or make a
> new environment before installing the rc from pip. 
>
>
> Documentation is available at:
> http://astropy.readthedocs.org/en/v1.0rc1/
>
> Of particular note is the what's new section for 1.0:
> http://astropy.readthedocs.org/en/v1.0rc1/whatsnew/1.0.html
>
> Assuming no major blocking bugs are found in the RC, we will aim to do
> the final 1.0 release in about a week.
>
>
> -- 
> Erik T
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150129/d467ab7c/attachment.html>