[AstroPy] Question re. astropy.table

[Aldcroft, Thomas]

Hi Brian,

I fixed the original code in the github thread and it works on your example
(also copied below).

- Tom

from astropy.io import ascii
import os

def read_chunks(filename, n_chunk, format='basic'):
    lines = []
    for i, line in enumerate(open(filename, 'r')):
        lines.append(line.rstrip(os.linesep))
        if i % n_chunk == n_chunk - 1:
            if i < n_chunk:  # first
                chunk = ascii.read(lines, format=format, guess=False)
                names = chunk.colnames
                yield chunk
            else:
                chunk = ascii.read(lines, format='no_header', names=names,
guess=False)
                yield chunk
            lines = []
    if lines:
        yield ascii.read(lines, format='no_header', names=names,
guess=False)

for dat in read_chunks('ipac.dat', 20, format='ipac'):
    print(dat)


On Fri, Feb 26, 2016 at 12:51 PM, Brian York <york at stsci.edu> wrote:

> On 2016/02/25 18:59 , "Aldcroft, Thomas" <aldcroft at head.cfa.harvard.edu>
> wrote:
>
> >Hi Brian,
> >
> >
> >If your input file an ASCII data table then please see:
> >   https://github.com/astropy/astropy/issues/3334
> >
> >
> >
> >There is a code snippet there that should work to read in a large table
> >in chunks.
>
> I've tried this, but I keep running into issues. The table is an ASCII
> IPAC format, and running the above code (with format='ipac') always gets
> me the following exception:
>
> Traceback (most recent call last):
>   File
> "/Users/york/ssbvirt/ssbdev-osx/lib/python2.7/site-packages/celery/app/trac
> e.py", line 240, in trace_task
>     R = retval = fun(*args, **kwargs)
>   File "/Users/york/JWST/pandeia/simulator.py", line 87, in __call__
>     return TaskBase.__call__(self, *args, **kwargs)
>   File
> "/Users/york/ssbvirt/ssbdev-osx/lib/python2.7/site-packages/celery/app/trac
> e.py", line 438, in __protected_call__
>     return self.run(*args, **kwargs)
>   File "/Users/york/JWST/pandeia/simulator.py", line 173, in observation
>     observed_catalogues.extend(obs.addCatalogue(cat))
>   File
> "/Users/york/JWST/pandeia/stips/stips/observation_module/observation_module
> .py", line 184, in addCatalogue
>     cats = self.instrument.addCatalogue(catalogue)
>   File "/Users/york/JWST/pandeia/stips/stips/instruments/instrument.py",
> line 218, in addCatalogue
>     cats.append(detector.addCatalogue(cat))
>   File "/Users/york/JWST/pandeia/stips/stips/astro_image/astro_image.py",
> line 348, in addCatalogue
>     for i, t in enumerate(read_table(cat)):
>   File "/Users/york/JWST/pandeia/stips/stips/utilities/utilities.py", line
> 109, in read_table
>     chunk = ascii.read(lines, format=format, guess=False)
>   File
> "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/ui.p
> y", line 339, in read
>     dat = reader.read(table)
>   File
> "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/core
> .py", line 1045, in read
>     self.header.get_cols(self.lines)
>   File
> "/usr/stsci/ssbdev/python/lib/python2.7/site-packages/astropy/io/ascii/ipac
> .py", line 180, in get_cols
>     raise ValueError('At least one header line beginning and ending with '
> ValueError: At least one header line beginning and ending with delimiter
> required
>
>
> I have no idea of what's causing this (for reference, the opening lines of
> the table are below, and this table opens just fine if I do a
> Table.read(format='ascii.ipac') (except that it takes up ~4GB of memory
> for a typical stellar population):
>
> \ Internal Format Catalogue
> \
> \ Parameters:
> \
> \type="internal"
> \filter="F115W"
> \
> |               RA|               DEC|          FLUX| TYPE|   N|  Re|
> Phi|Ratio|    ID| Notes|
> |           double|            double|        double| char|char|char|char|
> char|  long|  char|
> |                 |                  |              |     |    |    |    |
>     |      |      |
> |             null|              null|          null| null|null|null|null|
> null|  null|  null|
>  8.35744567067e-09  6.07703574397e-09  1.77083404307 point  N/A  N/A  N/A
>  N/A      1   None
>  1.82673672356e-10 -2.22541481358e-11  4.16120404729 point  N/A  N/A  N/A
>  N/A      2   None
>  1.67512697711e-08 -1.68243630672e-08  2.70787939571 point  N/A  N/A  N/A
>  N/A      3   None
>
>
> If anyone has any idea of how I'm using the code wrong, it would be much
> appreciated.
>
>         -Brian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20160227/a7e7b37e/attachment.html>