[AstroPy] Astropy and large VOTable files

[Michael Droettboom]

On 06/03/2015 05:48 PM, Demitri Muna wrote:
>
> On May 18, 2015, at 2:09 PM, Andrew Hearin <andrew.hearin at yale.edu 
> <mailto:andrew.hearin at yale.edu>> wrote:
>
>> Being able to read large data in chunks, make cuts on the chunks, and 
>> return a table of rows that pass the cuts is a pretty common data 
>> mining task that I think would be good to include in Astropy. I’m 
>> happy to (re-)raise a GitHub issue for this purpose, and contribute 
>> some code, but first: Jennifer, this is the functionality you are 
>> describing, right? If so: Mike, do you see any fundamental obstacles 
>> with this?
>
> If you are working with very large FITS files, I recommend using Erin 
> Sheldon's fitsio package. The underlying C (cfitsio) library is able 
> to directly read slices without reading the whole file into memory.

Just for completeness, this is also true of astropy.io.fits.

> As for XML... I wouldn't recommend XML for crazy large files. It 
> wasn't designed for that.

Ditto.

Mike

>
> Cheers,
> Demitri
>
> _________________________________________
> Demitri Muna
>
> Department of Astronomy
> Le Ohio State University
>
> http://trillianverse.org
> http://scicoder.org
>
>
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20150604/b7d3f967/attachment.html>