[AstroPy] Streaming binary data format for spectra

[G Jones]

Hello,
I am working on radio astronomy instrumentation. In particular I have
a spectrometer which dumps spectra very rapidly. I am using Python to
capture this data and plot this data. In evaluating the equipment, I
would like to make long running Allan variance measurements, and thus
I would like to save every spectrum to disk for post processing. I
realize I could dump the data to disk as just raw binary data, but if
possible I would like to use an existing data format to avoid
reinventing the wheel for reading and writing operations. I prefer to
not aggregate everything into RAM and then dump it all do disk at
once; it would be better to stream it to disk.
I looked into using FITS, but it doesn't seem suited to dynamically
increasing the size of the file. I tried using the pyfits streaming
HDU construct, but it seemed to require specifying an initial dataset
size first, then filling that dataset.
NetCDF seemed like a good solution since it allows for an unlimited
dimension to your data set, but I have not had any luck getting any of
the python wrappers to work with this library.
Does anyone have any suggestions of a data format that would fit my needs?
Thanks,
Glenn