Fastest way to store ints and floats on disk

[Matthew Woodcraft]

Laszlo Nagy <gandalf at shopzeus.com> writes:

> The facts table cannot be kept in memory because it is too big. I need to
> store it on disk, be able to read incrementally, and make statistics. In most
> cases, the "statistic" will be simple sum of the measures, and counting the
> number of facts affected. To be effective, reading the facts from disk should
> not involve complex conversions. For this reason, storing in CSV or XML or any
> textual format would be bad. I'm thinking about a binary format, but how can I
> interface that with Python?
>
> I already looked at:
>
> - xdrlib, which throws me DeprecationWarning when I store some integers
> - struct which uses format string for each read operation, I'm concerned about
> its speed
>
> What else can I use?

pytables (<http://www.pytables.org/>) looks like the right kind of
thing.

-M-