help with binary file io, perhaps with generators?

Marcus Goldfish magoldfish at gmail.com
Sat Apr 23 22:24:43 EDT 2005


I need to write a "fast" file reader in python for binary files structured as:

… x[0] y[0] z[0] x[1] y[1] z[1]  …

where c[k] is the k-th element from sequence c.  As mentioned, the
file is binary -- spaces above are just for visualization.  Each
element, c[k], is a 16-bit int.  I can assume I know the number of
sequences in the file a priori.  Files are stored and processed on a
WinXP machine (in case Endian-ness matters).

I'm looking for suggestions on how to tackle this problem, and how to
design my reader class.  Or if anyone has sample code: that would be
appreciated, too.  Some questions I have to start with:

   (i) should I use generators for iterating over the file?
   (ii) how can I handle the 16-bit word aspect of the binary data?
   (iii) ultimately, the data will need to be processed in chunks of
         M-values at a time... I assume this means I need some
         form of buffered io wrapper, but I'm not sure where to start
         with this.

Thanks!
Marcus

Ps -- this seems like a general stream processing problem… if anyone
can recommend good refs (web) on stream processing....


More information about the Python-list mailing list