Fast reading and unpacking of binary data (struct module)

[Neal Becker]

Daniel Platz wrote:

> Hi,
> 
> I have a Python newbie question about reading data from a binary file.
> I have an huge binary file from an external program. I want to read
> and process the data in this file in a reasonable time. It turns out
> that the reading of the data itself and the processing do not need
> most of the time. However, when using the read(bytes) method Python
> returns a string representing the binary information in hex. This
> string I have to "cast/translate" into a number (in my case a signed
> short). For this I am using the method struct.unpack from the struct
> module. This unpacking part of the program takes by far the most time.
> Is there a way to speed this up or to do it the unpacking more
> cleverly than with the struct module?
> 
> Thanks in advance.
> 
> With kind regards,
> 
> Daniel

Consider mmap
Consider numpy
Consider numpy+mmap