How to efficiently read binary files?
Robert Kern
robert.kern at gmail.com
Sun Apr 30 18:03:34 EDT 2006
Grant Edwards wrote:
> On 2006-04-30, David Lees <debl2NoSpam at verizon.net> wrote:
>
>>I want to process large binary files (>2GB) in Python. I have played
>>around with prototypes in pure Python and profiled the code. Most of
>>the time seems to be spent converting back and forth to and from strings
>>using the struct module. Is there a way to directly read into an array
>>of integers in Python?
>
> Perhaps the numarray module?
numpy for new code, please. In particular, numarray is limited by 32-bit APIs
even on 64-bit platforms, so a >2GB file will be difficult to process even when
using mmap. numpy removed this restriction on 64-bit platforms. 32-bit users
will still have to split up the file into <2GB chunks, though.
http://numeric.scipy.org/
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
--
Robert Kern
robert.kern at gmail.com
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
More information about the Python-list
mailing list