Manipulate Large Binary Files

[Derek Martin]

On Wed, Apr 02, 2008 at 10:59:57AM -0400, Derek Tracy wrote:
> I generated code that works wonderfully for files under 2Gb in size
> but the majority of the files I am dealing with are over the 2Gb
> limit
> 
> ary = array.array('H', INPUT.read())

You're trying to read the file all at once.  You need to break your
reads up into smaller chunks, in a loop.  You're essentially trying to
store more data in memory than your OS can actually access in a single
process...

Something like this (off the top of my head, I may have overlooked
some detail, but it should at least illustrate the idea):

# read a meg at a time
buffsize = 1048576
while true:
	buff = INPUT.read(buffsize)
	OUTPUT.write(buff)
	if len(buff) != buffsize:
		break

-- 
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20080402/0b300c39/attachment-0001.sig>