Unpacking byte strings from a file of unknown size

[Gabriel Genellina]

En Mon, 27 Oct 2008 19:03:37 -0200, Steven Clark  
<steven.p.clark at gmail.com> escribió:

> On Mon, Oct 27, 2008 at 4:29 PM, Mark <mseagoe at gmail.com> wrote:
>> Hi;
>>
>> I'm trying to use the struct.unpack to extract an int, int, char
>> struct info from a file.  I'm more accustomed to the file.readlines
>> which works well in a 'for' construct (ending loop after reaching
>> EOF).
>>
>> # This does OK at fetching one 10-byte string at a time:
>> # (4, 4, 2 ascii chars representing hex)
>> info1, info2, info3 = struct.unpack('<IIH', myfile.read(10))
>>
>> # Now to do the entire file, putting into a loop just gives error:
>> # TypeError: 'int' object is not iterable
>> for info1, info2, info3 in struct.unpack('<IIH', myfile.read(10)):
>>
>> In trying to shoehorn this into a 'for' loop I've been unsuccessful.
>> I also tried other variations that also didn't work but no point
>> wasting space.  Using Python 2.5, WinXP
>
> I usually do something like:
> s = myfile.read(10)
> while len(s) == 10:
>     info1, info2, info3 = struct.unpack('<IIH', s)
>     s = myfile.read(10)
> #might want to check that len(s) == 0 here

Pretty clear. Another alternative, using a for statement as the OP  
requested (and separating the "read" logic from the "process" part):

def chunked(f, size):
     while True:
         block = f.read(size)
         if not block: break
         yield block

for block in chunked(open(filename, 'rb'), 10):
     info1, info2, info3 = struct.unpack('<IIH', block)
     ...

A third one:

 from functools import partial

for block in iter(partial(open(filename,'rb').read, 10), ''):
     ...

(rather unreadable, I admit, if one isn't familiar with partial functions  
and the 2-argument iter variant)

-- 
Gabriel Genellina