[Baypiggies] Raw file input and python

[Jeremy Fishman]

A combination of struct and numpy can be useful as well.

>>> import struct, numpy
>>> f = open('foo', 'w+')
>>> f.write('deadbeaf' + 400 * '\0')
>>> f.seek(0)
>>>
>>> head = struct.unpack('Q', f.read(8))[0]
>>> head
7377289137874888036L
>>> tail = numpy.fromfile(f, 'int32')
>>> len(tail)
100

Upside is that you get your bulk data in a powerful numpy array.
  - Jeremy

On Sat, Feb 12, 2011 at 11:48 AM, Glen Jarvis <glen at glenjarvis.com> wrote:

> Yes, of course!  That's what I'm

>>> f = open('foo', 'w+')

>>> f.write('\0' * 408)

>>> f.seek(0)

>>> head = struct.unpack('Q', f.read(8))

>>> head

(0,)

>>> tail = numpy.fromfile(f, 'int32')

>>> len(tail)

100

>>>

 thinking of..  If I were "thinking in C" that would have come to me
> quicker... Oh well..
>
> Thanks for that!  :)
>
>
> Glen
>
>
> On Sat, Feb 12, 2011 at 11:28 AM, Michael Pittaro <
> mikeyp at lahondaresearch.org> wrote:
>
>>
>> On Sat, Feb 12, 2011 at 11:20 AM, Glen Jarvis <glen at glenjarvis.com>wrote:
>>
>>> Oh, and I'm constrained to Python 2.5 or lower only :(
>>>
>>> Glen
>>>
>>> On Sat, Feb 12, 2011 at 11:19 AM, Glen Jarvis <glen at glenjarvis.com>wrote:
>>>
>>>> I remember, about three years ago, reading someone's answer to a
>>>> question very similar to this. But, I can't find it now.
>>>>
>>>> I have a raw input file where the first "entity" is a 64-bit unsigned
>>>> integer and the remaining items in the file are 32-bit unsigned integers
>>>> (until the end of the file).
>>>>
>>>> I vaguely remember there was already a module to cleanly handle this. I
>>>> could, of course, open the file (f) in binary mode and do an f.read() on
>>>> each byte and figure thing out. But, one will have to think of byte
>>>> swapping, variable "sizes", etc. when running on different systems.
>>>>
>>>> I thought there was a layer above this that would take care of the
>>>> mapping of data in the file (i.e., a raw dump/raw read) to a set of
>>>> variables, taking care of byte swapping if needed, size of types, etc. as
>>>> long as I specified the format (i.e., first should be 64-bit unsigned,
>>>> second... etc.)
>>>>
>>>> Am I mistaking that this exists (i.e., did I just make this up)?
>>>>
>>>> Cheers,
>>>>
>>>>
>>>>
>> You probably are thinking of struct  - perfect for byte level unpacking of
>> records.
>>
>> http://www.doughellmann.com/PyMOTW/struct/
>>
>>
>> _______________________________________________
>> Baypiggies mailing list
>> Baypiggies at python.org
>> To change your subscription options or unsubscribe:
>> http://mail.python.org/mailman/listinfo/baypiggies
>>
>
>
>
> --
> Things which matter most must never be at the mercy of things which matter
> least.
>
> -- Goethe
>
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20110212/9e0fc8f6/attachment.html>