How do I use the unpack function?

[Gary Herron]

Marlin Rowley wrote:
> Gary,
>
> I'm getting streaming tile data from a renderer in order to allow the 
> user to see the rendering of tiles in real-time to create the total 
> image.  I'm NOT reading an entire image scanline-by-scanline.  The 
> renderer streams in a series of floats (for each tile) and I build 
> this tile up from each individual color component passed in.  I then 
> convert it to a small bitmap and blit() it to the window that will 
> eventually make up my entire image.  I'm just wanting to make the 
> process as fast as the renderer can render the tiles.  I've noticed on 
> scenes that aren't very complex for the renderer that my python script 
> is spending most of it's time drawing while the renderer is already 
> done.  What you've given me has sped up the drawing a lot, but I'm 
> hungry for more optimization!
>
> There is no file format to be read.  The data is raw bytes and can be 
> any format.

You are misinterpreting what I mean by a format.  All data is a string 
of bytes, and interpreting that string of bytes to have some particular 
form is applying a format to it.    Your particular format is: each 4 
bytes represents a float, and a string of such floats give the RGBA 
values for a rectangle (of some size) of pixels.  I doubt that you are 
the first ever to use that particular format to encode an image, but I 
also don't believe it matches any of the standard image formats.

So... There is still hope to use already written tools to decode your 
format. 

Here's a hint on how to use numpy for decoding a byte string into an 
array of floats.  My example byte string is a hand coded string of just 
12 bytes -- you should replace that with a whole row or better yet, a 
whole tile's worth of bytes.

 >>> import numpy
 >>> byteString = '\x00\x00\x80?\x00\x00\x00@\x00\x00@@'
 >>> b = numpy.frombuffer(byteString, dtype=numpy.float32)
 >>> b
array([ 1.,  2.,  3.], dtype=float32)


If you want to use the array module instead:

 >>> import array
 >>> a = array.array('f')
 >>> a.fromstring(byteString)
 >>> a
array('f', [1.0, 2.0, 3.0])


In either case ANY number of bytes can be decoded into an array of 
floats, in one highly efficient call from Python.

What you do with that array of float afterwards is up to you....

If I read your note correctly, here's what I'd do to turn the array of 
bytes into an image of a tile to be displayed on the screen.

  Get the whole tile's worth of bytes with one read into a single string.
  Decode that string into an array of floats (as above).
  Reshape the array into a 3D array (i.e., a rectangular array of RGBA 
values)
  Convert that array of floats into an array of 8-bit integers (scaling 
all by 255).
  Extract (via tostring) that array into a string of bytes.
  Send that string of bytes to the graphics system as an RGBA array of 
pixels.

Each of these calls is one Python call into a highly efficient library.  
This is using Python as a, so called, glue language. 

Gary Herron







>
> > Date: Thu, 15 May 2008 10:09:45 -0700
> > From: gherron at islandtraining.com
> > CC: python-list at python.org
> > Subject: Re: How do I use the unpack function?
> >
> > John Machin wrote:
> > > On May 16, 2:11 am, Gary Herron <gher... at islandtraining.com> wrote:
> > >
> > >> Marlin Rowley wrote:
> > >>
> > >>> All:
> > >>>
> > >>> I've got a script that runs really slow because I'm reading from a
> > >>> stream a byte at a time:
> > >>>
> > >>> // TERRIBLE
> > >>> for y in range( height ):
> > >>> for color in range(4):
> > >>> for x in range( width ):
> > >>> pixelComponent = fileIO.read(4)
> > >>> buffer = unpack("!f",pixelComponent) << unpacks ONE
> > >>>
> > >
> > > [snip]
> > > Perhaps the OP might be able to use the Python Imaging Library (PIL)
> > > instead of reinventing an image-file handler.
> > >
> >
> > Indeed. That's why my original answer included the line:
> > There are probably better ways overall, but this directly answers
> > your question.
> >
> > Other possibilities.
> >
> > I don't recognize the file format being read in here, but if it is a
> > standard image format, the PIL suggestion is a good way to go.
> >
> > Or, if it is an image of not too enormous size, read the *whole* thing
> > in at once.
> >
> > Or rather than manipulate the array by carving off 4 bytes at a time,
> > just index through it in 4 byte chunks:
> > for i in range(width):
> > buffer = unpack("!f", pixelComponent[4*i:4*i+4])
> >
> > Or
> > for i in range(0,4*width,4):
> > buffer = unpack("!f", pixelComponent[i:i+4])
> >
> > Or
> > Use numpy. Create an array of floats, and initialize it with the byte
> > string, making sure to take endianess int account. (I'm quite sure this
> > could be made to work, and then the whole operation is enormously fast
> > with only several lines of Python code and *no* Python loops.
> >
> > Or
> > ...?
> >
> > Gary Herron
> >
> > > --
> > > http://mail.python.org/mailman/listinfo/python-list
> > >
> >
> > --
> > http://mail.python.org/mailman/listinfo/python-list
>
> ------------------------------------------------------------------------
> Windows Live SkyDrive lets you share files with faraway friends. Start 
> sharing. 
> <http://www.windowslive.com/skydrive/overview.html?ocid=TXT_TAGLM_WL_Refresh_skydrive_052008> 
>
> ------------------------------------------------------------------------
>
> --
> http://mail.python.org/mailman/listinfo/python-list