How do I use the unpack function?

[Marlin Rowley]

Thanks.  I'll churn on this for awhile..

> Date: Thu, 15 May 2008 16:00:03 -0700> From: gherron at islandtraining.com> To: python-list at python.org> Subject: Re: How do I use the unpack function?> > Marlin Rowley wrote:> > Hey Gary!> > Please keep such discussions on the public python-list -- not personal > e-mail.> > Scroll down for an answer to your latest question.> >> > Here's what I have that renders fine but I see some optimization that > > can be done (as you mentioned):> >> > # Tile Generation> > # This is where all the drawing to the client window> > # will happen. > > def generateTile( txl, tyl, tileWidth, tileHeight, clientWindow ):> > # make rgba (8-bit) data structure and zero it out.> > rgb = zeros( tileWidth*tileHeight*3, UnsignedInt8 )> > alpha = zeros( tileWidth*tileHeight*3, UnsignedInt8 )> > > > #print 'tileWidth: %s' % tileWidth> > #print 'tileHeight: %s' % tileHeight> > # for each pixel in the tile> > # we must invert the rendering of each> > # tile for wxPython's Bitmap support.> > for y in range( (tileHeight-1),-1,-1 ):> > for color in range(4):> > > > # read per scanline> > pixelComp = clientWindow.fileIO.read(4*tileWidth) <<<< > > HERE'S YOUR OPTIMIZATION!!> > > > for x in range(tileWidth):> > # mental ray streams RGBA components across the width> > # of every tile. so it first does all the r's,> > # then all the g's, then all the b's, etc.. across> > # the width.. Then it streams the second row, etc..> > # However, wxPython Bitmap class accepts an array of> > # tuples or just a byte order of RGBARGBARGBA, etc..> > # so we convert, keeping track of an offset.> > if color < 3:> > index = (3*(y*tileWidth+x))+color> > else:> > index = (3*(y*tileWidth+x))> > > > > > # RGBA_FP> > if clientWindow.pixelCode == 13:> > > > # unpack the pixel> > #fourbytes = pixelComp[:4]> > #pixelComp = pixelComp[4:]> > buffer = unpack("!f", pixelComp[4*x:4*x+4]) > > <<<<<<<<<<<<<<<<<< YOUR OPTIMIZATION!!> > > > # convert from 32-bit to 8-bit precision> > gamma = clientWindow.gamma> > if gamma == 1.0:> > pixel = int(255 * buffer[0] + 0.5)> > pixel = clamp(pixel,0,255)> > if color == 3:> > alpha[index+0] = alpha[index+1] = > > alpha[index+2] = pixel> > else:> > rgb[index] = pixel> > else:> > pixel = int(buffer[0] * GAMMA_BIT_PRECISION + 0.5)> > pixel = clamp(pixel,0,GAMMA_BIT_PRECISION-1)> > # set the color and alpha> > if color == 3:> > alpha[index+0] = alpha[index+1] = > > alpha[index+2] = clientWindow.frame.gammaTable[pixel]> > else:> > rgb[index] = > > clientWindow.frame.gammaTable[pixel]> > > > > > # ...> >> > # create an empty rgb and alpha tile> > tileRGB = wx.BitmapFromBuffer( tileWidth, tileHeight, rgb )> > tileAlpha = wx.BitmapFromBuffer( tileWidth, tileHeight, alpha )> > > > # set up main device to render to the current> > # buffers> > dc = wx.BufferedDC( None,clientWindow.colorBuffer )> > dca = wx.BufferedDC( None,clientWindow.alphaBuffer )> > > > # draw tiles> > dc.DrawBitmap( tileRGB, txl, (clientWindow.height-tileHeight)-tyl )> > dca.DrawBitmap( tileAlpha, txl, (clientWindow.height-tileHeight)-tyl )> >> >> > I'm no python expert (as you can tell), but I'm trying.. :)> >> > I started to re-write this function but I'm confused on how to do:> >> > > Reshape the array into a 3D array (i.e., a rectangular array of RGBA> > > values)> >> > this without using a for loop.> > Yes, easily. No Python loops (although plenty of C-level loops.) (I > think you are using Numeric -- a ancient predecessor of numpy. However, > I think these operations work in Numeric.)> > > Again I create a small test case. The values will be in the order > rrrrggggbbbb to start with, and rgbrgbrgbrgb afterwards.> > Create a test array and examine it:> >>> a = numpy.frombuffer('rrrrggggbbbb', dtype='S1')> >>> a> array(['r', 'r', 'r', 'r', 'g', 'g', 'g', 'g', 'b', 'b', 'b', 'b'], > dtype='|S1')> > Isolate each color component (here height*width = 4)> >>> a.shape = (3,4)> >>> a> array([['r', 'r', 'r', 'r'],> ['g', 'g', 'g', 'g'],> ['b', 'b', 'b', 'b']],> dtype='|S1')> > Transpose it. (This creates a new array by copying efficiently.)> >>> b = a.transpose()> >>> b> array([['r', 'g', 'b'],> ['r', 'g', 'b'],> ['r', 'g', 'b'],> ['r', 'g', 'b']],> dtype='|S1')> > Reset it's shape to be a width*height array of rgb's.> >>> b.shape = (2,2,3)> >>> b> array([[['r', 'g', 'b'],> ['r', 'g', 'b']],> > [['r', 'g', 'b'],> ['r', 'g', 'b']]],> dtype='|S1')> > Put it out in byte array form:> >>> b.tostring()> 'rgbrgbrgbrgb'> > > Done.> > > Gary Herron> > > >> > -M> >> >> > > Date: Thu, 15 May 2008 13:46:06 -0700> > > From: gherron at islandtraining.com> > > CC: python-list at python.org> > > Subject: Re: How do I use the unpack function?> > >> > > Marlin Rowley wrote:> > > > Gary,> > > >> > > > I'm getting streaming tile data from a renderer in order to allow the> > > > user to see the rendering of tiles in real-time to create the total> > > > image. I'm NOT reading an entire image scanline-by-scanline. The> > > > renderer streams in a series of floats (for each tile) and I build> > > > this tile up from each individual color component passed in. I then> > > > convert it to a small bitmap and blit() it to the window that will> > > > eventually make up my entire image. I'm just wanting to make the> > > > process as fast as the renderer can render the tiles. I've noticed on> > > > scenes that aren't very complex for the renderer that my python > > script> > > > is spending most of it's time drawing while the renderer is already> > > > done. What you've given me has sped up the drawing a lot, but I'm> > > > hungry for more optimization!> > > >> > > > There is no file format to be read. The data is raw bytes and can be> > > > any format.> > >> > > You are misinterpreting what I mean by a format. All data is a string> > > of bytes, and interpreting that string of bytes to have some particular> > > form is applying a format to it. Your particular format is: each 4> > > bytes represents a float, and a string of such floats give the RGBA> > > values for a rectangle (of some size) of pixels. I doubt that you are> > > the first ever to use that particular format to encode an image, but I> > > also don't believe it matches any of the standard image formats.> > >> > > So... There is still hope to use already written tools to decode your> > > format.> > >> > > Here's a hint on how to use numpy for decoding a byte string into an> > > array of floats. My example byte string is a hand coded string of just> > > 12 bytes -- you should replace that with a whole row or better yet, a> > > whole tile's worth of bytes.> > >> > > >>> import numpy> > > >>> byteString = '\x00\x00\x80?\x00\x00\x00@\x00\x00@@'> > > >>> b = numpy.frombuffer(byteString, dtype=numpy.float32)> > > >>> b> > > array([ 1., 2., 3.], dtype=float32)> > >> > >> > > If you want to use the array module instead:> > >> > > >>> import array> > > >>> a = array.array('f')> > > >>> a.fromstring(byteString)> > > >>> a> > > array('f', [1.0, 2.0, 3.0])> > >> > >> > > In either case ANY number of bytes can be decoded into an array of> > > floats, in one highly efficient call from Python.> > >> > > What you do with that array of float afterwards is up to you....> > >> > > If I read your note correctly, here's what I'd do to turn the array of> > > bytes into an image of a tile to be displayed on the screen.> > >> > > Get the whole tile's worth of bytes with one read into a single string.> > > Decode that string into an array of floats (as above).> > > Reshape the array into a 3D array (i.e., a rectangular array of RGBA> > > values)> > > Convert that array of floats into an array of 8-bit integers (scaling> > > all by 255).> > > Extract (via tostring) that array into a string of bytes.> > > Send that string of bytes to the graphics system as an RGBA array of> > > pixels.> > >> > > Each of these calls is one Python call into a highly efficient library.> > > This is using Python as a, so called, glue language.> > >> > > Gary Herron> > >> > >> > >> > >> > >> > >> > >> > > >> > > > > Date: Thu, 15 May 2008 10:09:45 -0700> > > > > From: gherron at islandtraining.com> > > > > CC: python-list at python.org> > > > > Subject: Re: How do I use the unpack function?> > > > >> > > > > John Machin wrote:> > > > > > On May 16, 2:11 am, Gary Herron <gher... at islandtraining.com> > > wrote:> > > > > >> > > > > >> Marlin Rowley wrote:> > > > > >>> > > > > >>> All:> > > > > >>>> > > > > >>> I've got a script that runs really slow because I'm reading > > from a> > > > > >>> stream a byte at a time:> > > > > >>>> > > > > >>> // TERRIBLE> > > > > >>> for y in range( height ):> > > > > >>> for color in range(4):> > > > > >>> for x in range( width ):> > > > > >>> pixelComponent = fileIO.read(4)> > > > > >>> buffer = unpack("!f",pixelComponent) << unpacks ONE> > > > > >>>> > > > > >> > > > > > [snip]> > > > > > Perhaps the OP might be able to use the Python Imaging Library > > (PIL)> > > > > > instead of reinventing an image-file handler.> > > > > >> > > > >> > > > > Indeed. That's why my original answer included the line:> > > > > There are probably better ways overall, but this directly answers> > > > > your question.> > > > >> > > > > Other possibilities.> > > > >> > > > > I don't recognize the file format being read in here, but if it is a> > > > > standard image format, the PIL suggestion is a good way to go.> > > > >> > > > > Or, if it is an image of not too enormous size, read the *whole* > > thing> > > > > in at once.> > > > >> > > > > Or rather than manipulate the array by carving off 4 bytes at a > > time,> > > > > just index through it in 4 byte chunks:> > > > > for i in range(width):> > > > > buffer = unpack("!f", pixelComponent[4*i:4*i+4])> > > > >> > > > > Or> > > > > for i in range(0,4*width,4):> > > > > buffer = unpack("!f", pixelComponent[i:i+4])> > > > >> > > > > Or> > > > > Use numpy. Create an array of floats, and initialize it with the > > byte> > > > > string, making sure to take endianess int account. (I'm quite > > sure this> > > > > could be made to work, and then the whole operation is > > enormously fast> > > > > with only several lines of Python code and *no* Python loops.> > > > >> > > > > Or> > > > > ...?> > > > >> > > > > Gary Herron> > > > >> > > > > > --> > > > > > http://mail.python.org/mailman/listinfo/python-list> > > > > >> > > > >> > > > > --> > > > > http://mail.python.org/mailman/listinfo/python-list> > > >> > > > > > ------------------------------------------------------------------------> > > > Windows Live SkyDrive lets you share files with faraway friends. > > Start> > > > sharing.> > > > > > <http://www.windowslive.com/skydrive/overview.html?ocid=TXT_TAGLM_WL_Refresh_skydrive_052008> > >> > > >> > > > > > ------------------------------------------------------------------------> > > >> > > > --> > > > http://mail.python.org/mailman/listinfo/python-list> > >> > > --> > > http://mail.python.org/mailman/listinfo/python-list> >> > ------------------------------------------------------------------------> > Get Free (PRODUCT) RED™ Emoticons, Winks and Display Pics. Check it > > out! > > <http://joinred.spaces.live.com?ocid=TXT_HMTG_prodredemoticons_052008>> > --> http://mail.python.org/mailman/listinfo/python-list
_________________________________________________________________
Stay in touch when you're away with Windows Live Messenger.
http://www.windowslive.com/messenger/overview.html?ocid=TXT_TAGLM_WL_Refresh_messenger_052008
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20080515/ee78868d/attachment-0001.html>