Building Very Large Records

[Dave Brueck]

Greg Lindstrom wrote:
> Hello-
> 
> I am working on a routine to pull information from an Oracle database and
> format it into fixed-length records.  My problem is that the record layout
> is quite long (over 500 bytes) and contains dozens of fields.  How would
> *you* go about building such a beast?  I know that using the += operator is
> a bad idea, as it creates a new copy of the string each time.  How about the
> struct.pack() method?  A very big '%-12.12s%-50.50s.......' statement?
> Other ideas?  It does not have to be blazingly fast, but it doesn't have to
> suck, either.  I don't know enough about Oracle SQL yet, could I format the
> data in the SQL call?

Here's what I do, maybe somebody can suggest some improvements upon it:

Derive your record from ctypes' Structure class:

class XHeader(Structure):
     _pack_ = 1 # No alignment padding
     _fields_ = [('ucid', c_ulonglong),
                 ('majorVer', c_ubyte),
                 ('minorVer', c_ubyte),
                 ('extent', c_ulonglong),
                 ('future', c_byte),
                 ('titleLen', c_ubyte)]

Then to serialize it to a string:

def ObjAsString(obj):
     'Takes a ctypes object (works on structures too) and returns it as a string'
     # There is probably a better way, but this seems to work ok
     sz = sizeof(obj)
     RawType = c_char * sz
     ptr = RawType.from_address(addressof(obj))
     return ptr.raw[:sz]

To go the other direction:

def StringAsObject(s, ObjType):
     'Takes a string and returns it as a ctypes object'
     z = c_buffer(s)
     x = ObjType.from_address(addressof(z))
     x.__internal = z #ugly hack - save a ref to the c_buffer!
     return x

Basic usage is:

x = XHeader()
x.majorVer = 5
...etc...

StuffIntoDB(ObjAsString(x))

newX = StringAsObject(ReadFromDB(), XHeader)

I haven't used it with tons of object instances, so you may need to see if you 
end up with memory leaks.

-Dave