[SciPy-user] NumPy Newcomer Questions: Reading Data Stream & Building Array

[Johannes Loehnert]

Hi,

I must admit I did not understand the data format correctly, however maybe I 
can help.

>    How to slice the list of data from the scanner stream into two-byte
> chunks (leaving the end-of-record byte to be dealt with separately),

Since you want it as array in the end, you could convert it straight away 
using fromstring:

#code
s = '\x00\x01\x00\x02\x00\x03'   # some binary data
a = fromstring(s, dtype='>u2')   # >: Big Endian, u: uint, 2: bytes/val
#endcode

You will have to strip the EOR byte. Choose appropriate byteorder.

>
>    How to build the array so that each form read creates a new row in the
> array. While the 31 columns are fixed, the number of rows is indeterminate
> until all submitted forms have been read.

Since the row count is undetermined, the best way is to build a list (linked 
list, builtin type) containing all the columns and convert it to an array 
when finished with reading:

# code
rows = []  # create empty list
while still_rows_left:
     rows.append(row_that_was_just_read)
# now rows is [array(1,2,3,..), array(4,5,6,...), ...]
mydata = array(rows)  # make array out of list
#endcode

HTH, Johannes