[SciPy-user] NumPy Newcomer Questions: Reading Data Stream & Building Array

[Rich Shepard]

On Fri, 8 Sep 2006, Johannes Loehnert wrote:

> I must admit I did not understand the data format correctly, however maybe I
> can help.

Johannes,

   The data format is a stream of ASCII bytes. Each two-byte combination
represents either a string (for two columns) or an integer (for the other 28
data columns). The conversion from position-specific bit values to the ASCII
characters is accomplished using the data mapping dictionaries. I accept
that I wasn't as clear as I could have been.

> Since you want it as array in the end, you could convert it straight away
> using fromstring:
>
> #code
> s = '\x00\x01\x00\x02\x00\x03'   # some binary data
> a = fromstring(s, dtype='>u2')   # >: Big Endian, u: uint, 2: bytes/val
> #endcode

   After sending the message it occurred to me that the string.Split()
function is probably what I want. I'm not receiving Hex values from the
scanner. I'll play with this over the weekend.

   You are correct, however, that I do want to take that incoming string and
store it into an array in the fewest possible steps. I'll have to look at
'fromstring()' to see just what that does. I assumed that each row in the
array was a list, is that incorrect?

> You will have to strip the EOR byte. Choose appropriate byteorder.

   If the incoming data is stored as a list, I can slice off the last byte.

   On AMD processors (and Intel, too) I believe that the byte order has
always been littleendian.

> Since the row count is undetermined, the best way is to build a list (linked
> list, builtin type) containing all the columns and convert it to an array
> when finished with reading:
>
> # code
> rows = []  # create empty list
> while still_rows_left:
>     rows.append(row_that_was_just_read)
> # now rows is [array(1,2,3,..), array(4,5,6,...), ...]
> mydata = array(rows)  # make array out of list
> #endcode

   Ah! I think that's just what I need. I've started reading Travis' book,
but haven't hit this part yet.

   Many thanks,

Rich

-- 
Richard B. Shepard, Ph.D.               |    The Environmental Permitting
Applied Ecosystem Services, Inc.(TM)    |            Accelerator
<http://www.appl-ecosys.com>     Voice: 503-667-4517      Fax: 503-667-8863