[SciPy-user] NumPy Newcomer Questions: Reading Data Stream & Building Array
Rich Shepard
rshepard at appl-ecosys.com
Fri Sep 8 17:48:42 EDT 2006
On Fri, 8 Sep 2006, Johannes Loehnert wrote:
> I must admit I did not understand the data format correctly, however maybe I
> can help.
Johannes,
The data format is a stream of ASCII bytes. Each two-byte combination
represents either a string (for two columns) or an integer (for the other 28
data columns). The conversion from position-specific bit values to the ASCII
characters is accomplished using the data mapping dictionaries. I accept
that I wasn't as clear as I could have been.
> Since you want it as array in the end, you could convert it straight away
> using fromstring:
>
> #code
> s = '\x00\x01\x00\x02\x00\x03' # some binary data
> a = fromstring(s, dtype='>u2') # >: Big Endian, u: uint, 2: bytes/val
> #endcode
After sending the message it occurred to me that the string.Split()
function is probably what I want. I'm not receiving Hex values from the
scanner. I'll play with this over the weekend.
You are correct, however, that I do want to take that incoming string and
store it into an array in the fewest possible steps. I'll have to look at
'fromstring()' to see just what that does. I assumed that each row in the
array was a list, is that incorrect?
> You will have to strip the EOR byte. Choose appropriate byteorder.
If the incoming data is stored as a list, I can slice off the last byte.
On AMD processors (and Intel, too) I believe that the byte order has
always been littleendian.
> Since the row count is undetermined, the best way is to build a list (linked
> list, builtin type) containing all the columns and convert it to an array
> when finished with reading:
>
> # code
> rows = [] # create empty list
> while still_rows_left:
> rows.append(row_that_was_just_read)
> # now rows is [array(1,2,3,..), array(4,5,6,...), ...]
> mydata = array(rows) # make array out of list
> #endcode
Ah! I think that's just what I need. I've started reading Travis' book,
but haven't hit this part yet.
Many thanks,
Rich
--
Richard B. Shepard, Ph.D. | The Environmental Permitting
Applied Ecosystem Services, Inc.(TM) | Accelerator
<http://www.appl-ecosys.com> Voice: 503-667-4517 Fax: 503-667-8863
More information about the SciPy-User
mailing list