Efficient handling of fast, real-time hex data

[Rob Gaddi]

jladasky at itu.edu wrote:

> Greetings everyone,
>
> I'm developing small embedded systems.  I can't use Python to program them, I have to program the little CPU's in C.  
>
> I want a peripheral I've designed to talk over USB to a Python program on the host computer.  The peripheral is spewing data at a reasonably high rate, about 5,000 16-bit unsigned integers per second.  For now, I'm trying to print these numbers as they come in.  Even though I'm working with a fast, up-to-date computer, I am finding it difficult for the host computer to keep up with the data stream.
>
> The details of my setup: running Python 3.4 on Ubuntu 15.04 x64.  Using PySerial to handle UART over USB.  Intel Core i7-4790K CPU @ 4.00GHz.
>
> At first I had the peripheral transmit the numbers as base-10 encoded strings, separated by spaces, just so that I could see that I was receiving the right values from the peripheral.  My serial port reads look fine -- until they fall behind.  But in this first case, I was using up to six bytes to send a two-byte value.  Obviously that's wasteful.  So I rewrote the peripheral to send bundles of C uint_16's, with each bundle of three numbers separated by a newline.
>
> Now the challenge is to unpack and display those numbers on the Python side.  I know that print statements themselves are probably slowing things down, but I still want to see that those numbers are correct.  
>
> So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that Serial.readline() returns to me, and QUICKLY turn it into three integer values, 258, 772, and 1286?  Better yet, can I write these bytes directly into an array (numpy is an option), since I'm going to have a lot of them?
>
> I never thought I would find a task that's easier for me to imagine doing in C than in Python, but at the moment this is one.  Thanks for any suggestions!

You'll probably want to process it in blocks.  Allocate a 3kB
bytearray, assign into it from the data coming in off Serial (less
the newlines) and when you fill it, call numpy.from_buffer to rip it.

Then just print the first line of it.  Decimating the data
rate radically will help with the print-induced load, and it's not
like you need to see every sample.

Are you going to be trying to use this data realtime, or are you just
trying to datalog it and deal with it offline?  Because at some point
you'll need to decide, all in, how much data you're willing to try to
hold in memory and what you intend to do with the rest of it.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.