Efficient handling of fast, real-time hex data

[jladasky at itu.edu]

Greetings everyone,

I'm developing small embedded systems.  I can't use Python to program them, I have to program the little CPU's in C.  

I want a peripheral I've designed to talk over USB to a Python program on the host computer.  The peripheral is spewing data at a reasonably high rate, about 5,000 16-bit unsigned integers per second.  For now, I'm trying to print these numbers as they come in.  Even though I'm working with a fast, up-to-date computer, I am finding it difficult for the host computer to keep up with the data stream.

The details of my setup: running Python 3.4 on Ubuntu 15.04 x64.  Using PySerial to handle UART over USB.  Intel Core i7-4790K CPU @ 4.00GHz.

At first I had the peripheral transmit the numbers as base-10 encoded strings, separated by spaces, just so that I could see that I was receiving the right values from the peripheral.  My serial port reads look fine -- until they fall behind.  But in this first case, I was using up to six bytes to send a two-byte value.  Obviously that's wasteful.  So I rewrote the peripheral to send bundles of C uint_16's, with each bundle of three numbers separated by a newline.

Now the challenge is to unpack and display those numbers on the Python side.  I know that print statements themselves are probably slowing things down, but I still want to see that those numbers are correct.  

So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that Serial.readline() returns to me, and QUICKLY turn it into three integer values, 258, 772, and 1286?  Better yet, can I write these bytes directly into an array (numpy is an option), since I'm going to have a lot of them?

I never thought I would find a task that's easier for me to imagine doing in C than in Python, but at the moment this is one.  Thanks for any suggestions!