Efficient handling of fast, real-time hex data

[jladasky at itu.edu]

On Tuesday, May 31, 2016 at 9:37:18 PM UTC-7, Gregory Ewing wrote:
> > So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that
> > Serial.readline() returns to me,
> 
> Using readline() to read binary data doesn't sound like
> a good idea -- what happens if one of the data bytes
> happens to be 0x0a?
> 
> If you're going binary, it would be better to ditch the
> newlines and read a fixed number of bytes each time.

Hi Greg,

Of course you're correct.  In my very first version of this system, I piggybacked the transmission of data on the routines that I used to transmit text.  That won't be a permanent state of affairs.

I guess that I should have posted my example string without the newline, which I will of course strip off before parsing the string into 16-bit integers.  Also, now that I've verified that the CPU's at both ends of the line are little-endian, I should have swapped every other byte.  :^)

One common data transmission error I've seen in other systems is added/dropped bytes.  I'm not sure whether to expect this problem with a USB/UART setup.  I may add a CRC-8 error-checking byte in place of the newline.

> Regarding speed, if struct.unpack or numpy isn't fast
> enough, you may need to deal with the data in bigger
> chunks.
> 
> Although if you're not doing a huge amount of processing
> with them, I'd expect it to be plenty fast enough.
> I just did a quick test, and I was able to unpack
> about 700,000 random 6-byte strings per second on
> a 2.8GHz Xeon.

That's an encouraging speed.  I'm sure that I'm not doing everything as efficiently as possible yet.

I've done some Multiprocessing work before.  To speed things up further, I might implement the live graphics in one process and the I/O in another.