Efficient handling of fast, real-time hex data

[jladasky at itu.edu]

On Tuesday, May 31, 2016 at 6:02:07 PM UTC-7, Rob Gaddi wrote:

> You'll probably want to process it in blocks.  Allocate a 3kB
> bytearray, assign into it from the data coming in off Serial (less
> the newlines) and when you fill it, call numpy.from_buffer to rip it.

Thanks Rob, numpy.frombuffer (no underscore in my version of Numpy, v.1.8) looks like it will be very helpful!

> Then just print the first line of it.  Decimating the data
> rate radically will help with the print-induced load, and it's not
> like you need to see every sample.
> 
> Are you going to be trying to use this data realtime, or are you just
> trying to datalog it and deal with it offline?  Because at some point
> you'll need to decide, all in, how much data you're willing to try to
> hold in memory and what you intend to do with the rest of it.

No, I don't need to print all the data... in fact, I don't plan to print any of it when I'm done.  I do want to save every data packet for offline analysis.  I also want to display real-time histograms of at least some of the data.  I've already cobbled together (but have not optimized) a Matplotlib program that displays mock data at 16 FPS, which is fast enough.

I used to work on flow cytometers.  Look them up if you're interested in multi-dimensional, real-time data display.