Parsing data from pyserial

[John Machin]

Lone Wolf wrote:
> I'm trying to get data through my serial port from a CMUcam.
> This gizmo tracks a color and returns a packet of data. The
> packet has nine data points (well, really eight since the first
> point is just a packet header) separated by spaces as follows: M
> xxx xxx xxx xxx xxx xxx xxx xxx
>
> Here is the code I am using (python v24):
>
> import serial
>
> ser=serial.Serial('com1',baudrate=115200, bytesize=8,
> parity='N', stopbits=1,xonxoff=0, timeout=1)
>
> ser.write("PM 1") #This sets the CMUcam to poll mode
>
> for i in range(0,100,1):
> 	ser.write("TC 016 240 100 240 016 240\r\n")
> 	reading = ser.read(40)

You are asking for 40 bytes of data. You will get 40 bytes of data.

However your packets are (presumably) variable length, (presumably)
terminated by CR and/or LF. What does the documentation for the device
tell you?

> 	print reading

What you see from the print statement is not necessarily what you've
got.
Change that to print repr(reading) and show us what you then see.

> 	components = reading.split()
> 	print components
> ser.close
>
> Here is an example output:
>
> M 37 79 3 4 59 124 86 25
> ['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
> '124', '86', '25', 'M
> ']
> M 38 77 3 2 59 124 86 25
> ['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124', '86',
> '25', 'M', '38'
> , '7']
>

Let's try to reconstruct "reading":

| >>> a =  ['59', '123', '87', '25', 'M', '37', '79', '3', '4', '59',
| ... '124', '86', '25', 'M']
| >>> astrg = ' '.join(a)
| >>> astrg
| '59 123 87 25 M 37 79 3 4 59 124 86 25 M'
| >>> len(astrg)
| 39 <<<<<==== ooh! almost 40!!
| >>> b =  ['39', '85', '26', 'M', '38', '77', '3', '2', '59', '124',
'86',
| ... '25', 'M', '38'
| ... , '7']
| >>> bstrg = ' '.join(b)
|  >>> bstrg
| '39 85 26 M 38 77 3 2 59 124 86 25 M 38 7'
| >>> len(bstrg)
| 40 <<<<<==== ooh! exactly 40!!!

My guess: the device is pumping out packets faster than you can handle
them. So you are getting 40-byte snatches of bytes. A snatch is long
enough to cover a whole packet with possible fragments of packets at
each end. You will need to discard the fragments. If you need all the
data, you'd better get some help on how to implement flow control --
I've never used pyserial and I'm not going to read _all_ the docs for
you :-)

I'm very interested to see what print repr(reading) actually shows. I'm
strongly suspecting there is a CR (no LF) at the end of each packet; in
the two cases shown, this would cause the "print reading" to appear as
only one packet ... think about it: carriage return, with no linefeed,
would cause overwriting. It is a coincidence with those two samples
that the first part of the line doesn't appear strange, with a 4, 5, or
6-digit number showing up where the trailing fragment ends


> My problem is that I am trying to get each data point of the
> packet into a separate variable. Ordinarily, this would be easy,
> as I would just parse the packet, read the array and assign each
> element to a variable eg. mx = components[1].

better would be:

mx, foo, bar, ......, eighth_vbl = components[start:start + 8]
once you have worked out what start should be, e.g. start =
components.index('M') + 1

> However, that
> doesn't work here because the original packet and the array that
> I got from using the split() method are different. If I were to
> try read the array created in the first example output, mx would
> be 123 instead of 37 like it is in the packet. In the second
> example, the array is 85 while the packet is 38.
>
> As near as I can figure out, pyserial is reading a stream of
> data and helpfully rearranging it so that it fits the original
> packet format M xxx xxx xxx xxx xxx xxx xxx xxx.

How, if you've read the docstring for the Serial.read() method, did you
come to that conclusion?

pyserial knows nothing about your packet format.

> I would have
> thought the split() method that I used on original packet (ie
> the "reading" variable) would have just returned an array with
> nine elements like the packet has. This is not the case, and I
> am at a loss about how to fix this.
>
> I've searched the archive here and elsewhere with no luck. Any
> help REALLY appreciated!
>

With a bit of repr() and a bit of RTFM, one can often manage without
help :-)

Cheers,
John