Efficient handling of fast, real-time hex data

[Gene Heskett]

On Thursday 02 June 2016 04:13:51 alister wrote:

> On Thu, 02 Jun 2016 18:50:34 +1200, Gregory Ewing wrote:
> > jladasky at itu.edu wrote:
> >> One common data transmission error I've seen in other systems is
> >> added/dropped bytes. I may add a CRC-8 error-checking byte in place
> >> of the newline.
> >
> > Also maybe add a start byte with a known value at the beginning of
> > each packet to help resynchronise if you get out of step.
>
> No maybe about it
> if you are sending a binary stream you need to be able to reliably
> signal the start AND finish of the data stream (either send the length
> in the message start or have a fixed msg. length)
>
> after a lot of experimenting to ensure reliability you will probably
> have reinvented something like intelhex or x-modem
>
Neither of which can handle that last packet well unless the last packet 
is padded out to be a fill packet and the filler bytes thrown away in 
the receiver.

For that reason alone, zmodem wins because it does both..

zmodem in its present linux implementation must have in window size set 
explicitely on the invocation command line to match the default packet 
size of the receiving device, which is usually the size of its disk 
sector.  That way, when its talking to rzsz on a 256 byte sector box, it 
will be forced to do the checksum checks to match what rzsz can do.

Otherwise the linux version of zmodem will only check the checksums every 
8 kilobytes.  That of course will fail if the line speed overruns the 
receiver, and of course the receiver is sending error restart requests, 
not a good way to make any real speed.  My target box is running nitros9 
and somehow bit rot has destroyed the 7 wire protocol flow controls, so 
I have to restrict the data rate to 4800 baud.  Doing both the window 
size limit and the low baud rate, I have moved an 80 track disk image 
built on this machine, to that machine without any errors several times. 
Slow, and hard on the coffee pot staying awake while effectively 
watching paint dry, but it worked.

Now we have a better method that works at 112 kbaud, much better.  You 
can load a program direct from a virtual disk image "mounted" for use 
with disk descriptors that point at the disk image on this hard drive at 
a load speed approaching loading it from a real floppy on that machine. 

> The work [of software development] is becoming far easier (i.e. the
> tools we're using work at a higher level, more removed from machine,
> peripheral and operating system imperatives) than it was twenty years
> ago, and because
> of this, knowledge of the internals of a system may become less
> accessible.
> We may be able to dig deeper holes, but unless we know how to build
> taller ladders, we had best hope that it does not rain much.
> 		-- Paul Licker

Paul is absolutely correct.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>