[SciPy-User] Numpy pickle format

[Robert Kern]

On Mon, Nov 29, 2010 at 10:04, Francesc Alted <faltet at pytables.org> wrote:
> A Monday 29 November 2010 15:09:30 Robert Kern escrigué:
>> Rather than "adapting the format" per se, just wrap your format
>> around it. Send a message containing the version number of your
>> blocked format, the number of header blocks, the number of data
>> blocks, and any information about the compression of the data. Then
>> send the NPY header in its own message. Then start send the possibly
>> compressed data messages.
>
> Well, I was thinking basically in extending NPY for incorporating
> compression information, but your approach is feasible too (although it
> requires sending one additional message).  Which advantage would have
> your suggestion?

It's standard best practice for developing stacks of network protocols
(c.f. UDP and TCP over IP over Ethernet). Not least, it keeps the two
protocols orthogonal to each other. If I change the NPY format
slightly (i.e. adding another header key but not changing the
header/data separation), you don't have to change your protocol at
all. At least with ZeroMQ, adding an additional block is incredibly
cheap (you should probably err on the side of more blocks rather than
fewer).

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco