[Python-Dev] Accepting PEP 3154 for 3.4?

[Antoine Pitrou]

On Mon, 18 Nov 2013 16:18:21 -0600
Tim Peters <tim.peters at gmail.com> wrote:
> 
> But it has a different kind of advantage:  PREFETCH was optional.  As
> Guido said, it's annoying to bloat the size of small pickles (which
> may, although individually small, occur in great numbers) by 8 bytes
> each.  There's really no point to framing small chunks of data, right?

You can't know how much space the pickle will take until the pickling
ends, though, which makes it difficult to decide whether you want to
emit a PREFETCH opcode or not.

> Which leads to another idea:  after the PROTO opcode, there is, or is
> not, an optional PREFETCH opcde with an 8-byte argument.  If the
> PREFETCH opcode exists, then it gives the number of bytes up to and
> including the pickle's STOP opcode.  So there's exactly 0 or 1
> PREFETCH opcodes per pickle.
> 
> Is there an advantage to spraying multiple 8-byte "frame counts"
> throughout a pickle stream?

Well, yes: much better memory usage for large pickles.
Some people use pickles to store huge data, which was the motivation to
add the 8-byte-size opcodes after all.

> I'm not thinking of DOS at all, just general sanity as data objects
> get larger & larger.  Pickles have almost no internal checks now.  But
> I've seen my share of corrupted pickles!

IMO, any validation should use a dedicated CRC-like scheme, rather than
relying on the fact that correct pickles are statistically unlikely :-)

(or you can implicitly rely on TCP or UDP checksums, or you disk
subsystem's own sanity checks)

Regards

Antoine.