[Python-Dev] Accepting PEP 3154 for 3.4?

[Antoine Pitrou]

On Mon, 18 Nov 2013 16:44:59 -0600
Tim Peters <tim.peters at gmail.com> wrote:
> [Tim]
> >> But it has a different kind of advantage:  PREFETCH was optional.  As
> >> Guido said, it's annoying to bloat the size of small pickles (which
> >> may, although individually small, occur in great numbers) by 8 bytes
> >> each.  There's really no point to framing small chunks of data, right?
> 
> [Antoine]
> > You can't know how much space the pickle will take until the pickling
> > ends, though, which makes it difficult to decide whether you want to
> > emit a PREFETCH opcode or not.
> 
> Ah, of course.  Presumably the outgoing pickle stream is first stored
> in some memory buffer, right?  If pickling completes before the buffer
> is first flushed, then you know exactly how large the entire pickle
> is.  If "it's small" (say, < 100 bytes), don't write out the PREFETCH
> part.  Else do.

Yet another possibility: keep framing but use a variable-length
encoding for the frame size:

- first byte: bits 7-5: N (= frame size bytes length - 1)
- first byte: bits 4-0: first 5 bits of frame size
- remaning N bytes: remaining bits of frame size

With this scheme, very small pickles have a one byte overhead; small
ones a two byte overhead; and the max frame size is 2**61 rather than
2**64, which should still be sufficient.

And the frame size is read using either one or two read() calls, which
is efficient.

Regards

Antoine.