Automatic segmenting of large Kafka messages?

[Skip Montanaro]

When using Apache Kafka, the maximum message size can be defined in
the configuration. If you have a lot of similarly sized messages, you
can probably make a good estimate of a max message size. When the
message sizes are highly variable, that's less certain.

This presentation from an engineer at LinkedIn:

https://www.slideshare.net/JiangjieQin/handle-large-messages-in-apache-kafka-58692297

describes a Kafka message segmentation facility they created. I poked
around looking for an implementation of this concept, but came up
short. Before I go off an reimplement the wheel, does anyone know if
there is something out there already? I'd be using it with Dana
Powers' Kafka package:

https://pypi.python.org/pypi/kafka/1.3.4.1

Thx,

Skip