[New-bugs-announce] [issue27237] Kafka Python Consumer Messages gets truncated

[Rahul]

New submission from Rahul:

Snippet code is below:
from kafka import KafkaConsumer
from kafka.client import KafkaClient
from kafka.consumer import SimpleConsumer

consumer = KafkaConsumer('eventdetails_ingestion' , group_id='1',bootstrap_servers=‘xxxx:9092', max_partition_fetch_bytes=10555555)
for msg in consumer:
    try:
        jValue = json.loads(str(msg.value))
   except ValueError:
        fileErr.write(str(msg.value)+"\n")

Steps:
We send/produce large sets of messages to Kafka of around 20 to 30 KB size each messages in JSON format and producing around 200 messages / sec for 1 hour duration. We have 3 Kafka Brokers running and I am trying to consume the messages from these 3 Kafka Brokers from the same topic using the above code. The problem is that sometimes some of the messages gets truncated, I am not sure why it happen ?

----------
components: Library (Lib)
messages: 267487
nosy: rgowda at threatmetrix.com
priority: normal
severity: normal
status: open
title: Kafka Python Consumer Messages gets truncated
type: performance
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27237>
_______________________________________