Kafkaesque 1.0.0 not compatible with Kafka 0.10.0

[domige]

I use following code with Kafkaesque 1.0.0 to send message to Kafka 0.10.0.

var kafka = require('kafkaesque');
var kafkaesque = kafka({
        brokers: [{ host: XX.XX.XX.XX, port: 29001 }],
                                maxBytes: 1024*1024
});

kafkaesque.produce('tip-tests-Topic', 'message form 1', function(err, res) {
                if (err) {
        console.error("Kafka saving errors: " + err);
    }
                else
                {
                                console.log('kafka sent');
                }

                console.log(JSON.stringify(res));
                console.log(JSON.stringify(err));
                kafkaesque.disconnect();
})

The Kafka server didn't receive the message.
The callback err is always true, res always null, and kafkaesque.disconnect() fail with
_groupCoordinator.tearDown();
^

TypeError: Cannot read property 'tearDown' of undefined
at Object.disconnect (C:\ReachClient\Shield\node_modules\kafkaesque\lib\kafk
aesque.js:805:24)
at C:\ReachClient\Shield\artifacts\build\lib\custom-reporters\test_kafka.js:
19:13
at C:\ReachClient\Shield\node_modules\kafkaesque\lib\kafkaesque.js:480:18
at Object.callback (C:\ReachClient\Shield\node_modules\kafkaesque\lib\kafkae
sque.js:135:27)
at Socket. (C:\ReachClient\Shield\node_modules\kafkaesque\lib\api
.js:404:17)
at emitOne (events.js:77:13)
at Socket.emit (events.js:169:7)
at readableAddChunk (_stream_readable.js:146:16)
at Socket.Readable.push (_stream_readable.js:110:10)
at TCP.onread (net.js:529:20)

I suspect the message version change on 0.10.0 caused this? Current Kafkaesque may not able to support the v1 Kafka message?

v0

Message => Crc MagicByte Attributes Key Value

Crc => int32

MagicByte => int8

Attributes => int8

Key => bytes

Value => bytes

v1 (supported since 0.10.0)

Message => Crc MagicByte Attributes Key Value

Crc => int32

MagicByte => int8

Attributes => int8

Timestamp => int64

Key => bytes

Value => bytes

[mattkirwan]

Hi domige,

I'm wondering if you've got anywhere with this error?

I ask because I'm receiving the same error on both a kafka 0.10.0.1 and 0.8.2 install.

[GlenTiki]

I've been digging into this, will try resolve this if I can reproduce.

[PHameete]

I can confirm this error as well. I can use kafkaesque to consume messages of another producer fine, but messages produced by my kafkaesque producer are not accepted by the 0.10.0 broker.

[mattkirwan]

👍 Cheers PHameete, I thought I was going mad.

[PHameete]

It is probably indeed related to the breaking changes in the message format (see http://kafka.apache.org/documentation.html#upgrade_10). You can work around this for now by setting the log.message.format.version broker config to a 0.9.x version or earlier.
In my personal case I would actually like to use the new timestamp field so an upgrade of Kafkaesque to support message format v1 would be great.

[PHameete]

I've updated my broker configuration by setting log.message.format.version to 0.9.0.0 and unfortunately the issue remains. Kafkaesque isn't compatible with the new message format introduced with 0.10, but this issue does not seem to be related to that.

[eugef]

@domige @PHameete i ran into the same issue. Are there any workarounds to fix it?

[PHameete]

@eugef I didn't dive into the problem further. Im using the kafka-node client now which works fine for me.

[eugef]

@PHameete i've also found another client no-kafka that both supports brokers (no need for zookeeper) and plays nicely with kafka 0.10.

[peebles]

I am seeing this issue too. Specifically that the cb() passed to produce() which seems like its supposed to get (err,res) (see samples), the err is the boolean true. Also, when I subscribe to a topic, I get "error 22" from connect 'error' event, which also hints at a protocol version mismatch.

I am using the published npm module for kafkaesque, and the ches/kafka dockerhub image for kafka service. I have tried the dockerhub image tags 0.10.1.0 and 0.9.0.0 which I am guessing correspond to kafka broker versions, and both give me the same behavior.

So main questions is, exactly what version(s) of the kafka broker are compatible with the npm published version of kafkaesque?

Is there a better/different dockerhub kafka image I should be using with kafkaesque?