lz4 is available at Kafka should be available at Dory

[CogumelosMaravilha]

No flame war between snappy and lz4, just another option available for those who want to use lz4 compression.

[dspeterson]

Ok, I'll add it to my to-do list. Thanks!

[dspeterson]

The latest release of dory (2.1.0) contains working but disabled support for LZ4 compression. It works only with Kafka versions >= 0.10.0.0, due to a bug in an earlier Kafka version (see https://cwiki.apache.org/confluence/display/KAFKA/KIP-57+-+Interoperable+LZ4+Framing for details). The intent is to not support LZ4 on prior Kafka versions, since this would require a messy workaround. Wire protocol work is still needed to detect what version the brokers are, and refuse to enable LZ4 support for older brokers. Hopefully this will be finished in the not too distant future, and I will then enable LZ4 support. In the meantime, you can build a version of dory with LZ4 enabled by looking for the following line of code in src/dory/dory.cc:

          bool enable_lz4 = false;

and setting the value to true. Just be sure not to use it with Kafka brokers older than 0.10.0.0. To illustrate how to enable LZ4 compression for a topic, I have updated the example config file config/dory_conf.xml.