zstd (Zstandard) compression

[MrMoronIV]

After reading this: https://blog.cloudflare.com/squeezing-the-firehose/
I wonder if zstd is or becomes an option in dory?

[dspeterson]

It definitely looks like something that would be good to add to dory. I added a to-do item for zstd to dory's wiki page http://dory.wikidot.com/start . I also took a look at KIP-110 which covers zstd compression for Kafka: https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression . Based on the information there, it looks like adding zstd support to dory would require two areas of work:

1. Updating Dory to support Kafka's v2 wire protocol. This is needed because according to KIP-110, Kafka only supports zstd compression for v2 clients.
2. Adding a compression codec for zstd to dory. Compression codecs live here: https://github.com/dspeterson/dory/tree/master/src/dory/compress

Most of the work is in item 1. I'd call that a medium-sized project. I've been intending to add v2 wire protocol support to dory for a while and hope to do it in the not too distant future although it's hard to predict when. Once item 1 is done, item 2 should be very easy.

[MrMoronIV]

Looking forward to this although I can see it could be a lot of work!

On another note, does v2 wire protocol give any other advantages that dory is missing atm?

[dspeterson]

The v2 protocol adds some changes that support exactly once delivery semantics, as well as a timestamp field. There may be some other things I'm leaving out. I haven't yet had a chance to look at it in depth.