Dory SSL

[20Rishabh20]

Hi,

Looking to implement TLS connection with Kafka using Dory.
Please explain how we can achieve this.

[dspeterson]

Hi 20Rishabh20,

TLS support is something I've been wanting to add to Dory for quite a while. If you're interested in implementing it, I would welcome the contribution! Below are some pointers to places in the code to start looking at. This should help to get you started, but if you have more questions or anything looks confusing, please don't hesitate to ping me.

Dory opens a TCP connection to each Kafka broker for the purpose of sending produce requests (i.e. batches of messages) and receiving acknowledgements. Each connection is handled by a dedicated thread. In class TConnector in src/dory/msg_dispatch/connector.h, see following member:

      /* TCP socket to Kafka broker. */
      Base::TFd Sock;

This is the socket file descriptor for the TCP connection to a single broker. So adding TLS support would involve reworking things so that here you could optionally have a TLS-enabled socket, created through the appropriate OpenSSL library call(s).

Also dory uses class TMetadataFetcher to periodically send metadata requests to the Kafka brokers. That class contains a socket file descriptor, so you would want to add some OpenSSL calls in there (see TMetadataFetcher::Connect()).

There will also be some TLS-related options you will probably want to add to dory's config specifying things such as paths to certificate files. For this purpose, you can find the config file processing code in src/dory/conf.

To facilitate unit testing your code, it would be good to also add TLS support to the mock Kafka server which uses threads to simulate Kafka brokers. Then you can write unit test code where TLS-enabled dory sends requests to the TLS-enabled mock kafka server. I apologize in advance because the mock Kafka server is kind of a mess, but a good place to get started is to look at TConnectHandler::OnEvent() in src/dory/mock_kafka_server/connect_handler.cc. Here you can see where the mock kafka server accepts incoming connections from clients (i.e. dory). So you would want to add some OpenSSL calls somewhere in there.

I hope this helps!

Thanks,
Dave

[20Rishabh20]

Thanks for your response.
Will try this out.
Currently we are using dory version 2.0.5. So do we need to upgrade dory version?

[dspeterson]

Yes I would definitely upgrade to the latest version. The version you're using is 5 years old, and a lot of changes have been made since then, including bug fixes. My comments about where to find stuff in the code are all based on the latest version (currently 3.2.4).