streams-perf

A bare bones example of how to setup streams (Kafka) v1.2 on openshift using an external TLS client which uses client side TLS for authorization. Note these instructions use the following configuration:

• Red Hat internal env

• project/namespace: kafka

• kafka cluster name: my-cluster

• topics: my-topic (20 partitions / 3 replicas)

Prereqs:

• JDK 1.8

• OpenSSL

Note	In order to install the cluster operator a user with cluster administrator

For the purpose of this testing we used these settings

• memory (min) = 4 * CPU (1 CPU = 4 GB of RAM)

• java memory

  • Xms = Xmx

  • Xmx is roughly 1/4 of the total pod memory

    • pod: 4 GB, Xmx = 1 GB

Install cluster operator

After logging into openshift via the cli create a new project

oc new-project kafka

Create the cluster operator in the kafka project/namespace

oc apply -f install/cluster-operator -n kafka
oc apply -f examples/templates/cluster-operator -n kafka

Install and configure kafka cluster

Create the kafka cluster in the name space

oc apply -f examples/kafka/kafka-persistent.yaml

Create a topic (make sure the 3 zookeeper and 3 kafka pods are started)

oc apply -f examples/topic/kafka-topic.yaml

Create TLS based users

oc apply -f examples/user/kafka-user.yaml

Install and configure perf clients

Use the client directory for all perf client activity

cd client/bin

Install openshift secrets into key and trust store

rm  -f *.crt *.key *.jks *.pkcs12
oc extract secret/my-cluster-cluster-ca-cert --keys ca.crt
keytool -keystore client-truststore.jks -alias CARoot -import -file ca.crt
oc extract secret/my-user --confirm
openssl pkcs12 -inkey user.key -in user.crt -export -name client -out client.pkcs12
keytool -importkeystore -srckeystore client.pkcs12 -srcstoretype pkcs12 -destkeystore client-keystore.jks -deststoretype pkcs12

Modify Configuration

Modify the consumer.properties and producer.properties to meet your environment configuration

Execute the producer performance test

Note: You must have kafka installed locally and its bin directory is included in the PATH environment variable

./kafka-producer-perf-test.sh --producer.config producer.properties --throughput 15000 --num-records 150000 --record-size 5000 --topic my-topic
./kafka-consumer-perf-test.sh --consumer.config consumer.properties --topic my-topic --group my-group --messages 150000 --timeout 9999999999 --threads 20 --broker-list=https://my-cluster-kafka-0-kafka.apps.cluster-e6db.sandbox239.opentlc.com:443,https://my-cluster-kafka-1-kafka.apps.cluster-e6db.sandbox239.opentlc.com:443,https://my-cluster-kafka-2-kafka.apps.cluster-e6db.sandbox239.opentlc.com:443

Delete topics and clusters

Delete topic, user and cluster with sample below

oc delete kafkauser my-user
oc delete kafkatopic my-topic
oc delete kafka my-cluster

Running within container

Download your service account from https://access.redhat.com/terms-based-registry/ as an OpenShift secret and reference it below.

oc create -f my-service-account-pull-secret.yaml
oc secrets link builder my-service-account-pull-secret
oc new-app --name perf-client https://github.com/rediverson/streams-perf.git
oc rsh perf-client-1-t7tn7
cd /tmp/home/streams-perf/client/bin
/tmp/home/streams-perf/client/bin/oc login -u xxxx -p xxxxx https://api.cluster-dal-0975.dal-0975.open.redhat.com:6443
/tmp/home/streams-perf/client/bin/oc extract secret/my-cluster-cluster-ca-cert --keys ca.crt
keytool -keystore client-truststore.jks -alias CARoot -import -file ca.crt
/tmp/home/streams-perf/client/bin/oc extract secret/my-user --confirm
openssl pkcs12 -inkey user.key -in user.crt -export -name client -out client.pkcs12
keytool -importkeystore -srckeystore client.pkcs12 -srcstoretype pkcs12 -destkeystore client-keystore.jks -deststoretype pkcs12

Config modification and running the tests

Start with the Modify Configuration step above and use the fully qualified service addresses within openshift. Example below:

bootstrap.servers=https://my-cluster-kafka-bootstrap.kafka.svc.cluster.local:9093

Execute the tests using the instructions above under the Execute the producer performance test section.