P2. Basic commands

Basic commands

The script files are located in the bin folder for OSX/Unix users and in the bin/windows folder for Windows users. All the next steps assume that you are in the directory where the script files are located.

To start a Zookeeper instance:

• OSX/Unix users:

./zookeeper-server-start.sh ../config/zookeeper.properties

• Windows users:

zookeeper-server-start.bat ../../config/zookeeper.properties

The next step is to start a broker. Before doing so you should open the broker server.properties file in the config folder and search for the property:

advertised.listeners

This property should be set like this:

advertised.listeners=PLAINTEXT://localhost:9092

with the proper host and port.

To start a Kafka broker:

• OSX/Unix users:

./kafka-server-start.sh ../config/server.properties

• Windows users:

kafka-server-start.bat ../../config/server.properties

To create a topic:

• OSX/Unix users:

./kafka-topics.sh --create --topic my-first-topic --zookeeper localhost:2181 --replication-factor 1 --partitions 1

• Windows users:

kafka-topics.bat --create --topic my-first-topic --zookeeper localhost:2181 --replication-factor 1 --partitions 1

To alter the partitions of a topic:

• OSX/Unix users:

./kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-first-topic --partitions 4

• Windows users:

kafka-topics.bat --zookeeper localhost:2181 --alter --topic my-first-topic --partitions 4

To list all the available topics:

• OSX/Unix users:

./kafka-topics.sh --describe --zookeeper localhost:2181

• Windows users:

kafka-topics.bat --describe --zookeeper localhost:2181

To instantiate a Console Producer:

• OSX/Unix users:

./kafka-console-producer.sh --broker-list localhost:9092 --topic my-first-topic

• Windows users:

kafka-console-producer.bat --broker-list localhost:9092 --topic my-first-topic

To instantiate a Console Consumer:

• OSX/Unix users:

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-first-topic --from-beginning

• Windows users:

kafka-console-consumer.bat --zookeeper localhost:2181 --topic my-first-topic --from-beginning

You can omit --from-beginning and in this case the consumer won't read the old messages

Configuring multiple brokers

Each server.properties file represents a specific broker. So in order to have multiple brokers we need to have multiple server.properties files. Let us assume we have three brokers in the cluster:Broker, Broker1, Broker2. We need to have three server.properties files:

• server.properties for Broker
• server1.properties for Broker1
• server2.properties for Broker2

In each of the files we need to set a unique id identifying the broker:

• broker.id = 0 in server.properties
• broker.id = 1 in server1.properties
• broker.id = 2 in server2.properties

We should also set:

• advertised.listeners = PLAINTEXT://9092 in server.properties

• advertised.listeners = PLAINTEXT://9093 in server1.properties

• advertised.listeners = PLAINTEXT://9094 in server2.properties

• log.dirs =/tmp/kafka-logs in server.properties

• log.dirs =/tmp/kafka-logs1 in server1.properties

• log.dirs =/tmp/kafka-logs2 in server2.properties

The commands to start the brokers will be:

• ./kafka-server-start.sh ../config/server.properties
• ./kafka-server-start.sh ../config/server1.properties
• ./kafka-server-start.sh ../config/server2.properties

Kafka Commit LOG

In the server.properties file of config folder there is a property:

log.dirs=/tmp/kafka-logs

The above setting shows where Kafka logs are written for the broker which is associated to the server.properties file.

If you type the command ls /tmp/kafka-logs on OSX/Unix or DIR /tmp/kafka-logs on Windows you can see there is a directory: my-first-topic-0

Kafka created a directory with name topicName-partitionNumber.

For the broker associated to the server.properties file Kafka creates a log directory per each topic partition. In our case there is just one broker with one topic with a single partition. If you look at the command to create a topic you can see that we specified --partitions 1. If the topic was created specifying two partitions there would be one more directory my-first-topic-1 under the /tmp/kafka-logs directory of the broker. For each record that gets published, there will be a log and so the log will contain the ordered sequence of records published to the topic partition.