Kafka Cluster with Replication for Failover

For the replication configuration, the prerequisite is you should know the basics of Kafka with producer and consumer exchanging the data.

Kafka topics can be replicated across a number of nodes(broker) running in parallel. This is used for Fault Tolerance.
The leader is responsible for all Reads and Writes and the followers will replicate the data for each partition. Kafka distributes the partition across brokers and each has its own leadership for the topic partition.

The leader is chosen by the Zookeeper and the followers called in-sync (ISR) called in sync replica. If the leader fails it chooses a new ISR as a leader.

For setup, you can follow the given guidelines.

wget https://dist.apache.org/repos/dist/release/kafka/2.1.0/kafka_2.11-2.1.0.tgz

tar -xvf kafka_2.11-2.1.0.tgz

cd kafka_2.11-2.1.0/

Now we will copy the server properties and create two copies for the broker's

cp config/server.properties config/server1.properties
cp config/server.properties config/server2.properties

In server1.properties
broker.id=1
port=9093
log.dirs=./logs/kafka1

In server2.properties
broker.id=2
port=9094
log.dirs=./logs/kafka1

After configuring the server properties we can start the three brokers by command

./kafka-server-start.sh ../config/server.properties
./kafka-server-start.sh ../config/server1.properties
./kafka-server-start.sh ../config/server2.properties

And then we need to create a replicated topic.

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3  --partitions 13 --topic my-topic

To run the consumer we need to run the following command

./kafka-console-consumer.sh --bootstrap-server localhost:9093,localhost:9094,localhost:9094 --topic test1 --consumer-property group.id=mygroup

--bootstrap-server = list of servers

To run the producer we need to run the following command

./kafka-console-producer.sh --broker-list localhost:9094,localhost:9093,localhost:9098 --topic my-topic

Now you can send the data from the producer and it will be received on consumer side.

Also for failover, you can kill any broker process and the request from the producer will be served accordingly.

Producer Console:-

Hi Mom
How are you?
How are things going?
Good!

Consumer Console:-

Hi Mom
How are you?
How are things going?
Good!

Kafka Consumer Failover

Consumer failover can be checked by executing any of the consumers by sending more messages. Kafka should isolate up the work to the consumers that are running.