Apache Kafka Broker Cluster Setup on Docker
Posted By : Shubham Jain | 31-Dec-2018
High availability and network balance of a service
- Zero downtime.
- Work as a single unit.
- Data Redundancy.
In this blog, I will tell you how to set up an Apache Kafka broker cluster on Docker so that serving of data can be continuous. Why we are setting this on docker? Because it is easier to set up a node without any problem and without depending on the base environment.
For now, we are setting up the environment with a three node cluster, as it is a basic requirement so that a cluster can choose a master node to manage the environment. So we require three servers to set up the cluster.
- Docker ( to run containers)
- Docker-compose (to run a docker compose file)
- The network connection between the servers.
A Kafka broker cluster also consists of Zookeeper to manage the nodes master and kept metadata between them that which node is containing data and how many replicas are there. Each Node contains both Kafka and zookeeper as a pair. All zookeeper nodes will be connected to each other, also all Kafka servers will be connected to all the zookeeper nodes.
Here is the docker-compose file to set up the cluster.
version: '2' services: zoo: image: zookeeper:3.4 restart: always ports: - 2181:2181 - 2888:2888 - 3888:3888 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=<server 2 IP>:2888:3888 server.3=<server 3 IP>:2888:3888 kafka: image: wurstmeister/kafka:2.11-1.0.1 restart: always ports: - 9092:9092 environment: KAFKA_ADVERTISED_HOST_NAME: <host IP> KAFKA_PORT_NUMBER: 9092 KAFKA_ZOOKEEPER_CONNECT: <server 1 IP>:2181,<server 2 IP> :2181,<server 3 IP>:2181 KAFKA_ADVERTISED_LISTENERS: INSIDE://:19092,OUTSIDE://<Host IP>:9092 KAFKA_LISTENERS: INSIDE://:19092,OUTSIDE://:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_BROKER_ID: 1
The ZOO_MY_ID and KAFKA_BROKER_ID value is unique from the other server. For server1 we are using 1, for server2 we are using 2 and for server3 we are using 3 ID for both ZOO_MY_ID and KAFKA_BROKER_ID.
Change the <server IP> according to that server host IP.
After creating docker-compose.yaml file in all three servers run the service of both kafka and Zookeeper by using folloqing command.
docker-compose up -d zoo kafka
Now the kafka Broker cluster is fully setup.
To connect to the kafka cluster you have to provide all the three kafka server IPs with broker port i.e. 9092 in comma separated form so that the listener and producer can connect to all the brokers.