Apache Kafka is a free and open-source streaming platform used for streaming analytics, data integration, and mission-critical applications. Compared to ActiveMQ and RabbitMQ, Kafka has better throughput, built-in partitioning, replication, and fault-tolerance. Kafka is a distributed message agent that can deal with huge volumes of real-time information effectively.
CMAK also known as a “Cluster Manager” is used for managing Kafka cluster developed by Yahoo. Using CMAK, you can manage multiple clusters and inspect cluster state including, topics, consumers, offsets, brokers, replica distribution, partition distribution and many more.
In this tutorial, we will show how to install Apache Kafka streaming platform with CMAK on CentOS 8 server.
Prerequisites
- A server running CentOS 8.
- A root password is configured the server.
Getting Started
First, updating your system packages to the latest version is recommended. You can update all with the following command:
dnf update -y
Once all the packages are updated, install other required dependencies with the following command:
dnf install git unzip -y
Once you are finished, you can proceed to the next step.
Install Java
Kafka is based on Java, so Java must be installed in your server. If not installed, you can install if with the following command:
dnf install java-11-openjdk-devel -y
Once the Java is installed, you can verify the installed version of Java using the following command:
java --version
You should get the following output:
openjdk 11.0.8 2020-07-14 OpenJDK Runtime Environment (build 11.0.8 10-post-CentOS-0centos8) OpenJDK 64-Bit Server VM (build 11.0.8 10-post-CentOS-0centos8, mixed mode, sharing)
Once you are finished, you can proceed to the next step.
Download Kafka
Before downloading Kafka, create a directory to store Kafka. You can create it with the following command:
mkdir /usr/local/kafka-server
Next, change the directory to kafka-server and download the latest version of Kafka using the following command:
cd /usr/local/kafka-server
wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz
Once the download is completed, extract the downloaded file with the following command:
tar -xvzf kafka_2.13-2.6.0.tgz --strip 1
Once you are finished, you can proceed to the next step.
Create a Systemd File for Kafka and Zookeeper
Next, you will need to create a systemd service file to manage the Zookeeper and Kafka service.
First, create a Zookeeper service file using the following command:
nano /etc/systemd/system/zookeeper.service
Add the following lines:
[Unit] Description=Apache Zookeeper Server Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save and close the file when you are finished.
Next, create a Kafka service file with the following command:
nano /etc/systemd/system/kafka.service
Add the following lines:
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service After=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk" ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save and close the file then reload the systemd daemon with the following command:
systemctl daemon-reload
Next, start both services and enable them to start at boot with the following command:
systemctl enable --now zookeeper
systemctl enable --now kafka
systemctl start zookeeper
systemctl start kafka
Next, verify the status of both services using the following command:
systemctl status zookeeper kafka
You should see the following output:
? zookeeper.service - Apache Zookeeper Server
Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2020-08-29 08:48:42 EDT; 30s ago
Main PID: 26722 (java)
Tasks: 31 (limit: 12523)
Memory: 68.8M
CGroup: /system.slice/zookeeper.service
??26722 java -Xmx512M -Xms512M -server -XX: UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX: ExplicitGCIn>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,326] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,327] INFO Created server with tickTime 3000 minSessionTimeout 6>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,344] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactor>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,348] INFO Configuring NIO connection handler with 10s sessionle>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,355] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zook>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,391] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zooke>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,397] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapsho>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,401] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapsho>
Aug 29 08:48:44 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:44,431] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.a>
Aug 29 08:48:53 centos8 zookeeper-server-start.sh[26722]: [2020-08-29 08:48:53,030] INFO Creating new log file: log.1 (org.apache.zookeeper.se>
? kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2020-08-29 08:48:50 EDT; 22s ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 27100 (java)
Tasks: 66 (limit: 12523)
Memory: 306.4M
CGroup: /system.slice/kafka.service
??27100 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX: UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancy>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,733] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.c>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,747] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.De>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,846] INFO [/config/changes-event-process-thread]: Starting (kafka.c>
Aug 29 08:48:56 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:56,877] INFO [SocketServer brokerId=0] Starting socket server acceptor>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,236] INFO [SocketServer brokerId=0] Started data-plane acceptor and>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,237] INFO [SocketServer brokerId=0] Started socket server acceptors>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,246] INFO Kafka version: 2.6.0 (org.apache.kafka.common.utils.AppIn>
Aug 29 08:48:57 centos8 kafka-server-start.sh[27100]: [2020-08-29 08:48:57,246] INFO Kafka commitId: 62abe01bee039651 (org.apache.kafka.common>
Install and Configure CMAK
Next, you will need to install CMAK for managing Apache Kafka cluster. You can download it from the Git repository with the following command:
cd /root
git clone https://github.com/yahoo/CMAK.git
Once downloaded, you will need to edit the application.conf file and define the zookeeper hosts.
nano ~/CMAK/conf/application.conf
Change the following lines:
cmak.zkhosts="localhost:2181"
Save and close the file when you are finished. Then, create a zip file to deploy the application.
cd ~/CMAK/
./sbt clean dist
You should see the following output:
model contains 640 documentable templates [info] Main Scala API documentation successful. [info] LESS compiling on 1 source(s) [success] All package validations passed [info] Your package is ready in /root/CMAK/target/universal/cmak-3.0.0.5.zip [success] Total time: 150 s (02:30), completed 29-Aug-2020, 8:55:40 AM
Next, unzip the zip file which is created in the previous step with the following command:
cd ~/CMAK/target/universal
unzip cmak-3.0.0.5.zip
Once you are finished, you can proceed to the next step.
Start CMAK Services
Now, change the directory to ~/CMAK/target/universal/cmak-3.0.0.5 and start the CMAK service using the following command:
cd ~/CMAK/target/universal/cmak-3.0.0.5
bin/cmak
At this point, CMAK is started and listening on port 9000.
Configure SELinux and Firewall
By default, SELinux is enable in CentOS 8. So you will need to configure SELinux for CMAK. You can configure it with the following command:
chcon -t httpd_sys_rw_content_t ~/CMAK/target/universal/cmak-3.0.0.5 -R
setsebool -P httpd_can_network_connect 1
Next, you will need to allow port 9000 through firewalld. You can do it with the following command:
firewall-cmd --permanent --zone public --add-port 9000/tcp
firewall-cmd --reload
Once you are finished, you can proceed to the next step.
Access CMAK Web UI
Now, open your web browser and access the CMAK using the URL http://your-server-ip:9000. You should see the following page:
In the above screen, you should see that there is no cluster available. So you will need to create a new cluster first.
Click on the Cluster => Add CLuster. You should see the following screen:
Provide your cluster name, zookeeper host, other required information and click on the Save button. You should see the following screen:
Once you are done, you can proceed to the next step.
Add Your First Topic on Kafka
Next, you will need to create a topic on Kafka to test the functionality. To do so, open another terminal, change the directoru to kafka-server and create a new topic by specifying a topic name and replication factor by running the following command:
cd /usr/local/kafka-server
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic MyNewTopic
Now, go to the CMAK dashboard and click on the Topic. You should see the Topic which you have created earlier in the following screen:
Click on the Topic number. You should see your newly created topic in the following screen:
Now, click on your Topic name. You should see all the information of your topic in the following screen:
Conclusion
Congratulations! you have successfully installed Apache Kafka with CMAK on CentOS 8 server. You can now start exploring CMAK for more functionality. Feel free to ask me if you have any questions.
