‍

Apache Kafka is a powerful streaming platform that is widely used for streaming analytics, data integration, and mission-critical applications. In this guide, we will walk you through the step-by-step process of installing Apache Kafka on CentOS, along with the Cluster Manager (CMAK) for managing your Kafka cluster. By the end of this tutorial, you will have a fully functional Kafka setup, ready to process and analyze real-time data.

Prerequisites

Before we dive into the installation process, let’s make sure we have all the necessary prerequisites in place:

• A server running CentOS 8.
• Root access to the server.

Getting Started

To begin, let’s update our system packages to ensure we have the latest versions. Open a terminal and run the following command:

dnf update -y

Once the update process is complete, we can proceed to install the dependencies. Run the following command to install Git and Unzip:

dnf install git unzip -y

With the prerequisites taken care of, we can now move on to installing Java, as Kafka is based on Java.

Install Java

To install Java, run the following command:

dnf install java-11-openjdk-devel -y

Once the installation is complete, verify the installed version of Java by running:

java --version

You should see the Java version displayed, confirming a successful installation.

Download Kafka

Now that Java is installed, let’s proceed with downloading Kafka. First, create a directory to store Kafka by running the following command:

mkdir /usr/local/kafka-server

Next, navigate to the Kafka directory and download the latest version of Kafka:

cd /usr/local/kafka-server
wget https://downloads.apache.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz

Once the download is complete, extract the downloaded file:

tar -xvzf kafka_2.13-2.6.0.tgz --strip1

Congratulations! You now have Kafka installed on your CentOS server.

Create a Systemd File for Kafka and Zookeeper

To manage the Kafka and Zookeeper services, we need to create systemd service files for them. Let’s start with Zookeeper. Run the following command to create the Zookeeper service file:

nano /etc/systemd/system/zookeeper.service

Paste the following content into the file:

[Unit]
Description=Apache Zookeeper Server
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties
ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file.

Next, create the Kafka service file by running the following command:

nano /etc/systemd/system/kafka.service

Paste the following content into the file:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties
ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Save and close the file.

Now, reload the systemd daemon to apply the changes:

systemctl daemon-reload

Start both the Zookeeper and Kafka services and enable them to start at boot:

systemctl enable --now zookeeper
systemctl enable --now kafka

Verify the status of both services:

systemctl status zookeeper kafka

You should see that both services are active and running.

Install and Configure CMAK

CMAK (Cluster Manager) is a powerful tool developed by Yahoo for managing Kafka clusters. To install CMAK, we need to clone the CMAK repository from Git. Run the following commands to download and configure CMAK:

cd /root
git clone https://github.com/yahoo/CMAK.git

Once the repository is cloned, navigate to the CMAK directory and edit the application.conf file:

nano ~/CMAK/conf/application.conf

In the file, locate the line that starts with cmak.zkhosts= and replace it with the following:

cmak.zkhosts="localhost:2181"

Save and close the file.

Next, we need to create a zip file to deploy the CMAK application. Run the following commands:

cd ~/CMAK/
./sbt clean dist

Once the zip file is created, navigate to the target directory and unzip it:

cd ~/CMAK/target/universal
unzip cmak-3.0.0.5.zip

Congratulations! CMAK is now installed and ready to be started.

Start CMAK Services

Navigate to the CMAK directory and start the CMAK service:

cd ~/CMAK/target/universal/cmak-3.0.0.5
bin/cmak

CMAK is now up and running, listening on port 9000.

Configure SELinux and Firewall

By default, SELinux is enabled on CentOS 8. We need to configure SELinux to allow CMAK to function properly. Run the following commands:

chcon -t httpd_sys_rw_content_t ~/CMAK/target/universal/cmak-3.0.0.5 -R
setsebool -P httpd_can_network_connect 1

Next, we need to allow port 9000 through the firewall. Run the following commands:

firewall-cmd --permanent --zonepublic --add-port 9000/tcp
firewall-cmd --reload

You have successfully configured SELinux and the firewall.

Access CMAK Web UI

Open your web browser and enter the URL http://your-server-ip:9000 to access the CMAK web interface.

On the CMAK dashboard, you will see that there are no clusters available. Let’s create a new cluster. Click on “Cluster” and then “Add Cluster”.

Provide a name for your cluster, along with the Zookeeper host and any other required information. Click “Save” to create the cluster.

You should now see your newly created cluster on the CMAK dashboard.

Add Your First Topic on Kafka

To test the functionality of Kafka, let’s create a topic. Open a new terminal window and navigate to the Kafka directory:

cd /usr/local/kafka-server

Create a new topic by running the following command, replacing MyNewTopic with the desired topic name:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic MyNewTopic

Switch back to the CMAK dashboard and click on “Topics”. You should now see the topic you just created listed.

Click on the topic number to view more details about the topic.

Conclusion

Congratulations! You have successfully installed Apache Kafka with CMAK on your CentOS 8 server. You now have a powerful streaming platform and a robust cluster manager at your disposal.

With Kafka and CMAK, you can handle real-time data streaming, perform data integration, and build mission-critical applications. Explore the CMAK interface to unleash the full potential of your Kafka cluster.

For reliable and scalable cloud hosting solutions, consider Shape.host. Shape.host offers Linux SSD VPS with top-notch performance and security features. Take your Kafka deployment to the next level with Shape.host’s hosting services.

Remember to always keep your Kafka cluster and CMAK installation up to date to benefit from the latest features and security patches.