Apache Kafka is a distributed event streaming platform capable of handling trillions of events per day. It is used for building real-time data pipelines and streaming applications. This guide will walk you through the steps to install Apache Kafka on an Ubuntu 22.04 server. The guide is designed to be technical yet easy to understand, making it accessible for newcomers.

Step 1: Deploying a Cloud Instance on Shape.host

• Log in to Shape.host Dashboard:

• Navigate to the Shape.host website and log in to your account.

• Create a New Instance:

• Click on the “Create” button located at the top right corner of the dashboard.
• From the dropdown menu, select “Instances”.

• Select Instance Location:

• Choose the desired location for your server. For this tutorial, we’ll select “New York, USA”.

• Choose a Plan:

• Select a plan that fits your requirements. For example, you might choose a plan with 2 cores CPU, 2 GB Memory, and 50 GB SSD disk space.

• Select an Operating System:

• Scroll down to the “Choose an image” section and select “Ubuntu 22.04”.

• Configure Additional Options:

• (Optional) You can configure additional options like User Data Configuration and IPv6 Networking.
• Enter a hostname for your instance, e.g., “Tutorial Ubuntu”.
• Click on the “Create instance” button to deploy the instance.

Step 2: Connecting to Your Instance

• Retrieve SSH Credentials:

• Note the IP address of your newly created instance from the Shape.host dashboard.

• Connect via SSH:

• Open a terminal on your local machine.
• Use the following command to connect to your instance:
  sh ssh root@your_instance_ip
• Replace your_instance_ip with the actual IP address of your instance.

Step 3: Update the System

First, update your system packages to the latest versions:

sudo apt update
sudo apt upgrade -y

Step 4: Install Java

Kafka requires Java to run. We will install OpenJDK, which is an open-source implementation of the Java Platform:

sudo apt install -y openjdk-11-jdk

Verify the installation by checking the Java version:

java -version

You should see an output similar to:

openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu1.22.04)
OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu1.22.04, mixed mode, sharing)

Step 5: Create Kafka User

For security purposes, it is recommended to create a dedicated user for Kafka:

sudo adduser --system --no-create-home --disabled-login --group kafka

Step 6: Download and Install Apache Kafka

1. Download Kafka:

Navigate to the Apache Kafka download page and get the latest version link. Use wget to download Kafka:

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz

2. Extract Kafka:

Extract the downloaded file:

tar -xvzf kafka_2.12-3.7.0.tgz

3. Move Kafka Directory:

Move the extracted Kafka directory to /opt and create a symbolic link for easier access:

sudo mv kafka_2.12-3.7.0 /opt/kafka
sudo ln -s /opt/kafka /opt/kafka-latest

4. Set Ownership:

Set the ownership of the Kafka directory to the Kafka user:

sudo chown -R kafka:kafka /opt/kafka

Step 7: Configure Kafka

Kafka requires a few configurations to run properly.

1. Edit Zookeeper Configuration:

Kafka uses Zookeeper for coordination. Edit the Zookeeper configuration file:

sudo nano /opt/kafka/config/zookeeper.properties

Ensure it has the following settings:

dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=0

2. Edit Kafka Configuration:

Next, edit the Kafka server configuration:

sudo nano /opt/kafka/config/server.properties

Make sure to set the following parameters:

broker.id=0
log.dirs=/var/lib/kafka
zookeeper.connect=localhost:2181

Step 8: Create Systemd Unit Files

To manage Kafka as a system service, create systemd unit files for both Zookeeper and Kafka.

1. Zookeeper Unit File:

Create a systemd unit file for Zookeeper:

sudo nano /etc/systemd/system/zookeeper.service

Add the following content:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

2. Kafka Unit File:

Create a systemd unit file for Kafka:

sudo nano /etc/systemd/system/kafka.service

Add the following content:

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Step 9: Start and Enable Services

Start and enable the Zookeeper service:

sudo systemctl start zookeeper
sudo systemctl enable zookeeper

Then, start and enable the Kafka service:

sudo systemctl start kafka
sudo systemctl enable kafka

Check the status of Kafka to ensure it is running correctly:

sudo systemctl status kafka

You should see an output indicating that Kafka is active and running.

Step 10: Testing the Installation

To verify that Kafka is installed and configured correctly, you can create a test topic and send messages.

1. Create a Topic:

Create a test topic named test-topic:

/opt/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

2. List Topics:

Verify the topic was created:

/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

3. Send Messages:

Start a producer to send messages:

/opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

Type a few messages and press Enter after each message.

4. Consume Messages:

Open another terminal session and start a consumer to read the messages:

/opt/kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

You should see the messages you typed in the producer terminal.

Step 11: Manage Kafka Logs

Kafka logs are stored in the directory specified in the log.dirs parameter in the server.properties file. By default, Kafka logs are stored in /var/lib/kafka.

To view Kafka logs, you can use the following command:

  tail -f server.log

Additional Tips

1. Configure Kafka for Remote Access:
   If you need to access Kafka from a remote machine, update the listeners property in server.properties:

   listeners=PLAINTEXT://your_server_ip:9092

2. Secure Kafka:
   Consider securing Kafka with SSL and authentication mechanisms for production environments. This requires additional configuration in server.properties.
   

Shape.host offers robust and flexible Linux SSD VPS services that are ideal for hosting applications like Apache Kafka. With their easy-to-use dashboard, you can quickly deploy and manage cloud instances tailored to your specific needs. Shape.host provides a variety of plans to choose from, ensuring you get the resources you need for your projects. Whether you are a beginner setting up your first Kafka instance or a professional managing a large-scale deployment, Shape.host’s reliable services and support make it an excellent choice for your cloud hosting needs.