Apache Kafka is a distributed event streaming platform capable of handling trillions of events per day. It is used for building real-time data pipelines and streaming applications. This guide will walk you through the steps to install Apache Kafka on an Ubuntu 22.04 server. The guide is designed to be technical yet easy to understand, making it accessible for newcomers.
Step 1: Deploying a Cloud Instance on Shape.host
- Log in to Shape.host Dashboard:
- Navigate to the Shape.host website and log in to your account.
- Create a New Instance:
- Click on the “Create” button located at the top right corner of the dashboard.
- From the dropdown menu, select “Instances”.
- Select Instance Location:
- Choose the desired location for your server. For this tutorial, we’ll select “New York, USA”.
- Choose a Plan:
- Select a plan that fits your requirements. For example, you might choose a plan with 2 cores CPU, 2 GB Memory, and 50 GB SSD disk space.
- Select an Operating System:
- Scroll down to the “Choose an image” section and select “Ubuntu 22.04”.
- Configure Additional Options:
- (Optional) You can configure additional options like User Data Configuration and IPv6 Networking.
- Enter a hostname for your instance, e.g., “Tutorial Ubuntu”.
- Click on the “Create instance” button to deploy the instance.
Step 2: Connecting to Your Instance
- Retrieve SSH Credentials:
- Note the IP address of your newly created instance from the Shape.host dashboard.
- Connect via SSH:
- Open a terminal on your local machine.
- Use the following command to connect to your instance:
sh ssh root@your_instance_ip
- Replace
your_instance_ip
with the actual IP address of your instance.
Step 3: Update the System
First, update your system packages to the latest versions:
sudo apt update
sudo apt upgrade -y
Step 4: Install Java
Kafka requires Java to run. We will install OpenJDK, which is an open-source implementation of the Java Platform:
sudo apt install -y openjdk-11-jdk
Verify the installation by checking the Java version:
java -version
You should see an output similar to:
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu1.22.04)
OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu1.22.04, mixed mode, sharing)
Step 5: Create Kafka User
For security purposes, it is recommended to create a dedicated user for Kafka:
sudo adduser --system --no-create-home --disabled-login --group kafka
Step 6: Download and Install Apache Kafka
- Download Kafka:
Navigate to the Apache Kafka download page and get the latest version link. Use wget
to download Kafka:
wget https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz
- Extract Kafka:
Extract the downloaded file:
tar -xvzf kafka_2.12-3.7.0.tgz
- Move Kafka Directory:
Move the extracted Kafka directory to /opt
and create a symbolic link for easier access:
sudo mv kafka_2.12-3.7.0 /opt/kafka
sudo ln -s /opt/kafka /opt/kafka-latest
- Set Ownership:
Set the ownership of the Kafka directory to the Kafka user:
sudo chown -R kafka:kafka /opt/kafka
Step 7: Configure Kafka
Kafka requires a few configurations to run properly.
- Edit Zookeeper Configuration:
Kafka uses Zookeeper for coordination. Edit the Zookeeper configuration file:
sudo nano /opt/kafka/config/zookeeper.properties
Ensure it has the following settings:
dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=0
- Edit Kafka Configuration:
Next, edit the Kafka server configuration:
sudo nano /opt/kafka/config/server.properties
Make sure to set the following parameters:
broker.id=0
log.dirs=/var/lib/kafka
zookeeper.connect=localhost:2181
Step 8: Create Systemd Unit Files
To manage Kafka as a system service, create systemd unit files for both Zookeeper and Kafka.
- Zookeeper Unit File:
Create a systemd unit file for Zookeeper:
sudo nano /etc/systemd/system/zookeeper.service
Add the following content:
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
- Kafka Unit File:
Create a systemd unit file for Kafka:
sudo nano /etc/systemd/system/kafka.service
Add the following content:
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Step 9: Start and Enable Services
Start and enable the Zookeeper service:
sudo systemctl start zookeeper
sudo systemctl enable zookeeper
Then, start and enable the Kafka service:
sudo systemctl start kafka
sudo systemctl enable kafka
Check the status of Kafka to ensure it is running correctly:
sudo systemctl status kafka
You should see an output indicating that Kafka is active and running.
Step 10: Testing the Installation
To verify that Kafka is installed and configured correctly, you can create a test topic and send messages.
- Create a Topic:
Create a test topic named test-topic
:
/opt/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
- List Topics:
Verify the topic was created:
/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
- Send Messages:
Start a producer to send messages:
/opt/kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
Type a few messages and press Enter
after each message.
- Consume Messages:
Open another terminal session and start a consumer to read the messages:
/opt/kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092
You should see the messages you typed in the producer terminal.
Step 11: Manage Kafka Logs
Kafka logs are stored in the directory specified in the log.dirs
parameter in the server.properties
file. By default, Kafka logs are stored in /var/lib/kafka
.
To view Kafka logs, you can use the following command:
tail -f server.log
Additional Tips
- Configure Kafka for Remote Access:
If you need to access Kafka from a remote machine, update thelisteners
property inserver.properties
:
listeners=PLAINTEXT://your_server_ip:9092
- Secure Kafka:
Consider securing Kafka with SSL and authentication mechanisms for production environments. This requires additional configuration inserver.properties
.
Shape.host offers robust and flexible Linux SSD VPS services that are ideal for hosting applications like Apache Kafka. With their easy-to-use dashboard, you can quickly deploy and manage cloud instances tailored to your specific needs. Shape.host provides a variety of plans to choose from, ensuring you get the resources you need for your projects. Whether you are a beginner setting up your first Kafka instance or a professional managing a large-scale deployment, Shape.host’s reliable services and support make it an excellent choice for your cloud hosting needs.