‍Apache Hadoop is an open-source, Java-based software platform that manages data processing and storage for big data applications. It is widely used by businesses to handle large datasets and perform complex analytics tasks. In this article, we will guide you through the process of installing and configuring Apache Hadoop on Debian 11, step by step.

System Update and Java Installation

Before we begin with the installation of Apache Hadoop, it is essential to update the system packages to the latest version. To do so, open the terminal and run the following command:

sudo apt-get update -y

Once the system packages are updated, we need to install Java, as Apache Hadoop is a Java-based application. Run the following command to install the default JDK and JRE:

sudo apt-get install default-jdk default-jre -y

To verify the Java installation, run the following command:

java-version

The output should display the installed Java version.

Creating a Hadoop User and Setting up Passwordless SSH

To ensure secure access and management of Apache Hadoop, it is recommended to create a dedicated user. Run the following command to create a Hadoop user:

sudo adduser hadoop

After creating the user, switch to the Hadoop user by running the following command:

su - hadoop

Next, generate an SSH key for the Hadoop user by running the following command:

ssh-keygen -t rsa

This command will generate a public and private key pair. Press Enter to accept the default file location and passphrase. The SSH key will be generated and saved in the specified location.

To enable passwordless SSH access, add the public key of your computer to the authorized_keys file of the Hadoop user. Run the following command:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

To verify the passwordless SSH connection, run the following command:

ssh [Server's_IP_Address]

Replace [Server’sIPAddress] with the IP address of the server you are connecting to.

Installing Apache Hadoop

Switch back to the Hadoop user and download the latest version of Apache Hadoop using the following command:

su - hadoop
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz

Once the download is complete, extract the downloaded tar file using the following command:

tar -xvzf hadoop-3.3.0.tar.gz

Now, switch back to the root user for the remaining commands:

su root

Move the extracted Hadoop files to the appropriate directory by running the following command:

cd /home/hadoop
mv hadoop-3.3.0 /usr/local/hadoop

Create a log directory to store Apache Hadoop logs:

mkdir /usr/local/hadoop/logs

Change the ownership of the /usr/local/hadoop directory to the Hadoop user:

chown -R hadoop:hadoop /usr/local/hadoop

Switch back to the Hadoop user:

su - hadoop

Configuring Hadoop Environment Variables

To configure the Hadoop environment variables, enter the edit mode of the .bashrc file by running the following command:

nano ~/.bashrc

Add the following configuration at the end of the file:

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

Save the changes and exit the editor.

To activate the added environment variables, run the following command:

source ~/.bashrc

Configuring Hadoop on a Single Node

If you are new to Hadoop and want to explore basic commands or test applications, you can configure Hadoop on a single node. To do so, follow the steps below:

Configure Java Environment Variables

Determine the path of the installed Java version by running the following command:

which javac

The output should display the path to the Java compiler.

Next, find the OpenJDK directory by running the following command:

readlink -f /usr/bin/javac

Make a note of the output, as we will need it in the next step.

Edit the hadoop-env.sh file by running the following command:

nano $HADOOP_HOME /etc/hadoop/hadoop-env.sh

Add the following configuration at the end of the file:

export JAVA_HOME=[Java_Path]
export HADOOP_CLASSPATH+=" $HADOOP_HOME/lib/*.jar"

Replace [Java_Path] with the path obtained from the previous step.

Save the changes and exit the editor.

Download the Javax Activation File

To download the Javax Activation file, run the following command:

cd /usr/local/hadoop/lib
sudo wget https://jcenter.bintray.com/javax/activation/javax.activation-api/1.2.0/javax.activation-api-1.2.0.jar

Verify the Hadoop Version

To verify the installed Hadoop version, run the following command:

hadoop version

The output should display the installed Hadoop version.

Configuring Hadoop Files

To configure Hadoop, we need to modify several XML files. Follow the steps below to configure each file:

Configure core-site.xml File

Open the core-site.xml file using a text editor:

vi $HADOOP_HOME/etc/hadoop/core-site.xml

Add the following configuration inside the <configuration> tags:

<property>
<name>fs.default.name</name>
<value>hdfs://0.0.0.0:9000</value>
<description>The default file system URI</description>
</property>

Save the changes and exit the editor.

Configure hdfs-site.xml File

Create a directory to store the node metadata:

mkdir -p /home/hadoop/hdfs/{namenode,datanode}
chown -R hadoop:hadoop/home/hadoop/hdfs

Open the hdfs-site.xml file:

vi $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Add the following configuration inside the <configuration> tags:

<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///home/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///home/hadoop/hdfs/datanode</value>
</property>

Save the changes and exit the editor.

Configure mapred-site.xml File

Open the mapred-site.xml file:

vi $HADOOP_HOME/etc/hadoop/mapred-site.xml

Add the following configuration inside the <configuration> tags:

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

Save the changes and exit the editor.

Configure yarn-site.xml File

Open the yarn-site.xml file:

vi $HADOOP_HOME/etc/hadoop/yarn-site.xml

Add the following configuration inside the <configuration> tags:

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

Save the changes and exit the editor.

Formatting HDFS NameNode

Before starting the Hadoop services for the first time, it is important to format the NameNode. Run the following command:

hdfs namenode-format

Starting the Hadoop Cluster

To start the Hadoop cluster, follow the steps below:

Start the NameNode and DataNode

Switch to the Hadoop user and start the NameNode and DataNode services by running the following command:

start-dfs.sh

Start the YARN Resource Manager and NodeManagers

Start the YARN resource manager and nodemanagers by running the following command:

start-yarn.sh

Verifying the Hadoop Cluster

To verify if all the Hadoop daemons are active and running as Java processes, run the following command:

jps

The output should display the list of running processes, including NameNode, DataNode, ResourceManager, and NodeManager.

Accessing the Hadoop Web Interface

To access the Hadoop web interface, follow the steps below:

Hadoop NameNode

Open a web browser and navigate to the following URL:

http://your-server-ip:9870

Replace your-server-ip with the IP address of your server.

Individual DataNodes

To access individual DataNodes, navigate to the following URL:

http://your-server-ip:9864

Replace your-server-ip with the IP address of your server.

YARN Resource Manager

To access the YARN resource manager, navigate to the following URL:

http://your-server-ip:8088

Replace your-server-ip with the IP address of your server.

Congratulations! You have successfully installed and configured Apache Hadoop on Debian 11. You can now leverage the power of Hadoop to process and analyze large datasets. If you are looking for reliable and scalable cloud hosting solutions to run your Hadoop clusters, consider Shape.host’s Linux SSD VPS services. Shape.host offers high-performance VPS hosting with excellent support and competitive pricing. Start unlocking the potential of big data with Apache Hadoop and Shape.host today!