Celonis Product Documentation

Apache Hadoop 3.x fully needs Java 8 → Install java 8:

$ sudo apt install openjdk-8-jdk -y    

Install OpenSSH on Ubuntu:

$ sudo apt install openssh-server openssh-client -y

Setup a non-root user for Hadoop and switch to the new user:

$ sudo adduser hdoop
 
$ su - hdoop

Enable Passwordless SSH for Hadoop User:

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsacat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keyschmod 0600 ~/.ssh/authorized_keysssh localhost

Download and Install Hadoop on Ubuntu:

$ wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
 
$ tar xzf hadoop-3.2.1.tar.gz

Configure Hadoop Environment Variables (bashrc):

#Edit the .bashrc shell configuration file using a text editor of your choice
$ sudo nano .bashrc

#Copy the following Hadoop Related Options
export HADOOP_HOME=/home/hdoop/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/nativ"

#Apply the changes to the current running environment
$ source ~/.bashrc

Edit the hadoop-env.sh file:

$ sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
 
#Uncomment the $JAVA_HOME variable and add the full path to the OpenJDK installation on your system
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Edit core-site.xml File to specify the URL for your NameNode, and the temporary directory:

$ sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml
 
#Copy the following to the file 
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hdoop/tmpdata</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://127.0.0.1:9000</value>
</property>
</configuration>

Create the temporary directory added in the previous step.

Edit hdfs-site.xmlFile to define the NameNode and DataNode storage directories:

$ sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml
 
#Add the following configuration to the file and, if needed, adjust the NameNode and DataNode directories to your custom locations.
<configuration>
<property>
<name>dfs.data.dir</name>
<value>/home/hdoop/dfsdata/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hdoop/dfsdata/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

Create the specific directories you defined for the dfs.data.dir value in above step.

Editmapred-site.xmlFile to define MapReduce values.

$ sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml
 
#Add the following to the file
<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
</configuration>

Edit the yarn-site.xml file to define settings relevant to YARN. It contains configurations for the Node Manager, Resource Manager, Containers, and Application Master.

$ sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml
 
#Append the following configuration to the file<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>127.0.0.1</value>
</property>
<property>
<name>yarn.acl.enable</name>
<value>0</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>

Format HDFS NameNode:

#It is important to format the NameNode before starting Hadoop services for the first time
 
$ hdfs namenode -format

Start Hadoop Cluster:

#Navigate to the directory
$ cd hadoop-3.2.1/sbin
 
#Execute
$ ./start-all.sh

Now you can verify and access NameNode , DataNodes and YARN Resource Manager over the browser:

#Hadoop NameNode UI
http://localhost:9870
 
#DataNodes
http://localhost:9864
 
#YARN Resource Manager
http://localhost:8088