This document describes how to set up and configure a scalable HDFS data architecture with Apache Hadoop, Hive, Spark, Nifi, and Livy. It will guide you through the setup and configuration process, enabling you to perform distributed storage, querying, processing, and data flow automation efficiently using Hadoop functionalities, Hive queries, Spark processing, and Nifi’s data integration capabilities.
Apache Hadoop
To initiate the HDFS setup, it is essential first to establish a Hadoop cluster. A Hadoop cluster is a special computational cluster designed specifically for storing and analyzing huge amounts of data in a distributed computing environment. This type of setup is pivotal as it allows for the distributed processing of large data sets across clusters of computers, providing an efficient and effective means of handling vast amounts of data.
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
Prerequisites
Required software for Linux include:
-
Java
Recommended Java versions are: Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only).
And export your JAVA_HOME path to the shell configuration file. If you’re using Bash, this file is usually
~/.bashrcor~/.bash_profile. If you’re using a different shell, the file might be different (e.g.,~/.zshrcfor Zsh).
Download the recommended Java JDK from repositories or official Java website.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Java JDK.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")This will update dynamically and works well with the alternatives system.
Save your shell configurations.
source ~/.bashrc-
ssh
ssh must be installed and sshd must be running to use the Hadoop scripts that manage remote Hadoop daemons if the optional start and stop scripts are to be used.
Install the OpenSSH server according to your operating system.
Ubuntu for example:
sudo apt install openssh-serverEnable the ssh service.
sudo systemctl enable sshd --nowStart the ssh service.
sudo systemctl start sshdDownload and Install Apache Hadoop
To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.
wget <https://dlcdn.apache.org/hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz>Extract the contents of the file to the desired location (/usr/local/ or /opt/ is recommended).
cd /usr/local/
tar -xvf hadoop-3.4.0.tar.gzRename the directory.
mv hadoop-3.4.0.tar.gz/ hadoopGive user execution permissions to the directory (superuser permissions might be required).
chmod 777 hadoop/Now we’ll add the Hadoop path to the environment variables of out shell.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Hadoop directory.
export HADOOP_HOME=/usr/local/hadoop
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbinSave your shell configurations.
source ~/.bashrcNow, we have successfully installed Apache Hadoop in the system.
Setting up the nodes in cluster
Create host file on each node
For each node to communicate with each other by name, edit the /etc/hosts file to add the private IP addresses of the three servers.
file: /etc/hosts
192.168.8.202 masternode
192.168.8.204 node1
192.168.8.201 node2
192.168.8.199 node3Note: Don’t forget to replace the above IPs with your IPs.
Here, the masternode will act as the Namenode and the nodes node1, node2, and node3 which are worker nodes will act as the Datanodes in the Hadoop cluster.
Set up keys for ssh connections.
The master node will use an SSH connection to connect to other nodes with key-pair authentication. This will allow the master node to actively manage the cluster.
Generate SSH key for masternode as well as the worker nodes. Keep all the options as default and keep pressing enter until the key is generated.
ssh-keygen -b 4096Now display and copy the generated key.
cat ~/.ssh/id_rsa.pubCopy your key file into the authorized key store.
cat ~/.ssh/master.pub >> ~/.ssh/authorized_keysFollow the steps for each of the nodes and add keys from each node. This should get you four keys in the authorized_keys in all nodes.
Configuring the Cluster and Hadoop
Edit the hadoop-env.sh file
Export the Java path in the Hadoop environment and replace this lines.
file: /usr/local/hadoop/etc/hadoop/hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")
# Location of Hadoop.
export HADOOP_HOME=/usr/local/hadoop
# Location of Hadoop configuration directory.
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Where (primarily) daemon log files are stored.
export HADOOP_LOG_DIR=${HADOOP_HOME}/logsSet NameNode Location
Update your core-site.xml file to set the NameNode location to masternode on port 9000:
file: /usr/local/hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://masternode:9000</value>
</property>
</configuration>Set path for HDFS
Edit *hdfs-site.xml to resemble the following configuration:
file: /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/data/nameNode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/data/dataNode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>The dfs.replication indicates how many times data is replicated in the cluster. You can set 2 to have all the data duplicated on the two nodes.
Set YARN as Job Scheduler
Edit the mapred-site.xml file, setting YARN as the default framework for MapReduce operations:
file: /usr/local/hadoop/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
</configuration>Configure YARN
Edit yarn-site.xml, which contains the configuration options for YARN:
file: /usr/local/hadoop/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.acl.enable</name>
<value>0</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>masternode</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>Configure Workers
The file workers is used by startup scripts to start required daemons on all nodes. Edit workers file to include both of the nodes:
file: /usr/local/hadoop/etc/hadoop/workers
masternode
node1
node2
node3Similarly, set up this configuration on all nodes with the same configurations.
Interacting with the cluster
HDFS needs to be formatted like any classical file system. On masternode, run the following command:
bin/hdfs namenode -formatYour Hadoop installation is now configured and ready to run.
Start HDFS and YARN
Start the HDFS by running the following script from masternode:
start-dfs.shThis will start NameNode and SecondaryNameNode on masternode, and DataNode on node1, node2, and node3 according to the configuration in the workers config file.
Start YARN with the script from masternode:
start-yarn.shThis will start ResourceManager and NodeManager on masternode, and DataNode on node1, node2, and node3 according to the configuration in the workers config file.
Check that every process is running with the jps command on each node.
jpsOn masternode, you should see the following (the PID number will be different):
19169 ResourceManager
18690 SecondaryNameNode
19412 NodeManager
29831 Jps
17886 NameNodeAnd on node1, node2, and node3 you should see the following:
654360 NodeManager
2254614 Jps
648239 DataNodeMonitor your HDFS Cluster and YARN
You can get useful information about running your HDFS cluster with the hdfs dfsadmin command.
hdfs dfsadmin -reportThis will print information (e.g., capacity and usage) for all running DataNodes.
Configured Capacity: 1118728294400 (1.02 TB)
Present Capacity: 924032229376 (860.57 GB)
DFS Remaining: 924032131072 (860.57 GB)
DFS Used: 98304 (96 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 192.168.8.199:9866 (node6)
Hostname: hadoop
Decommission Status : Normal
Configured Capacity: 490577010688 (456.89 GB)
DFS Used: 32768 (32 KB)
Non DFS Used: 19604914176 (18.26 GB)
DFS Remaining: 445976846336 (415.35 GB)
DFS Used%: 0.00%
DFS Remaining%: 90.91%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Apr 29 15:42:57 IST 2024
Last Block Report: Mon Apr 29 15:41:21 IST 2024
Num of Blocks: 0
Name: 192.168.8.201:9866 (node2)
Hostname: paresh-ThinkPad-L490
Decommission Status : Normal
Configured Capacity: 319656751104 (297.70 GB)
DFS Used: 32768 (32 KB)
Non DFS Used: 54415609856 (50.68 GB)
DFS Remaining: 248928411648 (231.83 GB)
DFS Used%: 0.00%
DFS Remaining%: 77.87%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Apr 29 15:42:59 IST 2024
Last Block Report: Mon Apr 29 13:54:43 IST 2024
Num of Blocks: 0
Name: 192.168.8.204:9866 (node4)
Hostname: pratik-HP-EliteBook-840-G3
Decommission Status : Normal
Configured Capacity: 308494532608 (287.31 GB)
DFS Used: 32768 (32 KB)
Non DFS Used: 63622209536 (59.25 GB)
DFS Remaining: 229126873088 (213.39 GB)
DFS Used%: 0.00%
DFS Remaining%: 74.27%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 0
Last contact: Mon Apr 29 15:42:58 IST 2024
Last Block Report: Mon Apr 29 13:54:39 IST 2024
Num of Blocks: 0You can also use a user-friendly web interface. Point your browser to http://masternode:9870, and you’ll get a user-friendly monitoring console.
As with HDFS, YARN provides a friendlier web UI, started by default on port 8088 of the Resource Manager. Point your browser to http://masternode:8088, and browse the UI:
Stop HDFS and YARN
To stop HDFS on master and worker nodes, run the following command from masternode:
stop-dfs.shTo stop YARN, run the following command on masternode:
stop-yarn.shHere, you have successfully configured and set up a 3-node cluster in Apache Hadoop!
Apache Hive
Apache Hive is a distributed, fault-tolerant data warehouse system that enables analytics at a massive scale. Hive Metastore(HMS) provides a central repository of metadata that can easily be analyzed to make informed, data driven decisions, and therefore it is a critical component of many data lake architectures. Hive is built on top of Apache Hadoop and supports storage on S3, adls, gs etc though hdfs. Hive allows users to read, write, and manage petabytes of data using SQL.
Prerequisites
-
Java
Recommended Java versions are Java 8 and Java 11 (runtime only).
And export your JAVA_HOME path to the shell configuration file. If you’re using Bash, this file is usually
~/.bashrcor~/.bash_profile. If you’re using a different shell, the file might be different (e.g.,~/.zshrcfor Zsh).Download the recommended Java JDK from repositories or official Java website.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Java JDK.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")This will update dynamically and works well with the alternatives system.
Save your shell configurations.
source ~/.bashrc -
Apache Hadoop
Apache Hadoop must be installed and configured as mentioned in the above section.
-
MySQL/MariaDB
A database metastore is required for Hive from which the queries are run from. The metastore is a central repository of metadata for Hive tables and databases. It stores information such as schema, location, and statistics about the data. MySQL or MariaDB can be used as the backend database for this metastore due to their reliability and performance. The Hive metastore service interfaces with the MySQL/MariaDB database to retrieve and manage metadata information, which is essential for query execution planning and optimization in Hive.
Note: MariaDB and MySQL packages conflict because they provide similar files. So, you can only install one of them, either MariaDB or MySQL, but not both.
Install from Oracle MySQL
Please download the release package provided by Oracle from: Oracle MYSQL. Once downloaded, please install it using dnf:
sudo dnf install mysql84-community-release-fc40-1.noarch.rpmOnce the repository is set, proceed to install MySQL:
sudo dnf install mysql-community-server Now, start and enable the service:
sudo systemctl start mysqld
sudo systemctl enable mysqld --nowFind the default password, for security reasons, MySQL generates a temporary root key.
sudo grep 'temporary password' /var/log/mysqld.logPlease note that MySQL has even stricter security policies than MariaDB.
Configure MySQL for first time use:
sudo mysql_secure_installationThen, answer the security questions as you prefer. or just say yes to all of them.
Then proceed to login to MySQL:
sudo mysql -u root -pNow that we have our prerequisites fulfilled we will proceed with the installation of Hive.
Installation of Apache Hive
To get a Hive distribution, download a recent stable release from one of the Apache Download Mirrors.
wget https://dlcdn.apache.org/hive/hive-4.0.0/apache-hive-4.0.0-bin.tar.gzExtract the contents of the file to the desired location (/usr/local/ or /opt/ is recommended).
cd /usr/local/
tar -xvf apache-hive-4.0.0-bin.tar.gzRename the directory.
mv apache-hive-4.0.0-bin.tar.gz/ hiveGive user execution permissions to the directory (superuser permissions might be required).
chmod 777 hive/Now we’ll add the Hive path to the environment variables of out shell.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Hadoop directory.
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/binSave your shell configurations.
source ~/.bashrcNow, we have successfully installed Apache Hive in the system.
Configuration of Apache Hive
We’ll create the necessary directories and files in the distributed file system and give them appropriate permissions.
#Create the following directories
hdfs dfs -mkdir -p /user/hive/warehouse
hdfs dfs -mkdir -p /tmp/hive
#Give the appropriate permissions to the users and groups
hdfs dfs -chmod -R 777 /user
hdfs dfs -chmod -R 777 /tmpConfiguring database and initializing the metastore
First, login into MySQL through your machine.
mysql -u root -p Check for databases and create a database for the metastore
show databases;
CREATE DATABASE metastore;
use metastore;Now we’ll create an user and set up permissions for the database.
CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;Before proceeding forward we’ll configure hive config with hadoop configurations.
Go to the $HIVE_HOME/bin directory and change the hive-config.sh as per follows.
# Export the configuration directory path of hive
HIVE_CONF_DIR="${HIVE_CONF_DIR:-$HIVE_HOME/conf}"
export HIVE_CONF_DIR=$HIVE_CONF_DIR
# Export the path for the hadoop home directory
export HADOOP_HOME=/usr/local/hadoopWe have created the database and given appropriate permissions to the database and the user, now we will proceed to set up hive configuration to connect with the metastore.
Locate or create hive-site.xml file in $HIVE_HOME/conf directory and add the following values to the properties.
# Location of default database for the warehouse in HDFS
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
# Connection URL to JDBC connect string for a JDBC metastore
# For example, jdbc:mysql://'username':3306/'database'
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://hive:3306/metastore?useSSL=true&requireSSL=true</value>
</property>
# Username to use against metastore database
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
# Password to use against metastore database
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>As later we are going to use Spark as our execution engine so we’ll configure to change the default execution engine from MapReduce(mr) to Spark in our hive-site.xml .
# Choose an execution engine, expects one of [mr, tez, spark]
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>As we are using MySQL for the metastore, we need to add the MySQL connector to the lib folder of hive.
# Download the appropriate package
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-j-8.4.0-1.fc40.noarch.rpm
# Extract the contents of the package
rpm2cpio ./mysql-connector-j-8.4.0-1.fc40.noarch.rpm | cpio -idmv
# Give approprite permissions to the jar file in the folder and copy it to the lib folder in hive
cd mysql-connector-j-8.4.0-1.fc40.noarch.rpm/
chmod +x mysql-connector-java.jar
cp mysql-connector-java.jar $HIVE_HOME/libNow we will make changes in the hive-site.xml.
# Driver class name for a JDBC metastore
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
</property>Restart the MySQL service.
systemctl restart mysqldNow, initialize the metastore schema.
schematool -initSchema -dbType mysqlThe output will look like -
Metastore connection URL: jdbc:mysql://hive:3306/metastore?useSSL=true&requireSSL=true
Metastore Connection Driver : com.mysql.cj.jdbc.Driver
Metastore connection User: hive
Starting metastore schema initialization to 3.1.3
Initialization script hive-schema-3.1.3.mysql.sql
Initialization script completed
schemaTool completedStart the hive metastore server.
$HIVE_HOME/bin/hive --service metastoreThe hive metastore server runs on the default 9083 port.
Start the hiveserver2 service.
$HIVE_HOME/bin/hive --service hiveserver2The hiveserver2 service runs on the default 10000 port.
Login to the Beeline client using the JDBC url connect to hiveserver2.
beeline -u jdbc:hive2://localhost:10000/For monitoring, you can access HiveServer WebUI on port 10002 http://localhost:10002 .
Or you can use the hive shell to perform queries.
hiveHere is the hiveserver2 web ui.
http://locahost:10002/
Hence, we have installed and configured Apache Hive in out HDFS data architecture.
Apache Spark
Apache Spark is a multi-language engine designed for executing data engineering, data science, and machine learning tasks on single-node machines or clusters. It unifies the processing of data in batches and real-time streaming, supporting your preferred languages such as Python, SQL, Scala, Java, and R. With Spark, you can execute fast, distributed ANSI SQL queries for dashboarding and ad-hoc reporting, offering performance that surpasses most data warehouses.
Prerequisites
Required software for Linux include:
-
Java
Recommended Java versions are: Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only).
And export your JAVA_HOME path to the shell configuration file. If you’re using Bash, this file is usually
~/.bashrcor~/.bash_profile. If you’re using a different shell, the file might be different (e.g.,~/.zshrcfor Zsh).
Download the recommended Java JDK from repositories or official Java website.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Java JDK.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")This will update dynamically and works well with the alternatives system.
Save your shell configurations.
source ~/.bashrc-
Scala
You should Scala language to implement Spark. Scala is needed for Apache Spark due to its original implementation in Scala, support for functional programming, and performance benefits.
To install Scala, it is recommended to use cs setup, the Scala installer powered by Coursier. It installs everything necessary to use the latest Scala release from a command line.
Run the following command in your terminal, following the on-screen instructions.
curl -fL https://github.com/coursier/coursier/releases/latest/download/cs-x86_64-pc-linux.gz | gzip -d > cs && chmod +x cs && ./cs setupCheck your setup with the command scala -version .
scala -versionNote: If that does not work, close and reopen your terminal; otherwise, you may need to log out and log back in (or reboot) in order for the changes to take effect.
Download and Install Apache Spark
To get a Spark distribution, download a recent stable release from one of the Apache Download Mirrors.
wget https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgzExtract the contents of the file to the desired location (/usr/local/ or /opt/ is recommended).
cd /usr/local/
tar -xvf spark-3.5.1-bin-hadoop3.tgzRename the directory.
mv spark-3.5.1-bin-hadoop3.tgz/ sparkGive user execution permissions to the directory (superuser permissions might be required).
chmod 777 spark/Now we’ll add the Spark path to the environment variables of out shell.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Hadoop directory.
export SPARK_HOME=/usr/local/spark
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbinSave your shell configurations.
source ~/.bashrcNow, we have successfully installed Apache Spark in the system.
Configuring Spark and Deploying the Spark Cluster
First, we need to configure Spark configuration file spark-env.sh in the $SPARK_HOME/conf directory. Add the following values according to your configurations.
export SPARK_MASTER_HOST=masternode
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")Note: The SPARK_MASTER_HOST is the master node of the cluster. Here it is masternode because it has been mapped to an IP address in /etc/hosts. So you should input the IP address of the master node.
Further make changes and append to the spark-defaults.conf in $SPARK_HOME/conf.
spark.master yarn
spark.eventLog.enabled true
spark.eventLog.dir hdfs://masternode:9000/spark-logs
spark.sql.catalogImplementation hive
spark.sql.warehouse.dir hdfs://masternode:9000/user/hive/warehouseFirst, we specify the Spark master to run on YARN by setting spark.master to yarn. Then, we enable Spark event logging by configuring spark.eventLog.enabled as true, and designate the directory for storing these logs as an HDFS location through spark.eventLog.dir. Subsequently, we ensure compatibility with Hive for Spark SQL operations by setting spark.sql.catalogImplementation to hive. Finally, we define the location of the Hive warehouse directory via spark.sql.warehouse.dir, also pointing to an HDFS location. These adjustments optimize Spark’s configuration for seamless integration within the specified environment, leveraging YARN for resource management, enabling event logging, integrating with Hive for SQL operations, and utilizing HDFS for data storage.
Now for setting up the cluster, we do not need to do anything excess as we have exported the hadoop conf directory which maps the hadoop cluster requirements with the spark cluster. Although we need to mention the nodes in the workers file.
masternode
node7
node8
node9Note: A Spark Worker will be started on each of the machines listed. Also, change the workers to IP addresses or whichever way is suitable.
Now that we have everything set, we can verify the installation. Write the following command for opening Spark shell.
spark-shellIf spark is installed successfully then you will find the following output.
Spark context Web UI available at http://masternode:4040
Spark context available as 'sc' (master = yarn, app id = application_1717663716562_0001).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.1
/_/
Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 11.0.23)
Type in expressions to have them evaluated.
Type :help for more information.
scala> The web ui for is available at http://localhost:4040/.
Spark Jobs
Spark Executors
Now that we have our Spark session running, we can start the spark cluster. Run the following command to start the master and the workers.
$SPARK_HOME/sbin/start-all.shThis will start all the workers and master listed on the yarn master.
The spark cluster ui is available at http://localhost:8080/ with spark master at spark://masternode:7077/.
Spark Cluster
Running a Spark Application on Yarn
First, package your Spark application into a JAR file, including all necessary dependencies. Next, submit your Spark application to the cluster using the spark-submit command, specifying the location of your JAR file, any command-line arguments, and any additional configurations required for your application. Spark will then distribute your application across the nodes in your Hadoop cluster, utilizing Hadoop’s distributed file system (HDFS) for data storage and processing.
Let’s assume there’s a file named wordcount.py available for processing. Here’s how you would submit it using spark-submit.
spark-submit \
--name WordCountJob \
--master yarn \
--conf spark.app.name=WordCount \
--driver-memory 1G \
--executor-memory 2G \
--executor-cores 2 \
hdfs://masternode:9000/sparkjobs/wordcount.py \
hdfs://masternode:9000/sparkjobs/inputFile.txt \
hdfs://masternode:9000/sparkjobs/output/wordcount_output This command submits the Spark job named “WordCountJob” to the YARN cluster. It sets the application name to “WordCount” and specifies the driver memory as 1G and executor memory as 2G with 2 executor cores. The wordcount.py file is located in HDFS at hdfs://masternode:9000/sparkjobs/. The input file inputFile.txt is also located in the same HDFS directory, and the output will be stored in the output/wordcount_output directory in HDFS.
Application on Apache Hadoop Yarn Manager
Application attempt to nodes
Running Spark Application
Here, we have successfully installed and configure Apache Spark and a cluster while running a Spark Application with HDFS.
Apache Livy
Apache Livy is a service that enables easy interaction with a Spark cluster over a REST interface. It enables easy submission of Spark jobs or snippets of Spark code, synchronous or asynchronous result retrieval, as well as Spark Context management, all via a simple REST interface or an RPC client library. Apache Livy also simplifies the interaction between Spark and application servers, thus enabling the use of Spark for interactive web/mobile applications.
Prerequisites
-
Java
Export your JAVA_HOME path to the shell configuration file. If you’re using Bash, this file is usually
~/.bashrcor~/.bash_profile. If you’re using a different shell, the file might be different (e.g.,~/.zshrcfor Zsh).
Download the recommended Java JDK from repositories or official Java website.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Java JDK.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")This will update dynamically and works well with the alternatives system.
Save your shell configurations.
source ~/.bashrc-
Apache Spark
Apache Livy relies on Apache Spark to perform distributed data processing tasks. Spark provides the underlying computation engine that Livy uses to manage and execute jobs via its REST interface. Properly configuring Java is essential for the seamless operation of both Spark and Livy, ensuring that the environment is ready for distributed data processing and interactive analytics.
Note: The installation and configuration steps for Apache Spark is been discussed above in this documentation.
-
Python
Python is an optional but highly recommended language for running PySpark jobs with Apache Spark. PySpark allows for easy and efficient integration of Spark with Python, enabling data processing and analysis using Python’s extensive libraries.
To install Python, follow the instructions below:
Update your package list and install Python 3.
sudo dnf update
sudo dnf install python3Verify the installation by checking the Python version.
python3 --versionIt is also recommended to install pip, the package installer for Python, to manage Python packages.
sudo dnf install python3-pipVerify the installation by checking the pip version.
pip3 --versionNote: If you face any issues, ensure your environment variables are correctly set, and your Python path is correctly configured in your shell configuration file (~/.bashrc, ~/.zshrc, etc.).
-
Apache Hadoop
Apache Hadoop is essential for setting up a distributed computing environment for big data processing. Apache Livy often integrates with Hadoop to leverage HDFS (Hadoop Distributed File System) and YARN (Yet Another Resource Negotiator) for resource management.
Note: The installation and configuration steps for Apache Hadoop is been discussed above in this documentation.
Download and Install Apache Livy
To get a Livy distribution, download a recent stable release from one of the Apache Download Mirrors.
wget https://dlcdn.apache.org/incubator/livy/0.8.0-incubating/apache-livy-0.8.0-incubating_2.11-bin.zipExtract the contents of the file to the desired location (/usr/local/ or /opt/ is recommended).
cd /usr/local/
unzip apache-livy-0.8.0-incubating_2.11-bin.zipRename the directory.
mv apache-livy-0.8.0-incubating_2.11-bin/ livyGive user execution permissions to the directory (superuser permissions might be required).
chmod 777 livy/Now we’ll add the Livy path to the environment variables of out shell.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Hadoop directory.
export LIVY_HOME=/usr/local/livy
export PATH=$PATH:$LIVY_HOME/binSave your shell configurations.
source ~/.bashrcNow, we have successfully installed Apache Livy in the system.
Setup and Configuration of Apache Livy
To run Livy with local sessions, first export these variables:
export SPARK_HOME=/usr/local/spark
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopLivy uses a few configuration files under the configuration directory, which by default is the conf directory under the Livy installation. An alternative configuration directory can be provided by setting the LIVY_CONF_DIR environment variable when starting Livy.
The configuration files used by Livy are:
livy.conf: contains the server configuration. The Livy distribution ships with a default configuration file template listing available configuration keys and their default values.spark-blacklist.conf: lists Spark configuration options that users are not allowed to override. These options will be restricted to either their default values, or the values set in the Spark configuration used by Livy.log4j.properties: configuration for Livy logging. Defines log levels and where log messages will be written to. The default configuration template will print log messages to stderr.
Once all the configurations are done, start the server with:
livy-server startOnce the Livy server is running, you can connect to it on port 8998 (this can be changed with the
livy.server.port config option in livy.conf).
Sending API request calls to Livy
To interact with Apache Livy and submit a job, you’ll need to send API requests to the Livy server.
We’ll try to run a python script with the help of Spark and create a batch session in Livy.
Here is an example of a python script for word count from a input file.
import sys
import os
from datetime import datetime
from pyspark.sql import SparkSession
def main():
# Check if the correct number of arguments have been passed
if len(sys.argv) < 2:
print("Usage: wordcount.py <inputFile>")
sys.exit(1)
# Extract input file from command line arguments
inputFile = sys.argv[1]
# Initialize Spark Session
spark = SparkSession.builder \
.appName("WordCount") \
.master("local") \
.getOrCreate()
# Read input file into an RDD
inputRDD = spark.sparkContext.textFile(inputFile)
# Split lines into words and collect into a new RDD
words = inputRDD.flatMap(lambda line: line.split())
# Transform words into key-value pairs and reduce by key to count occurrences
wordCounts = words.map(lambda word: (word, 1)).reduceByKey(lambda x, y: x + y)
# Create output directory with timestamp
timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
outputDirectory = os.path.join("output", f"wordcount_{timestamp}")
# Save word counts to the specified output directory
wordCounts.saveAsTextFile(outputDirectory)
# Stop the Spark session
spark.stop()
if __name__ == "__main__":
main()This is a simple python script to count the words in an input file.
To run this as a spark job we need to upload the required file on HDFS.
bin/hdfs dfs -put /home/vinayak/spark-applications/wordcount.py /spark
bin/hdfs dfs -put /home/vinayak/spark-applications/inputFile.txt /sparkThis copies the wordcount.py and inputFile.txt in HDFS in the spark directory.
Uploaded files in HDFS
Now we will run the job as a POST query with Spark and Livy.
curl -X POST http://masternode:8998/batches -H "Content-Type: application/json" -d '{
"file": "hdfs://masternode:9000/spark/wordcount.py",
"args": [
"hdfs://masternode:9000/spark/inputFile.txt",
"hdfs://masternode:9000/spark/output/"
],
"conf": {
"spark.executor.memory": "2g"
}
}'This curl command submits a Spark job to a Livy server, which provides a REST interface for interacting with a Spark cluster. The command sends a POST request to http://masternode:8998/batches with a JSON payload specifying the job details. The JSON includes the path to the Spark application script (wordcount.py) located on HDFS, input and output file paths also on HDFS, and a configuration setting to allocate 2GB of memory to each Spark executor. The Content-Type: application/json header indicates that the data being sent is in JSON format.
The batch will be created on Livy and can be seen on the Livy UI on http://masternode:8998/ .
Livy batch in running session
The batch logs can be seen as the spark application is running.
Livy batch logs
After the batch is successfully completed and the spark application is executed the word count will be output in a file and stored in HDFS.
Output of the Spark Application on HDFS
Here, we have successfully installed and configure Apache Livy and ran a Spark Application with HDFS.
Apache NiFi
Apache NiFi was built to automate the flow of data between systems. While the term ‘dataflow’ is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data. The problems and solution patterns that emerged have been discussed and articulated extensively.
Prerequisites
-
Java
Export your JAVA_HOME path to the shell configuration file. If you’re using Bash, this file is usually
~/.bashrcor~/.bash_profile. If you’re using a different shell, the file might be different (e.g.,~/.zshrcfor Zsh).
Download the recommended Java JDK from repositories or official Java website.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Java JDK.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")This will update dynamically and works well with the alternatives system.
Save your shell configurations.
source ~/.bashrcDownload and Install Apache NiFi
To get a NiFi distribution, download a recent stable release from one of the Apache Download Mirrors.
wget https://dlcdn.apache.org/nifi/1.26.0/nifi-1.26.0-bin.zipExtract the contents of the file to the desired location (/usr/local/ or /opt/ is recommended).
cd /usr/local/
unzip nifi-1.26.0-bin.zipRename the directory.
mv nifi-1.26.0-bin/ nifiGive user execution permissions to the directory (superuser permissions might be required).
chmod 777 nifi/Now we’ll add the NiFi path to the environment variables of out shell.
Open the configuration file in a text editor. For example, if you’re using Bash, you can use:
nano ~/.bashrcExport the path of your Hadoop directory.
export NIFI_HOME=/usr/local/nifi
export PATH=$PATH:$NIFI_HOME/binSave your shell configurations.
source ~/.bashrcNow, we have successfully installed Apache NiFi in the system.
Configuring Apache NiFi
Open the nifi-env.sh from the $NIFI_HOME/bin directory and make the necessary changes.
# The java implementation to use.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:/bin/java::")Also, make any necessary changes in the nifi.properties file in the $NIFI_HOME/conf directory.
Now, set up a single login credential for the NiFi server access.
nifi.sh set-single-user-credentials admin admin111222333Here, the username is admin and the password is admin111222333.
Once the configuration is done, start the NiFi server with the command.
nifi.sh startCheck the status of the NiFi server.
nifi.sh statusThe server is now running is accessible at https://127.0.0.1:8443/nifi/.
Apache NiFi Flow
Now, we have the NiFi server up and running.
Configuring a Data Flow in Apache Nifi
After setting up the NiFi server, we’ll set up a data flow to automate processes.
Here, we’ll try to set up a Livy batch by running a Spark application with integration of HDFS.
First configure the whole data flow in Apache NiFi under a process flow.
NiFi Data Flow for Livy and HDFS
Now that the data flow has been setup, the configuration of the processes will be initiated.
Use the GetHTTP processor in Apache NiFi to ingest the status of your batch from Livy’s Metrics REST API and store it in HDFS.
Ingest metrics REST API from Livy with Apache NiFi and HDFS
GetHTTP to ingest the status on your batch from Livy
Now to set up HDFS and local file directory.
HDFS configuration for PutHDFS processor
GetFile processor to fetch files from local directory
Here, this is a process flow file for the above configuration.
Livy batch by running a Spark application with integration of HDFS in Apache NiFi.
Hence, we have configured Apache NiFi and a process flow to run a Livy batch by a Spark application with integration of HDFS.
Picture our data setup as a powerhouse team of tools: Hadoop is like the sturdy foundation, storing and distributing our data across multiple nodes. Hive steps in with its SQL interface, letting us query and analyze the data without breaking a sweat. Spark is the high-performance engine, speeding up processing with its in-memory computations. Livy acts as the bridge, allowing us to interact with Spark clusters through REST APIs. And NiFi serves as the traffic manager, orchestrating data flow with its intuitive graphical interface and ensuring data integrity and security throughout the pipeline!