vlambda博客
学习文章列表

前置准备

CentOS7、jdk1.8、scala-2.11.12、spark-2.4.5、hadoop-2.7.7、zookeeper-3.5.7

想要完成本期视频中所有操作，需要以下准备：

一、集群规划

二、集群配置

2.1 spark-env.sh

[xiaokang@hadoop01 conf]$ cp spark-env.sh.template spark-env.sh

export JAVA_HOME=/opt/moudle/jdk1.8.0_191
export SCALA_HOME=/opt/moudle/scala-2.11.12
YARN_CONF_DIR=/opt/software/hadoop-2.7.7/etc/hadoop
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop01:2181,hadoop02:2181,hadoop03:2181 -Dspark.deploy.zookeeper.dir=/ha-spark"
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=24 -Dspark.history.fs.logDirectory=hdfs://hadoop01:9000/spark-jobhistory"

2.2 spark-defaults.conf

[xiaokang@hadoop01 conf]$ cp spark-defaults.conf.template spark-defaults.conf

#spark.master                     spark://hadoop01:7077
spark.master                     spark://hadoop01:7077,hadoop02:7077
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://hadoop01:9000/spark-jobhistory
spark.yarn.historyServer.address               hadoop01:18080

2.3 slaves

[xiaokang@hadoop01 conf]$ cp slaves.template slaves

hadoop01
hadoop02
hadoop03

2.4 yarn-site.xml

<!-- 由于测试环境的虚拟机内存太少, 防止将来任务被意外杀死, 做如下配置 -->
<!--是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<!--是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true -->
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.log.server.url</name>
    <value>http://hadoop01:19888/jobhistory/logs</value>
</property>

2.5 分发

[xiaokang@hadoop01 ~]$ distribution.sh /opt/software/spark-2.4.5
[xiaokang@hadoop01 ~]$ distribution.sh /opt/software/hadoop-2.7.7/etc/hadoop/yarn-site.xml

三、启动集群

3.1 启动ha-hadoop集群

[xiaokang@hadoop01 ~]$ ha-hadoop.sh start

3.2 在hadoop01上启动spark集群

# 进入/opt/software/spark-2.4.5/sbin目录并启动集群
[xiaokang@hadoop01 sbin]$ ./start-all.sh

3.3 在hadoop02上启动备Master

# 进入/opt/software/spark-2.4.5/sbin目录并启动备Master
[xiaokang@hadoop02 sbin]$ ./start-master.sh

3.4 在hadoop01上启动任务历史服务器

# 进入/opt/software/spark-2.4.5/sbin目录并启动任务历史服务器
[xiaokang@hadoop01 sbin]$ ./start-history-server.sh

四、查看集群

4.1 jps进程查看

[xiaokang@hadoop01 ~]$ call-cluster.sh jps
--------hadoop01--------
10881 NodeManager
11265 HistoryServer
13058 Master
10131 NameNode
10470 JournalNode
11176 Worker
10250 DataNode
10683 DFSZKFailoverController
10796 JobHistoryServer
12973 Jps
9935 QuorumPeerMain
--------hadoop02--------
7232 QuorumPeerMain
7648 DFSZKFailoverController
7505 JournalNode
7831 NodeManager
8183 Worker
7321 NameNode
7402 DataNode
9099 Jps
7743 ResourceManager
8255 Master
--------hadoop03--------
7313 DataNode
8273 Jps
7234 QuorumPeerMain
7843 Worker
7545 ResourceManager
7419 JournalNode
7663 NodeManager

4.2 Web UI查看

[xiaokang@hadoop01 sbin]$ cat /opt/software/spark-2.4.5/logs/spark-xiaokang-org.apache.spark.deploy.master.Master-1-hadoop01.out | grep MasterWebUI

20/05/31 08:32:08 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hadoop01:8081

通过启动日志可以看到hadoop01的MasterWebUI的端口号为8081

[xiaokang@hadoop02 sbin]$ cat /opt/software/spark-2.4.5/logs/spark-xiaokang-org.apache.spark.deploy.master.Master-1-hadoop02.out | grep MasterWebUI

20/05/31 08:32:43 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hadoop02:8082

通过启动日志可以看到hadoop02的MasterWebUI的端口号为8082

hadoop01上的任务历史服务器端口号为18080

五、计算 PI (测试高可用)

[xiaokang@hadoop01 ~]$ spark-submit --master yarn --class org.apache.spark.examples.SparkPi /opt/software/spark-2.4.5/examples/jars/spark-examples_2.11-2.4.5.jar 1000

运行过程中将主Master给kill掉，测试是否高可用

[xiaokang@hadoop01 ~]$ kill -9 11091

杀掉主Master之后，程序还是同样在执行，可以看到hadoop02的MasterWebUI中状态显示为recovering

最终计算结果如下：

通过hadoop03上YARN的8088可以查看历史任务