Hadoop菜鸟入门指南_vlambda技术博客

vlambda
2022-04-13

Hadoop菜鸟入门指南

服务器Hadoop集群搭建与测试

各服务器（虚拟机）节点资源配置（离线数仓项目为例）

搭建与配置流程

systemctl start firewalld         # 启动
systemctl enable firewalld        # 开机启动
systemctl stop firewalld          # 关闭
systemctl disable firewalld       # 取消开机启动

8. ssh免密登陆
9. 安装JDK，Hadoop,配置环境变量等。环境变量位置：/etc/profile.d/my_env.sh
10. 修改$HADOOP_HOME/etc/hadoop/ 下的 core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml, workers五个配置文件
11. 配置历史服务器，修改mapred-site.xml配置文件
12. 将hadoop-3.1.3分发到其他主机，使用xsync脚本：~/bin/xsync ./hadoop-3.1.3

#!/bin/bash
#需求：循环复制文件到所有节点的相同目录下
#命令：xsync 要同步的文件名称

#1. 判断参数个数
if [ $# -lt 1 ]
then
    echo Not Enough Arguement!
    exit;
fi
#2. 遍历集群所有机器
for host in hadoop1 hadoop2 hadoop3
do
    echo $host 
    #3. 遍历所有目录，挨个发送
    for file in $@
    do
        #4. 判断文件是否存在
        if [ -e $file ]
            then
                #5. 获取父目录
                pdir=$(cd -P $(dirname $file); pwd)
                #6. 获取当前文件的名称
                fname=$(basename $file)
                ssh $host "mkdir -p $pdir"
                rsync -av $pdir/$fname $host:$pdir
            else
                echo $file does not exists!
        fi
    done
done

集群启动

常用命令

sbin/start-all.sh 启动所有的Hadoop守护进程。包括NameNode、 Secondary NameNode、DataNode、ResourceManager、NodeManager

sbin/stop-all.sh 停止所有的Hadoop守护进程。包括NameNode、 Secondary NameNode、DataNode、ResourceManager、NodeManager

sbin/start-dfs.sh 启动Hadoop HDFS守护进程NameNode、SecondaryNameNode、DataNode

sbin/stop-dfs.sh 停止Hadoop HDFS守护进程NameNode、SecondaryNameNode和DataNode

sbin/hadoop-daemons.sh start namenode 单独启动NameNode守护进程

sbin/hadoop-daemons.sh stop namenode 单独停止NameNode守护进程

sbin/hadoop-daemons.sh start datanode 单独启动DataNode守护进程

sbin/hadoop-daemons.sh stop datanode 单独停止DataNode守护进程

sbin/hadoop-daemons.sh start secondarynamenode 单独启动SecondaryNameNode守护进程

sbin/hadoop-daemons.sh stop secondarynamenode 单独停止SecondaryNameNode守护进程

sbin/start-yarn.sh 启动ResourceManager、NodeManager

sbin/stop-yarn.sh 停止ResourceManager、NodeManager

sbin/yarn-daemon.sh start resourcemanager 单独启动ResourceManager

sbin/yarn-daemons.sh start nodemanager 单独启动NodeManager

sbin/yarn-daemon.sh stop resourcemanager 单独停止ResourceManager

sbin/yarn-daemons.sh stopnodemanager 单独停止NodeManager

mapred --daemon start historyserver 手动启动历史服务器jobhistory

mapred --daemon stop historyserver 手动停止历史服务器jobhistory

集群启动脚本

~/bin/myhadoop.sh start：启动hdfs,yarn,historyserver
~/bin/myhadoop.sh stop：停止hdfs,yarn,historyserver

#!/bin/bash

if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit ;
fi

case $1 in
"start")
        echo "启动 hadoop集群"

        echo "启动 hdfs"
        ssh hadoop1 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"
        echo "启动 yarn"
        ssh hadoop2 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"
        echo "启动 historyserver"
        ssh hadoop1 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver"
;;
"stop")
        echo "关闭 hadoop集群"

        echo "关闭 historyserver"
        ssh hadoop1 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
        echo "关闭 yarn"
        ssh hadoop2 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"
        echo "关闭 hdfs"
        ssh hadoop1 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;;
*)
    echo "Input Args Error..."
;;
esac

查看各服务器进程脚本

~/bin/jpsall

#!/bin/bash

for host in hadoop102 hadoop103 hadoop104
do
        echo $host
        ssh $host jps 
done

Hadoop文件常用操作命令

查看帮助命令
hadoop fs -h

创建文件目录
hadoop fs -mkdir /input

删除单个文件
hdfs dfs -rm /input/word.txt

删除目录
hdfs dfs -rm -r [文件夹地址]

查看具体某一个文件目录下的文件层次
hadoop fs -ls /input

打开某个已存在的文件
hadoop dfs -cat /input/word.txt

上传单个文件到Hadoop目录
hadoop fs -put [本地目录] [Hadoop目录]

从Hadoop上下载文件到本地
hadoop fs -get [Hadoop目录] [本地目录]

各组件WEB端地址

Namenode: hadoop1:9870
Secondary Namenode: hadoop3:9868
yarn Resource Manager: hadoop2:8088
JobHistoryServer: hadoop1:19888

踩坑

在hadoop2上启动/关闭 sbin/start-yarn.sh 时出现此错误：

[root@hadoop2 hadoop-3.1.3]# sbin/start-yarn.sh
Starting resourcemanager
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
Starting nodemanagers
ERROR: Attempting to operate on yarn nodemanager as root
ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation.

解决办法：
在sbin下更改start-yarn.sh和stop-yarn.sh信息，在两个配置文件的第一行添加：

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

启动/关闭hdfs时出现此错误：

Stopping datanodes
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Stopping secondary namenodes [hadoop3]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.

解决办法：
在sbin下更改start-dfs.sh和stop-dfs.sh信息，在两个配置文件的第一行添加：

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

删错文件，集群崩溃的处理办法

先关闭Hadoop集群：sbin/stop-dfs.sh
删除每一个节点中$HADOOP_HOME下的data和logs：rm -rf data/ logs/
格式化：hdfs namenode -format

vlambda博客
学习文章列表