Hadoop 2.7.7 伪分布式安装
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了Hadoop 2.7.7 伪分布式安装，小编现在分享给大家，供广大互联网技能从业者学习和参考。文章包含16228字，纯文字阅读大概需要24分钟。
内容图文
Hadoop 2.7.7 伪分布式安装
伪分布式：一个集群的所有角色都分布式在一个节点。
注意区分一个概念：单机模式
一、Hadoop介绍
Hadoop从2.x开始，就开始分化了。逐渐演变成：HDFS、YARN、MapReduce三大应用模块，这三个应用模块分别的能力和作用是：
1、HDFS：分布式文件系统，用来解决海量大文件的存储问题
2、MapReduce：一套通用的用来解决海量大文件计算的编程模型API
3、YARN：资源调度/管理系统
其中需要注意的是：这三者之间的关系。彼此独立，又相互依赖。使用MapReduce的分布式编程API编写分布式计算应用程序，读取存储在HDFS上的海量大文件进行计算，由YARN提供计算资源。HDFS和YARN可以独立运行。主要表现在：
1、使用MapReduce编写的应用程序也可以运行在其他资源调度系统之上
2、使用其他编程模型编写的应用程序，比如Storm、Spark、Flink等也可以运行在YARN集群上
所以称Hadoop是一个分布式的成熟解决方案。
所以安装Hadoop,其实就是安装HDFS和YARN两个集群。HDFS和YARN都是一个一主多从的集群。
HDFS集群：
一个NameNode主节点/管理节点
多个DataNode从节点/工作节点
YARN集群：
一个ResourceManager主节点/管理节点
多个NodeManager从节点/工作节点
二、安装环境
CentOS7.6 + Hadoop 2.7.7
| 节点名称 | HDFS | YARN |
|---|---|---|
| master | NameNode + DataNode + SecondaryNamenode | ResourceManager+NodeManager |
安装：
[root@master ~]# yum -y install redhat-lsb-core-4.1-27.el7.centos.1.x86_64
[root@master ~]# lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.6.1810 (Core)
Release: 7.6.1810
Codename: Core
三、环境准备
1、修改主机名
1.1修改主机名称
[root@master ~]# hostnamectl set-hostname master
1.2查看主机名
[root@master ~]# hostnamectl
Static hostname: master
Icon name: computer-vm
Chassis: vm
Machine ID: a21c09986dee4158905b391d0d5e0d3f
Boot ID: 86a0d4893c5a411a92d11d4e1cfc70bc
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-957.el7.x86_64
Architecture: x86-64
2、添加hadoop用户
本次部署使用hadoop用户，需要添加hadoop用户
#创建hadoop组
[root@master ~]# groupadd hadoop
#创建一个hadoop组下的hadoop用户，并使用/bin/bash作为shell
[root@master ~]# useradd -m hadoop -g hadoop -s /bin/bash
#useradd参数
# -m:自动建立用户的登入目录
# -g:指定用户所属的附加群组
# -s:指定用户登入后所使用的shell
#设置hadoop用户密码
[root@master ~]# passwd hadoop
Changing password for user hadoop.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
可以为hadoop用户增加管理员权限
[root@master ~]# visudo
#在里面添加
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
hadoop ALL=(ALL) ALL
3、配置免密登录
集群、伪分布式模式的都需要用到SSH登录，一般情况下，CentOS默认安装了SSH client、SSH server，打开终端执行如下命令进行检验，查看是否包含了SSH client跟SSH server。
[root@master ~]# rpm -qa |grep ssh
openssh-server-7.4p1-16.el7.x86_64
openssh-clients-7.4p1-16.el7.x86_64
libssh2-1.4.3-12.el7.x86_64
openssh-7.4p1-16.el7.x86_64
如果不包含，可以通过yum进行安装
[root@master ~]# yum install openssh-clients
[root@master ~]# yum install openssh-server
测试ssh是否可用
#按照提示输入密码，就可以登录本机
[root@master ~]# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:wnXk424rDwk9tKvMjS2pWcqXnJ6B+cSq3MZVxPI69mE.
ECDSA key fingerprint is MD5:62:51:10:f6:78:10:76:b9:8c:91:fc:9b:b2:e5:8d:18.
Are you sure you want to continue connecting (yes/no)?
生成密钥
#若没有该目录，请先执行一次ssh master
[hadoop@master ~]$ cd ~/.ssh/
-bash: cd: /home/hadoop/.ssh/: No such file or directory
# pwd查看当前目录，应为"/home/hadoop/"
[hadoop@master ~]$ pwd
/home/hadoop
#查看.ssh目录已经存在
[hadoop@master ~]$ cd .ssh
[hadoop@master .ssh]$ pwd
/home/hadoop/.ssh
#修改/etc/hosts文件
[hadoop@master ~]$ cat /etc/hosts
添加内容如下：
192.168.1.100 master
#生成密钥
[hadoop@master .ssh]$ ssh-keygen -t rsa #需要输入三次密码
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:JVm7S4NqjrJyeZY9wPAY58WAA94EBqhPTnLMT0isuQA hadoop@master
The key's randomart image is:
+---[RSA 2048]----+
|==.o . |
|E B . o . |
|oB + o o o |
|* X o o + . |
|.O @ . S + |
|. + * . . o |
| . +o . |
|. + ++o |
| o.=. .. |
+----[SHA256]-----+
[hadoop@master .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@master .ssh]$ chmod 600 authorized_keys
#此时再使用ssh master命令，无需输入密码就可以直接登录
[hadoop@master .ssh]$ ssh master
Last login: Wed Apr 14 08:34:34 2021 from localhost
[hadoop@master ~]$ exit
logout
Connection to master closed.
[hadoop@master .ssh]$
4、关闭防火墙
#关闭防火墙
[root@master local]# systemctl stop firewalld.service
#查看防火墙状态
[root@master local]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
5、关闭SELinux
#查看状态
[root@master local]# getenforce
Disabled
[root@master local]# /usr/sbin/sestatus -v
SELinux status: disabled
#临时关闭
[root@master local]# setenforce 0
#永久关闭
[root@master local]# vim /etc/selinux/config
#将SELINUX=disabled改为SELINUX=disabled
#设置后需要重启才能生效
6、安装JDK1.8环境
6.1下载
略
6.2 解压到/usr/local/下
略
6.3配置JDK环境变量
- 编辑/etc/profile
[root@master local]# vim /etc/profile
- 添加内容如下
export JAVA_HOME=/usr/local/jdk1.8.0_211
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
- 使配置立即生效
[root@master local]# source /etc/profile
6.4验证java是否安装成功
[root@master local]# java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
四、安装配置Hadoop
1、说明
- 本次安装的是hadoop-2.7.7，使用hadoop用户安装
2、下载
略
3、上传/home/hadoop/apps下，并解压
[hadoop@master apps]$ tar -zxvf hadoop-2.7.7-centos7.tar.gz -C /home/hadoop/apps
4、修改hadoop-env.sh
#添加jdk安装目录
[hadoop@master hadoop-2.7.7]$ cd /home/hadoop/apps/hadoop-2.7.7/etc/hadoop/
[hadoop@master hadoop]$ vim hadoop-env.sh
#修改export JAVA_HOME=${JAVA_HOME}为：
export JAVA_HOME=/usr/local/jdk1.8.0_211
5、修改core-site.xml
[hadoop@master hadoop]$ vim core-site.xml
#添加配置如下：
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data/hadoopdata</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
6、修改slaves配置文件
该文件的作用，用来指定Hadoop集群的从节点有哪些。由于我们搭建伪分布式，所以只有一个
[hadoop@master hadoop]$ vim slaves
#添加配置如下：
master
7、修改hdfs-site.xml
[hadoop@master hadoop]$ vim hdfs-site.xml
#添加配置如下：
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/hadoopdata/name</value>
<description>为了保证元数据的安全一般配置多个不同目录</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/hadoopdata/data</value>
<description>datanode 的数据存储目录</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>HDFS 的数据块的副本存储个数, 默认是3</description>
</property>
</configuration>
8、修改mapred-site.xml
[hadoop@master hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@master hadoop]$ vim mapred-site.xml
#添加配置如下：
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
9、修改yarn-site.xml
[hadoop@master hadoop]$ vim yarn-site.xml
#添加配置如下：
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>YARN 集群为 MapReduce 程序提供的 shuffle 服务</description>
</property>
</configuration>
10、Hadoop环境变量配置
执行命令：
[hadoop@master ~]$ vim .bash_profile
#添加配置如下：
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.7.7
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH
#使配置立即生效
[hadoop@master ~]$ source .bash_profile
[hadoop@master ~]$ echo $HADOOP_HOME
/home/hadoop/apps/hadoop-2.7.7
11、创建hdfs-site.xml里配置的路径
[hadoop@master ~]$ mkdir -p /home/hadoop/data/hadoopdata/name
[hadoop@master ~]$ mkdir -p /home/hadoop/data/hadoopdata/data
12、格式化HDFS集群
在namenode所在节点执行命令
[hadoop@master ~]$ hadoop namenode -format
21/04/15 08:31:13 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
21/04/15 08:31:13 INFO namenode.NameNode: createNameNode [-format]
21/04/15 08:31:15 WARN common.Util: Path /home/hadoop/data/hadoopdata/name should be specified as a URI in configuration files. Please update hdfs configuration.
21/04/15 08:31:15 WARN common.Util: Path /home/hadoop/data/hadoopdata/name should be specified as a URI in configuration files. Please update hdfs configuration.
Formatting using clusterid: CID-dbd003a2-5380-45b0-a965-5e80285f26fc
21/04/15 08:31:15 INFO namenode.FSNamesystem: No KeyProvider found.
21/04/15 08:31:15 INFO namenode.FSNamesystem: fsLock is fair: true
21/04/15 08:31:15 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
21/04/15 08:31:15 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
21/04/15 08:31:15 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
21/04/15 08:31:15 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
21/04/15 08:31:15 INFO blockmanagement.BlockManager: The block deletion will start around 2021 Apr 15 08:31:15
21/04/15 08:31:15 INFO util.GSet: Computing capacity for map BlocksMap
21/04/15 08:31:15 INFO util.GSet: VM type = 64-bit
21/04/15 08:31:15 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
21/04/15 08:31:15 INFO util.GSet: capacity = 2^21 = 2097152 entries
21/04/15 08:31:15 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
21/04/15 08:31:15 INFO blockmanagement.BlockManager: defaultReplication = 2
21/04/15 08:31:15 INFO blockmanagement.BlockManager: maxReplication = 512
21/04/15 08:31:15 INFO blockmanagement.BlockManager: minReplication = 1
21/04/15 08:31:15 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
21/04/15 08:31:15 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
21/04/15 08:31:15 INFO blockmanagement.BlockManager: encryptDataTransfer = false
21/04/15 08:31:15 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
21/04/15 08:31:15 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)
21/04/15 08:31:15 INFO namenode.FSNamesystem: supergroup = supergroup
21/04/15 08:31:15 INFO namenode.FSNamesystem: isPermissionEnabled = true
21/04/15 08:31:15 INFO namenode.FSNamesystem: HA Enabled: false
21/04/15 08:31:15 INFO namenode.FSNamesystem: Append Enabled: true
21/04/15 08:31:16 INFO util.GSet: Computing capacity for map INodeMap
21/04/15 08:31:16 INFO util.GSet: VM type = 64-bit
21/04/15 08:31:16 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
21/04/15 08:31:16 INFO util.GSet: capacity = 2^20 = 1048576 entries
21/04/15 08:31:16 INFO namenode.FSDirectory: ACLs enabled? false
21/04/15 08:31:16 INFO namenode.FSDirectory: XAttrs enabled? true
21/04/15 08:31:16 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
21/04/15 08:31:16 INFO namenode.NameNode: Caching file names occuring more than 10 times
21/04/15 08:31:16 INFO util.GSet: Computing capacity for map cachedBlocks
21/04/15 08:31:16 INFO util.GSet: VM type = 64-bit
21/04/15 08:31:16 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
21/04/15 08:31:16 INFO util.GSet: capacity = 2^18 = 262144 entries
21/04/15 08:31:16 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
21/04/15 08:31:16 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
21/04/15 08:31:16 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
21/04/15 08:31:16 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
21/04/15 08:31:16 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
21/04/15 08:31:16 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
21/04/15 08:31:16 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
21/04/15 08:31:16 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
21/04/15 08:31:16 INFO util.GSet: Computing capacity for map NameNodeRetryCache
21/04/15 08:31:16 INFO util.GSet: VM type = 64-bit
21/04/15 08:31:16 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
21/04/15 08:31:16 INFO util.GSet: capacity = 2^15 = 32768 entries
21/04/15 08:31:16 INFO namenode.FSImage: Allocated new BlockPoolId: BP-211112453-127.0.0.1-1618446676539
21/04/15 08:31:16 INFO common.Storage: Storage directory /home/hadoop/data/hadoopdata/name has been successfully formatted.
21/04/15 08:31:16 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/data/hadoopdata/name/current/fsimage.ckpt_0000000000000000000 using no compression
21/04/15 08:31:17 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/data/hadoopdata/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.
21/04/15 08:31:17 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
21/04/15 08:31:17 INFO util.ExitUtil: Exiting with status 0
21/04/15 08:31:17 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/
13、启动HDFS集群
[hadoop@master ~]$ start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /home/hadoop/apps/hadoop-2.7.7/logs/hadoop-hadoop-namenode-master.out
master: starting datanode, logging to /home/hadoop/apps/hadoop-2.7.7/logs/hadoop-hadoop-datanode-master.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:wnXk424rDwk9tKvMjS2pWcqXnJ6B+cSq3MZVxPI69mE.
ECDSA key fingerprint is MD5:62:51:10:f6:78:10:76:b9:8c:91:fc:9b:b2:e5:8d:18.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/apps/hadoop-2.7.7/logs/hadoop-hadoop-secondarynamenode-master.out
[hadoop@master ~]$ jps
107667 Jps
107239 NameNode
107544 SecondaryNameNode
107340 DataNode
14、启动YARN集群
[hadoop@master ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.7.7/logs/yarn-hadoop-resourcemanager-master.out
master: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.7.7/logs/yarn-hadoop-nodemanager-master.out
[hadoop@master ~]$ jps
108179 Jps
107842 NodeManager
107239 NameNode
107544 SecondaryNameNode
107340 DataNode
107742 ResourceManager
五、系统验证
1、查看Hadoop版本
[hadoop@master ~]$ hadoop version
Hadoop 2.7.7
Subversion Unknown -r Unknown
Compiled by root on 2020-04-02T23:53Z
Compiled with protoc 2.5.0
From source with checksum 792e15d20b12c74bd6f19a1fb886490
This command was run using /home/hadoop/apps/hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar
2、检查集群启动是否成功
两种方式：
1、利用jps工具检测各进程是否启动成功
[hadoop@master ~]$ jps
107842 NodeManager
107239 NameNode
108455 Jps
107544 SecondaryNameNode
107340 DataNode
107742 ResourceManager
2、利用webui来进行查看
HDFS：http://192.168.1.100:50070
YARN: http://192.168.1.100:8088
参考:
Hadoop伪分布式搭建:https://www.cnblogs.com/zingp/p/11223220.html#_label1
Hadoop 2.7 伪分布式环境搭建（超详细）:https://www.pianshen.com/article/3948129026/
内容总结
以上是互联网集市为您收集整理的Hadoop 2.7.7 伪分布式安装全部内容，希望文章能够帮你解决Hadoop 2.7.7 伪分布式安装所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错，欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 gblab@vip.qq.com 举报，一经查实，本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。
来源：【匿名】