xixiNote

Appearance

hadoop 集群搭建 ​

环境版本描述/补充下载地址
CentOS7-Download (centos.org)
JDK8openlogic-openjdk-8u382-b05-linux-x64.tar.gzopenlogic.com
hadoop2.7.7hadoop-2.7.7.tar.gzApache Projects Releases
工具版本描述/补充下载地址
Xshell7-NetSarang Website (xshell.com)
Xftp7-NetSarang Website (xshell.com)
VMware17-VMwareDownload
目录

TIP

本次使用的网段为192.168.128.1/24

本次搭建节点ip:
master 192.168.128.130

slave1 192.168.128.131

slave2 192.168.128.132

slave3 192.168.128.133

TIP

虚拟机安装配置详见本文最下方
虚拟机安装配置
本次搭建的所有操作均由root用户完成
本次搭建所用到的软件都放于/root家用户目录中
本次搭建所有解压的软件都放于/opt/目录下
本次搭建集群是在四台未安装hadoop的虚拟机上进行，使用scp分发文件

添加IP映射 ​

修改/etc/目录下的hosts文件

shell
vi /etc/hosts
vi /etc/hosts

添加以下内容

txt
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com
192.168.128.133 slave3 slave3.centos.com
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com
192.168.128.133 slave3 slave3.centos.com

安装Java ​

因为jdk8的获取需要在Oracle公司注册认证，我们这里使用的是openjdk

请不要安装高于8的版本，可能会有不兼容的情况

进入/root目录解压tar.gz压缩包,并指定解压到的目录/opt下

shell
cd /root
tar -xzvf openlogic-openjdk-8u382-b05-linux-x64.tar.gz -C /opt/
cd /root
tar -xzvf openlogic-openjdk-8u382-b05-linux-x64.tar.gz -C /opt/

进入/opt目录，创建软链接,这里使用软链接单纯是习惯问题，不创建也不影响，但是在后续操作中需要格外注意名字

shell
cd /opt
ln -s openlogic-openjdk-8u382-b05-linux-x64 jdk
cd /opt
ln -s openlogic-openjdk-8u382-b05-linux-x64 jdk

修改/etc/profile文件，添加环境变量

shell
vi /etc/profile
vi /etc/profile

在文件最后方添加如下示例

sh
export JAVA_HOME=/opt/jdk
export PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME=/opt/jdk
export PATH=$JAVA_HOME/bin:$PATH

使配置生效

shell
source /etc/profile
source /etc/profile

检查是否安装成功,在任意路径执行以下命令

shell
java -version
java -version

若出现如下内容则说明安装成功

INFO

[root@master opt]# java -version
openjdk version "1.8.0_382-382"
OpenJDK Runtime Environment (build 1.8.0_382-382-b05)
OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)

hadoop安装 ​

进入/root目录解压tar.gz压缩包,并指定解压到的目录/opt下

shell
cd /root
tar -xzvf hadoop-2.7.7.tar.gz -C /opt/
cd /root
tar -xzvf hadoop-2.7.7.tar.gz -C /opt/

进入/opt目录，创建软链接

shell
cd /opt
ln -s hadoop-2.7.7 hadoop
cd /opt
ln -s hadoop-2.7.7 hadoop

修改/etc/profile文件，添加环境变量

sh
export HADOOP_HOME=/opt/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_HOME=/opt/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

使配置生效

shell
source /etc/profile
source /etc/profile

检查其是否安装成功

shell
hadoop version
hadoop version

若出现如下内容则说明安装成功

INFO

Hadoop 2.7.7
Subversion Unknown -r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3ac
Compiled by stevel on 2018-07-18T22:47Z
Compiled with protoc 2.5.0
From source with checksum 792e15d20b12c74bd6f19a1fb886490
This command was run using /opt/hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar

hadoop配置 ​

在master节点中进入hadoop配置文件目录

shell
cd /opt/hadoop/etc/hadoop
cd /opt/hadoop/etc/hadoop

core-site.xml ​

修改/opt/hadoop/etc/hadoop目录下的core-site.xml文件

shell
vi core-site.xml
vi core-site.xml

修改内容如下,只需要修改<configuration></configuration>标签下的内容,后续不再提示

xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 指定HDFS中NameNode的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:8020</value>
    </property>
    <!-- 指定hadoop运行时产生文件的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop/tmp</value>
    </property>
</configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <!-- 指定HDFS中NameNode的地址 -->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:8020</value>
    </property>
    <!-- 指定hadoop运行时产生文件的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/hadoop/tmp</value>
    </property>
</configuration>

hdfs-xite.xml ​

修改/opt/hadoop/etc/hadoop目录下的hdfs-site.xml文件

shell
vi hdfs-xite.xml
vi hdfs-xite.xml

修改内容如下

xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>slave1:50090</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/opt/hadoop/tmp/data</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/opt/hadoop/tmp/name</value>
    </property>
</configuration>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>slave1:50090</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/opt/hadoop/tmp/data</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/opt/hadoop/tmp/name</value>
    </property>
</configuration>

mapred-site.xml ​

修改/opt/hadoop/etc/hadoop目录下的mapred-site.xml文件

先进行复制改名，因为本身是没有mapred-site.xml这个文件的

shell
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
cp mapred-site.xml.template mapred-site.xml
vi mapred-site.xml

修改内容如下

xml
<configuration>
    <!-- 指定mr运行在yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
    </property>
</configuration>
<configuration>
    <!-- 指定mr运行在yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>master:19888</value>
    </property>
</configuration>

yarn-site.xml ​

修改/opt/hadoop/etc/hadoop目录下的yarn-site.xml文件

shell
vi yarn-site.xml
vi yarn-site.xml

修改内容如下，二选一

这是第一版

xml
<configuration>
    <!-- Site specific YARN configuration properties -->
    <!-- reducer获取数据的方式 -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 指定YARN的ResourceManager的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <!-- 关闭虚拟内存的检测 -->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
</configuration>
<configuration>
    <!-- Site specific YARN configuration properties -->
    <!-- reducer获取数据的方式 -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 指定YARN的ResourceManager的地址 -->
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <!-- 关闭虚拟内存的检测 -->
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
</configuration>

这是第二版

xml
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>${yarn.resourcemanager.hostname}:8090</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/data/hadoop/yarn/local</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/data/tmp/logs</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://master:19888/jobhistory/logs/</value>
        <description>URL for job history server</description>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>512</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
    </property>
</configuration>
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>${yarn.resourcemanager.hostname}:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>${yarn.resourcemanager.hostname}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>${yarn.resourcemanager.hostname}:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>${yarn.resourcemanager.hostname}:8090</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>${yarn.resourcemanager.hostname}:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>${yarn.resourcemanager.hostname}:8033</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/data/hadoop/yarn/local</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/data/tmp/logs</value>
    </property>
    <property>
        <name>yarn.log.server.url</name>
        <value>http://master:19888/jobhistory/logs/</value>
        <description>URL for job history server</description>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>512</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
    </property>
</configuration>

slaves ​

修改/opt/hadoop/etc/hadoop目录下的slaves文件

shell
vi slaves
vi slaves

删除文件中的localhost,并添加你要配置的集群节点名称

这里添加master也是可以的，添加master后，master节点也会作为数据节点存在

txt
slave1
slave2
slave3
slave1
slave2
slave3

hadoop-env.sh ​

修改/opt/hadoop/etc/hadoop目录下的hadoop-env.sh文件

shell
vi hadoop-env.sh
vi hadoop-env.sh

注释掉原本文件中的export JAVA_HOME=${JAVA_HOME},添加你配置的正确的JAVA_HOME

注意：这里的JAVA_HOME需要与你安装配置的，写在/etc/profile文件下的一致

sh
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=opt/jdk
# export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=opt/jdk

yarn-env.sh ​

修改/opt/hadoop/etc/hadoop目录下的yarn-env.sh文件

shell
vi yarn-env.sh
vi yarn-env.sh

与hadoop-env.sh的配置相同，注释掉原本的export JAVA_HOME=/home/y/libexec/jdk1.6.0/,并在其下方导入正确的JAVA_HOME

sh
# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
export JAVA_HOME=/opt/jdk
# export JAVA_HOME=/home/y/libexec/jdk1.6.0/
export JAVA_HOME=/opt/jdk

配置ssh无密码登录 ​

首先在master节点中生成密钥,连续按下三次Enter键

shell
ssh-keygen -t rsa
ssh-keygen -t rsa

会获得以下部分内容，注意，只作参考

INFO

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a6:13:5a:7b:54:eb:77:58:bd:56:ef:d0:64:90:66:d4 root @ master.centos.com
The key's randomart image is:

你将在/root/.ssh/目录下获得私有密钥id_rsa和公有密钥id_rsa.pub两个文件

将master公钥复制到远程机器中,根据提示选择yes和输入目标机器的root用户密码

shell
ssh-copy-id -i /root/.ssh/id_rsa.pub master
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
ssh-copy-id -i /root/.ssh/id_rsa.pub master
ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
ssh-copy-id -i /root/.ssh/id_rsa.pub slave3

验证是否成功,不用输入密码登录进去了即可视为成功

shell
ssh slave1
ssh slave2
ssh slave3
ssh slave1
ssh slave2
ssh slave3

配置时间同步 ​

在所有的节点安装ntp服务

shell
yum install -y ntp
yum install -y ntp

若出错则需先重新挂载、跟新yum源,再执行安装操作

shell
mount /dev/cdrom /media
yum clean all
yum -y install ntp
mount /dev/cdrom /media
yum clean all
yum -y install ntp

在master节点中设置master节点为ntp服务主节点

shell
vim /etc/ntp.conf
vim /etc/ntp.conf

注释掉以server开头的行，并添加以下内容

cnf
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 10
restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 10

在所有slave节点中配置ntp，同样修改/etc/ntp.conf文件，注释掉server开头的行，并添加以下内容

cnf
server master
server master

所有节点永久关闭防火墙

shell
systemctl stop firewalld && systemctl disable firewalld
systemctl stop firewalld && systemctl disable firewalld

在master节点执行ntp服务永久启动

shell
systemctl start ntpd && systemctl enable ntpd
systemctl start ntpd && systemctl enable ntpd

在数据节点slave1、slave2、slave3上执行命令同步时间

shell
ntpdate master
ntpdate master

在各从节点也即数据节点上执行永久启动ntp服务

shell
systemctl start ntpd && systemctl enable ntpd
systemctl start ntpd && systemctl enable ntpd

分发文件 ​

INFO

由于我们使用的是配置网络的四台未安装hadoop的虚拟机，完成到这个地方只有master安装配置了hadoop和java

所以我们使用scp命令进行分发，如果是配置完master后直接克隆的可以忽略这步

分发步骤完全在master上执行

分发/etc/profile文件

shell
scp -r /etc/profile slave1:/etc/
scp -r /etc/profile slave2:/etc/
scp -r /etc/profile slave3:/etc/
scp -r /etc/profile slave1:/etc/
scp -r /etc/profile slave2:/etc/
scp -r /etc/profile slave3:/etc/

记得在各个节点source以下/etc/profile

shell
source /etc/profile
source /etc/profile

分发jdk

shell
scp -r /opt/jdk slave1:/opt/
scp -r /opt/jdk slave2:/opt/
scp -r /opt/jdk slave3:/opt/
scp -r /opt/jdk slave1:/opt/
scp -r /opt/jdk slave2:/opt/
scp -r /opt/jdk slave3:/opt/

分发hadoop

shell
scp -r /opt/hadoop slave1:/opt/
scp -r /opt/hadoop slave2:/opt/
scp -r /opt/hadoop slave3:/opt/
scp -r /opt/hadoop slave1:/opt/
scp -r /opt/hadoop slave2:/opt/
scp -r /opt/hadoop slave3:/opt/

启动集群 注意 ​

WARNING

注意：不要多次格式化，会出现各类进程丢失的情况

如果出现了此类情况，需要将你配置的tmp缓存文件夹和hadoop目录下的logs文件夹下的文件删除

本次搭建若出现此类情况只需在hadoop目录下删除tmp目录和logs目录下的文件

进行格式化，只在master中进行

shell
hdfs namenode -format
hdfs namenode -format

启动集群,只需要在主节点上执行命令

shell
start-all.sh
start-all.sh

如果有需要开启日志服务也可以开启(可选)

shell
mr-jobhistory-daemon.sh start historyserver
mr-jobhistory-daemon.sh start historyserver

jps查看进程

shell
jps
jps