CentOS 7 安装Hadoop 2.7.1-白红宇

CentOS 7 安装Hadoop 2.7.1

阅读量：6594 次

发布时间：2019-06-24

本文共 11654 字，大约阅读时间需要 38 分钟。

两台机器 CentOS7（机器名分别为master-CentOS7、slave-CentOS7）内存2G （笔记本开虚拟机快撑不住了╮(╯-╰)╭

CentOS7 与 CetnOS6 有一些区别

网络配置

master-CentOS7

[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736TYPE=EthernetBOOTPROTO=staticDEFROUTE=yesPEERDNS=yesPEERROUTES=yesIPV4_FAILURE_FATAL=noIPV6INIT=yesIPV6_AUTOCONF=yesIPV6_DEFROUTE=yesIPV6_PEERDNS=yesIPV6_PEERROUTES=yesIPV6_FAILURE_FATAL=noNAME=eno16777736UUID=b30f5765-ecd7-4dba-a0ed-ebac92c836bdDEVICE=eno16777736ONBOOT=yesIPADDR=192.168.1.182NETMASK=255.255.255.0GATEWAY=192.168.1.1DNS1=114.114.114.114DNS2=8.8.4.4

网络信息根据自己实际的网络情况配置。

[root@localhost ~]# systemctl restart network[root@localhost ~]# ifconfig

slave-CentOS7

[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736TYPE=EthernetBOOTPROTO=staticDEFROUTE=yesPEERDNS=yesPEERROUTES=yesIPV4_FAILURE_FATAL=noIPV6INIT=yesIPV6_AUTOCONF=yesIPV6_DEFROUTE=yesIPV6_PEERDNS=yesIPV6_PEERROUTES=yesIPV6_FAILURE_FATAL=noNAME=eno16777736UUID=b30f5765-ecd7-4dba-a0ed-ebac92c836bdDEVICE=eno16777736ONBOOT=yesIPADDR=192.168.1.183NETMASK=255.255.255.0GATEWAY=192.168.1.1DNS1=114.114.114.114DNS2=8.8.4.4

网络信息根据自己实际的网络情况配置。

[root@localhost ~]# systemctl restart network[root@localhost ~]# ifconfig

设置hosts、hostname

master-CentOS7

[root@localhost ~]# vi /etc/hosts

添加

192.168.1.182 master192.168.1.183 slave

[root@localhost ~]# vi /etc/hostnamelocalhost.localdomain

内容修改为

master

slave-CentOS7

[root@localhost ~]# vi /etc/hosts

添加

192.168.1.182 master192.168.1.183 slave

[root@localhost ~]# vi /etc/hostnamelocalhost.localdomain

内容修改为

slave

关闭selinux

master-CentOS7

[root@master ~]# getenforceEnforcing[root@master ~]# vi /etc/selinux/configSELINUX=enforcing

修改为

SELINUX=disabled

保存重启

[root@master ~]# getenforceDisabled

slave-CentOS7

[root@slave ~]# getenforceEnforcing[root@slave ~]# vi /etc/selinux/configSELINUX=enforcing

修改为

SELINUX=disabled

保存重启

[root@slave ~]# getenforceDisabled

关闭firewalld

master-CentOS7

[root@master ~]# systemctl disable firewalldRemoved symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.[root@master ~]# systemctl stop firewalld[root@master ~]# iptables -nvLChain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destinationChain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destinationChain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destination[root@master ~]# yum install -y iptables-services[root@master ~]# service iptables saveiptables: Saving firewall rules to /etc/sysconfig/iptables:[  确定  ][root@master ~]# systemctl enable iptablesCreated symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.

slave-CentOS7

[root@slave ~]# systemctl disable firewalldRemoved symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.[root@slave ~]# systemctl stop firewalld[root@slave ~]# iptables -nvLChain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destination        Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destination        Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target     prot opt in     out     source               destination        [root@slave ~]# yum install -y iptables-services[root@slave ~]# service iptables saveiptables: Saving firewall rules to /etc/sysconfig/iptables:[  确定  ][root@slave ~]# systemctl enable iptablesCreated symlink from /etc/systemd/system/basic.target.wants/iptables.service to /usr/lib/systemd/system/iptables.service.

密钥登陆

master-CentOS7

[root@master ~]# ssh-keygen

一直回车

[root@master ~]# cat .ssh/id_rsa.pub

复制~/.ssh/id_rsa.pub 内容

slave-CentOS7

[root@slave ~]# vi .ssh/authorized_keys

复制master的~/.ssh/id_rsa.pub 内容到 slave的~/.ssh/authorized_keys时报错
“.ssh/authorized_keys” E212: Can’t open file for writing

解决方案

[root@slave ~]# ls -ld .sshls: 无法访问.ssh: 没有那个文件或目录[root@slave ~]# mkdir .ssh; chmod 700 .ssh[root@slave ~]# ls -ld .sshdrwx------ 2 root root 6 8月  28 15:59 .ssh[root@slave ~]# vi .ssh/authorized_keys

复制master的~/.ssh/id_rsa.pub 内容到 slave的~/.ssh/authorized_keys

[root@slave ~]# ls -l !$ls -l .ssh/authorized_keys-rw-r--r-- 1 root root 418 8月  28 16:02 .ssh/authorized_keys

测试

master-CentOS7

[root@master ~]# ssh master[root@master ~]# exit[root@master ~]# ssh slave[root@slave ~]# exit

安装JDK

hadoop2.7 需要安装jdk1.7版本，下载地址

先卸载CetnOS7自带的JDK

以slave-CentOS7为例（master-CetnOS7、slave-CentOS7上都需要卸载CetnOS7自带的JDK）

[root@slave ~]# java -versionopenjdk version "1.8.0_101"OpenJDK Runtime Environment (build 1.8.0_101-b13)OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)[root@master ~]# rpm -qa |grep jdkjava-1.7.0-openjdk-headless-1.7.0.111-2.6.7.2.el7_2.x86_64java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64java-1.8.0-openjdk-headless-1.8.0.101-3.b13.el7_2.x86_64java-1.7.0-openjdk-1.7.0.111-2.6.7.2.el7_2.x86_64[root@slave ~]# yum -y remove java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.2.el7_2.x86_64[root@slave ~]# yum -y remove java-1.8.0-openjdk-1.8.0.101-3.b13.el7_2.x86_64[root@slave ~]# yum -y remove java-1.8.0-openjdk-headless-1.8.0.101-3.b13.el7_2.x86_64[root@slave ~]# java -version-bash: /usr/bin/java: 没有那个文件或目录

master-CentOS7

[root@master ~]# wget http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz[root@master ~]# tar zxvf jdk-7u79-linux-x64.tar.gz[root@master ~]# mv jdk1.7.0_79 /usr/local/[root@master ~]# vi /etc/profile.d/java.sh

添加

export JAVA_HOME=/usr/local/jdk1.7.0_79export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/bin[root@master ~]# source !$source /etc/profile.d/java.sh[root@master ~]# java -versionjava version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)[root@master ~]# scp jdk-7u79-linux-x64.tar.gz slave:/root/[root@master ~]# scp /etc/profile.d/java.sh slave:/etc/profile.d/

slave-CentOS7

[root@slave ~]# tar zxvf jdk-7u79-linux-x64.tar.gz[root@slave ~]# mv jdk1.7.0_79 /usr/local/[root@slave ~]# source /etc/profile.d/java.sh[root@slave ~]# java -versionjava version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

安装Hadoop

master-CentOS7

[root@master ~]# wget 'http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz'[root@master ~]# tar zxvf hadoop-2.7.1.tar.gz[root@master ~]# mv hadoop-2.7.1 /usr/local/Hadoop[root@master ~]# ls !$ls /usr/local/hadoopbin  include  libexec      NOTICE.txt  sbinetc  lib      LICENSE.txt  README.txt  share[root@master ~]# mkdir /usr/local/hadoop/tmp  /usr/local/hadoop/dfs  /usr/local/hadoop/dfs/data  /usr/local/hadoop/dfs/name[root@master ~]# ls /usr/local/hadoopbin  dfs  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin  share  tmp[root@master ~]# rsync -av /usr/local/hadoop  slave:/usr/local

slave-CentOS7

[root@slave ~]# ls /usr/local/hadoopbin  etc      lib      LICENSE.txt  README.txt  sharedfs  include  libexec  NOTICE.txt   sbin        tmp

配置Hadoop

master-CentOS7

[root@master ~]# vi /usr/local/hadoop/etc/hadoop/core-site.xml

添加


         
              
       
        fs.defaultFS
               
       
        hdfs://master:9000
           
          
              
       
        hadoop.tmp.dir
               
       
        file:/usr/local/hadoop/tmp
           
          
              
       
        io.file.buffer.size
               
       
        131702

[root@master ~]# vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml

添加


         
              
       
        dfs.namenode.name.dir
               
       
        file:/usr/local/hadoop/dfs/name
           
          
              
       
        dfs.datanode.data.dir
               
       
        file:/usr/local/hadoop/dfs/data
           
          
              
       
        dfs.replication
               
       
        1
           
          
              
       
        dfs.namenode.secondary.http-address
               
       
        master:9001
           
          
          
       
        dfs.webhdfs.enabled
           
       
        true

replication默认为3，如果不修改，DataNode少于3台就会报错。我的实验环境就一台从机（一台slave），所以值为1。

[root@master ~]# mv /usr/local/hadoop/etc/hadoop/mapred-site.xml.template  /usr/local/hadoop/etc/hadoop/mapred-site.xml[root@master ~]# vi /usr/local/hadoop/etc/hadoop/mapred-site.xml

添加


         
              
       
        mapreduce.framework.name
               
       
        yarn
           
          
              
       
        mapreduce.jobhistory.address
               
       
        master:10020
           
          
              
       
        mapreduce.jobhistory.webapp.address
               
       
        master:19888

[root@master ~]# vi /usr/local/hadoop/etc/hadoop/yarn-site.xml

添加


     
          
              
       
        yarn.nodemanager.aux-services
               
       
        mapreduce_shuffle
           
          
              
       
        yarn.nodemanager.auxservices.mapreduce.shuffle.class
               
       
        org.apache.hadoop.mapred.ShuffleHandler
           
          
              
       
        yarn.resourcemanager.address
               
       
        master:8032
           
          
              
       
        yarn.resourcemanager.scheduler.address
               
       
        master:8030
           
          
              
       
        yarn.resourcemanager.resource-tracker.address
               
       
        master:8031
           
          
              
       
        yarn.resourcemanager.admin.address
               
       
        master:8033
           
          
              
       
        yarn.resourcemanager.webapp.address
               
       
        master:8088

[root@master ~]# cd /usr/local/hadoop/etc/hadoop[root@master hadoop]# vi hadoop-env.sh

更改

export JAVA_HOME=/usr/local/jdk1.7.0_79

[root@master hadoop]# vi yarn-env.sh

更改

export JAVA_HOME=/usr/local/jdk1.7.0_79

[root@master hadoop]# vi slaves

更改为

slave

注意slave-CentOS7的IP

[root@master hadoop]# rsync -av /usr/local/hadoop/etc/ slave:/usr/local/hadoop/etc/

slave-CentOS7

[root@slave ~]# cd /usr/local/hadoop/etc/hadoop/[root@slave hadoop]# cat slavesslave

检查slave没问题

启动Hadoop

master-CentOS7

[root@master hadoop]# /usr/local/hadoop/bin/hdfs namenode -format[root@master hadoop]# echo $?0[root@master hadoop]# /usr/local/hadoop/sbin/start-all.sh[root@master hadoop]# jps19907 ResourceManager19604 SecondaryNameNode19268 NameNode20323 Jps

在执行格式化-format命令时，要避免NameNode的namespace ID与DataNode的namespace ID的不一致。这是因为每格式化一次就会产生Name、Data、temp等临时文件记录信息，多次格式化会产生很多的Name、Data、temp，这样容易导致ID的不同，使Hadoop不能正常运行。每次执行格式化-format命令时，就需要将DataNode和NameNode上原来的data、temp文件删除。

slave-CentOS7

[root@slave hadoop]# jps18113 NodeManager18509 Jps17849 DataNode

浏览器打开

测试Hadoop

master-CentOS7

[root@master hadoop]# cd /usr/local/hadoop/[root@master hadoop]# bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 10 10

停止服务

master-CentOS7

[root@master hadoop]# cd /usr/local/hadoop[root@master hadoop]# sbin/stop-all.sh

如果提示 copyFromLocal: Cannot create directory /123/. Name node is in safe mode.
这是因为开启了安全模式
解决方法：
cd /usr/local/Hadoop
bin/hdfs dfsadmin -safemode leave

转载于:https://www.cnblogs.com/Genesis2018/p/8304729.html

你可能感兴趣的文章

java 中字符串比较equals()和equalsIgnoreCase()的区别