hadoop cdh5.15部署

Posted by llussy on January 29, 2019

hadoop cdh 5.15 部署

[TOC]

安装前准备

初始化

主机初始化:

wget http://10.21.8.20/init/puppetclient
sh puppetclient colligationhadoop ''

磁盘初始化:

wget http://10.21.8.20/init/centos-disk-init.sh
sh centos-disk-init.sh

hosts

修改各个节点的: /etc/hosts

配置时区

 mv /etc/localtime /etc/localtime.bak
 ln -s  /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

JDK

yum install jdk-1.8* -y
source /etc/profile.d/jdk.sh

设置SWAP

机器初始化后vm.swappiness的值就是10

echo vm.swappiness = 10 >> /etc/sysctl.conf

sysctl vm.swappiness=10

设置透明大页面

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

添加到开机自启:

将如下脚本添加到/etc/rc.d/rc.local文件中:

cat  >> /etc/rc.d/rc.local << EOF
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
EOF

/data/opt 软链到/opt下 (/opt空间太小,CDH默认会找/opt目录)

mv /opt /opt.bak
mkdir /data/opt
ln -s /data/opt /opt 

新建scm账号(所有主机)

useradd --system --home=/var/lib/cloudera-scm-server --no-create-home --shell=/sbin/nologin --comment "Cloudera Manager" cloudera-scm

安装配置mysql主从

Cloudera Manager 表引擎为InnoDB,如果是MyISAM engine会出现集群无法起来的情况;
mysql5.6版本如果开启了gtid,会出现如下报错:ERROR Unable to create/drop a table.
java.sql.SQLException: CREATE TABLE ... SELECT is forbidden when @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1.
所以最好关闭gtid ;

mysql软件包安装

安装mysql

yum install -y http://10.80.21.156/rpm/temporary/mysql-devel-5.6.26-1.el6.x86_64.rpm
yum install -y http://10.80.21.156/rpm/temporary/mysql-5.6.26-1.el6.x86_64.rpm

修改配置文件

mv /etc/my.cnf /etc/my.cnf.bak
ln -s /usr/local/mysql/etc/my.cnf /etc/my.cnf

修改innodb_buffer_pool_size参数和 server-id参数 关闭gtid

增加环境变量

cd /etc/profile.d/ ; wget http://10.21.8.20/mysql/mysql5.6.sh
source /etc/profile.d/mysql5.6.sh

启动

/etc/init.d/mysqld start

设置root密码

update mysql.user set password=password('xxxxxxxx') where user='root';

主从配置

grant replication slave on *.* to 'repl'@'%' identified by 'xxxxxx';
show master status;
change master to master_host='10.90.11.21',master_user='repl',master_password='xxxxxx',master_log_file='mysql-bin.000003',master_log_pos=1557;

备份

软件包安装

yum install -y http://10.80.21.156/rpm/all/xtrabackup-2.1.6-1.x86_64.rpm

增加备份账号

####参考 percona官网 https://www.percona.com/doc/percona-xtrabackup/2.4/using_xtrabackup/privileges.html
mysql> CREATE USER 'bkpuser'@'localhost' IDENTIFIED BY 'xxxxxx';
mysql> GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO 'bkpuser'@'localhost';
mysql> FLUSH PRIVILEGES;

备份脚本

#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/mysql/bin:/root/bin:/data/bin/xtrabackup/bin
export PATH
fulldir="/data/backup/full"
DATE=`date +%F`
LDIR="/data/backup/full"
cd /data/bin/xtrabackup/bin
/data/bin/xtrabackup/bin/innobackupex  --user=bkpuser --password=xxxxxx --defaults-file=/etc/my.cnf  $fulldir >>/tmp/xtraback.log 
find $fulldir  -maxdepth 1 -type d -mtime +5 -exec echo "removeing:"  {} \;|xargs rm -rf
#bash /data/script/cron/backup_center.sh  >>/tmp/rsync.log

建立CM,Hive等需要的表

create database hive default character set utf8; 
CREATE USER 'hive'@'%' IDENTIFIED BY 'xxxxxxxx';  
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';  
FLUSH PRIVILEGES; 

安装mysql-connector

下载 mysql-connector-java jar包 【MySQL 5.6 requires a driver version 5.1.26 or higher.】

在安装了mysql的两台机器操作

cd /usr/share/java
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.34/mysql-connector-java-5.1.34.jar 
ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar 

安装Cloudera Manager Server 和Agent

所有节点下载CM软件包:

cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz

节点解压安装

cloudera manager的目录默认位置在/opt下,解压:tar xzvf cloudera-manager*.tar.gz将解压后的cm-5.15.0和cloudera目录放到/opt目录下。

所有节点下载cm软件包

cd /opt
wget http://10.21.8.20/cdh/5.15/cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz
tar zxvf cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz 
rm cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz -f

主节点创建和配置SCM所需的数据库

# /opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host % scm scm xxxxxx
Enter database password: 
JAVA_HOME=/usr/java/jdk1.8.0_121/
Verifying that we can write to /data/opt/cm-5.15.0/etc/cloudera-scm-server
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
Creating SCM configuration file in /data/opt/cm-5.15.0/etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_121//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/data/opt/cm-5.15.0/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /data/opt/cm-5.15.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
[2018-10-12 09:57:14,027] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database.
All done, your SCM database is configured correctly!

所有节点修改config.ini配置文件

sed -i 's/server_host=localhost/server_host=10.90.11.21/g' /opt/cm-5.15.0/etc/cloudera-scm-agent/config.ini

启动cloudera-scm-server 和 agent

# /opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
# /opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]

CDH 下载

下载地址:http://archive.cloudera.com/cdh5/parcels

选择版本后进入目录下载:parcel、parcel.sha1、manifest.json三个文件,下载完将sha1后缀改为sha

CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel

CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha

manifest.json

cd /opt/cloudera/parcel-repo; #将parcel、parcel.sha、manifest.json三个文件放入这个目录中

cd /opt/cloudera/parcel-repo
wget http://10.21.8.20/cdh/5.15/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
wget http://10.21.8.20/cdh/5.15/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha
wget http://10.21.8.20/cdh/5.15/manifest.json

所有节点:mkdir -p /opt/cloudera/parcels;chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels

解析下:CM将CDH从主节点的/opt/cloudera/parcel-repo目录中抽取出来,分发解压激活到各个节点的/opt/cloudera/parcels (亦可以手工复制进去,这样可以加快速度)

CDH安装配置

访问主节点的7180端口。默认密码admin/admin

登录界面

继续

由于节点都启动了cloudera-scm-agent ,节点会出现在当前管理的主机。选中所有主机,继续。

这里要注意cdh的版本,要与下载的一致。

此步骤需要等待一段时间:

完成了继续

检测主机情况,根据提示处理

选择安装的服务,这里选择自定义服务:

规划主机的角色:

继续

服务安装后,可以看到各服务的状态,有问题的会标黄或者标红。根据具体情况处理。

其中需要调整的是:

1、堆转储目录 默认在/tmp, 可以设置在/opt/tmp或者/data/tmp

2、java堆栈大小, hdfs、yarn的都需要修改。

调整后状态:

启动hdfs HA

输入nameservice的名称:

分配角色:

输入角色目录:

启动:

启动成功后,可以手动故障转移namenode 。

启动yarn ha