hadoop cdh 5.15 部署
[TOC]
安装前准备
初始化
主机初始化:
wget http://10.21.8.20/init/puppetclient
sh puppetclient colligationhadoop ''
磁盘初始化:
wget http://10.21.8.20/init/centos-disk-init.sh
sh centos-disk-init.sh
hosts
修改各个节点的: /etc/hosts
配置时区
mv /etc/localtime /etc/localtime.bak
ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
JDK
yum install jdk-1.8* -y
source /etc/profile.d/jdk.sh
设置SWAP
机器初始化后vm.swappiness的值就是10
echo vm.swappiness = 10 >> /etc/sysctl.conf
sysctl vm.swappiness=10
设置透明大页面
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
添加到开机自启:
将如下脚本添加到/etc/rc.d/rc.local文件中:
cat >> /etc/rc.d/rc.local << EOF
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
EOF
/data/opt 软链到/opt下 (/opt空间太小,CDH默认会找/opt目录)
mv /opt /opt.bak
mkdir /data/opt
ln -s /data/opt /opt
新建scm账号(所有主机)
useradd --system --home=/var/lib/cloudera-scm-server --no-create-home --shell=/sbin/nologin --comment "Cloudera Manager" cloudera-scm
安装配置mysql主从
Cloudera Manager 表引擎为InnoDB,如果是MyISAM engine会出现集群无法起来的情况;
mysql5.6版本如果开启了gtid,会出现如下报错:ERROR Unable to create/drop a table.
java.sql.SQLException: CREATE TABLE ... SELECT is forbidden when @@GLOBAL.ENFORCE_GTID_CONSISTENCY = 1.
所以最好关闭gtid ;
mysql软件包安装
安装mysql
yum install -y http://10.80.21.156/rpm/temporary/mysql-devel-5.6.26-1.el6.x86_64.rpm
yum install -y http://10.80.21.156/rpm/temporary/mysql-5.6.26-1.el6.x86_64.rpm
修改配置文件
mv /etc/my.cnf /etc/my.cnf.bak
ln -s /usr/local/mysql/etc/my.cnf /etc/my.cnf
修改innodb_buffer_pool_size
参数和 server-id
参数 关闭gtid
增加环境变量
cd /etc/profile.d/ ; wget http://10.21.8.20/mysql/mysql5.6.sh
source /etc/profile.d/mysql5.6.sh
启动
/etc/init.d/mysqld start
设置root密码
update mysql.user set password=password('xxxxxxxx') where user='root';
主从配置
grant replication slave on *.* to 'repl'@'%' identified by 'xxxxxx';
show master status;
change master to master_host='10.90.11.21',master_user='repl',master_password='xxxxxx',master_log_file='mysql-bin.000003',master_log_pos=1557;
备份
软件包安装
yum install -y http://10.80.21.156/rpm/all/xtrabackup-2.1.6-1.x86_64.rpm
增加备份账号
####参考 percona官网 https://www.percona.com/doc/percona-xtrabackup/2.4/using_xtrabackup/privileges.html
mysql> CREATE USER 'bkpuser'@'localhost' IDENTIFIED BY 'xxxxxx';
mysql> GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO 'bkpuser'@'localhost';
mysql> FLUSH PRIVILEGES;
备份脚本
#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/mysql/bin:/root/bin:/data/bin/xtrabackup/bin
export PATH
fulldir="/data/backup/full"
DATE=`date +%F`
LDIR="/data/backup/full"
cd /data/bin/xtrabackup/bin
/data/bin/xtrabackup/bin/innobackupex --user=bkpuser --password=xxxxxx --defaults-file=/etc/my.cnf $fulldir >>/tmp/xtraback.log
find $fulldir -maxdepth 1 -type d -mtime +5 -exec echo "removeing:" {} \;|xargs rm -rf
#bash /data/script/cron/backup_center.sh >>/tmp/rsync.log
建立CM,Hive等需要的表
create database hive default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'xxxxxxxx';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
FLUSH PRIVILEGES;
安装mysql-connector
下载 mysql-connector-java jar包 【MySQL 5.6 requires a driver version 5.1.26 or higher.】
在安装了mysql的两台机器操作
cd /usr/share/java
wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.34/mysql-connector-java-5.1.34.jar
ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar
安装Cloudera Manager Server 和Agent
所有节点下载CM软件包:
cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz
节点解压安装
cloudera manager的目录默认位置在/opt下,解压:tar xzvf cloudera-manager*.tar.gz
将解压后的cm-5.15.0和cloudera目录放到/opt目录下。
所有节点下载cm软件包
cd /opt
wget http://10.21.8.20/cdh/5.15/cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz
tar zxvf cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz
rm cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz -f
主节点创建和配置SCM所需的数据库
# /opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql -uroot -p --scm-host % scm scm xxxxxx
Enter database password:
JAVA_HOME=/usr/java/jdk1.8.0_121/
Verifying that we can write to /data/opt/cm-5.15.0/etc/cloudera-scm-server
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
Creating SCM configuration file in /data/opt/cm-5.15.0/etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_121//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/data/opt/cm-5.15.0/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /data/opt/cm-5.15.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
[2018-10-12 09:57:14,027] INFO 0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database.
All done, your SCM database is configured correctly!
所有节点修改config.ini配置文件
sed -i 's/server_host=localhost/server_host=10.90.11.21/g' /opt/cm-5.15.0/etc/cloudera-scm-agent/config.ini
启动cloudera-scm-server 和 agent
# /opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server: [ OK ]
# /opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start
Starting cloudera-scm-agent: [ OK ]
CDH 下载
下载地址:http://archive.cloudera.com/cdh5/parcels
选择版本后进入目录下载:parcel、parcel.sha1、manifest.json三个文件,下载完将sha1后缀改为sha
CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha
manifest.json
cd /opt/cloudera/parcel-repo; #将parcel、parcel.sha、manifest.json三个文件放入这个目录中
cd /opt/cloudera/parcel-repo
wget http://10.21.8.20/cdh/5.15/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel
wget http://10.21.8.20/cdh/5.15/CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha
wget http://10.21.8.20/cdh/5.15/manifest.json
所有节点:mkdir -p /opt/cloudera/parcels;chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels
解析下:CM将CDH从主节点的/opt/cloudera/parcel-repo目录中抽取出来,分发解压激活到各个节点的/opt/cloudera/parcels (亦可以手工复制进去,这样可以加快速度)
CDH安装配置
访问主节点的7180端口。默认密码admin/admin
继续
由于节点都启动了cloudera-scm-agent ,节点会出现在当前管理的主机。选中所有主机,继续。
这里要注意cdh的版本,要与下载的一致。
此步骤需要等待一段时间:
完成了继续
检测主机情况,根据提示处理
选择安装的服务,这里选择自定义服务:
规划主机的角色:
继续
服务安装后,可以看到各服务的状态,有问题的会标黄或者标红。根据具体情况处理。
其中需要调整的是:
1、堆转储目录 默认在/tmp, 可以设置在/opt/tmp或者/data/tmp
2、java堆栈大小, hdfs、yarn的都需要修改。
调整后状态:
启动hdfs HA
输入nameservice的名称:
分配角色:
输入角色目录:
启动:
启动成功后,可以手动故障转移namenode 。
启动yarn ha