llussy

kubernetes node可用内存

Wed, 16 Nov 2022 21:20:30 +0800

kubernetes node可用内存

kubectl top node 发现内存使用率很高（超过100%），但是实际上内存还有很多可用，这是为什么呢？

# kubectl top node 
NAME    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
node-1   553m         14%    11363Mi         91%       
node-2   400m         10%    10120Mi         81%       
node-3   224m         5%     10613Mi         85%       
node-4   1086m        27%    12535Mi         101% 

查看node可调度的内存

# kubectl describe node node-1 
Allocatable:
  cpu:                3900m
  ephemeral-storage:  190119300396
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             12670892Ki
  pods:               64

这是由于kubelet 可调度的内存不是所有的内存，需要去掉预留资源和eviction-threshold,这个预留资源是由kubelet的–kube-reserved和–system-reserved参数指定的,eviction临界点由kubelet的–eviction-hard参数指定的。

官方的一张图解释的很清楚

让我们来计算一下

kubelet的预留配置

# cat /etc/kubernetes/kubelet-customized-args.conf 
KUBELET_CUSTOMIZED_ARGS=" --eviction-hard=imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5% --system-reserved=cpu=50m,memory=1272Mi --kube-reserved=cpu=50m,memory=1272Mi --kube-reserved=pid=1000 --system-reserved=pid=1000 "

free -m 查看系统总内存

# free -m
              total        used        free      shared  buff/cache   available
Mem:          15217       12300         236           5        2680        2682
Swap:             0           0           0

系统总内存减去kubelet预留的内存和eviction，就是node可用的内存

# kubelet Allocatable内存转换成Mi 
12670892/1024=12373Mi

# 系统总内存减去kubelet预留的内存
15217-1272-1272-300=12373Mi

reference

reserve-compute-resources

Taints and Tolerations

Wed, 16 Nov 2022 20:20:30 +0800

Taints 污点 and Tolerations 容忍

Taints and Tolerations 是pod的一个属性，它将允许某些pod在指定的节点上或者不允许指定的pod到指定节点上或者必须要有某些的pod才能调度到指定节点上。

可以通过kubectl taint来执行 Taints和Tolerations和搭配使用的，Taints定义在Node节点上，声明污点及标准行为，Tolerations定义在Pod，声明可接受得污点。

# kubectl taint nodes node1 key1=value1:NoSchedule
kubectl taint nodes node1 key=value:NoSchedule 
 -------node1:节点名称 
-------key 就是key 
-------value 就是value 
-------effect效果是 NoSchedule 
意思是说只有拥有: key=value:NoSchedule这个属性的pod才能调度到node1上


kubectl taint nodes node1 name=lucky:NoSchedule

spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: name
    value: lucky
    effect: NoSchedule

operator可以定义为：

Equal 表示key是否等于value，默认
Exists 表示key是否存在，此时无需定义value

effect可以定义为：

NoSchedule 表示不允许调度，已调度的不影响
PreferNoSchedule 表示尽量不调度
NoExecute 表示不允许调度，已调度的在tolerationSeconds（定义在Tolerations上）后删除

看下Master节点上的annotations定义：

annotations: scheduler.alpha.kubernetes.io/taints: '[{"key":"dedicated","value":"master","effect":"NoSchedule"}]'volumes.kubernetes.io/controller-managed-attach-detach: "true"

Master节点上定义了Taints，它表达的是一个含义：此节点已被key=value污染，Pod调度不允许（Pod Tolerates Node Taints策略）或尽量不（Taint Toleration Priority策略）调度到此节点，除非是能够容忍（Tolerations）key=value污点的Pod。

Master节点上定义了Taints，声明：如果不是带有Tolerations定义为[{"key":"dedicated","value":"master","effect":"NoSchedule"}]的Pod，不允许调度到Master节点，PS：operator的默认值为Equal，所以可以不必显示声明。

reference

Taints and Tolerations

官方文档

K8S学习之污点容忍

kubernetes 常用操作

Wed, 09 Mar 2022 16:20:30 +0800

常用命令

kubectl logs mypod --previous
kubectl delete pod -n kube-system test-q2ngc --grace-period=180
kubectl delete pod <pod> --force --grace-period=0
kubectl exec -it -n monitoring test-f7bdb9769-ctxbs -- env COLUMNS=210 LINES=60 bash
kubectl scale deployments/kubernetes-bootcamp --replicas=3
kubectl rollout status ds/<daemonset-name>  # status
kubectl rollout undo deployments/kubernetes-bootcamp #回退
kubectl  get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq
kubectl -v=8 top node
kubectl -v=8 get svc kubernetes

kubectl apply -f ds.yaml --dry-run -o go-template='{{.spec.updateStrategy.type}}{{"\\\\n"}}'


# doc
kubectl api-resources
kubectl explain statefulsets

journalctl -xef -u kubelet

# 打印使用的api
kubectl get pod -v=9

资源使用情况

kubectl get deployments.apps -n xxx -o custom-columns=:.metadata.namespace,:.metadata.name,:.spec.replicas,:.spec.template.spec.containers[0].resources.limits.cpu,:.spec.template.spec.containers[0].resources.limits.memory

运行测试 pod

kubectl run busytest --rm -it --image=busybox sh
kubectl run dig --rm -it --image=docker.io/azukiapp/dig /bin/sh
kubectl run centos --rm -it --image=centos bash

查看某个app pod分布情况

kubectl  get pods -o wide -l app="nginx-server" | awk '{print $7}'| \\
awk '{ count[$0]++  }
 END {
   printf("%-35s: %s\\n","Word","Count");
   for(ind in count){
    printf("%-35s: %d\\n",ind,count[ind]);
   }
 }'

label master

# label
kubectl get node --show-labels
kubectl label node yztest-ops-k8s24v8-yz role=master
kubectl label node yztest-ops-k8s24v8-yz role-
# spec.template.spec.nodeSelector:  node label位置

# 标记为node
kubectl label node hostname node-role.kubernetes.io/node=

# master 充当node
kubectl taint node yztest-ops-k8s24v8-yz node-role.kubernetes.io/master-

# 恢复master角色
kubectl taint node yztest-ops-k8s24v8-yz node-role.kubernetes.io/master="":NoSchedule

k8s自动补全

source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

patch

如果一个容器已经在运行，这时需要对一些容器属性进行修改，又不想删除容器，或不方便通过replace的方式进行更新。kubernetes还提供了一种在容器运行时，直接对容器进行修改的方式，就是patch命令。如前面创建pod的label是app=nginx-2，如果在运行过程中，需要把其label改为app=nginx-3，这patch命令如下：

kubectl patch pod rc-nginx-2-kpiqt -p '{"metadata":{"labels":{"app":"nginx-3"}}}'

deployment

rolling-update

rolling-update是一个非常重要的命令，对于已经部署并且正在运行的业务，rolling-update提供了不中断业务的更新方式。rolling-update每次起一个新的pod，等新pod完全起来后删除一个旧的pod，然后再起一个新的pod替换旧的pod，直到替换掉所有的pod。

rolling-update需要确保新的版本有不同的name，Version和label，否则会报错。

kubectl rolling-update rc-nginx-2 -f rc-nginx.yaml

如果在升级过程中，发现有问题还可以中途停止update，并回滚到前面版本

kubectl rolling-update rc-nginx-2 —rollback

scale

kubectl scale --replicas=3 rs/foo                                 # Scale a replicaset named 'foo' to 3
kubectl scale --replicas=3 -f foo.yaml                            # Scale a resource specified in "foo.yaml" to 3
kubectl scale --current-replicas=2 --replicas=3 deployment/mysql  # If the deployment named mysql's current size is 2, scale mysql to 3
kubectl scale --replicas=5 rc/foo rc/bar rc/baz                   # Scale multiple replication controllers

rollout

kubectl rollout history deployment/frontend                      # Check the history of deployments including the revision 
kubectl rollout undo deployment/frontend                         # Rollback to the previous deployment
kubectl rollout undo deployment/frontend --to-revision=2         # Rollback to a specific revision
kubectl rollout status -w deployment/frontend                    # Watch rolling update status of "frontend" deployment until completion
kubectl rollout restart deployment/frontend                      # Rolling restart of the "frontend" deployment

pod 与主机文件传输

kubectl cp foo-pod:/var/log/foo.log foo.log
kubectl cp localfile foo-pod:/etc/remotefile

kubeadm

kubeadm config print init-defaults
kubeadm config view

快速查看pid的容器名字

podinfo() {
  CID=$(cat /proc/$1/cgroup | awk -F '/' '{print $5}')
  CID=$(echo ${CID:7:10})
  crictl inspect $CID | jq '.status.labels["io.kubernetes.pod.name"]'
}

增加到~/.bashrc

# podinfo 16529
"test-6cc546b5-75q9n"

另一种方法

#!/bin/bash
forid=$(ps -efl|grep log4j-core|grep -v grep|awk '{print $4}'|xargs -i pstree -sg {} |awk 'NR==1{print $0}'|grep -E -o "[0-9]{1,10}"|grep -v ^1)
for I in $forid; do
docker ps -q | xargs docker inspect --format '{{.State.Pid}}, {{.Name}}' | grep "$I"|grep -v formatMsgNoLookups;
done

nsenter

快速脚本，需进入进入pod所在node,增加下面函数到~/.bashrc

function e() {
      set -eu
      ns=${2-"default"}
      pod=`kubectl -n $ns describe pod $1 | grep -A10 "^Containers:" | grep -Eo 'docker://.*$' | head -n 1 | sed 's/docker:\\/\\/\\(.*\\)$/\\1/'`
      pid=`docker inspect -f {{.State.Pid}} $pod`
      echo "entering pod netns for $ns/$1"
      cmd="nsenter -n --target $pid"
      echo $cmd
      $cmd
  }

进入pod所在netns

e istio-galley-58c7c7c646-m6568 istio-system
e proxy-5546768954-9rxg6 # 省略 NAMESPACE 默认为 default
tcpdump -i eth0 -w test.pcap port 80

原文地址： Kubernetes 问题定位技巧：容器内抓包

metrics

cadvisor

kubectl get --raw /api/v1/nodes/10.66.240.101/proxy/metrics/cadvisor

kube-apiserver

kubectl get --raw /metrics

参考

kubelet常用命令

linux strace

Sat, 03 Oct 2020 22:09:30 +0800

[toc]

strace

参数

-tt 在每行输出的前面，显示毫秒级别的时间 -T 显示每次系统调用所花费的时间 -v 对于某些相关调用，把完整的环境变量，文件stat结构等打出来。 -f 跟踪目标进程，以及目标进程创建的所有子进程 -e 控制要跟踪的事件和跟踪行为,比如指定要跟踪的系统调用名称 -o 把strace的输出单独写到指定的文件 -s 当系统调用的某个参数是字符串时，最多输出指定长度的内容，默认是32个字节 -p 指定要跟踪的进程pid, 要同时跟踪多个pid, 重复多次-p选项即可。

# 常用方法

strace -tt -T -f -e trace=file -o /data/log/strace.log -s 1024 ./nginx

strace  -tt -p 21298

从性能分析大师 Brendan Gregg 的测试结果得知，被 strace 追踪的目标进程的运行速度会降低 100 倍以上，这对生产环境来说将是个灾难。

perf

# 调用 syscall 数量的 top 排行榜
perf top -F 49 -e raw_syscalls:sys_enter --sort comm,dso --show-nr-samples

# 显示超过一定延迟的系统调用信息
perf trace --duration 200

# 统计某个进程一段时间内系统调用的开销
perf trace -p $PID  -s

# 高延迟的调用栈信息
perf trace record --call-graph dwarf -p $PID -- sleep 10

Flame Graph

perf record -e cpu-clock -g -p 222
-g 选项是告诉perf record额外记录函数的调用关系
-e cpu-clock 指perf record监控的指标为cpu周期
-p 指定需要record的进程pid


# 使用火焰图展示结果 https://www.cnblogs.com/felixzh/p/8932984.html

1、Flame Graph项目位于GitHub上：https://github.com/brendangregg/FlameGraph
2、可以用git将其clone下来：git clone https://github.com/brendangregg/FlameGraph.git
 注意：git clone之后，下面用到的*.pl文件先给+x可执行权限，注意路径

我们以perf为例，看一下flamegraph的使用方法：
1、第一步
$perf record -e cpu-clock -g -p 28591
Ctrl+c结束执行后，在当前目录下会生成采样数据perf.data.
2、第二步
用perf script工具对perf.data进行解析
perf script -i perf.data &> perf.unfold
3、第三步
将perf.unfold中的符号进行折叠：
./stackcollapse-perf.pl perf.unfold &> perf.folded
注意：该命令可能有错误，错误提示在perf.folded
4、最后生成svg图：
./flamegraph.pl perf.folded > perf.svg

参考

强大的strace命令用法详解

线上环境 Linux 系统调用追踪

database troubleshooting

Sat, 05 Sep 2020 23:03:30 +0800

mysql

[TOC]

向mysql导入数据失败（MySQL server has gone away）

mysql日志中报错[Warning] Aborted connection 5 to db: 'iamstigris' user: 'root' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)

mysql>**show VARIABLES like '%max_allowed_packet%';**

显示：
+--------------------------+---------------------------------+
 | Variable_name            | Value                       |
 +--------------------------+---------------------------------+
 | max_allowed_packet       | 4194304            |
 | slave_max_allowed_packet | 1073741824 |
 +--------------------------+----------------------------------+
 2 rows in set

（1）显示：主最大允许包（max_allowed_packet）等于4M，从最大允许包（slave_max_allowed_packet）等于1G；

max_allowed_packet 针对的是一个事务中的一行记录大小，当一行记录超过了限制的大小，将会报错。sql文件中每次insert完进同一张表的所有数据被称为一个数据包（packet），max_allowed_packet就是来限制这个的大小的阈值，大于这个值，mysql的I/O连接会关闭，就会报这个错。

Got a packet bigger than ‘max_allowed_packet’ bytes

# 命令行设置
set global max_allowed_packet = 2*1024*1024*10 # 20M

ERROR 1839 (HY000) at line 24: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_MODE = ON.

恢复mysqldump备份的数据库报错,可以在sql文件中注释掉

也可以加-f

 nohup  /usr/local/mysql/bin/mysql -f -uroot -pssss-HGmHFDHyyM feather < /data1/feather_1008.sql > /dev/null 2>&1 &
[1] 107160

Error: Couldn’t read status information for table innodb_index_stats ()

mysqldump: Couldn’t execute ‘show create table innodb_index_stats’: Table ‘mysql.innodb_index_stats’ doesn’t exist (1146)

mysqldump发生报错

在日志中查看,mysql数据库下以下表都损坏

-rw-rw---- 1 mysql mysql  11K Jul 27  2017 innodb_index.stats.frm
-rw-rw---- 1 mysql mysql  11K Jul 27  2017 innodb_table_stats.frm
-rw-rw---- 1 mysql mysql  11K Jul 27  2017 slave_master_info.frm
-rw-rw---- 1 mysql mysql 9.2K Jul 27  2017 slave_relay_log_info.frm
-rw-rw---- 1 mysql mysql 9.1K Jul 27  2017 slave_worker_info.frm

在$MYSQL_HOME/share/mysql_system_tables.sql，search到建表语句

在datadir中删除损坏的文件（上面的那五个）,重启mysql，重建那写表。

建表语句可以在其它数据库中show create table tablename;

参考：InnoDB: Error: Table “mysql”.”innodb_table_stats” not found.

ERROR 1794 (HY000): Slave is not configured or failed to initialize properly. You must at least set –server-id to enable either a master or a slave. Additional error messages can be found in the MySQL error log.

测试发现以下几个系统表不存在.

innodb_table_stats
innodb_index_stats
slave_master_info
slave_relay_log_info
slave_worker_info

解决的方法.

1.删除上述系统表
drop table mysql.innodb_index_stats;
drop table mysql.innodb_table_stats;
drop table mysql.slave_master_info;
drop table mysql.slave_relay_log_info;
drop table mysql.slave_worker_info;



2.删除相关的.frm .ibd文件
rm -f innodb_index_stats*
rm -f innodb_table_stats*
rm -f slave_master_info*
rm -f slave_relay_log_info*
rm -f slave_worker_info*



3.重新创建上述系统表

CREATE TABLE `innodb_index_stats` (
  `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `index_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `stat_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `stat_value` bigint(20) unsigned NOT NULL,
  `sample_size` bigint(20) unsigned DEFAULT NULL,
  `stat_description` varchar(1024) COLLATE utf8_bin NOT NULL,
  PRIMARY KEY (`database_name`,`table_name`,`index_name`,`stat_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;

CREATE TABLE `innodb_table_stats` (
  `database_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `table_name` varchar(64) COLLATE utf8_bin NOT NULL,
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `n_rows` bigint(20) unsigned NOT NULL,
  `clustered_index_size` bigint(20) unsigned NOT NULL,
  `sum_of_other_index_sizes` bigint(20) unsigned NOT NULL,
  PRIMARY KEY (`database_name`,`table_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;

CREATE TABLE `slave_master_info` (
  `Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file.',
  `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log currently being read from the master.',
  `Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last read event.',
  `Host` char(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'The host name of the master.',
  `User_name` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The user name used to connect to the master.',
  `User_password` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The password used to connect to the master.',
  `Port` int(10) unsigned NOT NULL COMMENT 'The network port used to connect to the master.',
  `Connect_retry` int(10) unsigned NOT NULL COMMENT 'The period (in seconds) that the slave will wait before trying to reconnect to the master.',
  `Enabled_ssl` tinyint(1) NOT NULL COMMENT 'Indicates whether the server supports SSL connections.',
  `Ssl_ca` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Authority (CA) certificate.',
  `Ssl_capath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path to the Certificate Authority (CA) certificates.',
  `Ssl_cert` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL certificate file.',
  `Ssl_cipher` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the cipher in use for the SSL connection.',
  `Ssl_key` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL key file.',
  `Ssl_verify_server_cert` tinyint(1) NOT NULL COMMENT 'Whether to verify the server certificate.',
  `Heartbeat` float NOT NULL,
  `Bind` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'Displays which interface is employed when connecting to the MySQL server',
  `Ignored_server_ids` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The number of server IDs to be ignored, followed by the actual server IDs',
  `Uuid` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The master server uuid.',
  `Retry_count` bigint(20) unsigned NOT NULL COMMENT 'Number of reconnect attempts, to the master, before giving up.',
  `Ssl_crl` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Revocation List (CRL)',
  `Ssl_crlpath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path used for Certificate Revocation List (CRL) files',
  `Enabled_auto_position` tinyint(1) NOT NULL COMMENT 'Indicates whether GTIDs will be used to retrieve events from the master.',
  PRIMARY KEY (`Host`,`Port`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Master Information';

CREATE TABLE `slave_relay_log_info` (
  `Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file or rows in the table. Used to version table definitions.',
  `Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the current relay log file.',
  `Relay_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The relay log position of the last executed event.',
  `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log file from which the events in the relay log file were read.',
  `Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last executed event.',
  `Sql_delay` int(11) NOT NULL COMMENT 'The number of seconds that the slave must lag behind the master.',
  `Number_of_workers` int(10) unsigned NOT NULL,
  `Id` int(10) unsigned NOT NULL COMMENT 'Internal Id that uniquely identifies this record.',
  PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Relay Log Information';

CREATE TABLE `slave_worker_info` (
  `Id` int(10) unsigned NOT NULL,
  `Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `Relay_log_pos` bigint(20) unsigned NOT NULL,
  `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `Master_log_pos` bigint(20) unsigned NOT NULL,
  `Checkpoint_relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `Checkpoint_relay_log_pos` bigint(20) unsigned NOT NULL,
  `Checkpoint_master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `Checkpoint_master_log_pos` bigint(20) unsigned NOT NULL,
  `Checkpoint_seqno` int(10) unsigned NOT NULL,
  `Checkpoint_group_size` int(10) unsigned NOT NULL,
  `Checkpoint_group_bitmap` blob NOT NULL,
  PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Worker Information';

4.重启数据库

一次mysql 5.5升级到5.6导致的ERROR 1805

主从错误信息 Last_SQL_Errno: 1805

Worker 0 failed executing transaction '' at master log mysql-bin.000063, end_log_pos 1007288192; Error 'Column count of mysql.user is wrong. Expected 43, found 42. The table is probably corrupted' on query. Default database: ''. Query: 'grant replication slave on *.* to 'repl'@'10.80.139.193' identified by '5R23YNJxrQzetZLg''

前阵子将mysql数据库由5.5.14升级到5.6.36，升级后所有的业务数据都正常。运行了几天后，发现在主库上添加用户失败，错误提示为：ERROR 1805 (HY000): Column count of mysql.user is wrong，提示mysql.user表列的数目不对。还真是个坑。下面是其解决方案。

从上面的描述来看，说列的数目不对，因此我们升级后环境mysql.user表的建表语句
(robin@localhost)[(none)]>show variables like 'version';
+---------------+------------+
| Variable_name | Value |
+---------------+------------+
| version | 5.5.14-log |
+---------------+------------+
1 row in set (0.01 sec)

(robin@localhost)[(none)]>show create table mysql.user\G
*************************** 1. row ***************************
Table: user
Create Table: CREATE TABLE `user` (
`Host` char(60) COLLATE utf8_bin NOT NULL DEFAULT '',
............
`plugin` char(64) COLLATE utf8_bin DEFAULT '',
`authentication_string` text COLLATE utf8_bin,
PRIMARY KEY (`Host`,`User`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Users and global privileges'

安装一个新的mysql 5.6.36后查看mysql.user表的建表语句
mysql> show variables like 'version';
+---------------+--------+
| Variable_name | Value |
+---------------+--------+
| version | 5.6.36 |
+---------------+--------+
1 row in set (0.01 sec)

mysql> show create table mysql.user\G
*************************** 1. row ***************************
Table: user
Create Table: CREATE TABLE `user` (
...........
`plugin` char(64) COLLATE utf8_bin DEFAULT 'mysql_native_password',
`authentication_string` text COLLATE utf8_bin,
`password_expired` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
PRIMARY KEY (`Host`,`User`)   ---上行为多出的列，即先前使用主从复制时password_expired列没有
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Users and global privileges'

--为升级后的mysql 5.6.36手工添加这个password_expired列
mysql> alter table mysql.user add column `password_expired` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N';
Query OK, 10 rows affected (0.00 sec)
Records: 10 Duplicates: 0 Warnings: 0

--再次创建用户成功
mysql> grant all privileges on *.* to 'henry'@'%' identified by 'henrypwd';
Query OK, 0 rows affected (0.00 sec)
--------------------- 
原文：https://blog.csdn.net/leshami/article/details/78521217 

Relay log write failure: could not queue event from master

show slave status\G中查看 Exec_Master_Log_Pos的位置，去master上相应的binlog日志中查找下一个正确的位置，然后change master改变正确的位置。

stop slave;
reset slave;
change master to master_host='10.90.9.110',master_user='repl',master_password='5R23YNrQzetZLg',master_log_file='mysql-bin.039927',master_log_pos=0;
start slave;
show slave status\G

不能启动

删掉 mysql-bin.index

Cannot execute the current event group in the parallel mode.

Cannot execute the current event group in the parallel mode. Encountered event Update_rows, relay-log name /data/site/mysqldata/mysql-relay-bin.001689, position 354 which prevents execution of this event group in parallel mode. Reason: the event is a part of a group that is unsupported in the parallel execution mode.

stop slave;
SET GLOBAL slave_parallel_workers=0;
start slave;

正常后
SET GLOBAL slave_parallel_workers=4;

MySQL error: The maximum column size is 767 bytes

set global innodb_file_format = BARRACUDA;
set global innodb_large_prefix = ON;
create table test (........) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;

MySQL数据库运维之主从复制延迟问题排查

MySQL innodb_table_stats表不存在的解决方法

redis

[TOC]

redis-cluster主从切换后，造成主从不同步

从库报错日志

S 22 Feb 13:26:39.269 # Connection with master lost.
S 22 Feb 13:26:39.269 * Caching the disconnected master state.
S 22 Feb 13:26:39.453 * Connecting to MASTER 10.80.97.147:6380
S 22 Feb 13:26:39.453 * MASTER <-> SLAVE sync started
S 22 Feb 13:26:39.453 * Non blocking connect for SYNC fired the event.
S 22 Feb 13:26:39.454 * Master replied to PING, replication can continue...
S 22 Feb 13:26:39.454 * Trying a partial resynchronization (request 4fcc29abe35fa7458ae69ee4a89a49c408f2f33b:2511885719192).
S 22 Feb 13:29:20.358 * Full resync from master: 4fcc29abe35fa7458ae69ee4a89a49c408f2f33b:2512010876867
S 22 Feb 13:29:20.358 * Discarding previously cached master state.
S 22 Feb 13:33:10.840 * MASTER <-> SLAVE sync: receiving 8161106002 bytes from master
S 22 Feb 13:34:05.563 * MASTER <-> SLAVE sync: Flushing old data
S 22 Feb 13:36:39.114 * MASTER <-> SLAVE sync: Loading DB in memory
S 22 Feb 13:39:45.539 * MASTER <-> SLAVE sync: Finished with success
S 22 Feb 13:39:45.664 # Connection with master lost.

看到有很多的Connection with master lost的错误，然后slave尝试要求master进行部分同步：Trying a partial resynchronization (request 4fcc29abe35fa7458ae69ee4a89a49c408f2f33b:2511885719192)。但是master还是要求全量同步：Full resync from master: 4fcc29abe35fa7458ae69ee4a89a49c408f2f33b:2512010876867。然后就是全量同步的过程。上面的过程每几分钟就发生一次，一直重复着。

主库报错日志

M 22 Feb 13:26:39.454 * Slave 10.80.91.147:6380 asks for synchronization
M 22 Feb 13:26:39.454 * Unable to partial resync with slave 10.80.91.147:6380 for lack of backlog (Slave request was: 2511885719192).
M 22 Feb 13:26:39.454 * Can't attach the slave to the current BGSAVE. Waiting for next BGSAVE for SYNC
C 22 Feb 13:29:14.960 * DB saved on disk
C 22 Feb 13:29:15.940 * RDB: 9362 MB of memory used by copy-on-write
M 22 Feb 13:29:18.859 * Background saving terminated with success
M 22 Feb 13:29:18.859 * Starting BGSAVE for SYNC with target: disk
M 22 Feb 13:29:20.358 * Background saving started by pid 13956
C 22 Feb 13:33:07.109 * DB saved on disk
C 22 Feb 13:33:08.087 * RDB: 9396 MB of memory used by copy-on-write
M 22 Feb 13:33:10.839 * Background saving terminated with success
M 22 Feb 13:34:04.057 * Synchronization with slave 10.80.91.147:6380 succeeded
M 22 Feb 13:34:11.121 * 10000 changes in 60 seconds. Saving...
M 22 Feb 13:34:12.635 * Background saving started by pid 14533
M 22 Feb 13:34:23.959 * FAIL message received from 7dc97cd72cf74483287692ab111c887a7c8a94e7 about e6cd89fdcb009cc10dfc8a08d4ecfcaa3ec8086a
M 22 Feb 13:36:39.214 * Clear FAIL state for node e6cd89fdcb009cc10dfc8a08d4ecfcaa3ec8086a: slave is reachable again.
M 22 Feb 13:37:38.022 # Client id=219273 addr=10.80.91.147:56276 fd=54 name= age=659 idle=2 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=4742 o
mem=77774182 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.
M 22 Feb 13:37:38.027 # Connection with slave 10.80.91.147:6380 lost.

master日志中发现：scheduled to be closed ASAP for overcoming of output buffer limits.

这是由于client-output-buffer-limit限制太小造成的。

redis的默认配置为：

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

问题处理

增大client-output-buffer-limit限制

修改配置为`config set client-output-buffer-limit "slave 536870912 536870912 60"`(512mb 512mb 60)从库还是有问题 。

拷贝主库rbd文件到从库

主库执行bgsave，产生新的rbd文件，传到从库，重新启动从库。从库初始化后，主从同步状态正常。

正常后取消了client-output-buffer-limit的限制：

config set client-output-buffer-limit "slave 0 0 0"
config set client-output-buffer-limit "pubsub 0 0 0"

从master日志我们可以知道为什么主库不能对从库进行部分同步：`Unable to partial resync with slave 10.80.91.147:6380 for lack of backlog (Slave request was: 2511885719192)`。这个涉及到Redis的部分同步的实现。在2.8版本，redis使用了新的复制方式，引入了replication backlog以支持部分同步。

The master then starts background saving, and starts to buffer all new commands received that will modify the dataset. When the background saving is complete, the master transfers the database file to the slave, which saves it on disk, and then loads it into memory. The master will then send to the slave all buffered commands. This is done as a stream of commands and is in the same format of the Redis protocol itself.

当主服务器进行命令传播的时候，maser不仅将所有的数据更新命令发送到所有slave的replication buffer，还会写入replication backlog。当断开的slave重新连接上master的时候，slave将会发送psync命令（包含复制的偏移量offset），请求partial resync。如果请求的offset不存在，那么执行全量的sync操作，相当于重新建立主从复制。

replication backlog是一个环形缓冲区，整个master进程中只会存在一个，所有的slave公用。backlog的大小通过repl-backlog-size参数设置，默认大小是1M，其大小可以根据每秒产生的命令、（master执行rdb bgsave） +（master发送rdb到slave） + （slave load rdb文件）时间之和来估算积压缓冲区的大小，repl-backlog-size值不小于这两者的乘积。

所以在主从同步的时候，slave会落后master的时间＝（master执行rdb bgsave）+ (master发送rdb到slave) + (slave load rdb文件) 的时间之和。然后如果在这个时间master的数据变更非常巨大，超过了replication backlog，那么老的数据变更命令就会被丢弃，导致需要全量同步。

client-output-buffer-limit

处理办法

命令行

config set client-output-buffer-limit "slave 0 0 0"
config set client-output-buffer-limit "pubsub 0 0 0"

配置文件

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 0 0 0
client-output-buffer-limit pubsub 0 0 0

参考

https://blog.csdn.net/rongge2008/article/details/53503526

kubernetes troubleshooting

Fri, 04 Sep 2020 11:32:30 +0800

[toc]

node相关

node维护

由于硬件问题或其他原因，node节点需要停机维护。

1.禁止新调度 到node上
 kubectl cordon k8snode
 
2.驱逐这个node的容器 过程比较慢
kubectl drain k8snode --ignore-daemonsets

3.解锁禁止调度。恢复业务
kubectl uncordon k8snode

内核版本过低引起的bug

linux内核版本太低(centos7.6.1810 内核版本：3.10.0-957.el7.x86_64)，容器多次重启可能会导致整个OS Hang住无法执行任何命令,node 节点notready。 建议升级内核版本到4.19+。

详细问题：

3.10 内核 kmem account 内存泄漏 bug导致节点 NotReady（SLUB: Unable to allocate memory on node -1）
3.10 内核 Bug, 系统日志出现大量 kernel: unregister_netdevice: waiting for eth0 to become free 报错，节点 NotReady
kernel: NMI watchdog: BUG soft lockup -CPU#9 stuck for 22s! [migration/9:54]”

社区相关 Issue:

https://github.com/kubernetes/kubernetes/issues/61937
https://github.com/opencontainers/runc/issues/1725
https://support.mesosphere.com/s/article/Critical-Issue-KMEM-MSPH-2018-0006
https://github.com/opsnull/follow-me-install-kubernetes-cluster/issues/430

诊断修复 TiDB Operator 在 K8s 测试中遇到的 Linux 内核问题

K8S 问题排查： cgroup 内存泄露问题

node 资源不够,导致pod被驱逐

遇到的问题：node 磁盘空间不足，导致pod被驱逐到其他node上，造成其他node也磁盘不足，会有连锁反应。

解决：紧急升级node磁盘，当时买的云主机系统盘比较小。

node资源方面相关延伸

# kubelet配置

1：设置预留系统服务的资源 
--system-reserved=cpu=200m,memory=1G

2：设置预留给k8s组件的资源（主要组件）
--kube-reserved=cpu=200m,memory=1G
系统内存--system-reserved-kube-reserved 就是可以分配给pod的内存

3：驱逐条件#驱逐条件只对内存硬盘有效，对CPU无效
--eviction-hard=nodefs.available<10%,nodefs.inodesFree<5%,memory.available<100Mi

4：最小驱逐
--eviction-minimum-reclaim="memory.available=0Mi,nodefs.available=500Mi,imagefs.available=2Gi"   #暂时不用

5：节点状态更新时间
--node-status-update-frequency =10s    #(默认10S，不要轻易修改。很容器node再read和notread问题) 

6：驱逐等待时间
--eviction-pressure-transition-period=60s

kubeernetes节点资源限制

从一次集群雪崩看Kubelet资源预留的正确姿势

pod相关

容器时区问题

确保容器时区正确，可挂载宿主机localtime。

pod.spec.containers下：

    volumeMounts:
    - mountPath: /etc/localtime
      name: localtime
      readOnly: true

pod.spec.volumes下：

  volumes:
  - hostPath:
      path: /etc/localtime
      type: ""
    name: localtime

java可能还需在启动时指定时区

java -jar -Duser.timezone=GMT+08

QoS问题

Quality of Service

内存不足，oom calico优先被干掉，calico等系统组件没有设置request、limit。是BestEffort最低优先级，第一个被kill。

QoS：

Guaranteed

Every Container in the Pod must have a memory limit and a memory request, and they must be the same.
Every Container in the Pod must have a CPU limit and a CPU request, and they must be the same.

Burstable

The Pod does not meet the criteria for QoS class Guaranteed.
At least one Container in the Pod has a memory or CPU request.

Best-Effort

For a Pod to be given a QoS class of BestEffort, the Containers in the Pod must not have any memory or CPU limits or requests.

低版本java不能自动识别容器限制

低版本java自动识别到容器限制，获取正确的内存和CPU信息.

需要添加额外参数控制内存 java -Xms1G -Xmx1G -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

8u191版本已经修复这个问题。https://www.oracle.com/java/technologies/javase/8u191-relnotes.html

容器中的JVM资源该如何被安全的限制？

容器优雅关闭

容器中的 1 号进程是非常重要的，如果它不能正确的处理相关的信号，那么应用程序退出的方式几乎总是被强制杀死而不是优雅的退出。

K8S滚动更新时，会在启动新pod时慢慢终止旧pod，Kubernetes会向旧pod中的容器发送SIGTERM信号,并等待应用优雅的结束（k8s默认等待30s)。注意，只有容器中的 1 号进程能够收到信号，这一点非常关键! 究竟谁是 1 号进程则主要由 EntryPoint, CMD等指令的写法决定。

详细信息：Dockerfile中CMD和ENTRYPOINT

这里推荐使用exec模式的写法,这样1号进程就是你的程序，滚动升级时,它会接收到SIGTERM信号：才会优雅的结束进程。

不要使用CMD command param1 param2 这种形式，这种形式是shell模式，1号进程会是sh。

# 错误案例
CMD /start.sh  #这种形式是shell模式，1号进程会是sh   滚动升级是杀掉的是sh，应用进程是sh下属进程，所以应用进程还会卡在容器里不会关闭,无法优雅的stop进程

# 正确用法
CMD ["executable","param1","param2"]
ENTRYPOINT ["executable", "param1", "param2"]

优雅关闭如果还会出现5xx问题，可以在k8s层面增加lifecycle prostop

If one of the Pod’s containers has defined a preStop hook, the kubelet runs that hook inside of the container. If the preStop hook is still running after the grace period expires, the kubelet requests a small, one-off grace period extension of 2 seconds.

        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

也可用 supervisord管理容器去拦截sigteam信号。

#!/bin/sh
pidfile=/var/tmp/supervisord.pid
signal_term(){
    echo "docker start exit...\n"
    pid=`cat $pidfile`
    sleep 5;
    kill -s 15 ${pid}
    while true
    do
        if [ ! -f $pidfile ];then
            echo "docker end exit...\n"
            exit;
        fi
        sleep 1;
    done;
}
trap signal_term 15;
# start up
supervisord -c /etc/supervisord.conf &
wait $!

traefik节点与业务节点隔离

保障traefik性能最好和其他业务节点隔离开。traefik有问题大部分业务都会有影响。

可以将traefik节点禁止调度，kubectl cordon xxx

other

跨版本升级造成api不兼容

k8s跨版本版本升级时需要注意api的版本，一定要仔细阅读CHANGELOG.

https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG

如1.16的改动：

All resources under apps/v1beta1 and apps/v1beta2 - use apps/v1 instead daemonsets, deployments, replicasets resources under extensions/v1beta1 - use apps/v1 instead

hadoop troubleshooting

Thu, 03 Sep 2020 09:32:30 +0800

[toc]

hive

HIVEMETASTORE_PAUSE_DURATION

# 报错信息
The health test result for HIVEMETASTORE_PAUSE_DURATION has become bad: Average time spent paused was 37.1 second(s) (61.91%) per minute over the previous 5 minute(s). Critical threshold: 60.00%.

最近几天cdh集群hive总是在凌晨出现HIVEMETASTORE_PAUSE_DURATION错误。

问题排查

排查hive所在机器，发现负载特别高，进一步排查有个spark程序(程序有问题)，跑在hive所在机器负载特别高。经沟通次spark程序已经没用了忘记下线了。就kill掉了spark程序。
查看hive日志发现有canary_test_db_hive_HIVEMETASTORE相关报错，cdh日志中也发现The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a database.
进一步查看CDH监控，发现hive metastore 的JVM Heap Memory Usage基本跑满了，发现此参数配置的居然是1.3G，当有大量任务是这点内存肯定不够通，监控图上发现内存趋于跑满，很可能是内存不够造成的异常。

解决

调整Java Heap Size of Hive Metastore Server in Bytes 到8G
去除hive节点datanode和yarn角色。

datanode

Caught unexpected exception in main loop.

ValueError: too many values to unpack

集群扩容启动cloudera-scm-agent报错。

# 报错信息
[07/Sep/2020 11:50:39 +0000] 179811 MainThread agent        ERROR    Caught unexpected exception in main loop.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.1-py2.7.egg/cmf/agent.py", line 758, in start
    self._init_after_first_heartbeat_response(resp_data)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.1-py2.7.egg/cmf/agent.py", line 938, in _init_after_first_heartbeat_response
    self.client_configs.load()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.1-py2.7.egg/cmf/client_configs.py", line 682, in load
    new_deployed.update(self._lookup_alternatives(fname))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.1-py2.7.egg/cmf/client_configs.py", line 432, in _lookup_alternatives
    return self._parse_alternatives(alt_name, out)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.1-py2.7.egg/cmf/client_configs.py", line 444, in _parse_alternatives
    path, _, _, priority_str = line.rstrip().split(" ")
ValueError: too many values to unpack

网上查看资料都是让修改client_configs.py文件，之前也扩容过集群启动就没问题，感觉根本原因不是这个文件的问题。

查看java版本，发现是openjdk。 openjdk对cdh的支持不是很好，remove 掉openjdk,安装Java jdk启动正常。

# java -version  
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

#  java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

kakfa常用命令

Tue, 28 Jul 2020 09:32:30 +0800

[toc]

重启

/usr/local/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
sleep 3
/usr/local/kafka_2.11-2.0.0/bin/kafka-server-start.sh  -daemon /usr/local/kafka_2.11-2.0.0/config/server.properties

列出/删除topic

#列出topic
./bin/kafka-topics.sh --list --zookeeper localhost:2181

#列出topic详情
./bin/kafka-topics.sh --zookeeper lcoalhost:2181 --topic ecar-baojia --describe

#创建topic
./bin/kafka-topics.sh --create --topic headlineRecomUpdateSync4 --replication-factor 3 --partitions 5 --zookeeper  192.168.1.1:2181

#删除topic
./bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test

#连接到zookeeper彻底删除topic
./bin/zookeeper-shell.sh 192.168.1.1:2181
找到topic目录：ls /brokers/topics   删掉对应的topic :     rmr  /brokers/topic/topic-name
找到目录:  ls /config/topics        删除对应的topic:     rmr  /config/topics/topic-name

####从头开始消费数据

# old version
./bin/kafka-console-consumer.sh --zookeeper 192.168.1.1:2181 --topic ecar-baojia --from-beginning

# new version
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.1:9092 --topic testtest --from-beginning

查看/删除消费组

#查看消费组
./bin/kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:9092 --list

#删除
./bin/kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:9092 --group logstash-b --delete

####查看消费组的积压情况

# ./bin/kafka-consumer-groups.sh --bootstrap-server 192.168.1.1:9092 --describe --group logstash-a

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
test-a         8          503108030       505454999       2346969         logstash-0-946898a5-7b1f-4930-9ec3-0d5d977d31b8 /192.168.1.1   logstash-0
test-a         2          503256691       505374825       2118134         logstash-0-946898a5-7b1f-4930-9ec3-0d5d977d31b8 /192.168.1.1   logstash-0
test-a         3          503152179       505364622       2212443         logstash-0-946898a5-7b1f-4930-9ec3-0d5d977d31b8 /192.168.1.1   logstash-0

展示的几个参数: TOPIC: 该消费者组消费的是哪些topic PARTITION: 表示该消费者消费的是哪些分区, 本例创建topic时只有一个分区, 所以输出只有一行, 分区号为0 CURRENT-OFFSET: 表示消费者组最新消费的位移值, 此值在消费过程中是变化的 LOG-END-OFFSET: 表示topic所有分区当前的日志终端位移值, 因为我们生产了1000万数据, 所以此处是1000万 LAG: 表示滞后进度, 此值为LOG-END-OFFSET 与 CURRENT-OFFSET的差值, 代表的是滞后情况, 此值越大表示滞后严重, 本例最终LAG为0 说明没有消费滞后.

检查消费者位置offset（0.9.0.0之前）

/usr/local/kafka_2.12-2.1.0/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker  --topic ugc_topic_test --zookeeper 192.168.1.1:2181

清除offset

登陆kafka集群所使用的zookeeper集群中任一台服务器（以10.90.11.32为例）：

./ zkCli.sh -server 10.90.11.32:12181

[zk: 10.90.11.19:12181(CONNECTED) 1] ls /consumers/${consumer_name}/offsets/${topic_name}

[zk: 10.90.11.19:12181(CONNECTED) 2] rmr /consumers/${consumer_name}/offsets/${topic_name}

注：清除的是${consumer_name}消费者分组对于${topic_name}该topic的offset，此分组的消费者重启后默认会从当前开始消费。

设置消费者offset

# 设置为最初偏移量
./kafka-consumer-groups.sh --bootstrap-server snn:6667 --group offsettest --topic offset-test --reset-offsets --to-earliest –execute

#设置任意偏移量
./kafka-consumer-groups.sh --bootstrap-server snn:6667 --group offsettest --topic offset-test --reset-offsets --to-offset 3 –execute

#设置最近偏移量
./kafka-consumer-groups.sh --bootstrap-server snn:6667 --group offsettest --topic offset-test --reset-offsets --to-latest --execute

修改topic持久化时间

./bin/kafka-topics.sh --zookeeper 192.168.1.1:2181 -topic test --alter --config retention.ms=259200000
# 259200000 3天
# 345600000 4天

增加partition分区数

./kafka-topics.sh --alter --topic recHeadLine --zookeeper  192.168.1.1:2181 --partitions 80

auto.offset.reset

earliest
当各分区下有已提交的offset时，从提交的offset开始消费；无提交的offset时，从头开始消费
latest
当各分区下有已提交的offset时，从提交的offset开始消费；无提交的offset时，消费新产生的该分区下的数据
none
topic各分区都存在已提交的offset时，从offset后开始消费；只要有一个分区不存在已提交的offset，则抛出异常

Kafka auto.offset.reset值详解

参考

prometheus结合consul

Tue, 19 May 2020 09:32:30 +0800

promethues结合consul可以实现自动发现,可以把一些需要收集的信息(metrics)上报到consul,prometheus会自动去consul中拉取。

启动一个consul

# 这里只作测试,线上确保consul高可用
consul agent -server -dev -ui -data-dir /data/consul -client=192.168.1.1 -bind=192.168.1.1 -config-dir /etc/consul.d -enable-script-checks=true

prometheus配置

- job_name: consul_collect
  metrics_path: /metrics
  scheme: http
  consul_sd_configs:
    - server: 192.168.1.1:8500
      services:
        - consul_test

consul创建service

# 写入配置文件
echo '{"service": {"name": "consul_test", "tags": ["exporter"], "port": 9100}}' \
    | sudo tee /etc/consul.d/consul_test.json
# 通过接口
curl -X PUT -d '{"id": "node-192.168.1.2","name": "consul_test","address": "192.168.1.2","port": 9100,"tags": ["exporter"],"checks": [{"http": "http://192.168.1.2:9100/metrics","interval": "35s"}]}' http://192.168.1.1:8500/v1/agent/service/register

hadoop常用命令

Mon, 30 Mar 2020 09:32:30 +0800

hdfs

sudo -u hdfs hdfs dfsadmin -report   #查看hdfs使用率
hdfs dfs -copyFromLocal /data/app_tmp/hippoZip/2018-05-29/ /hippo/hippo/hippoZip/ #上传
sudo -u hdfs hdfs dfs -mkdir /lisai  #创建目录

#检测损坏的block，然后删掉
hdfs fsck -list-corruptfileblocks
sudo -u hdfs hdfs dfs -rm  /tmp/hadoop-yarn/staging/history/done_intermediate/root/job_1486604935762_0002-1486605171955-root-Chukwa%2DDemux_20170209_09_52-1486605234028-0-0-FAILED-root.root-1486605178417.jhist
或者
hdfs fsck /xxx/xxx/xxx/xxx -delete

#根目录下各个目录的大小
hdfs dfs -du -h /

# 平衡限速
sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 5242880  # 5MB/s
#count
hdfs dfs -count < hdfs path >   或  hadoop fs -count < hdfs path >
统计hdfs对应路径下的目录个数，文件个数，文件总计大小
显示为目录个数，文件个数，文件总计大小，输入路径

hadoop

hadoop fs -copyFromLocal 1.txt /user/lisai/
hadoop fs -ls /
hadoop fs -put /data/null_ad_imp/* /ids/no_ad_impl/
hadoop fs -get [hadoop源文件路径路径] [linux下载目的地路径]
hadoop fs -du -h /
hadoop classpath

集群间迁移数据

hadoop distcp hdfs://10.90.4.199:8020/newread/logbak/android/20170101 hdfs://10.21.8.65:8020/newread
hadoop distcp hdfs://10.90.4.199:8020/lijun hdfs://10.21.8.65:8020/lijun

设置副本数

setrep

Usage: hadoop fs -setrep [-R] [-w] <numReplicas> <path>

Changes the replication factor of a file. If path is a directory then the command recursively
changes the replication factor of all files under the directory tree rooted at path.

Options:

The -w flag requests that the command wait for the replication to complete. This can potentially take a very long time.
The -R flag is accepted for backwards compatibility. It has no effect.

Example:

    hadoop fs -setrep -w 3 /user/hadoop/dir1

Exit Code:

Returns 0 on success and -1 on error.
---------------------
原文：https://blog.csdn.net/chengyuqiang/article/details/79139432