K8s集群重启与恢复-Node节点启停方式

来自:网络
时间:2024-09-10
阅读:

1 应用场景

场景

  • 在实际工作中,可能某个 Worker 节点需要维护
  • 迁移,我们需要平滑的停止、启动该节点
  • 应尽量减少启停中对集群、业务造成的影响

注意

  • 剔除 Worker 节点操作中
  • 该 Worker 节点上的工作负载(Pod)将被驱逐到其他节点上
  • 请确保集群资源充足

2 操作步骤

2.1 停止 Worker 节点调度

# 查看信息
root@sh-gpu091:~# kubectl get node
NAME                 STATUS     ROLES   AGE    VERSION
172.19.13.31         Ready      node    403d   v1.14.1
sh-cpu032.aibee.cn   Ready      node    403d   v1.14.1
sh-cpu033.aibee.cn   Ready      node    403d   v1.14.1
sh-gpu091.aibee.cn   Ready      node    403d   v1.14.1
sh-gpu101.aibee.cn   Ready      node    403d   v1.14.1
# 停止 Worker 节点调度
root@sh-gpu091:~# kubectl cordon sh-gpu101.aibee.cn
node/sh-gpu101.aibee.cn cordoned
  • 查看节点状态
root@sh-gpu091:~# kubectl get node
NAME                 STATUS                     ROLES   AGE    VERSION
172.19.13.31         Ready                      node    403d   v1.14.1
sh-cpu032.aibee.cn   Ready                      node    403d   v1.14.1
sh-cpu033.aibee.cn   Ready                      node    403d   v1.14.1
sh-gpu091.aibee.cn   Ready                      node    403d   v1.14.1
sh-gpu101.aibee.cn   Ready,SchedulingDisabled   node    403d   v1.14.1

2.2 驱逐 Worker 节点上的工作负载

# --ignore-daemonsets 驱逐 pod 时忽略 daemonset
# --delete-local-data 驱逐 pod 时删除 pod 的临时数据,该参数不会删除持久化数据
root@sh-gpu091:~# kubectl drain sh-gpu101.aibee.cn --delete-local-data --ignore-daemonsets --force
node/sh-gpu101.aibee.cn already cordoned
WARNING: ignoring DaemonSet-managed Pods: cattle-system/cattle-node-agent-8wcvs, kube-system/kube-flannel-ds-kqzhc, kube-system/nvidia-device-plugin-daemonset-rr2lf, monitoring/prometheus-node-exporter-xtbxp
evicting pod "model-server-0"
evicting pod "singleview-proxy-client-pbdownloader-0"
evicting pod "singleview-proxy-service-0"
pod/singleview-proxy-client-pbdownloader-0 evicted
pod/singleview-proxy-service-0 evicted
pod/model-server-0 evicted
node/sh-gpu101.aibee.cn evicted

2.3 停止 Docker、Kubelet 等服务

systemctl stop kubelet 
systemctl stop docker
  • 查看节点上是否还有业务 pod
kubectl get pod -A -o wide |grep sh-gpu101.aibee.cn
  • 如果不需要恢复了,可以删除节点并确认节点信息
root@sh-gpu091:~# kubectl delete node sh-gpu101.aibee.cn
node "sh-gpu101.aibee.cn" deleted
root@sh-gpu091:~# kubectl get node
NAME                 STATUS     ROLES   AGE    VERSION
172.19.13.31         Ready      node    403d   v1.14.1
sh-cpu032.aibee.cn   Ready      node    403d   v1.14.1
sh-cpu033.aibee.cn   Ready      node    403d   v1.14.1
sh-gpu091.aibee.cn   Ready      node    403d   v1.14.1
root@sh-gpu091:~#

2.4 恢复 Worker 节点

systemctl start docker
systemctl status docker
systemctl start kubelet
systemctl status kubelet

2.5 允许 Worker 节点调度

# 取消不可调度
kubectl uncordon sh-gpu101.aibee.cn

总结

以上为个人经验,希望能给大家一个参考,也希望大家多多支持。

返回顶部
顶部