使用 kubeKey 快速安装 kubernetes 高可用集群
一、为何选择 KubeKey
以前基于 ansible 的安装程序依赖于许多软件，例如 Python。KubeKey 由 Go 语言开发，可以消除在多种环境中出现的问题，确保成功安装。
KubeKey 支持多种安装选项，例如 All-in-One、多节点安装以及离线安装。
KubeKey 使用 Kubeadm 在节点上尽可能多地并行安装 Kubernetes 集群，使安装更简便，提高效率。与旧版的安装程序相比，它极大地节省了安装时间。
KubeKey 提供内置高可用模式，支持一键安装高可用 Kubernetes 集群。
二、使用 KubeKey 快速安装高可用集群
2.1 部署拓扑图
本示例包括以下七台主机，其中一台机器为堡垒机，三个主节点和三个工作节点。
关闭交换分区
[root@tke-master1 ~]# swapoff -a
# 永久关闭交换分区
[root@tke-master1 ~]# sed -i '/ swap / s/^/#/' /etc/fstab关闭防火墙
[root@tke-master1 ~]# setenforce 0
# 永久关闭selinux
[root@tke-master1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@k8s-master ~]# systemctl stop firewalld.service
[root@k8s-master ~]# systemctl disable firewalld.service配置时间同步
# 设置时区
[root@k8s-master01 ~]# timedatectl set-timezone Asia/Shanghai
# 查看时间同步源，时间同步进度
[root@k8s-master01 ~]# chronyc -n sources -v
210 Number of sources = 4
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- 94.130.49.186 3 10 377 10 +4917us[+4917us] +/- 111ms
^- 178.215.228.24 2 10 377 799 -14ms[ -14ms] +/- 134ms
^* 202.118.1.81 1 10 377 664 +1283us[+1619us] +/- 30ms
^- 162.159.200.1 3 10 377 25 +11ms[ +11ms] +/- 112ms
# 查看时间是否同步
[root@k8s-master01 ~]# chronyc tracking
Reference ID : CA760151 (time.neu.edu.cn)
Stratum : 2
Ref time (UTC) : Mon Apr 11 12:01:54 2022
System time : 0.000179534 seconds fast of NTP time
Last offset : +0.000336160 seconds
RMS offset : 0.000334456 seconds
Frequency : 26.376 ppm slow
Residual freq : +0.010 ppm
Skew : 0.260 ppm
Root delay : 0.058298927 seconds
Root dispersion : 0.001503241 seconds
Update interval : 1031.4 seconds
Leap status : Normal
# 查看时间同步状态
[root@k8s-master01 ~]# timedatectl status
Local time: 一 2022-04-11 20:14:39 CST
Universal time: 一 2022-04-11 12:14:39 UTC
RTC time: 一 2022-04-11 12:14:38
Time zone: Asia/Shanghai (CST, +0800)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/aKubeKey 可以一同安装 Kubernetes 和 KubeSphere。根据要安装的 Kubernetes 版本，需要安装的依赖项可能会不同。
[root@k8s-master01 ~]# yum -y install conntrack-tools socat
已加载插件：fastestmirror
Determining fastest mirrors
* base: mirrors.nju.edu.cn
* extras: mirrors.dgut.edu.cn
* updates: mirrors.dgut.edu.cn
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
updates/7/x86_64/primary_db | 15 MB 00:00:13
正在解决依赖关系
--> 正在检查事务
---> 软件包 conntrack-tools.x86_64.0.1.4.4-7.el7 将被 安装
--> 正在处理依赖关系 libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit)，它被软件包 conntrack-tools-1.4.4-7.el7.x86_64 需要
--> 正在处理依赖关系 libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit)，它被软件包 conntrack-tools-1.4.4-7.el7.x86_64 需要
--> 正在处理依赖关系 libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit)，它被软件包 conntrack-tools-1.4.4-7.el7.x86_64 需要
--> 正在处理依赖关系 libnetfilter_queue.so.1()(64bit)，它被软件包 conntrack-tools-1.4.4-7.el7.x86_64 需要( 3 ) 配置系统密钥登录
创建密钥
[root@jumpserver ~]# ssh-keygen -b 4096 -t rsa -C k8s-kubekey
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:4wEWBGtUqeyOo1dU5osNANFf8WDb4Ww7PNIMAWiu+W4 k8s-kubekey
The key's randomart image is:
+---[RSA 4096]----+
|o+ .+=Oo. |
| =. o=X . |
| o +o*= * |
| ..*..B . |
| o o +..S |
|o + oo = |
| . + . |
| E . |
|.=.. |
+----[SHA256]-----+使用 ssh-copy-id 指令拷贝公钥到远程服务器
[root@jumpserver ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@k8s-master01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'k8s-master01 (192.168.14.70)' can't be established.
ECDSA key fingerprint is SHA256:ND6FQf8nVhwr0jjaufcZpLAmJgcMWoEe5cqca4MeV3E.
ECDSA key fingerprint is MD5:7c:b2:e0:61:0e:7d:5c:92:ed:b2:d9:6a:c2:76:a7:0e.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@k8s-master01's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@k8s-master01'"
and check to make sure that only the key(s) you wanted were added.2.3 安装 kubernetes 高可用集群
( 1 ) 下载 KubeKey
[root@k8s-master01 ~]# wget -c https://kubesphere.io/download/kubekey-v1.0.0-linux-amd64.tar.gz -O -|tar -xz( 2 ) 创建集群配置文件
# 通过 ./kk create config -h 获取帮助信息
[root@jumpserver ~]# ./kk create config -hCreate cluster configuration file
Usage:
kk create config [flags]
Flags:
-f, --filename string Specify a configuration file path
--from-cluster Create a configuration based on existing cluster
-h, --help help for config
--kubeconfig string Specify a kubeconfig file
--name string Specify a name of cluster object (default "sample")
--with-kubernetes string Specify a supported version of kubernetes
--with-kubesphere Deploy a specific version of kubesphere (default v3.1.0)
Global Flags:
--debug Print detailed information (default true)
--in-cluster Running inside the cluster
#创建集群配置文件
# 指定要安装的 Kubesphere 版本(--with-kubesphere v3.2.1),指定要安装的 Kubernetes 版本(--with-kubernetes v1.22.5)
[root@jumpserver ~]# ./kk create config --with-kubernetes v1.22.1 --with-kubesphere v3.2.1
您也可以运行 ./kk version --show-supported-k8s，查看能使用 KubeKey 安装的所有受支持的 Kubernetes 版本。
能使用 KubeKey 安装的 Kubernetes 版本与 KubeSphere v3.0.0 支持的 Kubernetes 版本不同。如需在现有 Kubernetes 集群上安装 KubeSphere 3.2.1，您的 Kubernetes 版本必须为 v1.17.x，v1.18.x，v1.19.x 或 v1.20.x。
( 3 ) 修改集群配置文件
[root@jumpserver ~]# cat config-sample.yaml
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: k8s-master01, address: 192.168.14.70, internalAddress: 192.168.14.70, privateKeyPath: "~/.ssh/id_rsa"}
- {name: k8s-master02, address: 192.168.14.57, internalAddress: 192.168.14.57, privateKeyPath: "~/.ssh/id_rsa"}
- {name: k8s-master03, address: 192.168.14.175, internalAddress: 192.168.14.175, privateKeyPath: "~/.ssh/id_rsa"}
- {name: k8s-node01, address: 192.168.14.99, internalAddress: 192.168.14.99, privateKeyPath: "~/.ssh/id_rsa"}
- {name: k8s-node02, address: 192.168.14.113, internalAddress: 192.168.14.113, privateKeyPath: "~/.ssh/id_rsa"}
- {name: k8s-node03, address: 192.168.14.12, internalAddress: 192.168.14.12, privateKeyPath: "~/.ssh/id_rsa"}
roleGroups:
etcd:
- k8s-master01
- k8s-master02
- k8s-master03
master:
- k8s-master01
- k8s-master02
- k8s-master03
worker:
- k8s-node01
- k8s-node02
- k8s-node03
controlPlaneEndpoint:
##Internal loadbalancer for apiservers
#internalLoadbalancer: haproxy
domain: k8s-api.tongtool.com
address: "192.168.14.70"
port: 6443
kubernetes:
version: v1.21.5
clusterName: company-pvt-cluster
maxPods: 120
nodeCidrMaskSize: 24
proxyMode: ipvs
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
registry:
registryMirrors: []
insecureRegistries: []
addons: []
---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.2.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
local_registry: ""
# dev_tag: ""
etcd:
monitoring: false
endpointIps: localhost
port: 2379
tlsEnable: true
common:
core:
console:
enableMultiLogin: true
port: 30880
type: NodePort
# apiserver:
# resources: {}
# controllerManager:
# resources: {}
redis:
enabled: false
volumeSize: 2Gi
openldap:
enabled: false
volumeSize: 2Gi
minio:
volumeSize: 20Gi
monitoring:
# type: external
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
GPUMonitoring:
enabled: false
gpu:
kinds:
- resourceName: "nvidia.com/gpu"
resourceType: "GPU"
default: true
es:
# master:
# volumeSize: 4Gi
# replicas: 1
# resources: {}
# data:
# volumeSize: 20Gi
# replicas: 1
# resources: {}
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchHost: ""
externalElasticsearchPort: ""
alerting:
enabled: false
# thanosruler:
# replicas: 1
# resources: {}
auditing:
enabled: false
# operator:
# resources: {}
# webhook:
# resources: {}
devops:
enabled: false
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsJavaOpts_MaxRAM: 2g
events:
enabled: false
# operator:
# resources: {}
# exporter:
# resources: {}
# ruler:
# enabled: true
# replicas: 2
# resources: {}
logging:
enabled: false
containerruntime: docker
logsidecar:
enabled: true
replicas: 2
# resources: {}
metrics_server:
enabled: false
monitoring:
storageClass: ""
# kube_rbac_proxy:
# resources: {}
# kube_state_metrics:
# resources: {}
# prometheus:
# replicas: 1
# volumeSize: 20Gi
# resources: {}
# operator:
# resources: {}
# adapter:
# resources: {}
# node_exporter:
# resources: {}
# alertmanager:
# replicas: 1
# resources: {}
# notification_manager:
# resources: {}
# operator:
# resources: {}
# proxy:
# resources: {}
gpu:
nvidia_dcgm_exporter:
enabled: false
# resources: {}
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: false
servicemesh:
enabled: false
kubeedge:
enabled: false
cloudCore:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
cloudhubPort: "10000"
cloudhubQuicPort: "10001"
cloudhubHttpsPort: "10002"
cloudstreamPort: "10003"
tunnelPort: "10004"
cloudHub:
advertiseAddress:
- ""
nodeLimit: "100"
service:
cloudhubNodePort: "30000"
cloudhubQuicNodePort: "30001"
cloudhubHttpsNodePort: "30002"
cloudstreamNodePort: "30003"
tunnelNodePort: "30004"
edgeWatcher:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
edgeWatcherAgent:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []-
name：实例的主机名。 -
address：任务机和其他实例通过 SSH 相互连接所使用的 IP 地址。根据您的环境，可以是公有 IP 地址或私有 IP 地址。例如，一些云平台为每个实例提供一个公有 IP 地址，用于通过 SSH 访问。在这种情况下，您可以在该字段填入这个公有 IP 地址。 -
internalAddress：实例的私有 IP 地址。
-
etcd：etcd 节点名称 -
master：主节点名称 -
worker：工作节点名称
( 4 ) 使用 SSH 无密码登录示例,并使用配置文件创建集群
[root@jumpserver ~]# ./kk create cluster -f config-sample.yaml
+--------------+------+------+---------+----------+-------+-------+-----------+---------+------------+-------------+------------------+--------------+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+--------------+------+------+---------+----------+-------+-------+-----------+---------+------------+-------------+------------------+--------------+
| k8s-node01 | y | y | y | y | y | y | y | 19.03.9 |||| CST 23:38:30 |
| k8s-node02 | y | y | y | y | y | y | y | 19.03.9 |||| CST 00:36:47 |
| k8s-master02 | y | y | y | y | y | y | y | 19.03.9 |||| CST 03:14:44 |
| k8s-node03 | y | y | y | y | y | y | y | 19.03.9 |||| CST 00:37:08 |
| k8s-master03 | y | y | y | y | y | y | y | 19.03.9 |||| CST 03:12:26 |
| k8s-master01 | y | y | y | y | y | y | y ||||| CST 00:14:46 |
+--------------+------+------+---------+----------+-------+-------+-----------+---------+------------+-------------+------------------+--------------+
This is a simple check of your environment.
Before installation, you should ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations
Continue this installation? [yes/no]: yes( 5 ) 验证安装
运行以下命令查看安装日志。
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f若您看到以下信息，您的高可用集群便已创建成功。
[k8s-master01 192.168.14.70] MSG:
namespace/kubesphere-system created
namespace/kubesphere-monitoring-system unchanged
[k8s-master01 192.168.14.70] MSG:
secret/kube-etcd-client-certs created
[k8s-master01 192.168.14.70] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
clusterconfiguration.installer.kubesphere.io/ks-installer created
#####################################################
### Welcome to KubeSphere! ###
#####################################################
Console: http://192.168.14.70:30880
Account: admin
Password: P@88w0rd
NOTES：
1. After you log into the console, please check the
monitoring status of service components in
"Cluster Management". If any service is not
ready, please wait patiently until all components
are up and running.
2. Please change the default password after login.
#####################################################
https://kubesphere.io 2022-04-13 00:18:35
#####################################################
INFO[00:18:42 CST] Installation is complete.
Please check the result using the command:
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f( 6 ) 查看集群节点信息
[root@k8s-master01 manifests]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready control-plane,master 28m v1.21.5
k8s-master02 Ready control-plane,master 26m v1.21.5
k8s-master03 Ready control-plane,master 26m v1.21.5
k8s-node01 Ready worker 26m v1.21.5
k8s-node02 Ready worker 26m v1.21.5
k8s-node03 Ready worker 26m v1.21.5( 7 ) 查看集群组件信息
[root@k8s-master01 manifests]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}( 8 ) 查看 kube-system 名称空间下的 pod 信息
[root@k8s-master01 manifests]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-75ddb95444-chh5x 1/1 Running 0 30m
calico-node-758jc 1/1 Running 0 30m
calico-node-96ssb 1/1 Running 0 30m
calico-node-mb7h6 1/1 Running 0 30m
calico-node-mv7wk1/1 Running 0 30m
calico-node-n7tdk 1/1 Running 0 30m
calico-node-xr874 1/1 Running 0 30m
coredns-5495dd7c88-hjg5n 1/1 Running 0 31m
coredns-5495dd7c88-zq6zf 1/1 Running 0 31m
kube-apiserver-k8s-master01 1/1 Running 0 31m
kube-apiserver-k8s-master02 1/1 Running 0 31m
kube-apiserver-k8s-master03 1/1 Running 0 31m
kube-controller-manager-k8s-master01 1/1 Running 0 5m43s
kube-controller-manager-k8s-master02 1/1 Running 0 31m
kube-controller-manager-k8s-master03 1/1 Running 0 31m
kube-proxy-2vn49 1/1 Running 0 30m
kube-proxy-7p2h6 1/1 Running 0 31m
kube-proxy-g5htp 1/1 Running 0 30m
kube-proxy-j5cvh 1/1 Running 0 30m
kube-proxy-qdrfx 1/1 Running 0 30m
kube-proxy-zfclc 1/1 Running 0 31m
kube-scheduler-k8s-master01 1/1 Running 0 5m50s
kube-scheduler-k8s-master02 1/1 Running 0 31m
kube-scheduler-k8s-master03 1/1 Running 1 31m
nodelocaldns-5fcgw 1/1 Running 0 30m
nodelocaldns-8cc7v 1/1 Running 0 31m
nodelocaldns-c8756 1/1 Running 0 30m
nodelocaldns-d9hmq 1/1 Running 0 31m
nodelocaldns-k5dmv 1/1 Running 0 31m
nodelocaldns-xgxks 1/1 Running 0 30m
openebs-localpv-provisioner-6c9dcb5c54-gzc59 1/1 Running 0 28m
snapshot-controller-0 1/1 Running 0 26m
三、集群卸载
卸载 KubeSphere 和 Kubernetes 意味着将其从您的机器上移除。该操作不可逆，且不会进行任何备份。请谨慎操作。
如需删除集群，请执行以下命令。
使用高级模式安装的 KubeSphere（使用配置文件创建）：
[root@k8s-master01 manifests]# ./kk delete cluster -f config-sample.yaml