一个全新的基于容器技术的分布式架构方案-Kubernetes

vlambda
2020-05-09

一个全新的基于容器技术的分布式架构方案-Kubernetes

文末搜集了几张非常不错的架构图，从k8s基础组件，网络，工作流以及工作模式四个方面介绍k8s的工作模式，对深入理解k8s有很好的帮助。(文章较长，建议先收藏)

k8s(1.14.0)+etcd(3.3.10)+flanneld(0.10)

安装和配置

前提

os：centos 7.5.1804

一、初始化环境

1.1 关闭防火墙和SeLinux

 
   
   
 
  
    
    
  systemctl stop firewalld
  
    
    
  setenforce 0 (临时关闭)
  
    
    
  vi /etc/selinux/config
  
    
    
  SELINUX=disabled

1.2 下载链接

 
   
   
 
  
    
    
  Client Binaries
  
    
    
  https://dl.k8s.io/v1.14.0/kubernetes-client-linux-amd64.tar.gz
  
    
    
  Server Binaries
  
    
    
  https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz
  
    
    
  Node Binaries
  
    
    
  https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz
  
    
    
  etcd
  
    
    
  https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
  
    
    
  flannel
  
    
    
  https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

二、 k8s-Master部署

2.1 下载资源(wget不到的话，可以用科学上网的机器直接download)

 
   
   
 
  
    
    
  wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz
  
    
    
  wget https://dl.k8s.io/v1.14.0/kubernetes-client-linux-amd64.tar.gz
  
    
    
  wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
  
    
    
  wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

2.2 cfssl安装(wget不到的话，可以用科学上网的机器直接download)

 
   
   
 
  
    
    
  wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
  
    
    
  wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
  
    
    
  wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
  
    
    
  chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
  
    
    
  mv cfssl_linux-amd64 /usr/local/bin/cfssl
  
    
    
  mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
  
    
    
  mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.3 创建etcd证书

 
   
   
 
  
    
    
  mkdir /k8s/etcd/{bin,cfg,ssl} -p
  
    
    
  mkdir /k8s/kubernetes/{bin,cfg,ssl} -p
  
    
    
  cd /k8s/etcd/ssl/

2.3.1 etcd ca配置

 
   
   
 
  
    
    
  cat << EOF | tee ca-config.json
  
    
    
  {
  
    
    
   "signing": {
  
    
    
   "default": {
  
    
    
   "expiry": "87600h"
  
    
    
   },
  
    
    
   "profiles": {
  
    
    
   "etcd": {
  
    
    
   "expiry": "87600h",
  
    
    
   "usages": [
  
    
    
   "signing",
  
    
    
   "key encipherment",
  
    
    
   "server auth",
  
    
    
   "client auth"
  
    
    
   ]
  
    
    
   }
  
    
    
   }
  
    
    
   }
  
    
    
  }
  
    
    
  EOF

2.3.2 etcd ca证书

 
   
   
 
  
    
    
  cat << EOF | tee ca-csr.json
  
    
    
  {
  
    
    
   "CN": "etcd CA",
  
    
    
   "key": {
  
    
    
   "algo": "rsa",
  
    
    
   "size": 2048
  
    
    
   },
  
    
    
   "names": [
  
    
    
   {
  
    
    
   "C": "CN",
  
    
    
   "L": "Beijing",
  
    
    
   "ST": "Beijing"
  
    
    
   }
  
    
    
   ]
  
    
    
  }
  
    
    
  EOF

2.3.3 etcd server证书

 
   
   
 
  
    
    
  cat << EOF | tee server-csr.json
  
    
    
  {
  
    
    
   "CN": "etcd",
  
    
    
   "hosts": [
  
    
    
   "192.168.4.71",
  
    
    
   "192.168.4.72",
  
    
    
   "192.168.4.76"
  
    
    
   ],
  
    
    
   "key": {
  
    
    
   "algo": "rsa",
  
    
    
   "size": 2048
  
    
    
   },
  
    
    
   "names": [
  
    
    
   {
  
    
    
   "C": "CN",
  
    
    
   "L": "Beijing",
  
    
    
   "ST": "Beijing"
  
    
    
   }
  
    
    
   ]
  
    
    
  }
  
    
    
  EOF

2.3.4 生成etcd ca证书和私钥初始化ca

 
   
   
 
  
    
    
  cfssl gencert -initca ca-csr.json | cfssljson -bare ca

2.3.5 生成server证书

 
   
   
 
  
    
    
  cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server

一个全新的基于容器技术的分布式架构方案-Kubernetes

2.4 etcd安装

2.4.1 解压

 
   
   
 
  
    
    
  tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
  
    
    
  cd etcd-v3.3.10-linux-amd64/
  
    
    
  cp etcd etcdctl /k8s/etcd/bin/

2.4.2 配置etcd的配置文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /k8s/etcd/cfg/etcd.conf
  
    
    
  

  
    
    
  #[Member]
  
    
    
  ETCD_NAME="etcd01"
  
    
    
  ETCD_DATA_DIR="/data1/etcd"
  
    
    
  ETCD_LISTEN_PEER_URLS="https://192.168.4.71:2380"
  
    
    
  ETCD_LISTEN_CLIENT_URLS="https://192.168.4.71:2379"
  
    
    
  

  
    
    
  #[Clustering]
  
    
    
  ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.4.71:2380"
  
    
    
  ETCD_ADVERTISE_CLIENT_URLS="https://192.168.4.71:2379"
  
    
    
  ETCD_INITIAL_CLUSTER="etcd01=https://192.168.4.71:2380"
  
    
    
  ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  
    
    
  ETCD_INITIAL_CLUSTER_STATE="new"
  
    
    
  

  
    
    
  #[Security]
  
    
    
  ETCD_CERT_FILE="/k8s/etcd/ssl/server.pem"
  
    
    
  ETCD_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
  
    
    
  ETCD_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
  
    
    
  ETCD_CLIENT_CERT_AUTH="true"
  
    
    
  ETCD_PEER_CERT_FILE="/k8s/etcd/ssl/server.pem"
  
    
    
  ETCD_PEER_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
  
    
    
  ETCD_PEER_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
  
    
    
  ETCD_PEER_CLIENT_CERT_AUTH="true"

2.4.3 配置etcd的启动文件

 
   
   
 
  
    
    
  mkdir /data1/etcd
  
    
    
  [root@t71 cfg]# vim /usr/lib/systemd/system/etcd.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Etcd Server
  
    
    
  After=network.target
  
    
    
  After=network-online.target
  
    
    
  Wants=network-online.target
  
    
    
  

  
    
    
  [Service]
  
    
    
  Type=notify
  
    
    
  WorkingDirectory=/data1/etcd/
  
    
    
  EnvironmentFile=-/k8s/etcd/cfg/etcd.conf
  
    
    
  # set GOMAXPROCS to number of processors
  
    
    
  ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /k8s/etcd/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" --cert-file=\"${ETCD_CERT_FILE}\" --key-file=\"${ETCD_KEY_FILE}\" --trusted-ca-file=\"${ETCD_TRUSTED_CA_FILE}\" --client-cert-auth=\"${ETCD_CLIENT_CERT_AUTH}\" --peer-cert-file=\"${ETCD_PEER_CERT_FILE}\" --peer-key-file=\"${ETCD_PEER_KEY_FILE}\" --peer-trusted-ca-file=\"${ETCD_PEER_TRUSTED_CA_FILE}\" --peer-client-cert-auth=\"${ETCD_PEER_CLIENT_CERT_AUTH}\""
  
    
    
  Restart=on-failure
  
    
    
  LimitNOFILE=65536
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.targe

2.4.4 启动

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl enabel etcd
  
    
    
  systemctl start etcd

2.4.5 检查服务

 
   
   
 
  
    
    
  [root@t71 bin]# etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://192.168.4.71:2379" cluster-health
  
    
    
  member ac829673d2b22824 is healthy: got healthy result from https://192.168.4.71:2379
  
    
    
  cluster is healthy
  
    
    
  [root@t71 bin]#

2.5 生成kubernets证书与私钥

2.5.1 制作kubernetes ca证书

 
   
   
 
  
    
    
  cd /k8s/kubernetes/ssl
  
    
    
  cat << EOF | tee ca-config.json
  
    
    
  {
  
    
    
   "signing": {
  
    
    
   "default": {
  
    
    
   "expiry": "87600h"
  
    
    
   },
  
    
    
   "profiles": {
  
    
    
   "kubernetes": {
  
    
    
   "expiry": "87600h",
  
    
    
   "usages": [
  
    
    
   "signing",
  
    
    
   "key encipherment",
  
    
    
   "server auth",
  
    
    
   "client auth"
  
    
    
   ]
  
    
    
   }
  
    
    
   }
  
    
    
   }
  
    
    
  }
  
    
    
  EOF

 
   
   
 
  
    
    
  cat << EOF | tee ca-csr.json
  
    
    
  {
  
    
    
   "CN": "kubernetes",
  
    
    
   "key": {
  
    
    
   "algo": "rsa",
  
    
    
   "size": 2048
  
    
    
   },
  
    
    
   "names": [
  
    
    
   {
  
    
    
   "C": "CN",
  
    
    
   "L": "Beijing",
  
    
    
   "ST": "Beijing",
  
    
    
   "O": "k8s",
  
    
    
   "OU": "System"
  
    
    
   }
  
    
    
   ]
  
    
    
  }
  
    
    
  EOF

 
   
   
 
  
    
    
  cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

2.5.2 制作apiserver证书

 
   
   
 
  
    
    
  cat << EOF | tee server-csr.json
  
    
    
  {
  
    
    
   "CN": "kubernetes",
  
    
    
   "hosts": [
  
    
    
   "10.254.0.1",
  
    
    
   "127.0.0.1",
  
    
    
   "192.168.4.71",
  
    
    
   "192.168.4.72",
  
    
    
   "192.168.4.76",
  
    
    
   "kubernetes",
  
    
    
   "kubernetes.default",
  
    
    
   "kubernetes.default.svc",
  
    
    
   "kubernetes.default.svc.cluster",
  
    
    
   "kubernetes.default.svc.cluster.local"
  
    
    
   ],
  
    
    
   "key": {
  
    
    
   "algo": "rsa",
  
    
    
   "size": 2048
  
    
    
   },
  
    
    
   "names": [
  
    
    
   {
  
    
    
   "C": "CN",
  
    
    
   "L": "Beijing",
  
    
    
   "ST": "Beijing",
  
    
    
   "O": "k8s",
  
    
    
   "OU": "System"
  
    
    
   }
  
    
    
   ]
  
    
    
  }
  
    
    
  EOF

 
   
   
 
  
    
    
  cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

2.5.3 制作kube-proxy证书

 
   
   
 
  
    
    
  cat << EOF | tee kube-proxy-csr.json
  
    
    
  {
  
    
    
   "CN": "system:kube-proxy",
  
    
    
   "hosts": [],
  
    
    
   "key": {
  
    
    
   "algo": "rsa",
  
    
    
   "size": 2048
  
    
    
   },
  
    
    
   "names": [
  
    
    
   {
  
    
    
   "C": "CN",
  
    
    
   "L": "Beijing",
  
    
    
   "ST": "Beijing",
  
    
    
   "O": "k8s",
  
    
    
   "OU": "System"
  
    
    
   }
  
    
    
   ]
  
    
    
  }
  
    
    
  EOF

 
   
   
 
  
    
    
   cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

一个全新的基于容器技术的分布式架构方案-Kubernetes

2.6 安装k8s服务

2.6.1 解压缩文件

 
   
   
 
  
    
    
  tar -zxvf kubernetes-server-linux-amd64.tar.gz
  
    
    
  cd kubernetes/server/bin/
  
    
    
  cp kube-scheduler kube-apiserver kube-controller-manager kubectl /k8s/kubernetes/bin/

2.6.2 安装kube-apiserver

2.6.2.1 部署kube-apiserver组件创建TLS Bootstrapping Token

 
   
   
 
  
    
    
  [root@elasticsearch01 bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
  
    
    
  663eb46fb81c4cf2a9bedb84bea03582
  
    
    
  

  
    
    
  vim /k8s/kubernetes/cfg/token.csv
  
    
    
  663eb46fb81c4cf2a9bedb84bea03582,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

2.6.2.2 创建Apiserver配置文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-apiserver
  
    
    
  

  
    
    
  KUBE_APISERVER_OPTS="--logtostderr=true \
  
    
    
  --v=4 \
  
    
    
  --etcd-servers=https://192.168.4.71:2379 \
  
    
    
  --bind-address=192.168.4.71 \
  
    
    
  --secure-port=6443 \
  
    
    
  --advertise-address=192.168.4.71 \
  
    
    
  --allow-privileged=true \
  
    
    
  --service-cluster-ip-range=10.254.0.0/16 \
  
    
    
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
  
    
    
  --authorization-mode=RBAC,Node \
  
    
    
  --enable-bootstrap-token-auth \
  
    
    
  --token-auth-file=/k8s/kubernetes/cfg/token.csv \
  
    
    
  --service-node-port-range=30000-50000 \
  
    
    
  --tls-cert-file=/k8s/kubernetes/ssl/server.pem \
  
    
    
  --tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem \
  
    
    
  --client-ca-file=/k8s/kubernetes/ssl/ca.pem \
  
    
    
  --service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
  
    
    
  --etcd-cafile=/k8s/etcd/ssl/ca.pem \
  
    
    
  --etcd-certfile=/k8s/etcd/ssl/server.pem \
  
    
    
  --etcd-keyfile=/k8s/etcd/ssl/server-key.pem"

2.6.2.3 创建apiserver systemd文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /usr/lib/systemd/system/kube-apiserver.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Kubernetes API Server
  
    
    
  Documentation=https://github.com/kubernetes/kubernetes
  
    
    
  

  
    
    
  [Service]
  
    
    
  EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver
  
    
    
  ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
  
    
    
  Restart=on-failure
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

2.6.2.4 启动服务

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl enable kube-apiserver
  
    
    
  systemctl start kube-apiserver

2.6.3 安装kube-scheduler

2.6.3.1 创建kube-scheduler配置文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-scheduler
  
    
    
  KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080

2.6.3.2 创建kube-scheduler systemd文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /usr/lib/systemd/system/kube-scheduler.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Kubernetes Scheduler
  
    
    
  Documentation=https://github.com/kubernetes/kubernetes
  
    
    
  

  
    
    
  [Service]
  
    
    
  EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler
  
    
    
  ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
  
    
    
  Restart=on-failure
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

2.6.3.3 启动服务

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl start kube-scheduler
  
    
    
  systemctl enable kube-scheduler

2.6.4 安装kube-controller-manager

2.6.3.1 创建kube-controller-managerr配置文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-controller-manager
  
    
    
  

  
    
    
  KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
  
    
    
  --v=4 \
  
    
    
  --master=127.0.0.1:8080 \
  
    
    
  --leader-elect=true \
  
    
    
  --address=127.0.0.1 \
  
    
    
  --service-cluster-ip-range=10.254.0.0/16 \
  
    
    
  --cluster-name=kubernetes \
  
    
    
  --cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
  
    
    
  --cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem \
  
    
    
  --root-ca-file=/k8s/kubernetes/ssl/ca.pem \
  
    
    
  --service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem"

2.6.3.2 创建kube-controller-manager systemd文件

 
   
   
 
  
    
    
  [root@t71 cfg]# vim /usr/lib/systemd/system/kube-controller-manager.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Kubernetes Controller Manager
  
    
    
  Documentation=https://github.com/kubernetes/kubernetes
  
    
    
  

  
    
    
  [Service]
  
    
    
  EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager
  
    
    
  ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
  
    
    
  Restart=on-failure
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

2.6.3.3 启动服务

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl start kube-controller-manager
  
    
    
  systemctl enable kube-controller-manager

2.7 验证kube server

2.7.1 添加环境变量

 
   
   
 
  
    
    
  vim /etc/profile
  
    
    
  export PATH=/k8s/kubernetes/bin:$PATH
  
    
    
  source /etc/profile

2.7.2 get

一个全新的基于容器技术的分布式架构方案-Kubernetes

三、k8s-slave部署

3.1 docker 环境安装

 
   
   
 
  
    
    
  yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  
    
    
  yum list docker-ce --showduplicates | sort -r
  
    
    
  yum install docker-ce -y
  
    
    
  systemctl start docker && systemctl enable docker

3.2 部署kubelet插件

kublet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，执行交互式命令，如exec、run、logs 等; kublet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况; 为确保安全，只开启接收 https 请求的安全端口，对请求进行认证和授权，拒绝未授权的访问(如apiserver、heapster)

3.2.1 安装二进制文件

 
   
   
 
  
    
    
  wget https://dl.k8s.io/v1.13.1/kubernetes-node-linux-amd64.tar.gz
  
    
    
  tar zxvf kubernetes-node-linux-amd64.tar.gz
  
    
    
  cd kubernetes/node/bin/
  
    
    
  cp kube-proxy kubelet kubectl /k8s/kubernetes/bin/

3.2.2 复制相关证书到node节点

 
   
   
 
  
    
    
  scp *.pem 10.2.8.65:$PWD

3.2.3 创建kubelet bootstrap kubeconfig文件通过脚本实现

 
   
   
 
  
    
    
  [root@t72 cfg]# vim environment.sh
  
    
    
  

  
    
    
  #!/bin/bash
  
    
    
  #创建kubelet bootstrapping kubeconfig
  
    
    
  BOOTSTRAP_TOKEN=663eb46fb81c4cf2a9bedb84bea03582
  
    
    
  KUBE_APISERVER="https://192.168.4.71:6443"
  
    
    
  #设置集群参数
  
    
    
  kubectl config set-cluster kubernetes \
  
    
    
   --certificate-authority=/k8s/kubernetes/ssl/ca.pem \
  
    
    
   --embed-certs=true \
  
    
    
   --server=${KUBE_APISERVER} \
  
    
    
   --kubeconfig=bootstrap.kubeconfig
  
    
    
  

  
    
    
  #设置客户端认证参数
  
    
    
  kubectl config set-credentials kubelet-bootstrap \
  
    
    
   --token=${BOOTSTRAP_TOKEN} \
  
    
    
   --kubeconfig=bootstrap.kubeconfig
  
    
    
  

  
    
    
  # 设置上下文参数
  
    
    
  kubectl config set-context default \
  
    
    
   --cluster=kubernetes \
  
    
    
   --user=kubelet-bootstrap \
  
    
    
   --kubeconfig=bootstrap.kubeconfig
  
    
    
  

  
    
    
  # 设置默认上下文
  
    
    
  kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
  
    
    
  

  
    
    
  #----------------------
  
    
    
  

  
    
    
  # 创建kube-proxy kubeconfig文件
  
    
    
  

  
    
    
  kubectl config set-cluster kubernetes \
  
    
    
   --certificate-authority=/k8s/kubernetes/ssl/ca.pem \
  
    
    
   --embed-certs=true \
  
    
    
   --server=${KUBE_APISERVER} \
  
    
    
   --kubeconfig=kube-proxy.kubeconfig
  
    
    
  

  
    
    
  kubectl config set-credentials kube-proxy \
  
    
    
   --client-certificate=/k8s/kubernetes/ssl/kube-proxy.pem \
  
    
    
   --client-key=/k8s/kubernetes/ssl/kube-proxy-key.pem \
  
    
    
   --embed-certs=true \
  
    
    
   --kubeconfig=kube-proxy.kubeconfig
  
    
    
  

  
    
    
  kubectl config set-context default \
  
    
    
   --cluster=kubernetes \
  
    
    
   --user=kube-proxy \
  
    
    
   --kubeconfig=kube-proxy.kubeconfig
  
    
    
  

  
    
    
  kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

3.2.4 执行脚本

 
   
   
 
  
    
    
  sh environment.sh

3.2.5 创建kubelet参数配置模板文件

 
   
   
 
  
    
    
  [root@t72 cfg]# vim kubelet.config
  
    
    
  

  
    
    
  kind: KubeletConfiguration
  
    
    
  apiVersion: kubelet.config.k8s.io/v1beta1
  
    
    
  address: 192.168.4.72
  
    
    
  port: 10250
  
    
    
  readOnlyPort: 10255
  
    
    
  cgroupDriver: cgroupfs
  
    
    
  clusterDNS: ["10.254.0.10"]
  
    
    
  clusterDomain: cluster.local.
  
    
    
  failSwapOn: false
  
    
    
  #authentication:
  
    
    
  # anonymous:
  
    
    
  # enabled: true
  
    
    
  authentication:
  
    
    
   anonymous:
  
    
    
   enabled: true # Defaults to false as of 1.10
  
    
    
   webhook:
  
    
    
   enabled: false # Deafults to true as of 1.10
  
    
    
  authorization:
  
    
    
   mode: AlwaysAllow

3.2.6 创建kubelet配置文件

 
   
   
 
  
    
    
  [root@t72 cfg]# vim kubelet
  
    
    
  

  
    
    
  KUBELET_OPTS="--logtostderr=true \
  
    
    
  --v=4 \
  
    
    
  --hostname-override=192.168.4.72 \
  
    
    
  --kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
  
    
    
  --bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
  
    
    
  --config=/k8s/kubernetes/cfg/kubelet.config \
  
    
    
  --cert-dir=/k8s/kubernetes/ssl \
  
    
    
  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

3.2.7 创建kubelet systemd文件

 
   
   
 
  
    
    
  [root@t72 cfg]# vim /usr/lib/systemd/system/kubelet.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Kubernetes Kubelet
  
    
    
  After=docker.service
  
    
    
  Requires=docker.service
  
    
    
  

  
    
    
  [Service]
  
    
    
  EnvironmentFile=-/k8s/kubernetes/cfg/kubelet
  
    
    
  ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
  
    
    
  Restart=on-failure
  
    
    
  KillMode=process
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

3.2.8 在master节点上面操作，将kubelet-bootstrap用户绑定到系统集群角色

 
   
   
 
  
    
    
  kubectl create clusterrolebinding kubelet-bootstrap \
  
    
    
   --clusterrole=system:node-bootstrapper \
  
    
    
   --user=kubelet-bootstrap

3.2.9 启动kubelet服务

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl start kubelet
  
    
    
  systemctl enable kubelet

3.2.10 Master接受kubelet CSR请求可以手动或自动 approve CSR 请求。推荐使用自动的方式，因为从 v1.8 版本开始，可以自动轮转approve csr 后生成的证书，如下是手动 approve CSR请求操作方法查看CSR列表

 
   
   
 
  
    
    
  # kubectl get csr
  
    
    
  NAME AGE REQUESTOR CONDITION
  
    
    
  node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc 102s kubelet-bootstrap Pending

接受node

 
   
   
 
  
    
    
  # kubectl certificate approve node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc
  
    
    
  certificatesigningrequest.certificates.k8s.io/node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc approved

再看CRS

 
   
   
 
  
    
    
  kubectl get csr
  
    
    
  NAME AGE REQUESTOR CONDITION
  
    
    
  node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc 5m13s kubelet-bootstrap Approved,Issued

3.3 安装kube-proxy

3.3.1 kube-proxy的配置文件

 
   
   
 
  
    
    
  [root@t72 cfg]# vim kube-proxy
  
    
    
  

  
    
    
  KUBE_PROXY_OPTS="--logtostderr=true \
  
    
    
  --v=4 \
  
    
    
  --hostname-override=192.168.4.72 \
  
    
    
  --cluster-cidr=10.254.0.0/16 \
  
    
    
  --kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"

3.3.2 kube-proxy的systemd 文件

 
   
   
 
  
    
    
  [root@t72 cfg]# vim /usr/lib/systemd/system/kube-proxy.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Kubernetes Proxy
  
    
    
  After=network.target
  
    
    
  

  
    
    
  [Service]
  
    
    
  EnvironmentFile=-/k8s/kubernetes/cfg/kube-proxy
  
    
    
  ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
  
    
    
  Restart=on-failure
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

3.3.3 启动

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl start kube-proxy
  
    
    
  systemctl enable kube-proxy

3.4 部署另外一个node和认证csr

注意: 安装期间，如果kubelet，kube-proxy配置错误，比如监听IP或者hostname错误导致node not found，需要删除kubelet-client证书，重启kubelet服务，重启认证csr即可

四、网络部署

4.1 etcd注册网段（master节点，因为只有master安装了etcd）

 
   
   
 
  
    
    
  /k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://192.168.4.71:2379" set /k8s/network/config '{ "Network": "10.254.0.0/16", "Backend": {"Type": "vxlan"}}'

4.2 Flanneld部署

4.2.1 解压

 
   
   
 
  
    
    
  tar -xvf flannel-v0.10.0-linux-amd64.tar.gz
  
    
    
  mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/

4.2.2 配置flanneld

 
   
   
 
  
    
    
  [root@t72 cfg]# vim flanneld
  
    
    
  

  
    
    
  FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.4.71:2379, -etcd-cafile=/k8s/etcd/ssl/ca.pem -etcd-certfile=/k8s/etcd/ssl/server.pem -etcd-keyfile=/k8s/etcd/ssl/server-key.pem -etcd-prefix=/k8s/network"

4.2.3 创建flanneld systemd文件

 
   
   
 
  
    
    
  [root@t72 cfg]# vim /usr/lib/systemd/system/flanneld.service
  
    
    
  

  
    
    
  [Unit]
  
    
    
  Description=Flanneld overlay address etcd agent
  
    
    
  After=network-online.target network.target
  
    
    
  Before=docker.service
  
    
    
  

  
    
    
  [Service]
  
    
    
  Type=notify
  
    
    
  EnvironmentFile=/k8s/kubernetes/cfg/flanneld
  
    
    
  ExecStart=/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
  
    
    
  #ExecStart=/k8s/kubernetes/bin/flanneld $FLANNEL_OPTIONS
  
    
    
  ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
  
    
    
  Restart=on-failure
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

注: mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件，后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥；flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口; flanneld 运行时需要 root 权限；

4.2.4 配置Docker启动指定子网

修改EnvironmentFile=/run/flannel/subnet.env，ExecStart=/usr/bin/dockerd $DOCKERNETWORKOPTIONS即可

 
   
   
 
  
    
    
  [root@t72 cfg]# cat /usr/lib/systemd/system/docker.service | grep -v "#" 
  
    
    
  [Unit]
  
    
    
  Description=Docker Application Container Engine
  
    
    
  Documentation=https://docs.docker.com
  
    
    
  BindsTo=containerd.service
  
    
    
  After=network-online.target firewalld.service containerd.service
  
    
    
  Wants=network-online.target
  
    
    
  Requires=docker.socket
  
    
    
  

  
    
    
  [Service]
  
    
    
  Type=notify
  
    
    
  EnvironmentFile=/run/flannel/subnet.env
  
    
    
  ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
  
    
    
  ExecReload=/bin/kill -s HUP $MAINPID
  
    
    
  TimeoutSec=0
  
    
    
  RestartSec=2
  
    
    
  Restart=always
  
    
    
  

  
    
    
  StartLimitBurst=3
  
    
    
  

  
    
    
  StartLimitInterval=60s
  
    
    
  

  
    
    
  LimitNOFILE=infinity
  
    
    
  LimitNPROC=infinity
  
    
    
  LimitCORE=infinity
  
    
    
  

  
    
    
  TasksMax=infinity
  
    
    
  

  
    
    
  Delegate=yes
  
    
    
  

  
    
    
  KillMode=process
  
    
    
  

  
    
    
  [Install]
  
    
    
  WantedBy=multi-user.target

4.2.5 启动服务

注：启动flannel前要关闭docker及相关的kubelet这样flannel才会覆盖docker0网桥(如果已经存在docker0，则先删除docker0 ip link delete docker0)

 
   
   
 
  
    
    
  systemctl daemon-reload
  
    
    
  systemctl stop docker
  
    
    
  systemctl start flanneld
  
    
    
  systemctl enable flanneld
  
    
    
  systemctl start docker
  
    
    
  systemctl restart kubelet
  
    
    
  systemctl restart kube-proxy

4.2.6 验证服务

一个全新的基于容器技术的分布式架构方案-Kubernetes

 
   
   
 
  
    
    
  kubectl get nodes -o wide

一个全新的基于容器技术的分布式架构方案-Kubernetes

五、后记

5.1 kubernetes 组件图

Master节点：Master节点上面主要由四个模块组成，APIServer，schedule,controller-manager,etcd

APIServer: APIServer负责对外提供RESTful的kubernetes API的服务，它是系统管理指令的统一接口，任何对资源的增删改查都要交给APIServer处理后再交给etcd，如图，kubectl(kubernetes提供的客户端工具，该工具内部是对kubernetes API的调用）是直接和APIServer交互的。
schedule: schedule负责调度Pod到合适的Node上，如果把scheduler看成一个黑匣子，那么它的输入是pod和由多个Node组成的列表，输出是Pod和一个Node的绑定。kubernetes目前提供了调度算法，同样也保留了接口。用户根据自己的需求定义自己的调度算法。
controller manager: 如果APIServer做的是前台的工作的话，那么controller manager就是负责后台的。每一个资源都对应一个控制器。而control manager就是负责管理这些控制器的，比如我们通过APIServer创建了一个Pod，当这个Pod创建成功后，APIServer的任务就算完成了。
etcd：etcd是一个高可用的键值存储系统，kubernetes使用它来存储各个资源的状态，从而实现了Restful的API。

Node节点：每个Node节点主要由三个模板组成：kublet, kube-proxy

kube-proxy: 该模块实现了kubernetes中的服务发现和反向代理功能。kube-proxy支持TCP和UDP连接转发，默认基Round Robin算法将客户端流量转发到与service对应的一组后端pod。服务发现方面，kube-proxy使用etcd的watch机制监控集群中service和endpoint对象数据的动态变化，并且维护一个service到endpoint的映射关系，从而保证了后端pod的IP变化不会对访问者造成影响，另外，kube-proxy还支持session affinity。
kublet：kublet是Master在每个Node节点上面的agent，是Node节点上面最重要的模块，它负责维护和管理该Node上的所有容器，但是如果容器不是通过kubernetes创建的，它并不会管理。本质上，它负责使Pod的运行状态与期望的状态一致。

5.2 Kubernetes工作流程

一个全新的基于容器技术的分布式架构方案-Kubernetes 以创建Deployment为例：

Devops人员通过Apiserver，访问k8s,发送创建Deployment请求
Apiserver将Deployment元数据存至etcd中
Controller manager 中的Deployment controller通过Apiserver watch 资源的变换情况，发现有新的Deployment需要创建
Deployment controller请求Apiserver创建ReplicaSet资源, Apiserver将ReplicaSet元数据存至etcd中
Controller manager 中的ReplicaSet controller通过Apiserver watch 资源的变换情况，发现有新的ReplicaSet需要创建
ReplicaSet controller请求Apiserver创建Pod资源, Apiserver将Pod元数据存至etcd中
Scheduler通过Apiserver watch 资源的变换情况，发现有新的Pod需要创建。综合节点资源通过（请求apiserver获取）和pod资源情况（请求apiserver获取）制定出调度信息返回给给Apiserver
Apiserver 根据Scheduler返回的调度信息，通过slave中kubelet组件创建pod资源
kubelet通过docker daemon 创建container资源（包括用户容器和pause容器）

5.3 kubernetes 网络架构图（flanneld）

一个全新的基于容器技术的分布式架构方案-Kubernetes

数据从源容器中发出后，经由所在主机的docker0虚拟网卡转发到flannel0虚拟网卡，这是个P2P的虚拟网卡，flanneld服务监听在网卡的另外一端。
Flannel通过Etcd服务维护了一张节点间的路由表，在稍后的配置部分我们会介绍其中的内容。
源主机的flanneld服务将原本的数据内容UDP封装后根据自己的路由表投递给目的节点的flanneld服务，数据到达以后被解包，然后直接进入目的节点的flannel0虚拟网卡，然后被转发到目的主机的docker0虚拟网卡，最后就像本机容器通信一下的有docker0路由到达目标容器。

vlambda博客
学习文章列表