Kubernetes 部署手册v1.12.2(6)

  1. 1. 05.部署 flannel 网络
    1. 1.1. 下载和分发 flanneld 二进制文件
    2. 1.2. 创建 flannel 证书和私钥
    3. 1.3. 向 etcd 写入集群 Pod 网段信息
    4. 1.4. 创建 flanneld 的 systemd unit 文件
    5. 1.5. 分发 flanneld systemd unit 文件到所有节点
    6. 1.6. 启动 flanneld 服务
    7. 1.7. 检查启动结果
    8. 1.8. 检查分配给各 flanneld 的 Pod 网段信息
    9. 1.9. 验证各节点能通过 Pod 网段互通

#Kubernetes部署手册v1.12.2(6)

05.部署 flannel 网络

kubernetes 要求集群内各节点(包括 master 节点)能通过 Pod 网段互联互通。flannel 使用 vxlan 技术为各节点创建一个可以互通的 Pod 网络。

flaneel 第一次启动时,从 etcd 获取 Pod 网段信息,为本节点分配一个未使用的地址段,然后创建 flannedl.1(也可能是其它名称,如 flannel1 等) 接口。

flannel 将分配的 Pod 网段信息写入 /run/flannel/docker 文件,docker 后续使用这个文件中的环境变量设置 docker0 网桥。

下载和分发 flanneld 二进制文件

https://github.com/coreos/flannel/releases 页面下载最新版本的发布包:

1
2
3
4
cd /opt/k8s/work
mkdir flannel
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel

分发 flanneld 二进制文件到集群所有节点:

1
2
3
4
5
6
7
8
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp flannel/{flanneld,mk-docker-opts.sh} k8s@${node_ip}:/opt/k8s/bin/
ssh k8s@${node_ip} "sudo chmod +x /opt/k8s/bin/*"
done

创建 flannel 证书和私钥

flannel 从 etcd 集群存取网段分配信息,而 etcd 集群启用了双向 x509 证书认证,所以需要为 flanneld 生成证书和私钥。

创建证书签名请求:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cd /opt/k8s/work
cat > flanneld-csr.json <<EOF
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "Lenovo"
}
]
}
EOF
  • 该证书只会被 kubectl 当做 client 证书使用,所以 hosts 字段为空;

生成证书和私钥:

1
2
3
4
5
cfssl gencert -ca=/opt/k8s/work/ca.pem \
-ca-key=/opt/k8s/work/ca-key.pem \
-config=/opt/k8s/work/ca-config.json \
-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
ls flanneld*pem

将生成的证书和私钥分发到所有节点(master 和 worker):

1
2
3
4
5
6
7
8
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo mkdir -p /etc/flanneld/cert"
scp flanneld*.pem k8s@${node_ip}:/etc/flanneld/cert
done

向 etcd 写入集群 Pod 网段信息

注意:本步骤只需执行一次

1
2
3
4
5
6
7
8
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/opt/k8s/work/ca.pem \
--cert-file=/opt/k8s/work/flanneld.pem \
--key-file=/opt/k8s/work/flanneld-key.pem \
set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'
  • flanneld 当前版本 (v0.10.0) 不支持 etcd v3,故使用 etcd v2 API 写入配置 key 和网段数据;
  • SubnetLen 设置为 21,而 ${CLUSTER_CIDR} 是 /16 段地址,故支持 32 个节点,每个节点最多 2048 个 Pods;

创建 flanneld 的 systemd unit 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
export IFACE=eth0
cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
-etcd-cafile=/etc/kubernetes/cert/ca.pem \\
-etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
-etcd-endpoints=${ETCD_ENDPOINTS} \\
-etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
-iface=${IFACE}
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
  • mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件,后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥;
  • flanneld 使用系统缺省路由所在的接口与其它节点通信,对于有多个网络接口(如内网和公网)的节点,可以用 -iface 参数指定通信接口,如上面的 eth0 接口;
  • flanneld 运行时需要 root 权限;

分发 flanneld systemd unit 文件到所有节点

1
2
3
4
5
6
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp flanneld.service k8s@${node_ip}:/etc/systemd/system/
done

启动 flanneld 服务

1
2
3
4
5
6
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo systemctl daemon-reload && sudo systemctl enable flanneld && sudo systemctl restart flanneld"
done

检查启动结果

1
2
3
4
5
6
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo systemctl status flanneld|grep Active"
done

确保状态为 active (running),否则查看日志,确认原因:

1
$ journalctl -u flanneld

检查分配给各 flanneld 的 Pod 网段信息

查看集群 Pod 网段(/16):

1
2
3
4
5
6
7
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/config

输出:

{"Network":"10.253.0.0/16", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}

查看已分配的 Pod 子网段列表(/24):

1
2
3
4
5
6
7
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
ls ${FLANNEL_ETCD_PREFIX}/subnets

输出:

1
2
3
4
5
6
/kubernetes/network/subnets/10.253.12.0-24
/kubernetes/network/subnets/10.253.2.0-24
/kubernetes/network/subnets/10.253.40.0-24
/kubernetes/network/subnets/10.253.74.0-24
/kubernetes/network/subnets/10.253.89.0-24
/kubernetes/network/subnets/10.253.9.0-24

查看某一 Pod 网段对应的节点 IP 和 flannel 接口地址:

1
2
3
4
5
6
7
source /opt/k8s/bin/environment.sh
etcdctl \
--endpoints=${ETCD_ENDPOINTS} \
--ca-file=/etc/kubernetes/cert/ca.pem \
--cert-file=/etc/flanneld/cert/flanneld.pem \
--key-file=/etc/flanneld/cert/flanneld-key.pem \
get ${FLANNEL_ETCD_PREFIX}/subnets/10.253.12.0-24

输出:

{"PublicIP":"10.12.11.4","BackendType":"vxlan","BackendData":{"VtepMAC":"0a:0f:61:91:80:8a"}}

验证各节点能通过 Pod 网段互通

各节点上部署 flannel 后,检查是否创建了 flannel 接口(名称可能为 flannel0、flannel.0、flannel.1 等):

1
2
3
4
5
6
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
done

输出:

1
2
3
4
5
6
>>> 10.12.11.4
inet 10.253.12.0/32 scope global flannel.1
>>> 10.12.11.5
inet 10.253.2.0/32 scope global flannel.1
>>> 10.12.11.6
inet 10.253.40.0/32 scope global flannel.1

在各节点上 ping 所有 flannel 接口 IP,确保能通:

1
2
3
4
5
6
7
8
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh ${node_ip} "ping -c 1 10.253.12.0"
ssh ${node_ip} "ping -c 1 10.253.2.0"
ssh ${node_ip} "ping -c 1 10.253.40.0"
done