Kubernetes 部署手册v1.12.2(13)

  1. 1. 07-1.部署 docker 组件
    1. 1.1. 安装依赖包
    2. 1.2. 下载和分发 docker 二进制文件
    3. 1.3. 创建和分发 systemd unit 文件
    4. 1.4. 配置和分发 docker 配置文件
    5. 1.5. 启动 docker 服务
    6. 1.6. 检查服务运行状态
      1. 1.6.1. 检查 docker0 网桥

#Kubernetes部署手册v1.12.2(13)

07-1.部署 docker 组件

docker 是容器的运行环境,管理它的生命周期。kubelet 通过 Container Runtime Interface (CRI) 与 docker 进行交互。

安装依赖包

参考 07-0.部署worker节点.md

下载和分发 docker 二进制文件

https://download.docker.com/linux/static/stable/x86_64/ 页面下载最新发布包:

1
2
3
cd /opt/k8s/work
wget https://download.docker.com/linux/static/stable/x86_64/docker-18.06.1-ce.tgz
tar -xvf docker-18.06.1-ce.tgz

分发二进制文件到所有 worker 节点:

1
2
3
4
5
6
7
8
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${WORKER_IPS[@]}
do
echo ">>> ${node_ip}"
scp docker/docker* k8s@${node_ip}:/opt/k8s/bin/
ssh k8s@${node_ip} "sudo chmod +x /opt/k8s/bin/*"
done

创建和分发 systemd unit 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
cd /opt/k8s/work
cat > docker.service <<"EOF"
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io

[Service]
WorkingDirectory=/mnt/disk0/docker
Environment="PATH=/opt/k8s/bin:/bin:/sbin:/usr/bin:/usr/sbin"
EnvironmentFile=-/run/flannel/docker
ExecStart=/opt/k8s/bin/dockerd --log-level=error $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF
  • EOF 前后有双引号,这样 bash 不会替换文档中的变量,如 $DOCKER_NETWORK_OPTIONS;
  • dockerd 运行时会调用其它 docker 命令,如 docker-proxy,所以需要将 docker 命令所在的目录加到 PATH 环境变量中;
  • flanneld 启动时将网络配置写入 /run/flannel/docker 文件中,dockerd 启动前读取该文件中的环境变量 DOCKER_NETWORK_OPTIONS ,然后设置 docker0 网桥网段;
  • 如果指定了多个 EnvironmentFile 选项,则必须将 /run/flannel/docker 放在最后(确保 docker0 使用 flanneld 生成的 bip 参数);
  • docker 需要以 root 用于运行;
  • docker 从 1.13 版本开始,可能将 iptables FORWARD chain的默认策略设置为DROP,从而导致 ping 其它 Node 上的 Pod IP 失败,遇到这种情况时,需要手动设置策略为 ACCEPT

    1
    $ sudo iptables -P FORWARD ACCEPT

    并且把以下命令写入 /etc/rc.local 文件中,防止节点重启iptables FORWARD chain的默认策略又还原为DROP

    1
    /sbin/iptables -P FORWARD ACCEPT

完整 unit 见 docker.service

分发 systemd unit 文件到所有 worker 机器:

1
2
3
4
5
6
7
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${WORKER_IPS[@]}
do
echo ">>> ${node_ip}"
scp docker.service k8s@${node_ip}:/etc/systemd/system/
done

配置和分发 docker 配置文件

使用国内的仓库镜像服务器以加快 pull image 的速度,同时增加下载的并发数 (需要重启 dockerd 生效):

1
2
3
4
5
6
7
8
9
cd /opt/k8s/work
cat > docker-daemon.json <<EOF
{
"registry-mirrors": ["https://hub-mirror.c.163.com", "https://docker.mirrors.ustc.edu.cn"],
"insecure-registries": ["docker02:35000"],
"max-concurrent-downloads": 20,
}
}
EOF

分发 docker 配置文件到所有 work 节点:

1
2
3
4
5
6
7
8
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${WORKER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo mkdir -p /etc/docker/"
scp docker-daemon.json k8s@${node_ip}:/etc/docker/daemon.json
done

启动 docker 服务

1
2
3
4
5
6
7
8
9
10
source /opt/k8s/bin/environment.sh
for node_ip in ${WORKER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo systemctl stop firewalld && sudo systemctl disable firewalld"
ssh k8s@${node_ip} "sudo /usr/sbin/iptables -F && sudo /usr/sbin/iptables -X && sudo /usr/sbin/iptables -F -t nat && sudo /usr/sbin/iptables -X -t nat"
ssh k8s@${node_ip} "sudo /usr/sbin/iptables -P FORWARD ACCEPT"
ssh k8s@${node_ip} "sudo systemctl daemon-reload && sudo systemctl enable docker && sudo systemctl restart docker"
ssh k8s@${node_ip} "sudo sysctl -p /etc/sysctl.d/kubernetes.conf"
done
  • 关闭 firewalld,否则可能会重复创建 iptables 规则;
  • 清理旧的 iptables rules 和 chains 规则;

检查服务运行状态

1
2
3
4
5
6
source /opt/k8s/bin/environment.sh
for node_ip in ${WORKER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo systemctl status docker|grep Active"
done

确保状态为 active (running),否则查看日志,确认原因:

1
$ journalctl -u docker

检查 docker0 网桥

1
2
3
4
5
6
source /opt/k8s/bin/environment.sh
for node_ip in ${WORKER_IPS[@]}
do
echo ">>> ${node_ip}"
ssh k8s@${node_ip} "sudo /usr/sbin/ip addr show flannel.1 && sudo /usr/sbin/ip addr show docker0"
done

确认各 work 节点的 docker0 网桥和 flannel.1 接口的 IP 处于同一个网段中(如下 172.30.112.0/32 位于 172.30.112.1/21 中):

1
2
3
4
5
6
7
8
9
10
11
12
13
>>> 10.12.11.7
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 9e:ea:06:f8:7a:bc brd ff:ff:ff:ff:ff:ff
inet 10.253.40.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::9cea:6ff:fef8:7abc/64 scope link
valid_lft forever preferred_lft forever
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 02:42:ec:01:98:94 brd ff:ff:ff:ff:ff:ff
inet 10.253.40.1/24 brd 10.253.40.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:ecff:fe01:9894/64 scope link
valid_lft forever preferred_lft forever