環境條件
- 升級核心
- 修改核心引數
- 配置cgroup
- 安裝CRI
核心引數
- 配置網路流量轉發
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF
- 啟用模組
modprobe overlay modprobe br_netfilter
- 調整核心引數net.ipv4.ip_forward
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF
- 啟用核心引數without reboot
[23:07:40] root@kubernetes-template-os:~ # sysctl --system * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... kernel.yama.ptrace_scope = 0 * Applying /usr/lib/sysctl.d/50-coredump.conf ... kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e kernel.core_pipe_limit = 16 * Applying /usr/lib/sysctl.d/50-default.conf ... kernel.sysrq = 16 kernel.core_uses_pid = 1 kernel.kptr_restrict = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.promote_secondaries = 1 net.core.default_qdisc = fq_codel fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ... net.core.optmem_max = 81920 * Applying /usr/lib/sysctl.d/50-pid-max.conf ... kernel.pid_max = 4194304 * Applying /etc/sysctl.d/60-gce-network-security.conf ... net.ipv4.tcp_syncookies = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 net.ipv4.conf.all.secure_redirects = 1 net.ipv4.conf.default.secure_redirects = 1 net.ipv4.ip_forward = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.conf.all.log_martians = 1 net.ipv4.conf.default.log_martians = 1 kernel.randomize_va_space = 2 kernel.panic = 10 * Applying /etc/sysctl.d/99-sysctl.conf ... * Applying /etc/sysctl.d/k8s.conf ... net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 * Applying /etc/sysctl.conf ... - 結果驗證
[23:09:10] root@kubernetes-template-os:~ # sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 ## 驗證 br_netfilter, overlay [23:09:14] root@kubernetes-template-os:~ # lsmod | grep br_netfilter br_netfilter 32768 0 bridge 393216 1 br_netfilter [23:10:06] root@kubernetes-template-os:~ # lsmod | grep overlay overlay 188416 0
配置cgroup驅動
在Linux中CGroup的功能是用於限制分配給程序的資源，Kubernetes中的Kubelet與CRI都需要對cgroup控制組強勢執行Pod的資源控制
注意：kubelet與CRI必須使用同一個cgroup，避免資源壓力大造成的不穩定
Kubernetes支援二種cgroup
cgroupfs
-
- cgroupfs是kubelet預設使用的cgroup驅動，但是當使用systemd初始化系統時，不建議使用cgroupfs
systemd- systemd 與 cgroup 整合緊密，並將為每個 systemd 單元分配一個 cgroup。 因此，如果你
systemd用作初始化系統，同時使用cgroupfs驅動，則系統中會存在兩個不同的 cgroup 管理器 - 修改kubelet的cgroup驅動，編輯
KubeletConfiguration的cgroupDriver選項，並將其設定為systemd。例如：apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration ... cgroupDriver: systemd
- systemd 與 cgroup 整合緊密，並將為每個 systemd 單元分配一個 cgroup。 因此，如果你
說明：從 v1.22 開始，在使用 kubeadm 建立叢集時，如果使用者沒有在
KubeletConfiguration下設定cgroupDriver欄位，kubeadm 預設使用systemd
CRI安裝
CRI版本說明
Kubernetes 從 1.26 版本開始僅適用於 v1 版本的容器執行時（CRI）API。早期版本預設為 v1 版本， 但是如果容器執行時不支援 v1 版本的 API， 則 kubelet 會回退到使用（已棄用的）v1alpha2 版本的 API。
CRI支援選擇
- containerd
- CRI-O
- Docker Engine
- Mirantis Container Runtime
Containerd配置
選擇使用containerd作為CRI，官方文件提供了三種安裝方式
- 官方二進位制，https://github.com/containerd/containerd/blob/main/docs/getting-started.md#option-1-from-the-official-binaries，共需要安裝三個元件
- Download the
containerd-<VERSION>-<OS>-<ARCH>.tar.gzarchive from https://github.com/containerd/containerd/releases , verify its sha256sum, and extract it under/usr/local - Download the
runc.<ARCH>binary from https://github.com/opencontainers/runc/releases , verify its sha256sum, and install it as/usr/local/sbin/runc. - Download the
cni-plugins-<OS>-<ARCH>-<VERSION>.tgzarchive from https://github.com/containernetworking/plugins/releases , verify its sha256sum, and extract it under/opt/cni/bin
- Download the
- 使用apt-get 或者 dnf（docker官方YUM源）
- Docker官方YUM源的Containerd整合了runcf，但是不包含cni-plugins
- 原始碼安裝
YUM安裝
- 配置YUM倉庫
[01:29:01] root@kubernetes-template-os:~ # dnf install -y yum-utils Last metadata expiration check: 1:32:41 ago on Sun 15 Oct 2023 11:56:31 PM CST. Dependencies resolved. ============================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================ Installing: yum-utils noarch 4.0.21-23.el8 baseos 80 k Transaction Summary ============================================================================================================================================================================================================ Install 1 Package Total download size: 80 k Installed size: 23 k Downloading Packages: yum-utils-4.0.21-23.el8.noarch.rpm 760 kB/s | 80 kB 00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 107 kB/s | 80 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : yum-utils-4.0.21-23.el8.noarch 1/1 Running scriptlet: yum-utils-4.0.21-23.el8.noarch 1/1 Verifying : yum-utils-4.0.21-23.el8.noarch 1/1 Installed: yum-utils-4.0.21-23.el8.noarch Complete! [01:33:07] root@kubernetes-template-os:~ # yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
- 安裝最新版本Docker engine
[01:33:16] root@kubernetes-template-os:~ # dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin Docker CE Stable - x86_64 509 kB/s | 50 kB 00:00 Last metadata expiration check: 0:00:01 ago on Mon 16 Oct 2023 01:38:21 AM CST. Dependencies resolved. ============================================================================================================================================================================================================ Package Architecture Version Repository Size ============================================================================================================================================================================================================ Installing: containerd.io x86_64 1.6.24-3.1.el8 docker-ce-stable 34 M docker-buildx-plugin x86_64 0.11.2-1.el8 docker-ce-stable 13 M docker-ce x86_64 3:24.0.6-1.el8 docker-ce-stable 24 M docker-ce-cli x86_64 1:24.0.6-1.el8 docker-ce-stable 7.2 M docker-compose-plugin x86_64 2.21.0-1.el8 docker-ce-stable 13 M Installing dependencies: checkpolicy x86_64 2.9-1.el8 baseos 348 k container-selinux noarch 2:2.221.0-1.module_el8+653+feef7bfe appstream 69 k docker-ce-rootless-extras x86_64 24.0.6-1.el8 docker-ce-stable 4.9 M fuse-common x86_64 3.3.0-17.el8 baseos 22 k fuse-overlayfs x86_64 1.12-1.module_el8+454+d7ef4b8d appstream 70 k fuse3 x86_64 3.3.0-17.el8 baseos 54 k fuse3-libs x86_64 3.3.0-17.el8 baseos 95 k libcgroup x86_64 0.41-19.el8 baseos 70 k libslirp x86_64 4.4.0-1.module_el8+487+8e42a277 appstream 70 k policycoreutils-python-utils noarch 2.9-24.el8 baseos 260 k python3-audit x86_64 3.0.7-5.el8 baseos 87 k python3-libsemanage x86_64 2.9-9.el8 baseos 128 k python3-policycoreutils noarch 2.9-24.el8 baseos 2.3 M python3-setools x86_64 4.3.0-5.el8 baseos 627 k slirp4netns x86_64 1.2.1-1.module_el8+661+d1afb926 appstream 56 k Enabling module streams: container-tools rhel8 Transaction Summary ============================================================================================================================================================================================================ Install 20 Packages
- 也可以安裝指定版本
[01:41:23] root@kubernetes-template-os:~ # dnf list docker-ce --showduplicates | sort -r Last metadata expiration check: 0:03:11 ago on Mon 16 Oct 2023 01:38:21 AM CST. Installed Packages docker-ce.x86_64 3:24.0.6-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.6-1.el8 @docker-ce-stable docker-ce.x86_64 3:24.0.5-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.4-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.3-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.2-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.1-1.el8 docker-ce-stable docker-ce.x86_64 3:24.0.0-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.6-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.5-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.4-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.3-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.2-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.1-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.0-1.el8 docker-ce-stable dnf install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io docker-buildx-plugin docker-compose-plugin
Containerd配置檔案
- 生成預設配置城檔案
[03:04:54] root@kubernetes-template-os:~ # containerd config default > /etc/containerd/config.toml
- 配置systemd cgroup
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] ... [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] SystemdCgroup = true
安裝Kubernetes軟體包
https://v1-27.docs.kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl
- 配置YUM源
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl EOF ## 檢視YUM源 [03:11:26] root@kubernetes-template-os:~ # cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl EOF ## 檢視YUM源 [03:11:26] root@kubernetes-template-os:~ # cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl
- 關閉selinux
- 方法一，setenforce 0
- 方法二，sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config
- 安裝kubelet、kubectl、kubeadm
dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable --now kubelet
- Kubernetes shell自動補全
echo 'source <(kubectl completion bash)' >>~/.bashrc echo 'source <(kubeadm completion bash)' >>~/.bashrc source ~/.bashrc