Flannel Wireguard 模式

evescn發表於2024-08-22

Flannel WireGuard 模式

一、環境資訊

主機	IP
ubuntu	172.16.94.141

軟體	版本
docker	26.1.4
helm	v3.15.0-rc.2
kind	0.18.0
clab	0.54.2
kubernetes	1.23.4
ubuntu os	Ubuntu 20.04.6 LTS
kernel	5.11.5 核心升級文件

二、安裝服務

`kind` 配置檔案資訊

$ cat install.sh

#!/bin/bash
date
set -v

# 1.prep noCNI env
cat <<EOF | kind create cluster --name=flannel-wireguard --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  # kind 預設使用 rancher cni，cni 我們需要自己建立
  disableDefaultCNI: true
  # 定義節點使用的 pod 網段
  podSubnet: "10.244.0.0/16"
nodes:
  - role: control-plane
  - role: worker
  - role: worker

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.evescn.com"]
    endpoint = ["https://harbor.evescn.com"]
EOF

# 2.install cni
kubectl apply -f ./flannel.yaml

# 3.install necessary tools
# cd /opt/
# curl -o calicoctl -O -L "https://gh.api.99988866.xyz/https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz" 
# tar -zxvf cni-plugins-linux-amd64-v0.9.0.tgz

for i in $(docker ps -a --format "table {{.Names}}" | grep flannel) 
do
    echo $i
    docker cp /opt/bridge $i:/opt/cni/bin/
    docker cp /usr/bin/ping $i:/usr/bin/ping
    docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
    docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done

flannel.yaml

配置檔案

# flannel.yaml
---
kind: Namespace
apiVersion: v1
metadata:
  name: kube-flannel
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "wireguard"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: xtables-lock
          mountPath: /run/xtables.lock
        - name: tun
          mountPath: /dev/net/tun
      volumes:
      - name: tun
        hostPath:
          path: /dev/net/tun
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate

flannel.yaml 引數解釋

Backend.Type
- 含義: 用於指定 flannel 工作模式。
- wireguard: flannel 工作在 wireguard 模式。

安裝 k8s 叢集和 flannel 服務

# ./install.sh

Creating cluster "flannel-wireguard" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-flannel-wireguard"
You can now use your cluster with:

kubectl cluster-info --context kind-flannel-wireguard

Thanks for using kind! 😊

檢視安裝的服務

root@kind:~# kubectl get pods -A
NAMESPACE            NAME                                                      READY   STATUS    RESTARTS   AGE
kube-system          coredns-64897985d-dwbzg                                   1/1     Running   0          7m
kube-system          coredns-64897985d-wxhwb                                   1/1     Running   0          7m
kube-system          etcd-flannel-wireguard-control-plane                      1/1     Running   0          7m15s
kube-system          kube-apiserver-flannel-wireguard-control-plane            1/1     Running   0          7m13s
kube-system          kube-controller-manager-flannel-wireguard-control-plane   1/1     Running   0          7m13s
kube-system          kube-flannel-ds-4pn8v                                     1/1     Running   0          6m43s
kube-system          kube-flannel-ds-62dvf                                     1/1     Running   0          6m43s
kube-system          kube-flannel-ds-dnvbq                                     1/1     Running   0          6m43s
kube-system          kube-proxy-hnww9                                          1/1     Running   0          6m44s
kube-system          kube-proxy-lr88h                                          1/1     Running   0          6m44s
kube-system          kube-proxy-vbhrl                                          1/1     Running   0          7m1s
kube-system          kube-scheduler-flannel-wireguard-control-plane            1/1     Running   0          7m13s
local-path-storage   local-path-provisioner-5ddd94ff66-l9hxt                   1/1     Running   0          7m

`k8s` 叢集安裝 `Pod` 測試網路

root@kind:~# cat cni.yaml

apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: cni
  name: cni
spec:
  #replicas: 1
  selector:
    matchLabels:
      app: cni
  template:
    metadata:
      labels:
        app: cni
    spec:
      containers:
      - image: harbor.dayuan1997.com/devops/nettool:0.9
        name: nettoolbox
        securityContext:
          privileged: true

---
apiVersion: v1
kind: Service
metadata:
  name: serversvc
spec:
  type: NodePort
  selector:
    app: cni
  ports:
  - name: cni
    port: 80
    targetPort: 80
    nodePort: 32000

root@kind:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created

root@kind:~# kubectl run net --image=harbor.dayuan1997.com/devops/nettool:0.9
pod/net created

檢視安裝服務資訊

root@kind:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE                        NOMINATED NODE   READINESS GATES
cni-2zz99   1/1     Running   0          13s   10.244.1.2   flannel-wireguard-worker    <none>           <none>
cni-8c9np   1/1     Running   0          13s   10.244.2.5   flannel-wireguard-worker2   <none>           <none>
net         1/1     Running   0          10s   10.244.1.3   flannel-wireguard-worker    <none>           <none>

root@kind:~# kubectl get svc 
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        7m49s
serversvc    NodePort    10.96.105.87   <none>        80:32000/TCP   11s

三、測試網路

同節點 `Pod` 網路通訊

可以檢視此文件 Flannel UDP 模式中，同節點網路通訊，資料包轉發流程一致

Flannel 同節點通訊透過 l2 網路通訊， 2 層交換機完成

不同節點 `Pod` 網路通訊

Pod 節點資訊

## ip 資訊
root@kind:~# kubectl exec -it net -- ip a l
3: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc noqueue state UP group default 
    link/ether fa:f0:48:65:c5:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.1.3/24 brd 10.244.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f8f0:48ff:fe65:c528/64 scope link 
       valid_lft forever preferred_lft forever

## 路由資訊
root@kind:~# kubectl exec -it net -- ip r s
default via 10.244.1.1 dev eth0 
10.244.0.0/16 via 10.244.1.1 dev eth0 
10.244.1.0/24 dev eth0 proto kernel scope link src 10.244.1.3

Pod 節點所在 Node 節點資訊

root@kind:~# docker exec -it flannel-wireguard-worker bash

## ip 資訊
root@flannel-wireguard-worker:/# ip a l 
2: flannel-wg: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default 
    link/none 
    inet 10.244.1.0/32 scope global flannel-wg
       valid_lft forever preferred_lft forever
3: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:37:f3:a0:8e:a0 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.1/24 brd 10.244.1.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c37:f3ff:fea0:8ea0/64 scope link 
       valid_lft forever preferred_lft forever
4: veth7ff69c99@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc noqueue master cni0 state UP group default 
    link/ether 06:f8:d3:64:ef:e4 brd ff:ff:ff:ff:ff:ff link-netns cni-ebbfc658-feb1-eca0-91aa-ac815a1a8504
    inet6 fe80::4f8:d3ff:fe64:efe4/64 scope link 
       valid_lft forever preferred_lft forever
5: vethd38faacd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc noqueue master cni0 state UP group default 
    link/ether ea:cf:56:fa:c2:da brd ff:ff:ff:ff:ff:ff link-netns cni-e938b0d2-ea2f-f1cf-0823-af51a9ec0f7b
    inet6 fe80::e8cf:56ff:fefa:c2da/64 scope link 
       valid_lft forever preferred_lft forever
9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.4/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::4/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:4/64 scope link 
       valid_lft forever preferred_lft forever

## 路由資訊
root@flannel-wireguard-worker:/# ip r s
default via 172.18.0.1 dev eth0 
10.244.0.0/16 dev flannel-wg scope link 
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.4

Pod 節點進行 ping 包測試，訪問 cni-8c9np Pod 節點

root@kind:~# kubectl exec -it net -- ping 10.244.2.5 -c 1
PING 10.244.2.5 (10.244.2.5): 56 data bytes
64 bytes from 10.244.2.5: seq=0 ttl=62 time=2.926 ms

--- 10.244.2.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 2.926/2.926/2.926 ms

Pod 節點 eth0 網路卡抓包

net~$ tcpdump -pne -i eth0
10:20:20.910458 fa:f0:48:65:c5:28 > 3e:37:f3:a0:8e:a0, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.5: ICMP echo request, id 74, seq 0, length 64
10:20:20.910777 3e:37:f3:a0:8e:a0 > fa:f0:48:65:c5:28, ethertype IPv4 (0x0800), length 98: 10.244.2.5 > 10.244.1.3: ICMP echo reply, id 74, seq 0, length 64

資料包源 mac 地址: fa:f0:48:65:c5:28 為 eth0 網路卡 mac 地址，而目的 mac 地址: 3e:37:f3:a0:8e:a0 為 net Pod 節點 cni0 網路卡對應的網路卡 mac 地址，cni0
網路卡 ip 地址為網路閘道器地址 10.244.2.1 ， flannel 為 2 層網路模式透過路由送往資料到閘道器地址

net~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.244.1.1               ether   3e:37:f3:a0:8e:a0   C                     eth0

而透過 veth pair 可以確定 Pod 節點 eth0 網路卡對應的 veth pair 為 vethd38faacd@if3 網路卡

flannel-wireguard-worker 節點 vethd38faacd 網路卡抓包

root@flannel-wireguard-worker:/# tcpdump -pne -i vethd38faacd
10:21:30.939970 fa:f0:48:65:c5:28 > 3e:37:f3:a0:8e:a0, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.5: ICMP echo request, id 90, seq 0, length 64
10:21:30.940112 3e:37:f3:a0:8e:a0 > fa:f0:48:65:c5:28, ethertype IPv4 (0x0800), length 98: 10.244.2.5 > 10.244.1.3: ICMP echo reply, id 90, seq 0, length 64

因為他們互為 veth pair 所以抓包資訊相同

flannel-wireguard-worker 節點 cni0 網路卡抓包

root@flannel-wireguard-worker:/# tcpdump -pne -i cni0
10:21:30.939970 fa:f0:48:65:c5:28 > 3e:37:f3:a0:8e:a0, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.5: ICMP echo request, id 90, seq 0, length 64
10:21:30.940112 3e:37:f3:a0:8e:a0 > fa:f0:48:65:c5:28, ethertype IPv4 (0x0800), length 98: 10.244.2.5 > 10.244.1.3: ICMP echo reply, id 90, seq 0, length 64

資料包源 mac 地址: fa:f0:48:65:c5:28 為 net Pod 節點 eth0 網路卡 mac 地址，而目的 mac 地址: 3e:37:f3:a0:8e:a0 為 cni0 網路卡 mac 地址

檢視 flannel-wireguard-worker 主機路由資訊，發現並在資料包會在透過 10.244.0.0/16 dev flannel-wg scope link 路由資訊轉發

flannel-wireguard-worker 節點 flannel-wg 網路卡抓包

root@flannel-wireguard-worker:/# tcpdump -pne -i flannel-wg icmp
listening on flannel-wg, link-type RAW (Raw IP), snapshot length 262144 bytes
10:22:36.317228 ip: 10.244.1.3 > 10.244.2.5: ICMP echo request, id 98, seq 0, length 64
10:22:36.317538 ip: 10.244.2.5 > 10.244.1.3: ICMP echo reply, id 98, seq 0, length 64

flannel-wg 網路卡資料包中沒有 mac 層，沒有 mac 地址，只有 ip 層往上的資料層。因為這個介面模式為 link-type RAW (Raw IP)

flannel-wireguard-worker 節點安裝 wireguard-tools 軟體包，檢視 wg 資訊

root@flannel-wireguard-worker:/# apt install -y wireguard-tools

# 檢視 wg 資訊
root@flannel-wireguard-worker:/# ip -d link show
2: flannel-wg: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/none  promiscuity 0 minmtu 0 maxmtu 2147483552 
    wireguard addrgenmode none numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 

root@flannel-wireguard-worker:/# wg
interface: flannel-wg
  public key: 1umq7/lWVbly2PKshmKxLOYq+NI3MAsCYr3OiIVuSG8=
  private key: (hidden)
  listening port: 51820

peer: 8UiTuiJtms7jN89M45aRg/RHu69CSheMM5JLmO30RxM=
  endpoint: 172.18.0.2:51820
  allowed ips: 10.244.2.0/24
  latest handshake: 8 seconds ago
  transfer: 916 B received, 1.18 KiB sent

peer: D67/74RE+5kgAGYJdU7MFgbOZRJ0kNntKoMK9u8dFA4=
  endpoint: 172.18.0.3:51820
  allowed ips: 10.244.0.0/24

flannel-wireguard-worker 節點 eth0 網路卡抓包

資料包外層源 mac 地址: 02:42:ac:12:00:04 為 Node 節點 eth0 網路卡 mac 地址，而目的 mac 地址: 02:42:ac:12:00:02 為對端 Node 節點 eth0 網路卡 mac 地址

root@flannel-wireguard-worker:/# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.18.0.2               ether   02:42:ac:12:00:02   C                     eth0

資料包流向

資料從 pod 服務發出，透過檢視本機路由表，送往 10.244.2.1 網路卡。路由: 10.244.0.0/16 via 10.244.2.1 dev eth0
透過 veth pair 網路卡 vethd38faacd 傳送資料到 flannel-wireguard-worker 主機上，在轉送到 cni0: 10.244.2.1 網路卡
flannel-wireguard-worker 主機檢視自身路由後，會送往 flannel-wg 介面，因為目的地址為 10.244.2.5。路由: 10.244.0.0/16 dev flannel-wg scope link
flannel-wg 介面重新封裝為 wireguard 資料包並送往 eth0 介面
對端 flannel-ipsec-worker2 主機接受到資料包後，發現這個是一個 wireguard 資料包，將資料包核心模組處理。
解封裝後發現目的地址為 10.244.2.5 ，透過檢視本機路由表，送往 cni0 網路卡。路由: 10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
透過 cni0 網路卡 brctl showmacs cni0 mac 資訊，最終會把資料包送到 cni-8c9np 主機

`Service` 網路通訊

可以檢視此文件 Flannel UDP 模式中，Service 網路通訊，資料包轉發流程一致

Flannel IPIP 模式
2024-08-21
模式
Flannel VxLAN DR 模式
2024-08-20
模式
Flannel IPIP DR 模式
2024-08-21
模式
WireGuard 教程：WireGuard 的工作原理
2020-07-06
Wireguard 全互聯模式（full mesh）配置指南
2021-03-01
模式
WireGuard Use Notes
2024-07-17
wireguard進行組網
2024-10-07
深入 K8s 網路原理（一）- Flannel VXLAN 模式分析
2023-12-19
K8S模式
flannel網路概述
2020-11-08
[實踐]wireguard安裝和配置
2024-08-01
詳解：Flannel安裝與配置
2020-06-20
docker k8s 1.3.8 + flannel
2016-10-09
DockerK8S
四、建立覆蓋網路--Flannel
2017-11-15
Kubernetes網路分析之Flannel
2019-10-18
WireGuard結合WsTunnel實現UDP OVER TCP組網
2024-10-21
UDPTCP
WireGuard 教程：使用 DNS-SD 進行 NAT-to-NAT 穿透
2021-02-01
DNS穿透
docker筆記31-網路外掛flannel
2018-10-01
Docker筆記
容器中跨主機的網路方案-flannel
2018-05-01
Docker網路解決方案 - Flannel部署記錄
2017-05-16
Docker
企業網路服務搭建（一）OpenWRT uhttpd ddns firewall wireguard
2024-10-12
httpdDNS
部署 k8s 服務， flannel pod 報錯
2020-04-15
K8S
flannel網路在kubernetes中的運用
2016-08-24
Flannel和Calico網路外掛工作流程對比
2021-02-26
kubernetes應用flannel失敗 Connection refused.
2020-11-29
什麼是 WireGuard？為什麼 Linux 使用者為它瘋狂？
2020-02-21
Linux
flannel vxlan工作基本原理及常見排障方法
2019-06-11
059、安裝配置flannel（2019-03-28 週四）
2019-03-30
容器編排系統K8s之flannel網路模型
2021-01-03
K8S模型
k8s容器互聯-flannel host-gw原理篇
2023-03-16
K8S
kubernetes實踐之四：Flannel網路外掛安裝
2018-03-15
使用multus實現管理網和業務網分離——calico和flannel共存
2021-06-09
ubuntu 16.04 下安裝kubernetes 1.6 之flannel網路模型
2017-09-27
Ubuntu模型
CentOS 7安裝etcd和flannel實現docker跨物理機通訊
2018-08-21
CentOSDocker
深入淺出Kubernetes網路：跨節點網路通訊之Flannel
2019-04-10
【kubernetes】網路虛擬網路卡對veth pair、flannel網路模型實現原理
2020-12-06
AI模型
設計模式-單例模式、多例模式
2020-12-14
設計模式單例
設計模式 #5 （策略模式、代理模式）
2020-09-20
設計模式
建立型模式-單例模式-工廠模式
2021-02-02
模式單例

Flannel Wireguard 模式