Flannel VxLAN DR 模式

evescn發表於2024-08-20

Flannel VxLAN DR 模式

一、環境資訊

主機	IP
ubuntu	172.16.94.141

軟體	版本
docker	26.1.4
helm	v3.15.0-rc.2
kind	0.18.0
clab	0.54.2
kubernetes	1.23.4
ubuntu os	Ubuntu 20.04.6 LTS
kernel	5.11.5 核心升級文件

二、安裝服務

`kind` 配置檔案資訊

$ cat install.sh

#!/bin/bash
date
set -v

# 1.prep noCNI env
cat <<EOF | kind create cluster --name=clab-flannel-vxlan-directrouting --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableDefaultCNI: true
  podSubnet: "10.244.0.0/16"
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.5.10
        node-labels: "rack=rack0"

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.5.11
        node-labels: "rack=rack0"

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.8.10
        node-labels: "rack=rack1"

- role: worker
  kubeadmConfigPatches:
  - |
    kind: JoinConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-ip: 10.1.8.11
        node-labels: "rack=rack1"

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.dayuan1997.com"]
    endpoint = ["https://harbor.dayuan1997.com"]
EOF

# 2.remove taints
controller_node=`kubectl get nodes --no-headers  -o custom-columns=NAME:.metadata.name| grep control-plane`
kubectl taint nodes $controller_node node-role.kubernetes.io/master:NoSchedule-
kubectl get nodes -o wide

# 3.install necessary tools
# cd /opt/
# curl -o calicoctl -O -L "https://gh.api.99988866.xyz/https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz" 
# tar -zxvf cni-plugins-linux-amd64-v0.9.0.tgz

for i in $(docker ps -a --format "table {{.Names}}" | grep flannel) 
do
    echo $i
    docker cp /opt/bridge $i:/opt/cni/bin/
    docker cp /usr/bin/ping $i:/usr/bin/ping
    docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
    docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done

安裝 k8s 叢集

root@kind:~# ./install.sh

Creating cluster "clab-flannel-vxlan-directrouting" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-clab-flannel-vxlan-directrouting"
You can now use your cluster with:

kubectl cluster-info --context kind-clab-flannel-vxlan-directrouting

Have a nice day! 👋

root@kind:~# kubectl get node -o wide
NAME                                             STATUS     ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION          CONTAINER-RUNTIME
clab-flannel-vxlan-directrouting-control-plane   NotReady   control-plane,master   2m48s   v1.23.4   <none>        <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker          NotReady   <none>                 2m14s   v1.23.4   <none>        <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker2         NotReady   <none>                 2m14s   v1.23.4   <none>        <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker3         NotReady   <none>                 2m14s   v1.23.4   <none>        <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10

建立 `clab` 容器環境

建立網橋

root@kind:~# brctl addbr br-leaf0
root@kind:~# ifconfig br-leaf0 up
root@kind:~# brctl addbr br-leaf1
root@kind:~# ifconfig br-leaf1 up

root@kind:~# ip a l
19: br-pool0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:c1:ab:14:8f:99 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e8df:fcff:fed4:3e17/64 scope link 
       valid_lft forever preferred_lft forever
20: br-pool1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:c1:ab:08:cc:9d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::88c:adff:fef2:f336/64 scope link 
       valid_lft forever preferred_lft forever

建立這兩個網橋主要是為了讓 kind 上節點透過虛擬交換機連線到 containerLab ，為什麼不直連線 containerLab ，如果 10.1.5.10/24 使用 vethPair 和 containerLab 進行連線， 10.1.5.11/24 就沒有額外的埠進行連線

`clab` 網路拓撲檔案

# flannel.vxlan.directrouting.clab.yml
name: flannel-vxlan-directrouting
topology:
  nodes:
    gw0:
      kind: linux
      image: vyos/vyos:1.2.8
      cmd: /sbin/init
      binds:
        - /lib/modules:/lib/modules
        - ./startup-conf/gw0-boot.cfg:/opt/vyatta/etc/config/config.boot

    br-pool0:
      kind: bridge
  
    br-pool1:
      kind: bridge


    server1:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 複用節點網路，共享網路名稱空間
      network-mode: container:clab-flannel-vxlan-directrouting-control-plane
      # 配置是為了設定節點上的業務網路卡，同時將預設路由的閘道器進行更改，使用業務網路卡為出介面。
      exec:      
      - ip addr add 10.1.5.10/24 dev net0      
      - ip route replace default via 10.1.5.1

    server2:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 複用節點網路，共享網路名稱空間
      network-mode: container:clab-flannel-vxlan-directrouting-worker
      # 配置是為了設定節點上的業務網路卡，同時將預設路由的閘道器進行更改，使用業務網路卡為出介面。
      exec:
      - ip addr add 10.1.5.11/24 dev net0
      - ip route replace default via 10.1.5.1

    server3:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 複用節點網路，共享網路名稱空間
      network-mode: container:clab-flannel-vxlan-directrouting-worker2
      # 配置是為了設定節點上的業務網路卡，同時將預設路由的閘道器進行更改，使用業務網路卡為出介面。
      exec:
      - ip addr add 10.1.8.10/24 dev net0
      - ip route replace default via 10.1.8.1

    server4:
      kind: linux
      image: harbor.dayuan1997.com/devops/nettool:0.9
      # 複用節點網路，共享網路名稱空間
      network-mode: container:clab-flannel-vxlan-directrouting-worker3
      # 配置是為了設定節點上的業務網路卡，同時將預設路由的閘道器進行更改，使用業務網路卡為出介面。
      exec:
      - ip addr add 10.1.8.11/24 dev net0
      - ip route replace default via 10.1.8.1


  links:
    - endpoints: ["br-pool0:br-pool0-net0", "server1:net0"]
    - endpoints: ["br-pool0:br-pool0-net1", "server2:net0"]
    - endpoints: ["br-pool1:br-pool1-net0", "server3:net0"]
    - endpoints: ["br-pool1:br-pool1-net1", "server4:net0"]

    - endpoints: ["gw0:eth1", "br-pool0:br-pool0-net2"]
    - endpoints: ["gw0:eth2", "br-pool1:br-pool1-net2"]

`VyOS` 配置檔案

gw0-boot.cfg

配置檔案

# ./startup-conf/gw0-boot.cfg
interfaces {
    ethernet eth1 {
        address 10.1.5.1/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    ethernet eth2 {
        address 10.1.8.1/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    loopback lo {
    }
}
# 配置 nat 資訊，gw0 網路下的其他伺服器可以訪問外網
nat {
    source {
        rule 100 {
            outbound-interface eth0
            source {
                address 10.1.0.0/16
            }
            translation {
                address masquerade
            }
        }
    }
}
system {
    config-management {
        commit-revisions 100
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
                plaintext-password ""
            }
            level admin
        }
    }
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "wanloadbalance@3:l2tp@1:pptp@1:ntp@1:mdns@1:webgui@1:conntrack@1:ipsec@5:cluster@1:dhcp-server@5:nat@4:dhcp-relay@2:webproxy@1:system@10:pppoe-server@2:dns-forwarding@1:ssh@1:quagga@7:broadcast-relay@1:qos@1:snmp@1:firewall@5:zone-policy@1:config-management@1:webproxy@2:vrrp@2:conntrack-sync@1" === */
/* Release version: 1.2.8 */

部署服務

# tree -L 2 ./
./
├── flannel.vxlan.directrouting.clab.yml
└── startup-conf
    └── gw0-boot.cfg

# clab deploy -t flannel.vxlan.directrouting.clab.yml

INFO[0000] Containerlab v0.54.2 started                 
INFO[0000] Parsing & checking topology file: clab.yaml  
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU=1500 
INFO[0000] Creating lab directory: /root/wcni-kind/flannel/4-flannel-vxlan-directrouting/clab-flannel-vxlan-directrouting 
WARN[0000] node clab-flannel-vxlan-directrouting-control-plane referenced in namespace sharing not found in topology definition, considering it an external dependency. 
WARN[0000] node clab-flannel-vxlan-directrouting-worker referenced in namespace sharing not found in topology definition, considering it an external dependency. 
WARN[0000] node clab-flannel-vxlan-directrouting-worker2 referenced in namespace sharing not found in topology definition, considering it an external dependency. 
WARN[0000] node clab-flannel-vxlan-directrouting-worker3 referenced in namespace sharing not found in topology definition, considering it an external dependency. 
INFO[0000] Creating container: "gw0"                    
INFO[0001] Created link: gw0:eth1 <--> br-pool0:br-pool0-net2 
INFO[0001] Created link: gw0:eth2 <--> br-pool1:br-pool1-net2 
INFO[0003] Creating container: "server2"                
INFO[0003] Creating container: "server1"                
INFO[0003] Created link: br-pool0:br-pool0-net1 <--> server2:net0 
INFO[0003] Created link: br-pool0:br-pool0-net0 <--> server1:net0 
INFO[0004] Creating container: "server3"                
INFO[0004] Creating container: "server4"                
INFO[0004] Created link: br-pool1:br-pool1-net0 <--> server3:net0 
INFO[0004] Created link: br-pool1:br-pool1-net1 <--> server4:net0 
INFO[0005] Executed command "ip addr add 10.1.5.10/24 dev net0" on the node "server1". stdout: 
INFO[0005] Executed command "ip route replace default via 10.1.5.1" on the node "server1". stdout: 
INFO[0005] Executed command "ip addr add 10.1.8.11/24 dev net0" on the node "server4". stdout: 
INFO[0005] Executed command "ip route replace default via 10.1.8.1" on the node "server4". stdout: 
INFO[0005] Executed command "ip addr add 10.1.8.10/24 dev net0" on the node "server3". stdout: 
INFO[0005] Executed command "ip route replace default via 10.1.8.1" on the node "server3". stdout: 
INFO[0005] Executed command "ip addr add 10.1.5.11/24 dev net0" on the node "server2". stdout: 
INFO[0005] Executed command "ip route replace default via 10.1.5.1" on the node "server2". stdout: 
INFO[0005] Adding containerlab host entries to /etc/hosts file 
INFO[0005] Adding ssh config for containerlab nodes     
INFO[0005] 🎉 New containerlab version 0.56.0 is available! Release notes: https://containerlab.dev/rn/0.56/
Run 'containerlab version upgrade' to upgrade or go check other installation options at https://containerlab.dev/install/ 
+---+------------------------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
| # |                   Name                   | Container ID |                  Image                   | Kind  |  State  |  IPv4 Address  |     IPv6 Address     |
+---+------------------------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+
| 1 | clab-flannel-vxlan-directrouting-gw0     | 2ac429824caa | vyos/vyos:1.2.8                          | linux | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-flannel-vxlan-directrouting-server1 | c3cc494ff542 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
| 3 | clab-flannel-vxlan-directrouting-server2 | bc24767595b4 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
| 4 | clab-flannel-vxlan-directrouting-server3 | 71c0daf892e3 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
| 5 | clab-flannel-vxlan-directrouting-server4 | b8361f60cfe6 | harbor.dayuan1997.com/devops/nettool:0.9 | linux | running | N/A            | N/A                  |
+---+------------------------------------------+--------------+------------------------------------------+-------+---------+----------------+----------------------+

檢查 k8s 叢集資訊

root@kind:~# kubectl get node -o wide
NAME                                             STATUS     ROLES                  AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION          CONTAINER-RUNTIME
clab-flannel-vxlan-directrouting-control-plane   NotReady   control-plane,master   8m38s   v1.23.4   10.1.5.10     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker          NotReady   <none>                 8m4s    v1.23.4   10.1.5.11     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker2         NotReady   <none>                 8m4s    v1.23.4   10.1.8.10     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker3         NotReady   <none>                 8m4s    v1.23.4   10.1.8.11     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10

# 檢視 node 節點 ip 資訊
root@kind:~# docker exec -it clab-flannel-vxlan-directrouting-control-plane ip a l
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.5/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::5/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:5/64 scope link 
       valid_lft forever preferred_lft forever
22: net0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default 
    link/ether aa:c1:ab:ab:fa:3b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.5.10/24 scope global net0
       valid_lft forever preferred_lft forever
    inet6 fe80::a8c1:abff:feab:fa3b/64 scope link 
       valid_lft forever preferred_lft forever

# 檢視 node 節點路由資訊
root@kind:~# docker exec -it clab-flannel-vxlan-directrouting-control-plane ip r s
default via 10.1.5.1 dev net0 
10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10 
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.5

檢視 k8s 叢集發現 node 節點 ip 地址分配了，登陸容器檢視到了新的 ip 地址，並且預設路由資訊調整為了 10.1.5.0/24 dev net0 proto kernel scope link src 10.1.5.10

安裝 `flannel` 服務

flannel.yaml

配置檔案

# flannel.yaml
---
kind: Namespace
apiVersion: v1
metadata:
  name: kube-flannel
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan",
        "DirectRouting": true
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
       #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
       #image: 192.168.2.100:5000/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
       #image: flannelcni/flannel:v0.19.2 for ppc64le and mips64le (dockerhub limitations may apply)
       #image: 192.168.2.100:5000/rancher/mirrored-flannelcni-flannel:v0.19.2
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
       #image: flannelcni/flannel:v0.19.2 for ppc64le and mips64le (dockerhub limitations may apply)
       #image: 192.168.2.100:5000/rancher/mirrored-flannelcni-flannel:v0.19.2
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: xtables-lock
          mountPath: /run/xtables.lock
        - name: tun
          mountPath: /dev/net/tun
      volumes:
      - name: tun
        hostPath:
          path: /dev/net/tun
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate

flannel.yaml 引數解釋

Backend.Type
- 含義: 用於指定 flannel 工作模式。
- vxlan: flannel 工作在 vxlan 模式。
Backend.DirectRouting
- 含義: 用於指定 vxlan 模式同網段使用 host-gw 模式。
- true: vxlan 模式中，同網段的 node 節點之間資料轉發使用 host-gw 模式。

root@kind:~# kubectl apply -f flannel.yaml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

檢視 k8s 叢集和 flannel 服務

root@kind:~# kubectl get node -o wide
NAME                                             STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION          CONTAINER-RUNTIME
clab-flannel-vxlan-directrouting-control-plane   Ready    control-plane,master   13m   v1.23.4   10.1.5.10     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker          Ready    <none>                 13m   v1.23.4   10.1.5.11     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker2         Ready    <none>                 13m   v1.23.4   10.1.8.10     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10
clab-flannel-vxlan-directrouting-worker3         Ready    <none>                 13m   v1.23.4   10.1.8.11     <none>        Ubuntu 21.10   5.11.5-051105-generic   containerd://1.5.10

檢視安裝的服務

root@kind:~# kubectl get pods -A
NAMESPACE            NAME                                                                     READY   STATUS    RESTARTS   AGE
kube-system          coredns-64897985d-8dt79                                                  1/1     Running   0          13m
kube-system          coredns-64897985d-sprng                                                  1/1     Running   0          13m
kube-system          etcd-clab-flannel-vxlan-directrouting-control-plane                      1/1     Running   0          14m
kube-system          kube-apiserver-clab-flannel-vxlan-directrouting-control-plane            1/1     Running   0          14m
kube-system          kube-controller-manager-clab-flannel-vxlan-directrouting-control-plane   1/1     Running   0          14m
kube-system          kube-flannel-ds-6b4sk                                                    1/1     Running   0          3m52s
kube-system          kube-flannel-ds-6vqf2                                                    1/1     Running   0          3m52s
kube-system          kube-flannel-ds-76czr                                                    1/1     Running   0          3m52s
kube-system          kube-flannel-ds-xhw4f                                                    1/1     Running   0          3m52s
kube-system          kube-proxy-d9kjr                                                         1/1     Running   0          13m
kube-system          kube-proxy-fl2v7                                                         1/1     Running   0          13m
kube-system          kube-proxy-mgv2m                                                         1/1     Running   0          13m
kube-system          kube-proxy-xcl4n                                                         1/1     Running   0          13m
kube-system          kube-scheduler-clab-flannel-vxlan-directrouting-control-plane            1/1     Running   0          14m
local-path-storage   local-path-provisioner-5ddd94ff66-lk6kc                                  1/1     Running   0          13m

`k8s` 叢集安裝 `Pod` 測試網路

root@kind:~# cat cni.yaml

apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: cni
  name: cni
spec:
  #replicas: 1
  selector:
    matchLabels:
      app: cni
  template:
    metadata:
      labels:
        app: cni
    spec:
      containers:
      - image: harbor.dayuan1997.com/devops/nettool:0.9
        name: nettoolbox
        securityContext:
          privileged: true

---
apiVersion: v1
kind: Service
metadata:
  name: serversvc
spec:
  type: NodePort
  selector:
    app: cni
  ports:
  - name: cni
    port: 80
    targetPort: 80
    nodePort: 32000

root@kind:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created

root@kind:~# kubectl run net --image=harbor.dayuan1997.com/devops/nettool:0.9
pod/net created

檢視安裝服務資訊

root@kind:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE                                             NOMINATED NODE   READINESS GATES
cni-5sxkl   1/1     Running   0          18s   10.244.2.2   clab-flannel-vxlan-directrouting-worker2         <none>           <none>
cni-vwgqx   1/1     Running   0          18s   10.244.0.2   clab-flannel-vxlan-directrouting-control-plane   <none>           <none>
cni-w79ph   1/1     Running   0          18s   10.244.3.5   clab-flannel-vxlan-directrouting-worker3         <none>           <none>
cni-x2rxr   1/1     Running   0          18s   10.244.2.2   clab-flannel-vxlan-directrouting-worker          <none>           <none>
net         1/1     Running   0          13s   10.244.1.3   clab-flannel-vxlan-directrouting-worker2         <none>           <none>

root@kind:~# kubectl get svc 
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP        15m
serversvc    NodePort    10.96.50.15   <none>        80:32000/TCP   27s

三、測試網路

同節點 `Pod` 網路通訊

可以檢視此文件 Flannel UDP 模式中，同節點網路通訊，資料包轉發流程一致

Flannel 同節點通訊透過 l2 網路， 2 層交換機完成

跨節點同 `Node` 網段 `Pod` 網路通訊

可以檢視此文件 Flannel HOST-GW 模式中，不同節點 Pod 網路通訊，資料包轉發流程一致

跨節點不同 `Node` 網段 `Pod` 網路通訊

Pod 節點資訊

## ip 資訊
root@kind:~# kubectl exec -it net -- ip a l
3: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9450 qdisc noqueue state UP group default 
    link/ether 26:a8:8d:74:59:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.1.3/24 brd 10.244.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::24a8:8dff:fe74:593f/64 scope link 
       valid_lft forever preferred_lft forever

## 路由資訊
root@kind:~# kubectl exec -it net -- ip r s
default via 10.244.1.1 dev eth0 
10.244.0.0/16 via 10.244.1.1 dev eth0 
10.244.1.0/24 dev eth0 proto kernel scope link src 10.244.1.3

Pod 節點所在 Node 節點資訊

root@kind:~# docker exec -it clab-flannel-vxlan-directrouting-worker2 bash

## ip 資訊
root@clab-flannel-vxlan-directrouting-worker2:/# ip a l 
2: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9450 qdisc noqueue state UNKNOWN group default 
    link/ether ca:7d:71:63:e2:a2 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::c87d:71ff:fe63:e2a2/64 scope link 
       valid_lft forever preferred_lft forever
3: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9450 qdisc noqueue state UP group default qlen 1000
    link/ether d2:3c:81:f3:9c:6f brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.1/24 brd 10.244.2.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::d03c:81ff:fef3:9c6f/64 scope link 
       valid_lft forever preferred_lft forever
4: vethd3fcbe42@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9450 qdisc noqueue master cni0 state UP group default 
    link/ether 92:01:1c:7e:82:65 brd ff:ff:ff:ff:ff:ff link-netns cni-baf1f367-320c-1afa-a624-88ba3fc51a48
    inet6 fe80::9001:1cff:fe7e:8265/64 scope link 
       valid_lft forever preferred_lft forever
5: veth25e540eb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9450 qdisc noqueue master cni0 state UP group default 
    link/ether b6:10:4f:07:c1:0e brd ff:ff:ff:ff:ff:ff link-netns cni-07b75239-d571-5b75-5257-e3c3e6ed5c01
    inet6 fe80::b410:4fff:fe07:c10e/64 scope link 
       valid_lft forever preferred_lft forever
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::3/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:3/64 scope link 
       valid_lft forever preferred_lft forever
28: net0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default 
    link/ether aa:c1:ab:84:fd:ec brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.1.8.10/24 scope global net0
       valid_lft forever preferred_lft forever
    inet6 fe80::a8c1:abff:fe84:fdec/64 scope link 
       valid_lft forever preferred_lft forever

## 路由資訊
root@clab-flannel-vxlan-directrouting-worker2:/# ip r s
default via 10.1.8.1 dev net0 
10.1.8.0/24 dev net0 proto kernel scope link src 10.1.8.10 
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink 
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 
10.244.3.0/24 via 10.1.8.11 dev net0 
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.3

Pod 節點進行 ping 包測試，訪問 cni-x2rxr Pod 節點

root@kind:~# kubectl exec -it net -- ping 10.244.2.2 -c 1
PING 10.244.2.2 (10.244.2.2): 56 data bytes
64 bytes from 10.244.2.2: seq=0 ttl=62 time=1.402 ms

--- 10.244.2.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.402/1.402/1.402 ms

Pod 節點 eth0 網路卡抓包

net~$ tcpdump -pne -i eth0
08:23:24.189370 26:a8:8d:74:59:3f > d2:3c:81:f3:9c:6f, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 54, seq 14, length 64
08:23:24.189496 d2:3c:81:f3:9c:6f > 26:a8:8d:74:59:3f, ethertype IPv4 (0x0800), length 98: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 54, seq 14, length 64

資料包源 mac 地址: 26:a8:8d:74:59:3f 為 eth0 網路卡 mac 地址，而目的 mac 地址: d2:3c:81:f3:9c:6f 為 net Pod 節點 cni0 網路卡對應的網路卡 mac 地址，cni0
網路卡 ip 地址為網路閘道器地址 10.244.2.1 ， flannel 為 2 層網路模式透過路由送往資料到閘道器地址

net~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.244.1.1               ether   d2:3c:81:f3:9c:6f   C                     eth0

而透過 veth pair 可以確定 Pod 節點 eth0 網路卡對應的 veth pair 為 veth25e540eb@if3 網路卡

clab-flannel-vxlan-directrouting-worker2 節點 veth25e540eb 網路卡抓包

root@clab-flannel-vxlan-directrouting-worker2:/# tcpdump -pne -i veth25e540eb
08:26:28.784300 26:a8:8d:74:59:3f > 42:a5:47:f5:4b:9f, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.1.2: ICMP echo request, id 23, seq 1576, length 64
08:26:28.784371 42:a5:47:f5:4b:9f > 26:a8:8d:74:59:3f, ethertype IPv4 (0x0800), length 98: 10.244.1.2 > 10.244.1.3: ICMP echo reply, id 23, seq 1576, length 64

因為他們互為 veth pair 所以抓包資訊相同

clab-flannel-vxlan-directrouting-worker2 節點 cni0 網路卡抓包

root@clab-flannel-vxlan-directrouting-worker2:/# tcpdump -pne -i cni0
08:26:45.256754 26:a8:8d:74:59:3f > d2:3c:81:f3:9c:6f, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 54, seq 215, length 64
08:26:45.256878 d2:3c:81:f3:9c:6f > 26:a8:8d:74:59:3f, ethertype IPv4 (0x0800), length 98: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 54, seq 215, length 64

資料包源 mac 地址: 26:a8:8d:74:59:3f 為 net Pod 節點 eth0 網路卡 mac 地址，而目的 mac 地址: d2:3c:81:f3:9c:6f 為 cni0 網路卡 mac 地址

檢視 clab-flannel-vxlan-directrouting-worker2 主機路由資訊，發現並在資料包會在透過 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink 路由資訊轉發

clab-flannel-vxlan-directrouting-worker2 節點 flannel.1 網路卡抓包

root@clab-flannel-vxlan-directrouting-worker2:/# tcpdump -pne -i flannel.1 icmp
08:30:06.337766 ca:7d:71:63:e2:a2 > 3e:0a:47:72:56:37, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 54, seq 416, length 64
08:30:06.337895 3e:0a:47:72:56:37 > ca:7d:71:63:e2:a2, ethertype IPv4 (0x0800), length 98: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 54, seq 416, length 64

資料包源 mac 地址: ca:7d:71:63:e2:a2 為 clab-flannel-vxlan-directrouting-worker2 節點 flannel.1 網路卡 mac 地址，而目的 mac 地址: 3e:0a:47:72:56:37 是誰的 mac 地址？檢視宿主機 arp 資訊，目的 mac 地址: 3e:0a:47:72:56:37 是 10.244.2.0 網段 mac 地址，這個地址如何學習到的？可以檢視 FDB 自動學習繫結過程檢測

root@clab-flannel-vxlan-directrouting-worker2:/# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.244.2.0               ether   3e:0a:47:72:56:37   CM                    flannel.1

檢視 fdb 資訊

root@clab-flannel-vxlan-directrouting-worker2:/# bridge fdb show
3e:0a:47:72:56:37 dev flannel.1 dst 10.1.5.11 self permanent

透過檢視 fdb 表資訊可以看到 3e:0a:47:72:56:37 dev flannel.1 dst 10.1.5.11 self permanent 標示了 mac 地址 3e:0a:47:72:56:37 所在的主機為 10.1.5.11 ，因此 vxlan 封裝的外層資料的目的 ip 是使用 10.1.5.11，而 vxlan 封裝的外層資料的源的 ip 是本機 net0 網路卡 ip

clab-flannel-vxlan-directrouting-worker2 節點 net0 網路卡抓包

request 資料包資訊資訊
- icmp 包中，外部 mac 資訊中，源 mac: aa:c1:ab:84:fd:ec 為 clab-flannel-vxlan-directrouting-worker2 的 net0 網路卡 mac ，目的 mac: aa:c1:ab:d2:3d:96 為 gw0 主機的 eth2 網路卡 mac。使用 udp 協議 8472 埠進行資料傳輸，vxlan 資訊中 vni 為 1。
- icmp 包中，內部 mac 資訊中，源 mac: ca:7d:71:63:e2:a2 為 clab-flannel-vxlan-directrouting-worker2 的 flannel.1 網路卡 mac ，目的 mac: 3e:0a:47:72:56:37 為對端 clab-flannel-vxlan-directrouting-worker 主機的 flannel.1 網路卡 mac。
clab-flannel-vxlan-directrouting-worker2 節點 vxlan 資訊

root@clab-flannel-vxlan-directrouting-worker2:/# ip -d link show
2: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether ca:7d:71:63:e2:a2 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    vxlan id 1 local 10.1.8.10 dev net0 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

資料包流向

資料從 pod 服務發出，透過檢視本機路由表，送往 10.244.1.1 網路卡。路由: 10.244.0.0/16 via 10.244.1.1 dev eth0
透過 veth pair 網路卡 veth25e540eb 傳送資料到 clab-flannel-vxlan-directrouting-worker2 主機上，在轉送到 cni0: 10.244.1.1 網路卡
clab-flannel-vxlan-directrouting-worker2 主機檢視自身路由後，會送往 flannel.1 介面，因為目的地址為 10.244.2.2。路由: 10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
flannel.1 介面為 vxlan 模式，會重新封裝資料包，封裝資訊檢視 arp 資訊 10.244.2.0 ether 3e:0a:47:72:56:37 CM flannel.1、fdb 資訊 3e:0a:47:72:56:37 dev flannel.1 dst 10.1.5.11 self permanent
資料封裝完成後，會送往 net0 介面，並送往 gw0 主機。
gw0 主機接受到資料包後，發目的地址為 10.1.5.11，會檢視路由表，送往 eth1 介面。路由: 10.1.5.0/24 dev eth1 proto kernel scope link src 10.1.5.1
透過 gw0 主機 eth1 網路卡重新封裝資料包後，最終會把資料包送到 clab-flannel-vxlan-directrouting-worker 主機
對端 clab-flannel-vxlan-directrouting-worker 主機接受到資料包後，發現這個是本機資料包資訊，在解封裝資料包過程中發現這是一個送往 UDP 8472 介面的 vxlan 資料包，將資料包交給監聽 UDP 8472 埠的應用程式或核心模組處理。
解封裝後發現內部的資料包，目的地址為 10.244.2.2 ，透過檢視本機路由表，送往 cni0 網路卡。路由: 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1
透過 cni0 網路卡 brctl showmacs cni0 mac 資訊，最終會把資料包送到 cni-x2rxr 主機

`Service` 網路通訊

可以檢視此文件 Flannel UDP 模式中，Service 網路通訊，資料包轉發流程一致

Flannel IPIP DR 模式
2024-08-21
模式
深入 K8s 網路原理（一）- Flannel VXLAN 模式分析
2023-12-19
K8S模式
Flannel IPIP 模式
2024-08-21
模式
Flannel Wireguard 模式
2024-08-22
模式
flannel vxlan工作基本原理及常見排障方法
2019-06-11
LVS - 直接路由(DR)模式示例
2018-05-17
路由模式
LVS DR模式單網段案例
2020-11-06
模式
LVS-DR直連路由模式
2020-10-27
路由模式
[VXLAN]
2024-08-07
VXLAN
2024-05-25
LVS 負載均衡之 VS/DR 模式
2019-08-05
負載模式
什麼是 VxLAN？
2019-07-04
LVS + Keepalived + Nginx基於DR模式構建高可用方案
2019-08-01
Nginx模式
VxLAN協議詳解
2020-05-23
協議
kube-flannel.yml
2020-11-19
flannel網路概述
2020-11-08
OpenvSwitch系列之八 vxlan隧道
2020-09-08
作業八-實現DR跨網段模式和nfs自動掛載
2020-11-08
模式NFS
Kubernetes網路分析之Flannel
2019-10-18
DR和BDR詳解
2019-10-11
LVS的DR模型配置
2019-04-27
模型
詳解：Flannel安裝與配置
2020-06-20
Dr.Elephant mysql connection error
2020-09-21
MySqlError
網路虛擬化VXLAN網路架構
2020-04-20
架構
vxlan基本原理及裸搭過程
2024-06-21
058、flannel概述（2019-03-27 週三）
2019-03-30
kubernetes應用flannel失敗 Connection refused.
2020-11-29
docker筆記31-網路外掛flannel
2018-10-01
Docker筆記
容器中跨主機的網路方案-flannel
2018-05-01
一次DB2 DR演練
2018-10-13
DB2
059、安裝配置flannel（2019-03-28 週四）
2019-03-30
部署 k8s 服務， flannel pod 報錯
2020-04-15
K8S
Kubernetes 叢集網路：Flannel 與 Calico 的區別
2024-11-30
快速簡單的瞭解VLAN（VXLAN）和埠鏈路型別
2021-07-01
型別
DB，DR倒換後的一點問題
2019-07-10
C4DR26: Maxon Cinema 4D
2022-05-14
Maxon Cinema 4D C4DR26
2022-05-09
Cinema 4D for mac(C4DR26)
2022-05-26
Mac

Flannel VxLAN DR 模式