Flannel IPIP 模式

evescn發表於2024-08-21

Flannel IPIP 模式

一、環境資訊

主機	IP
ubuntu	172.16.94.141

軟體	版本
docker	26.1.4
helm	v3.15.0-rc.2
kind	0.18.0
clab	0.54.2
kubernetes	1.23.4
ubuntu os	Ubuntu 20.04.6 LTS
kernel	5.11.5 核心升級文件

二、安裝服務

`kind` 配置檔案資訊

$ cat install.sh

#!/bin/bash
date
set -v

# 1.prep noCNI env
cat <<EOF | kind create cluster --name=flannel-vxlan --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  # kind 預設使用 rancher cni，cni 我們需要自己建立
  disableDefaultCNI: true
  # 定義節點使用的 pod 網段
  podSubnet: "10.244.0.0/16"
nodes:
  - role: control-plane
  - role: worker
  - role: worker

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.evescn.com"]
    endpoint = ["https://harbor.evescn.com"]
EOF

# 2.install cni
kubectl apply -f ./flannel.yaml

# 3.install necessary tools
# cd /opt/
# curl -o calicoctl -O -L "https://gh.api.99988866.xyz/https://github.com/containernetworking/plugins/releases/download/v0.9.0/cni-plugins-linux-amd64-v0.9.0.tgz" 
# tar -zxvf cni-plugins-linux-amd64-v0.9.0.tgz

for i in $(docker ps -a --format "table {{.Names}}" | grep flannel) 
do
    echo $i
    docker cp /opt/bridge $i:/opt/cni/bin/
    docker cp /usr/bin/ping $i:/usr/bin/ping
    docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
    docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done

flannel.yaml

配置檔案

# flannel.yaml
---
kind: Namespace
apiVersion: v1
metadata:
  name: kube-flannel
  labels:
    pod-security.kubernetes.io/enforce: privileged
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "ipip"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        command:
        - cp
        args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: /opt/cni/bin
      - name: install-cni
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: harbor.dayuan1997.com/devops/rancher/mirrored-flannelcni-flannel:v0.19.2
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
        - name: xtables-lock
          mountPath: /run/xtables.lock
        - name: tun
          mountPath: /dev/net/tun
      volumes:
      - name: tun
        hostPath:
          path: /dev/net/tun
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni-plugin
        hostPath:
          path: /opt/cni/bin
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate

flannel.yaml 引數解釋

Backend.Type
- 含義: 用於指定 flannel 工作模式。
- ipip: flannel 工作在 ipip 模式。

安裝 k8s 叢集和 flannel 服務

# ./install.sh

Creating cluster "flannel-ipip" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-flannel-ipip"
You can now use your cluster with:

kubectl cluster-info --context kind-flannel-ipip

Have a nice day! 👋

檢視安裝的服務

root@kind:~# kubectl get pods -A -w
NAMESPACE            NAME                                                 READY   STATUS    RESTARTS   AGE
kube-system          coredns-64897985d-m2fln                              1/1     Running   0          3m45s
kube-system          coredns-64897985d-wcxz8                              1/1     Running   0          3m45s
kube-system          etcd-flannel-ipip-control-plane                      1/1     Running   0          4m
kube-system          kube-apiserver-flannel-ipip-control-plane            1/1     Running   0          4m
kube-system          kube-controller-manager-flannel-ipip-control-plane   1/1     Running   0          4m4s
kube-system          kube-flannel-ds-845gs                                1/1     Running   0          3m26s
kube-system          kube-flannel-ds-gm7n2                                1/1     Running   0          3m26s
kube-system          kube-flannel-ds-k5lkj                                1/1     Running   0          3m26s
kube-system          kube-proxy-4hngw                                     1/1     Running   0          3m26s
kube-system          kube-proxy-v8zmv                                     1/1     Running   0          3m26s
kube-system          kube-proxy-znb29                                     1/1     Running   0          3m46s
kube-system          kube-scheduler-flannel-ipip-control-plane            1/1     Running   0          4m1s
local-path-storage   local-path-provisioner-5ddd94ff66-x8jjt              1/1     Running   0          3m45s

`k8s` 叢集安裝 `Pod` 測試網路

root@kind:~# cat cni.yaml

apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: cni
  name: cni
spec:
  #replicas: 1
  selector:
    matchLabels:
      app: cni
  template:
    metadata:
      labels:
        app: cni
    spec:
      containers:
      - image: harbor.dayuan1997.com/devops/nettool:0.9
        name: nettoolbox
        securityContext:
          privileged: true

---
apiVersion: v1
kind: Service
metadata:
  name: serversvc
spec:
  type: NodePort
  selector:
    app: cni
  ports:
  - name: cni
    port: 80
    targetPort: 80
    nodePort: 32000

root@kind:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created

root@kind:~# kubectl run net --image=harbor.dayuan1997.com/devops/nettool:0.9
pod/net created

檢視安裝服務資訊

root@kind:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
cni-f58n6   1/1     Running   0          19s   10.244.1.2   flannel-ipip-worker2   <none>           <none>
cni-p548p   1/1     Running   0          19s   10.244.2.2   flannel-ipip-worker    <none>           <none>
net         1/1     Running   0          14s   10.244.1.3   flannel-ipip-worker2   <none>           <none>

root@kind:~# kubectl get svc 
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        4m46s
serversvc    NodePort    10.96.82.171   <none>        80:32000/TCP   30s

三、測試網路

同節點 `Pod` 網路通訊

可以檢視此文件 Flannel UDP 模式中，同節點網路通訊，資料包轉發流程一致

Flannel 同節點通訊透過 l2 網路通訊， 2 層交換機完成

不同節點 `Pod` 網路通訊

Pod 節點資訊

## ip 資訊
root@kind:~# kubectl exec -it net -- ip a l
4: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default 
    link/ether 3a:b2:db:1a:fc:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.1.3/24 brd 10.244.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::38b2:dbff:fe1a:fcd7/64 scope link 
       valid_lft forever preferred_lft forever

## 路由資訊
root@kind:~# kubectl exec -it net -- ip r s
default via 10.244.1.1 dev eth0 
10.244.0.0/16 via 10.244.1.1 dev eth0 
10.244.1.0/24 dev eth0 proto kernel scope link src 10.244.1.3

Pod 節點所在 Node 節點資訊

root@kind:~# docker exec -it flannel-ipip-worker2 bash

## ip 資訊
root@flannel-ipip-worker2:/# ip a l 
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: flannel.ipip@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default 
    link/ipip 172.18.0.3 brd 0.0.0.0
    inet 10.244.1.0/32 scope global flannel.ipip
       valid_lft forever preferred_lft forever
    inet6 fe80::5efe:ac12:3/64 scope link 
       valid_lft forever preferred_lft forever
4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
    link/ether 42:94:41:c6:b2:23 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.1/24 brd 10.244.1.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::4094:41ff:fec6:b223/64 scope link 
       valid_lft forever preferred_lft forever
5: vethff7a3f2c@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue master cni0 state UP group default 
    link/ether ee:0b:49:ab:46:91 brd ff:ff:ff:ff:ff:ff link-netns cni-e18d1e60-364c-9ff3-9e8f-0367e78fd563
    inet6 fe80::ec0b:49ff:feab:4691/64 scope link 
       valid_lft forever preferred_lft forever
6: veth753e0660@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue master cni0 state UP group default 
    link/ether 8a:69:94:d2:56:6b brd ff:ff:ff:ff:ff:ff link-netns cni-81b6e21f-62ae-df9b-7fe7-501253072833
    inet6 fe80::8869:94ff:fed2:566b/64 scope link 
       valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::3/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:3/64 scope link 
       valid_lft forever preferred_lft forever

## 路由資訊
root@flannel-ipip-worker2:/# ip r s
default via 172.18.0.1 dev eth0 
10.244.0.0/24 via 172.18.0.4 dev flannel.ipip onlink 
10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 
10.244.2.0/24 via 172.18.0.2 dev flannel.ipip onlink 
172.18.0.0/16 dev eth0 proto kernel scope link src 172.18.0.3

Pod 節點進行 ping 包測試，訪問 cni-p548p Pod 節點

root@kind:~# kubectl exec -it net -- ping 10.244.2.2 -c 1
PING 10.244.2.2 (10.244.2.2): 56 data bytes
64 bytes from 10.244.2.2: seq=0 ttl=62 time=2.661 ms

--- 10.244.2.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 2.661/2.661/2.661 ms

Pod 節點 eth0 網路卡抓包

net~$ tcpdump -pne -i eth0
06:45:42.959427 3a:b2:db:1a:fc:d7 > 42:94:41:c6:b2:23, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 64, seq 0, length 64
06:45:42.959519 42:94:41:c6:b2:23 > 3a:b2:db:1a:fc:d7, ethertype IPv4 (0x0800), length 98: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 64, seq 0, length 64

資料包源 mac 地址: 3a:b2:db:1a:fc:d7 為 eth0 網路卡 mac 地址，而目的 mac 地址: 42:94:41:c6:b2:23 為 net Pod 節點 cni0 網路卡對應的網路卡 mac 地址，cni0
網路卡 ip 地址為網路閘道器地址 10.244.2.1 ， flannel 為 2 層網路模式透過路由送往資料到閘道器地址

net~$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
10.244.1.1               ether   42:94:41:c6:b2:23   C                     eth0

而透過 veth pair 可以確定 Pod 節點 eth0 網路卡對應的 veth pair 為 veth753e0660@if4 網路卡

flannel-ipip-worker2 節點 veth753e0660 網路卡抓包

root@flannel-ipip-worker2:/# tcpdump -pne -i veth753e0660
06:45:42.959427 3a:b2:db:1a:fc:d7 > 42:94:41:c6:b2:23, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 64, seq 0, length 64
06:45:42.959519 42:94:41:c6:b2:23 > 3a:b2:db:1a:fc:d7, ethertype IPv4 (0x0800), length 98: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 64, seq 0, length 64

因為他們互為 veth pair 所以抓包資訊相同

flannel-ipip-worker2 節點 cni0 網路卡抓包

root@flannel-ipip-worker2:/# tcpdump -pne -i cni0
06:45:42.959427 3a:b2:db:1a:fc:d7 > 42:94:41:c6:b2:23, ethertype IPv4 (0x0800), length 98: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 64, seq 0, length 64
06:45:42.959519 42:94:41:c6:b2:23 > 3a:b2:db:1a:fc:d7, ethertype IPv4 (0x0800), length 98: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 64, seq 0, length 64

資料包源 mac 地址: 3a:b2:db:1a:fc:d7 為 net Pod 節點 eth0 網路卡 mac 地址，而目的 mac 地址: 42:94:41:c6:b2:23 為 cni0 網路卡 mac 地址

檢視 flannel-ipip-worker2 主機路由資訊，發現並在資料包會在透過 10.244.2.0/24 via 172.18.0.2 dev flannel.ipip onlink 路由資訊轉發

flannel-ipip-worker2 節點 flannel.ipip 網路卡抓包

root@flannel-ipip-worker2:/# tcpdump -pne -i flannel.ipip icmp
listening on flannel.ipip, link-type RAW (Raw IP), snapshot length 262144 bytes
06:48:37.658204 ip: 10.244.1.3 > 10.244.2.2: ICMP echo request, id 91, seq 0, length 64
06:48:37.658260 ip: 10.244.2.2 > 10.244.1.3: ICMP echo reply, id 91, seq 0, length 64

icmp 包中，沒有 mac 資訊，只有源 ip 目的 ip 資訊，這也是 ipip 資料包的特性: IPIP 隧道的工作原理是將源主機的IP資料包封裝在一個新的IP資料包中

flannel-ipip-worker2 節點 eth0 網路卡抓包

request 資料包資訊資訊
- icmp 包中，外部 mac 資訊中，源 mac: 02:42:ac:12:00:03 為 flannel-ipip-worker2 的 eth0 網路卡 mac ，目的 mac: 02:42:ac:12:00:02 為對端 Pod 宿主機 flannel-ipip-worker 的 eth0 網路卡 mac。
flannel-ipip-worker2 節點 ipip 資訊

root@flannel-ipip-worker2:/# ip -d link show
3: flannel.ipip@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ipip 172.18.0.3 brd 0.0.0.0 promiscuity 0 minmtu 0 maxmtu 0 
    ipip any remote any local 172.18.0.3 ttl inherit nopmtudisc addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

資料包流向

資料從 pod 服務發出，透過檢視本機路由表，送往 10.244.2.1 網路卡。路由: 10.244.0.0/16 via 10.244.2.1 dev eth0
透過 veth pair 網路卡 veth753e0660 傳送資料到 flannel-ipip-worker2 主機上，在轉送到 cni0: 10.244.1.1 網路卡
flannel-ipip-worker2 主機檢視自身路由後，會送往 flannel.ipip 介面，因為目的地址為 10.244.2.2。路由: 10.244.2.0/24 via 172.18.0.2 dev flannel.ipip onlink
flannel.ipip 介面為 ipip 模式，會重新封裝資料包。
資料封裝完成後，會送往 eth0 介面，並送往對端 flannel-ipip-worker 主機。
對端 flannel-ipip-worker 主機接受到資料包後，發現這個是一個 ipip 資料包，接收端會將外層 IP 頭部去掉，提取內層的 IP 資料包。
內層資料包會被交給 flannel.ipip 介面進行處理，就像是接收到了一個普通的 IP 資料包一樣。
解封裝後發現內部的資料包，目的地址為 10.244.2.2 ，透過檢視本機路由表，送往 cni0 網路卡。路由: 10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
透過 cni0 網路卡 brctl showmacs cni0 mac 資訊，最終會把資料包送到 cni-p548p 主機

`Service` 網路通訊

可以檢視此文件 Flannel UDP 模式中，Service 網路通訊，資料包轉發流程一致

Flannel IPIP DR 模式
2024-08-21
模式
Flannel Wireguard 模式
2024-08-22
模式
Flannel VxLAN DR 模式
2024-08-20
模式
深入 K8s 網路原理（一）- Flannel VXLAN 模式分析
2023-12-19
K8S模式
flannel網路概述
2020-11-08
詳解：Flannel安裝與配置
2020-06-20
docker k8s 1.3.8 + flannel
2016-10-09
DockerK8S
四、建立覆蓋網路--Flannel
2017-11-15
Kubernetes網路分析之Flannel
2019-10-18
docker筆記31-網路外掛flannel
2018-10-01
Docker筆記
容器中跨主機的網路方案-flannel
2018-05-01
Docker網路解決方案 - Flannel部署記錄
2017-05-16
Docker
部署 k8s 服務， flannel pod 報錯
2020-04-15
K8S
flannel網路在kubernetes中的運用
2016-08-24
IPIP：2018 上半年全球雲服務 IP 地址分析報告
2018-08-02
Flannel和Calico網路外掛工作流程對比
2021-02-26
kubernetes應用flannel失敗 Connection refused.
2020-11-29
flannel vxlan工作基本原理及常見排障方法
2019-06-11
059、安裝配置flannel（2019-03-28 週四）
2019-03-30
容器編排系統K8s之flannel網路模型
2021-01-03
K8S模型
k8s容器互聯-flannel host-gw原理篇
2023-03-16
K8S
kubernetes實踐之四：Flannel網路外掛安裝
2018-03-15
使用multus實現管理網和業務網分離——calico和flannel共存
2021-06-09
ubuntu 16.04 下安裝kubernetes 1.6 之flannel網路模型
2017-09-27
Ubuntu模型
CentOS 7安裝etcd和flannel實現docker跨物理機通訊
2018-08-21
CentOSDocker
深入淺出Kubernetes網路：跨節點網路通訊之Flannel
2019-04-10
【kubernetes】網路虛擬網路卡對veth pair、flannel網路模型實現原理
2020-12-06
AI模型
在K8S中，cailico的ipip模型和ciliume的vxlan模型除了在具體的實現不同，在原理上有何區別？
2024-07-24
K8SAI模型
設計模式-單例模式、多例模式
2020-12-14
設計模式單例
設計模式 #5 （策略模式、代理模式）
2020-09-20
設計模式
建立型模式-單例模式-工廠模式
2021-02-02
模式單例
設計模式-單例模式，觀察者模式
2014-08-15
設計模式單例
設計模式--介面卡模式/代理模式
2024-08-28
設計模式
設計模式（二十）：訪問者模式、迭代器模式、觀察者模式、原型模式、模板方法
2016-01-25
設計模式原型
共享模式 & 專有模式
2019-11-13
模式
iOS模式分析策略模式
2019-04-04
iOS模式
設計模式----代理模式
2019-04-21
設計模式
設計模式----中介模式
2019-04-14
設計模式

Flannel IPIP 模式