Cilium Gateway API 特性（轉載）

evescn發表於2024-07-16
GatewayAPI

Cilium Gateway API 特性（轉載）

一、環境資訊

主機 IP
ubuntu 10.0.0.234
軟體 版本
docker 26.1.4
helm v3.15.0-rc.2
kind 0.18.0
kubernetes 1.23.4
ubuntu os Ubuntu 22.04.6 LTS
kernel 5.15.0-106

二、Cilium Gateway API 流程圖

img

Cilium 現在提供完全一致的 Gateway API 實現。 Gateway API 是 Kubernetes 叢集中南北負載均衡和流量路由的新標準，並且是 Ingress API 的長期繼承者。 Gateway API 代表了流量管理的未來。

GatewaAPI 從設計之初就是為了解決 Ingress API 的侷限性

  • Gateway API 的建立是源於 Ingress API 存在一些侷限性：首先，它不提供使用者需要定義的高階負載均衡功能。它本身僅支援簡單的基於路徑的 HTTP 流量請求路由
  • 其次，使用者管理變得不切實際：產品供應商透過註釋來解決 Ingress API 中功能缺乏的問題。但是註釋雖然非常強大，但最終會導致一個 Ingress 與另一個 Ingress 之間出現不一致
  • 第三，由於 Ingress API 是單一 API 資源，因此它操作受限：不適合具有共享負載均衡基礎設施的多團隊叢集

在 Cilium 1.13 中，Cilium Gateway API 透過了所有 Gateway API 一致性測試 (v0.5.1)[https://isovalent.com/blog/post/cilium-release-113/]

參考官方連結

三、Cilium Gateway API 模式環境搭建

安裝須知

  • 配置 Cilium 時，必須啟用 NodePort，使用 nodePort.enabled=true 或使用 kubeProxyReplacement 將 kube-proxy 替換為部分或嚴格。
  • 配置 Cilium 時，必須使用 --enable-l7-proxy 標誌（預設已啟用）啟用 L7 代理。
  • 必須預先安裝以下來自 Gateway API v0.5.1 的 CRD。(安裝步驟)(https://docs.cilium.io/en/v1.13/network/servicemesh/gateway-api/gateway-api/#prerequisites)
  • 與 Ingress 類似，Gateway API 控制器會建立一個 LoadBalancer 型別的服務，因此您的環境需要支援該服務。

kind 配置檔案資訊

root@KinD:~# cat install.sh

#!/bin/bash
date
set -v

# 1.prep noCNI env
cat <<EOF | kind create cluster --name=cilium-gatewayapi-http --image=kindest/node:v1.23.4 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  # kind 預設使用 rancher cni，cni 我們需要自己建立
  disableDefaultCNI: true
  # 此處使用 cilium 代替 kube-proxy 功能
  kubeProxyMode: "none"
nodes:
  - role: control-plane
  - role: worker
  - role: worker

containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.evescn.com"]
    endpoint = ["https://harbor.evescn.com"]
EOF

# 2.install necessary tools
for i in $(docker ps -a --format "table {{.Names}}" | grep cilium) 
do
    echo $i
    docker cp /usr/bin/ping $i:/usr/bin/ping
    docker exec -it $i bash -c "sed -i -e 's/jp.archive.ubuntu.com\|archive.ubuntu.com\|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list"
    docker exec -it $i bash -c "apt-get -y update >/dev/null && apt-get -y install net-tools tcpdump lrzsz bridge-utils >/dev/null 2>&1"
done
  • 安裝 k8s 叢集和 cilium 服務
root@KinD:~# ./install.sh

Creating cluster "cilium-gatewayapi-http" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Set kubectl context to "kind-cilium-gatewayapi-http"
You can now use your cluster with:

kubectl cluster-info --context kind-cilium-gatewayapi-http

Thanks for using kind! 😊
  • 安裝 gateway api
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
customresourcedefinition.apiextensions.k8s.io/gatewayclasses.gateway.networking.k8s.io created
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
customresourcedefinition.apiextensions.k8s.io/gateways.gateway.networking.k8s.io created
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
customresourcedefinition.apiextensions.k8s.io/httproutes.gateway.networking.k8s.io created
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.5.1/config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml
customresourcedefinition.apiextensions.k8s.io/referencegrants.gateway.networking.k8s.io created
  • 安裝 cilium 服務
#/bin/bash
set -v
controller_node_ip=`kubectl get node -o wide --no-headers | grep -E "control-plane|bpf1" | awk -F " " '{print $6}'`

helm repo add cilium https://helm.cilium.io > /dev/null 2>&1
helm repo update > /dev/null 2>&1

helm install cilium cilium/cilium \
  --set k8sServiceHost=$controller_node_ip \
  --set k8sServicePort=6443 \
  --version 1.13.0-rc5 \
  --namespace kube-system \
  --set debug.enabled=true \
  --set debug.verbose=datapath \
  --set monitorAggregation=none \
  --set ipam.mode=cluster-pool \
  --set cluster.name=cilium-gatewayapi-http \
  --set kubeProxyReplacement=strict \
  --set bpf.masquerade=true \
  --set gatewayAPI.enabled=true

--set 引數解釋

  1. --set kubeProxyReplacement=strict

    • 含義: 啟用 kube-proxy 替代功能，並以嚴格模式執行。
    • 用途: Cilium 將完全替代 kube-proxy 實現服務負載均衡，提供更高效的流量轉發和網路策略管理。

  2. --set bpf.masquerade

    • 含義: 啟用 eBPF 功能。
    • 用途: 使用 eBPF 實現資料路由，提供更高效和靈活的網路地址轉換功能。

  3. --set gatewayAPI.enable=true:

    • 啟用 Gateway API 支援，Cilium 將支援並管理 Gateway API 資源。
  • 檢視安裝的服務
root@KinD:~# kubectl get pods -A
NAMESPACE            NAME                                                           READY   STATUS    RESTARTS   AGE
kube-system          cilium-55g65                                                   1/1     Running   0          103s
kube-system          cilium-bb4s7                                                   1/1     Running   0          103s
kube-system          cilium-operator-58959f76d6-46b62                               1/1     Running   0          103s
kube-system          cilium-operator-58959f76d6-92kwx                               1/1     Running   0          103s
kube-system          cilium-r5v9v                                                   1/1     Running   0          103s
kube-system          coredns-64897985d-l5rqq                                        1/1     Running   0          131m
kube-system          coredns-64897985d-qx7bl                                        1/1     Running   0          131m
kube-system          etcd-cilium-gatewayapi-http-control-plane                      1/1     Running   0          132m
kube-system          kube-apiserver-cilium-gatewayapi-http-control-plane            1/1     Running   0          132m
kube-system          kube-controller-manager-cilium-gatewayapi-http-control-plane   1/1     Running   0          132m
kube-system          kube-scheduler-cilium-gatewayapi-http-control-plane            1/1     Running   0          132m
local-path-storage   local-path-provisioner-5ddd94ff66-7t65r                        1/1     Running   0          131m

安裝 metallb 服務，提供 LoadBanlencer 功能

  • 安裝 metallb 服務
root@KinD:~# kubectl apply -f https://gh.api.99988866.xyz/https://raw.githubusercontent.com/metallb/metallb/v0.13.10/config/manifests/metallb-native.yaml

namespace/metallb-system created
customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
configmap/metallb-excludel2 created
secret/webhook-server-cert created
service/webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
  • 建立 metalLb layer2 的 IPAddressPool
root@KinD:~# cat metallb-l2-ip-config.yaml
---
# metallb 分配給 loadbanlencer 的 ip 地址段定義
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ippool
  namespace: metallb-system
spec:
  addresses:
  - 172.18.0.200-172.18.0.210

---
# 建立 L2Advertisement 進行 IPAddressPool 地址段
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ippool
  namespace: metallb-system
spec:
  ipAddressPools:
  - ippool


root@KinD:~# kubectl apply -f metallb-l2-ip-config.yaml
ipaddresspool.metallb.io/ippool unchanged
l2advertisement.metallb.io/ippool created

root@KinD:~# kubectl get ipaddresspool -n metallb-system
NAME     AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
ippool   true          false             ["172.18.0.200-172.18.0.210"]
root@KinD:~# kubectl get l2advertisement -n metallb-system
NAME     IPADDRESSPOOLS   IPADDRESSPOOL SELECTORS   INTERFACES
ippool   ["ippool"]

k8s 叢集安裝 Pod 和 Service

root@KinD:~# cat cni.yaml
---
apiVersion: apps/v1
kind: DaemonSet
#kind: Deployment
metadata:
  labels:
    app: cni
  name: cni
spec:
  #replicas: 1
  selector:
    matchLabels:
      app: cni
  template:
    metadata:
      labels:
        app: cni
    spec:
      containers:
      - image: harbor.dayuan1997.com/devops/nettool:0.9
        name: nettoolbox
        securityContext:
          privileged: true

---
apiVersion: v1
kind: Service
metadata:
  name: cni
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  type: ClusterIP
  selector:
    app: cni

root@KinD:~# kubectl apply -f cni.yaml
daemonset.apps/cni created
service/serversvc created
  • 檢視安裝服務資訊
root@KinD:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP          NODE                             NOMINATED NODE   READINESS GATES
cni-9c8h5   1/1     Running   0          7s    10.0.2.2    cilium-gatewayapi-http-worker    <none>           <none>
cni-j9bzr   1/1     Running   0          7s    10.0.1.69   cilium-gatewayapi-http-worker2   <none>           <none>

root@KinD:~# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
cni          ClusterIP   10.96.145.109   <none>        80/TCP    20s
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   134m

四、測試 Gateway HTTP 特性

配置 Gateway HTTP

root@KinD:~# cat gateway.yamnl
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: basic-gateway
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    port: 80
    name: web-gateway
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: http-app-cni
spec:
  parentRefs:
  - name: basic-gateway
    namespace: default
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: cni
      port: 80


root@KinD:~# kubectl apply -f gateway.yaml
gateway.gateway.networking.k8s.io/basic-gateway created
httproute.gateway.networking.k8s.io/http-app-cni created

root@KinD:~# kubectl get svc
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
cilium-gateway-basic-gateway   LoadBalancer   10.96.30.37     172.18.0.200   80:30594/TCP   21s
cni                            ClusterIP      10.96.145.109   <none>         80/TCP         4m9s
kubernetes                     ClusterIP      10.96.0.1       <none>         443/TCP        138m

root@KinD:~# kubectl get gateway
NAME            CLASS    ADDRESS        READY   AGE
basic-gateway   cilium   172.18.0.200   True    37s

root@KinD:~# kubectl get HTTPRoute
NAME           HOSTNAMES   AGE
http-app-cni               54s

建立 Gateway 後 basic-gateway 在定義的地址池中自動獲取了一個 IP 地址，並且自動建立了一個 cilium-gateway-basic-gateway svc 服務，服務的 EXTERNAL-IP 為 basic-gateway 從地址池中自動獲取的 IP 地址

測試 Gateway HTTP

root@KinD:~# GATEWAY=$(kubectl get gateway basic-gateway -o jsonpath='{.status.addresses[0].value}')
root@KinD:~# curl -v http://"$GATEWAY"/
*   Trying 172.18.0.200:80...
* Connected to 172.18.0.200 (172.18.0.200) port 80 (#0)
> GET / HTTP/1.1
> Host: 172.18.0.200
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Tue, 16 Jul 2024 06:14:57 GMT
< content-type: text/html
< content-length: 46
< last-modified: Tue, 16 Jul 2024 06:09:16 GMT
< etag: "66960e8c-2e"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 0
< 
PodName: cni-j9bzr | PodIP: eth0 10.0.1.69/32
* Connection #0 to host 172.18.0.200 left intact

在宿主機 172.18.0.1 主機上訪問業務正常訪問，使用 ip 地址為 172.18.0.200

五、測試 Gateway HTTPS 特性

建立 tls 證書，此處使用了一個測試證書

root@KinD:~# kubectl  create secret tls demo-cert --cert=tls.crt --key=tls.key
secret/demo-cert created

配置 Gateway HTTPS

root@KinD:~# cat gateway-https.yamnl
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: tls-gateway
spec:
  gatewayClassName: cilium
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "cni.evescn.com"
    tls:
      certificateRefs:
      - kind: Secret
        name: test-cert
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: https-app-cni
spec:
  parentRefs:
  - name: tls-gateway
  hostnames:
  - "cni.evescn.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: cni
      port: 80


root@KinD:~# kubectl apply -f gateway-https.yaml
gateway.gateway.networking.k8s.io/tls-gateway created
httproute.gateway.networking.k8s.io/https-app-cni created

root@KinD:~#  kubectl get svc
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)         AGE
cilium-gateway-basic-gateway   LoadBalancer   10.96.30.37     172.18.0.200   80:30594/TCP    7m21s
cilium-gateway-tls-gateway     LoadBalancer   10.96.5.61      172.18.0.201   443:30197/TCP   19s
cni                            ClusterIP      10.96.145.109   <none>         80/TCP          11m
kubernetes                     ClusterIP      10.96.0.1       <none>         443/TCP         145m

root@KinD:~# kubectl get gateway
NAME            CLASS    ADDRESS        READY   AGE
basic-gateway   cilium   172.18.0.200   True    7m47s
tls-gateway     cilium   172.18.0.201   True    45s

root@KinD:~# kubectl get HTTPRoute
NAME            HOSTNAMES                AGE
http-app-cni                             7m56s
https-app-cni   ["cni.evescn.com"]   54s

建立 gateway 後 tls-gateway 在定義的地址池中自動獲取了一個 IP 地址，並且自動建立了一個 cilium-gateway-tls-gateway svc 服務，服務的 EXTERNAL-IP 為 tls-gateway 從地址池中自動獲取的 IP 地址

測試 Gateway HTTPS

  • /etc/hosts 新增 dns 解析
root@KinD:~# cat /etc/hosts
......
172.18.0.201 cni.evescn.com
  • 測試 Gateway HTTPS
root@KinD:~# curl --cacert /root/evescn.com.pem -v https://cni.evescn.com/
*   Trying 172.18.0.201:443...
* Connected to cni.evescn.com (172.18.0.201) port 443 (#0)
> GET / HTTP/1.1
> Host: cni.evescn.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Tue, 16 Jul 2024 06:25:27 GMT
< content-type: text/html
< content-length: 46
< last-modified: Tue, 16 Jul 2024 06:09:16 GMT
< etag: "66960e8c-2e"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 0
< 
PodName: cni-j9bzr | PodIP: eth0 10.0.1.69/32
* Connection #0 to host cni.evescn.com left intact

在宿主機 172.18.0.1 主機上訪問業務正常訪問，使用 ip 地址為 172.18.0.201

六、轉載部落格

https://bbs.huaweicloud.com/blogs/418684

相關文章