Adrian Goins 最近舉辦了關於如何使用 K3s 和 Traefik 保護和控制邊緣的 Kubernetes 大師班,演示瞭如何訪問 K3s 的 Traefik Proxy 儀表板,可以通過以下途徑註冊觀看回放:https://more.suse.com/MC_Secure_Edge_K3s_Traefik.htmlRancher Desktop 建立了一個單節點 K3s 叢集,我非常好奇在使用 Rancher Desktop 時,是否可以訪問 Traefik Proxy 儀表板。我在 Adrian 的課程上提出了這個問題,他說應該可以,於是我便開始著手操作。
注意:本文使用的環境為 Linux 作業系統,如 Mac 或 Windows 需視情況調整引數。
本文參考了 Adrian 在 GitHub 上釋出的一些課程: https://github.com/traefik-workshops/k3s-and-traefik-proxy
首先,克隆 Adrian 的 repo:
> git clone https://github.com/traefik-workshops/k3s-and-traefik-proxy.git
> cd k3s-and-traefik-proxy/
第一課: 暴露 Traefik 儀表盤
注意:01-Expose-the-Dashboard 中的所有檔案目前都沒有在 Adrian 的課程中使用。
將叢集 IP 設定為變數
Adrian 建議檢查 kubeconfig 檔案中的叢集 IP 地址,Rancher Desktop 會在主機上建立一個 ~/.kube/config 檔案:
> grep server ~/.kube/config
server: https://127.0.0.1:6443
> export CLUSTERIP=127.0.0.1
此時,Adrian 繼續他的課程,但目前 Linux 上的 Rancher Desktop 存在一個問題:特權埠(低於 1024 的埠)無法訪問。請參考 https://github.com/rancher-sandbox/rancher-desktop/issues/576
相反,Linux 上的 Rancher Desktop 使用者必須瞭解 HTTP (80) 和 HTTPS (443) 埠已轉發到哪些 Ingress 埠:
> kubectl get service -n kube-system traefik
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik LoadBalancer 10.43.146.37 192.168.5.15 80:30876/TCP,443:30614/TCP 26
我們將 Ingress 埠儲存到變數中,以便可以在整個課程中使用:
> export CLUSTERHTTP=`kubectl get service -n kube-system traefik -o json | jq '.spec.ports[0].nodePort'`
> export CLUSTERHTTPS=`kubectl get service -n kube-system traefik -o json | jq '.spec.ports[1].nodePort'`
切換當前 Namespace 為 kube-system
> kubectl config set-context --current --namespace kube-system
Context "rancher-desktop" modified.
建立 Service
> kubectl expose deploy/traefik -n kube-system --port=9000 --target-port=9000 --name=traefik-dashboard
service/traefik-dashboard exposed
建立 Ingress
> kubectl create ingress traefik-dashboard --rule="dashboard.traefik.$CLUSTERIP.sslip.io/*=traefik-dashboard:9000"
ingress.networking.k8s.io/traefik-dashboard created
訪問儀表板
與 Adrian 步驟不同的是,我們需要在 URL 中包含 HTTP 的 Ingress 埠:
> curl -si http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/dashboard/ | head -n 1
HTTP/1.1 200 OK
> echo http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/dashboard/
http://dashboard.traefik.127.0.0.1.sslip.io:30876/dashboard/
新增 Annotations
> kubectl annotate ingress traefik-dashboard traefik.ingress.kubernetes.io/router.entrypoints=web
ingress.networking.k8s.io/traefik-dashboard annotated
第 2 課:使用 Middleware 保護儀表板
> cd 02-Secure-the-Dashboard-With-Middleware
建立使用者檔案
注意 Adrian 已根據研討會提供了使用者檔案設定:
> cat users
user@example.com:$apr1$nWlieTS.$pbESld2QB5uYuUTAfFICr.
admin@example.com:$apr1$XMtXkoUy$IwIKiM./ujfaYf6/MsCaf1
從使用者檔案中建立儀表板 dashboard-users Secret
> kubectl create secret generic dashboard-users --from-file=users
secret/dashboard-users created
從 middleware-auth.yaml 建立 Middleware
> cat middleware-auth.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: dashboard-auth
spec:
basicAuth:
secret: dashboard-users
> kubectl apply -f middleware-auth.yaml
middleware.traefik.containo.us/dashboard-auth created
將 Middleware 應用到 Ingress
> kubectl annotate ingress traefik-dashboard \
traefik.ingress.kubernetes.io/router.middlewares=kube-system-dashboard-auth@kubernetescrd
ingress.networking.k8s.io/traefik-dashboard annotated
請注意,如果你在瀏覽器中一直訪問儀表板,那麼現在應該提示你輸入使用者名稱和密碼:
測試 Middleware
> curl -si http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/dashboard/ | head -n 1
HTTP/1.1 401 Unauthorized
> curl -si -u 'admin@example.com:admin1234' http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/dashboard/ | head -n 1
HTTP/1.1 200 OK
建立 Middleware 以新增 /dashboard 字首
> cat middleware-rewrite.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: dashboard-rewrite
spec:
addPrefix:
prefix: /dashboard
> kubectl apply -f middleware-rewrite.yaml
middleware.traefik.containo.us/dashboard-rewrite created
將第二個 Middleware 應用到 Ingress
> kubectl annotate ingress traefik-dashboard \
traefik.ingress.kubernetes.io/router.middlewares=kube-system-dashboard-rewrite@kubernetescrd,kube-system-dashboard-auth@kubernetescrd \
--overwrite=true
ingress.networking.k8s.io/traefik-dashboard annotated
訪問沒有 /dashboard/ 的儀表板
> curl -si http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/ | head -n 1
HTTP/1.1 401 Unauthorized
修復儀表板
> kubectl create ingress traefik-dashboard-api --rule="dashboard.traefik.$CLUSTERIP.sslip.io/api/*=traefik-dashboard:9000"
ingress.networking.k8s.io/traefik-dashboard-api created
> kubectl annotate ingress traefik-dashboard-api \
traefik.ingress.kubernetes.io/router.middlewares=kube-system-dashboard-auth@kubernetescrd
ingress.networking.k8s.io/traefik-dashboard-api annotated
第 3 課:使用 IngressRoute 自定義資源
> cd ../03-Use-the-IngressRoute-Custom-Resource/
將 Ingress 更改為 IngressRoutes
移除之前建立的 Ingress:
> kubectl delete ingress/traefik-dashboard ingress/traefik-dashboard-api
ingress.networking.k8s.io "traefik-dashboard" deleted
ingress.networking.k8s.io "traefik-dashboard-api" deleted
建立新的 IngressRoute,我們需要更改 IP 地址:
> cat ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard-secure
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host("dashboard.traefik.10.68.0.70.sslip.io")
services:
- name: traefik-dashboard
port: 9000
middlewares:
- name: dashboard-auth
- name: dashboard-rewrite
- kind: Rule
match: Host("dashboard.traefik.10.68.0.70.sslip.io") && PathPrefix("/api")
services:
- name: traefik-dashboard
port: 9000
middlewares:
- name: dashboard-auth
> sed -i "s/10\.68\.0\.70/${CLUSTERIP}/" ingressroute.yaml
> cat ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard-secure
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host("dashboard.traefik.127.0.0.1.sslip.io")
services:
- name: traefik-dashboard
port: 9000
middlewares:
- name: dashboard-auth
- name: dashboard-rewrite
- kind: Rule
match: Host("dashboard.traefik.127.0.0.1.sslip.io") && PathPrefix("/api")
services:
- name: traefik-dashboard
port: 9000
middlewares:
- name: dashboard-auth
> kubectl apply -f ingressroute.yaml
ingressroute.traefik.containo.us/traefik-dashboard-secure created
檢視 IngressRoute
> kubectl get ingressroute traefik-dashboard -o yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
annotations:
helm.sh/hook: post-install,post-upgrade
creationTimestamp: "2022-02-11T16:01:09Z"
generation: 1
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-10.9.100
name: traefik-dashboard
namespace: kube-system
resourceVersion: "657"
uid: 7993457e-7cde-478b-82c9-76acc5eebbd9
spec:
entryPoints:
- traefik
routes:
- kind: Rule
match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
services:
- kind: TraefikService
name: api@internal
什麼是 TraefikService?
> kubectl patch ingressroute/traefik-dashboard-secure --type=json --patch-file patch-dashboard-service.yaml
ingressroute.traefik.containo.us/traefik-dashboard-secure patched
> kubectl delete service traefik-dashboard
service "traefik-dashboard" deleted
> curl -si -u 'admin@example.com:admin1234' http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/ | head -n 1
HTTP/1.1 200 OK
第 4 課:使用 TLS 保護儀表板
> cd ../04-Secure-the-Dashboard-With-TLS/
設定 cert-manager
我使用了最新版本的 cert-manager,目前是 1.7.1:
> kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
...
...
...
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
> kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-cainjector-d6cbc4d9-j8q8x 1/1 Running 0 70s
cert-manager-6d8d6b5dbb-ts2mq 1/1 Running 0 70s
cert-manager-webhook-85fb68c79b-ql658 1/1 Running 0 70s
建立 ClusterIssuer
> cat clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
> kubectl apply -f clusterissuer.yaml
clusterissuer.cert-manager.io/selfsigned created
為儀表板生成證書
我們需要更改 IP 地址:
> cat certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: dashboard
spec:
subject:
organizations:
- Traefik Academy
commonName: dashboard.traefik.10.68.0.70.sslip.io
issuerRef:
kind: ClusterIssuer
name: selfsigned
secretName: dashboard-crt
> sed -i "s/10\.68\.0\.70/${CLUSTERIP}/" certificate.yaml
> cat certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: dashboard
spec:
subject:
organizations:
- Traefik Academy
commonName: dashboard.traefik.127.0.0.1.sslip.io
issuerRef:
kind: ClusterIssuer
name: selfsigned
secretName: dashboard-crt
> kubectl apply -f certificate.yaml
certificate.cert-manager.io/dashboard created
> kubectl get secret | grep tls
k3s-serving kubernetes.io/tls 2 87m
dashboard-crt
將證書新增到 IngressRoute
> cat patch-dashboard-tls.yaml
- op: replace
path: /spec/entryPoints
value:
- websecure
- op: add
path: /spec/tls
value:
secretName: dashboard-crt
> kubectl patch ingressroute/traefik-dashboard-secure \
--type=json \
--patch-file patch-dashboard-tls.yaml
ingressroute.traefik.containo.us/traefik-dashboard-secure patched
> echo https://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTPS/
https://dashboard.traefik.127.0.0.1.sslip.io:30614/
新增 HTTP 重定向
> cat middleware-scheme.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: redirect-permanent
spec:
redirectScheme:
permanent: true
scheme: https
需要在 middleware-scheme.yaml 新增 HTTPS 埠,並在 ingressroute.yaml 中更改 IP 地址:
> echo " port: \"${CLUSTERHTTPS}\"" >> middleware-scheme.yaml
> cat middleware-scheme.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: redirect-permanent
spec:
redirectScheme:
permanent: true
scheme: https
port: "30614"
> kubectl apply -f middleware-scheme.yaml
middleware.traefik.containo.us/redirect-permanent created
> cat ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard-http
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host("dashboard.traefik.10.68.0.70.sslip.io")
services:
- name: api@internal
kind: TraefikService
middlewares:
- name: redirect-permanent
> sed -i "s/10\.68\.0\.70/${CLUSTERIP}/" ingressroute.yaml
> cat ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-dashboard-http
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host("dashboard.traefik.127.0.0.1.sslip.io")
services:
- name: api@internal
kind: TraefikService
middlewares:
- name: redirect-permanent
> kubectl apply -f ingressroute.yaml
ingressroute.traefik.containo.us/traefik-dashboard-http created
> curl -si http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/ | head -n 1
HTTP/1.1 301 Moved Permanently
如果我們刪除 head 命令,我們可以看到它被移動到了哪裡:
> curl -si http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/
HTTP/1.1 301 Moved Permanently
Location: https://dashboard.traefik.127.0.0.1.sslip.io:30614/
Date: Fri, 11 Feb 2022 17:40:15 GMT
Content-Length: 17
Content-Type: text/plain; charset=utf-8
該位置應包括 HTTPS 的 Ingress 埠:
> echo http://dashboard.traefik.$CLUSTERIP.sslip.io:$CLUSTERHTTP/
http://dashboard.traefik.127.0.0.1.sslip.io:30876/
如果我們在 Web 瀏覽器中開啟 URL,它應該重定向到 HTTPS 站點。如果沒有,你可能需要清除 Web 瀏覽器的快取。