K3S新增節點報錯

解決辦法：
在主節點/etc/systemd/system/k3s.service配置檔案中指定網路卡
ExecStart 之前：

ExecStart=/usr/local/bin/k3s \
    server \

ExecStart 之後：

ExecStart=/usr/local/bin/k3s \
    server --node-external-ip <這裡填寫對外的IP> \

使用以下命令在 master 上重新啟動 k3s，然後再次嘗試從節點加入主節點。

sudo systemctl daemon-reload
sudo systemctl restart k3s

https://github.com/k3s-io/k3s/issues/6879

root@xg:~# sudo journalctl -u k3s-agent
Mar 06 13:57:31 xg systemd[1]: Starting Lightweight Kubernetes...
Mar 06 13:57:31 xg sh[1227]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Mar 06 13:57:31 xg sh[1228]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Mar 06 13:57:31 xg k3s[1231]: time="2024-03-06T13:57:31Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Mar 06 13:57:31 xg k3s[1231]: time="2024-03-06T13:57:31Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/a3b46c0299091b71bfcc617b1e1fec1845c13bdd848584ceb39d2e700e702a4b"
Mar 06 13:57:34 xg k3s[1231]: time="2024-03-06T13:57:34Z" level=info msg="Starting k3s agent v1.28.7+k3s1 (051b14b2)"
Mar 06 13:57:34 xg k3s[1231]: time="2024-03-06T13:57:34Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: bj.com:6443"
Mar 06 13:57:34 xg k3s[1231]: time="2024-03-06T13:57:34Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [bj.com:6443] [default: bj.com:6443]"
Mar 06 13:57:53 xg k3s[1231]: time="2024-03-06T13:57:53Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 06 13:58:16 xg k3s[1231]: time="2024-03-06T13:58:16Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: CA cert validation failed: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting header>
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Module overlay was already loaded"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Module nf_conntrack was already loaded"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Module br_netfilter was already loaded"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Module iptable_filter was already loaded"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Mar 06 13:58:48 xg k3s[1231]: time="2024-03-06T13:58:48Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Mar 06 13:58:49 xg k3s[1231]: time="2024-03-06T13:58:49Z" level=info msg="containerd is now running"
Mar 06 13:58:49 xg k3s[1231]: time="2024-03-06T13:58:49Z" level=info msg="Getting list of apiserver endpoints from server"
Mar 06 13:58:55 xg k3s[1231]: time="2024-03-06T13:58:55Z" level=info msg="Updated load balancer k3s-agent-load-balancer default server address -> 10.0.24.2:6443"
Mar 06 13:58:55 xg k3s[1231]: time="2024-03-06T13:58:55Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.24.2:6443"
Mar 06 13:58:55 xg k3s[1231]: time="2024-03-06T13:58:55Z" level=info msg="Removing server from load balancer k3s-agent-load-balancer: bj.com:6443"
Mar 06 13:58:55 xg k3s[1231]: time="2024-03-06T13:58:55Z" level=info msg="Updated load balancer k3s-agent-load-balancer server addresses -> [10.0.24.2:6443] [default: 10.0.24.2:6443]"
Mar 06 13:58:55 xg k3s[1231]: time="2024-03-06T13:58:55Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgr>
Mar 06 13:58:55 xg k3s[1231]: time="2024-03-06T13:58:55Z" level=info msg="Connecting to proxy" url="wss://10.0.24.2:6443/v1-k3s/connect"
Mar 06 13:59:05 xg k3s[1231]: W0306 13:59:05.522594    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 13:59:05 xg k3s[1231]: I0306 13:59:05.522687    1231 trace.go:236] Trace[446973662]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 13:58:55.520) (total time: 10002ms):
Mar 06 13:59:05 xg k3s[1231]: Trace[446973662]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10002ms (13:59:05.522)
Mar 06 13:59:05 xg k3s[1231]: Trace[446973662]: [10.002526091s] [10.002526091s] END
Mar 06 13:59:05 xg k3s[1231]: E0306 13:59:05.522700    1231 reflector.go:147] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelecto>
Mar 06 13:59:15 xg k3s[1231]: time="2024-03-06T13:59:15Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting head>
Mar 06 13:59:16 xg k3s[1231]: W0306 13:59:16.732032    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 13:59:16 xg k3s[1231]: I0306 13:59:16.732099    1231 trace.go:236] Trace[235148201]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 13:59:06.730) (total time: 10001ms):
Mar 06 13:59:16 xg k3s[1231]: Trace[235148201]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10001ms (13:59:16.732)
Mar 06 13:59:16 xg k3s[1231]: Trace[235148201]: [10.001138329s] [10.001138329s] END
Mar 06 13:59:16 xg k3s[1231]: E0306 13:59:16.732115    1231 reflector.go:147] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelecto>
Mar 06 13:59:28 xg k3s[1231]: W0306 13:59:28.976597    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 13:59:28 xg k3s[1231]: I0306 13:59:28.976664    1231 trace.go:236] Trace[1351895730]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 13:59:18.975) (total time: 10000ms):
Mar 06 13:59:28 xg k3s[1231]: Trace[1351895730]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10000ms (13:59:28.976)
Mar 06 13:59:28 xg k3s[1231]: Trace[1351895730]: [10.000753606s] [10.000753606s] END
Mar 06 13:59:28 xg k3s[1231]: E0306 13:59:28.976684    1231 reflector.go:147] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelecto>
Mar 06 13:59:40 xg k3s[1231]: time="2024-03-06T13:59:40Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting head>
Mar 06 13:59:44 xg k3s[1231]: W0306 13:59:44.516370    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 13:59:44 xg k3s[1231]: I0306 13:59:44.516437    1231 trace.go:236] Trace[1738401379]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 13:59:34.515) (total time: 10001ms):
Mar 06 13:59:44 xg k3s[1231]: Trace[1738401379]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10001ms (13:59:44.516)
Mar 06 13:59:44 xg k3s[1231]: Trace[1738401379]: [10.001244698s] [10.001244698s] END
Mar 06 13:59:44 xg k3s[1231]: E0306 13:59:44.516454    1231 reflector.go:147] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelecto>
Mar 06 14:00:02 xg k3s[1231]: W0306 14:00:02.313355    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 14:00:02 xg k3s[1231]: I0306 14:00:02.313426    1231 trace.go:236] Trace[1621173419]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 13:59:52.311) (total time: 10001ms):
Mar 06 14:00:02 xg k3s[1231]: Trace[1621173419]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10001ms (14:00:02.313)
Mar 06 14:00:02 xg k3s[1231]: Trace[1621173419]: [10.001522652s] [10.001522652s] END
Mar 06 14:00:02 xg k3s[1231]: E0306 14:00:02.313443    1231 reflector.go:147] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelecto>
Mar 06 14:00:05 xg k3s[1231]: time="2024-03-06T14:00:05Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting head>
Mar 06 14:00:27 xg k3s[1231]: W0306 14:00:27.371194    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 14:00:27 xg k3s[1231]: I0306 14:00:27.371310    1231 trace.go:236] Trace[1259851656]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 14:00:17.370) (total time: 10000ms):
Mar 06 14:00:27 xg k3s[1231]: Trace[1259851656]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10000ms (14:00:27.371)
Mar 06 14:00:27 xg k3s[1231]: Trace[1259851656]: [10.000961066s] [10.000961066s] END
Mar 06 14:00:27 xg k3s[1231]: E0306 14:00:27.371329    1231 reflector.go:147] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelecto>
Mar 06 14:00:30 xg k3s[1231]: time="2024-03-06T14:00:30Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting head>
Mar 06 14:00:55 xg k3s[1231]: time="2024-03-06T14:00:55Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting head>
Mar 06 14:01:09 xg k3s[1231]: time="2024-03-06T14:01:09Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.24.2:6443: connect: connection timed out"
Mar 06 14:01:09 xg k3s[1231]: time="2024-03-06T14:01:09Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.24.2:6443: connect: connection timed out"
Mar 06 14:01:14 xg k3s[1231]: time="2024-03-06T14:01:14Z" level=info msg="Connecting to proxy" url="wss://10.0.24.2:6443/v1-k3s/connect"
Mar 06 14:01:20 xg k3s[1231]: time="2024-03-06T14:01:20Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting head>
Mar 06 14:01:28 xg k3s[1231]: W0306 14:01:28.109895    1231 reflector.go:535] k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&li>
Mar 06 14:01:28 xg k3s[1231]: I0306 14:01:28.109958    1231 trace.go:236] Trace[12605897]: "Reflector ListAndWatch" name:k8s.io/client-go@v1.28.7-k3s1/tools/cache/reflector.go:229 (06-Mar-2024 14:01:18.108) (total time: 10001ms):
Mar 06 14:01:28 xg k3s[1231]: Trace[12605897]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": net/http: TLS handshake timeout 10001ms (14:01:28.109)