k8s網路之calico學習

wilson排球發表於2018-11-07
K8S

一、知識準備

1.calico主要通過ipip協議與bgp協議來實現通訊。前者通過ipip隧道作為通訊基礎，後者則是純三層的路由交換
2.bgp協議主要由兩種方式：BGP Speaker 全互聯模式(node-to-node mesh)與BGP Speaker RR模式
3.本文主要探索一下calico bgp的兩種模式

二、環境準備

元件 版本
OS Ubuntu 18.04.1 LTS
docker 18.06.0-ce
k8s 1.10.1
calico 3.1.3

ip hostname
192.168.56.101 k8s-master
192.168.56.102 k8s-node1
192.168.56.103 k8s-node2
192.168.56.104 k8s-node3

三、安裝

k8s安裝

參考官網安裝以及社群諸多大神的安裝帖子，這裡就不班門弄斧了

本文的k8s的環境開啟了rbac，etcd加入了證書

calico安裝

主要參考官方文件 https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico

1.calico rbac

kubectl apply -f 
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml

2.下載calico.yaml

curl 
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml 
-O

3.填入etcd地址

ETCD_ENDPOINTS="https://192.168.56.101:2379"
sed -i "s#.*etcd_endpoints:.*#  etcd_endpoints: "${ETCD_ENDPOINTS}"#g" calico.yaml

sed -i "s#__ETCD_ENDPOINTS__#${ETCD_ENDPOINTS}#g" calico.yaml

4.將etcd證書資訊填入。我的etcd證書在/etc/etcd/ssl下

ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d `
``
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d `
``
ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d `
``

sed -i "s#.*etcd-cert:.*#  etcd-cert: ${ETCD_CERT}#g" calico.yaml
sed -i "s#.*etcd-key:.*#  etcd-key: ${ETCD_KEY}#g" calico.yaml
sed -i "s#.*etcd-ca:.*#  etcd-ca: ${ETCD_CA}#g" calico.yaml

sed -i `s#.*etcd_ca:.*#  etcd_ca: "/calico-secrets/etcd-ca"#g` calico.yaml
sed -i `s#.*etcd_cert:.*#  etcd_cert: "/calico-secrets/etcd-cert"#g` calico.yaml
sed -i `s#.*etcd_key:.*#  etcd_key: "/calico-secrets/etcd-key"#g` calico.yaml

sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico.yaml
sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-root-ca.pem#g" calico.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico.yaml

5.配置calico bgp 並且修改ip cidr:10.10.0.0/16

sed -i `/CALICO_IPV4POOL_IPIP/{n;s/Always/off/g}` calico.yaml
sed -i `/CALICO_IPV4POOL_CIDR/{n;s/192.168.0.0/10.10.0.0/g}` calico.yaml

6.kubectl安裝calico

kubectl apply -f calico.yaml

注意:因為calico-node需要獲取作業系統的許可權執行，所以要在apiserver、kubelet中加入--allow-privileged=true

檢視一下狀態:

root@k8s-master:/tmp# kubectl get pods -n kube-system -owide
NAME                                        READY     STATUS    RESTARTS   AGE       IP               NODE
calico-kube-controllers-98989846-b4n72      1/1       Running   0          18d       192.168.56.102   k8s-node1
calico-node-58pck                           2/2       Running   0          18d       192.168.56.103   k8s-node2
calico-node-s2txw                           2/2       Running   0          18d       192.168.56.101   k8s-master
calico-node-svmbp                           2/2       Running   0          18d       192.168.56.102   k8s-node1
...

7.kubelet配置calico

找到kubelet的配置檔案（我的環境在/etc/kubernetes/kubelet），加入
--network-plugin=cni

重啟kubelet

8.測試一個pod

cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: network-test
  namespace: test
spec:
  containers:
  - image: busybox:latest
    command:
      - sleep
      - "3600"
    name: network-test
EOF
root@k8s-master:~# kubectl -n test get pods -owide
NAME           READY     STATUS    RESTARTS   AGE       IP              NODE
network-test   1/1       Running   0          41s       10.10.169.139   k8s-node2

至此：calico安裝已經完成

四、calicoctl使用

1.下載calicoctl

https://github.com/projectcalico/calicoctl/releases/download/v3.1.3/calicoctl-linux-amd64

2.檢視當前的calico-node

root@k8s-master:/tmp# calicoctl get node
NAME
k8s-master
k8s-node1
k8s-node2

calicoctl get node -o yaml 檢視詳細資訊

3.檢視當前的ippool

root@k8s-master:/tmp# calicoctl get ippool
NAME                  CIDR
default-ipv4-ippool   10.10.0.0/16
default-ipv6-ippool   fdc6:1a69:2b39::/48

4.檢視當前模式

root@k8s-master:/tmp# calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+----------------+-------------------+-------+----------+-------------+
| 192.168.56.102 | node-to-node mesh | up    | 07:39:02 | Established |
| 192.168.56.103 | node-to-node mesh | up    | 07:39:02 | Established |
+----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@k8s-master:/tmp# netstat -anp | grep ESTABLISH | grep bird
tcp        0      0 192.168.56.101:33029    192.168.56.102:179      ESTABLISHED 26558/bird
tcp        0      0 192.168.56.101:58055    192.168.56.103:179      ESTABLISHED 26558/bird

當前執行在BGP Speaker 全互聯模式(node-to-node mesh)模式，calico叢集中的節點之間都會相互建立連線，用於路由交換。適合規模不大的叢集中執行，一旦叢集節點增大，mesh模式將形成一個巨大服務網格，連線數暴增

5.修改BGP Speaker RR模式

禁止mesh模式，配置bgpPeer

cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 61234
EOF

cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global
spec:
  peerIP: 192.168.56.103
  asNumber: 61234
EOF

檢視RR模式配置：

root@k8s-master:~# calicoctl get bgpconfig
NAME      LOGSEVERITY   MESHENABLED   ASNUMBER
default   Info          false         61234

root@k8s-master:~# calicoctl get bgppeer
NAME             PEERIP           NODE       ASN
bgppeer-global   192.168.56.103   (global)   61234

安裝routereflector

docker run --privileged --net=host -d                             
           --name=calico-rr                                       
           -e IP=192.168.56.104                                   
           -e ETCD_ENDPOINTS=https://192.168.56.101:2379          
           -v /etc/etcd/ssl:/etc/calico/ssl                     
           -e ETCD_CA_CERT_FILE=/etc/calico/ssl/etcd-root-ca.pem  
           -e ETCD_CERT_FILE=/etc/calico/ssl/etcd.pem             
           -e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem          
           calico/routereflector:v0.6.1

檢視效果：

root@k8s-master:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+----------------+-----------+-------+----------+-------------+
|  PEER ADDRESS  | PEER TYPE | STATE |  SINCE   |    INFO     |
+----------------+-----------+-------+----------+-------------+
| 192.168.56.103 | global    | up    | 09:13:23 | Established |
+----------------+-----------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@k8s-master:~# netstat -anp | grep ESTABLISH | grep bird
tcp        0      0 192.168.56.101:179      192.168.56.103:54903    ESTABLISHED 26558/bird

每臺機器都只會與rr建立一條連線，並且與rr通訊即可拿到所有路由，大大減少了連線數量

至此，本文結束
在下才疏學淺，有撒湯漏水的，請各位不吝賜教…

相關文章