一、知識準備
1.calico主要通過ipip協議與bgp協議來實現通訊。前者通過ipip隧道作為通訊基礎，後者則是純三層的路由交換
2.bgp協議主要由兩種方式：BGP Speaker 全互聯模式(node-to-node mesh)與BGP Speaker RR模式
3.本文主要探索一下calico bgp的兩種模式
二、環境準備
| 元件 | 版本 |
|---|---|
| OS | Ubuntu 18.04.1 LTS |
| docker | 18.06.0-ce |
| k8s | 1.10.1 |
| calico | 3.1.3 |
| ip | hostname |
|---|---|
| 192.168.56.101 | k8s-master |
| 192.168.56.102 | k8s-node1 |
| 192.168.56.103 | k8s-node2 |
| 192.168.56.104 | k8s-node3 |
三、安裝
k8s安裝
參考官網安裝以及社群諸多大神的安裝帖子，這裡就不班門弄斧了
本文的k8s的環境開啟了rbac，etcd加入了證書
calico安裝
主要參考官方文件 https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico
1.calico rbac
kubectl apply -f
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml
2.下載calico.yaml
curl
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml
-O
3.填入etcd地址
ETCD_ENDPOINTS="https://192.168.56.101:2379"
sed -i "s#.*etcd_endpoints:.*# etcd_endpoints: "${ETCD_ENDPOINTS}"#g" calico.yaml
sed -i "s#__ETCD_ENDPOINTS__#${ETCD_ENDPOINTS}#g" calico.yaml
4.將etcd證書資訊填入。我的etcd證書在/etc/etcd/ssl下
ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d `
``
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d `
``
ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d `
``
sed -i "s#.*etcd-cert:.*# etcd-cert: ${ETCD_CERT}#g" calico.yaml
sed -i "s#.*etcd-key:.*# etcd-key: ${ETCD_KEY}#g" calico.yaml
sed -i "s#.*etcd-ca:.*# etcd-ca: ${ETCD_CA}#g" calico.yaml
sed -i `s#.*etcd_ca:.*# etcd_ca: "/calico-secrets/etcd-ca"#g` calico.yaml
sed -i `s#.*etcd_cert:.*# etcd_cert: "/calico-secrets/etcd-cert"#g` calico.yaml
sed -i `s#.*etcd_key:.*# etcd_key: "/calico-secrets/etcd-key"#g` calico.yaml
sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico.yaml
sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-root-ca.pem#g" calico.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico.yaml
5.配置calico bgp 並且修改ip cidr:10.10.0.0/16
sed -i `/CALICO_IPV4POOL_IPIP/{n;s/Always/off/g}` calico.yaml
sed -i `/CALICO_IPV4POOL_CIDR/{n;s/192.168.0.0/10.10.0.0/g}` calico.yaml
6.kubectl安裝calico
kubectl apply -f calico.yaml注意:因為calico-node需要獲取作業系統的許可權執行，所以要在apiserver、kubelet中加入--allow-privileged=true
檢視一下狀態:
root@k8s-master:/tmp# kubectl get pods -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE
calico-kube-controllers-98989846-b4n72 1/1 Running 0 18d 192.168.56.102 k8s-node1
calico-node-58pck 2/2 Running 0 18d 192.168.56.103 k8s-node2
calico-node-s2txw 2/2 Running 0 18d 192.168.56.101 k8s-master
calico-node-svmbp 2/2 Running 0 18d 192.168.56.102 k8s-node1
...7.kubelet配置calico
找到kubelet的配置檔案（我的環境在/etc/kubernetes/kubelet），加入
--network-plugin=cni
重啟kubelet8.測試一個pod
cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: network-test
namespace: test
spec:
containers:
- image: busybox:latest
command:
- sleep
- "3600"
name: network-test
EOFroot@k8s-master:~# kubectl -n test get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE
network-test 1/1 Running 0 41s 10.10.169.139 k8s-node2至此：calico安裝已經完成
四、calicoctl使用
1.下載calicoctl
https://github.com/projectcalico/calicoctl/releases/download/v3.1.3/calicoctl-linux-amd64
2.檢視當前的calico-node
root@k8s-master:/tmp# calicoctl get node
NAME
k8s-master
k8s-node1
k8s-node2
calicoctl get node -o yaml 檢視詳細資訊3.檢視當前的ippool
root@k8s-master:/tmp# calicoctl get ippool
NAME CIDR
default-ipv4-ippool 10.10.0.0/16
default-ipv6-ippool fdc6:1a69:2b39::/484.檢視當前模式
root@k8s-master:/tmp# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.56.102 | node-to-node mesh | up | 07:39:02 | Established |
| 192.168.56.103 | node-to-node mesh | up | 07:39:02 | Established |
+----------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-master:/tmp# netstat -anp | grep ESTABLISH | grep bird
tcp 0 0 192.168.56.101:33029 192.168.56.102:179 ESTABLISHED 26558/bird
tcp 0 0 192.168.56.101:58055 192.168.56.103:179 ESTABLISHED 26558/bird當前執行在BGP Speaker 全互聯模式(node-to-node mesh)模式，calico叢集中的節點之間都會相互建立連線，用於路由交換。適合規模不大的叢集中執行，一旦叢集節點增大，mesh模式將形成一個巨大服務網格，連線數暴增
5.修改BGP Speaker RR模式
禁止mesh模式，配置bgpPeer
cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false
asNumber: 61234
EOF
cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: bgppeer-global
spec:
peerIP: 192.168.56.103
asNumber: 61234
EOF檢視RR模式配置：
root@k8s-master:~# calicoctl get bgpconfig
NAME LOGSEVERITY MESHENABLED ASNUMBER
default Info false 61234
root@k8s-master:~# calicoctl get bgppeer
NAME PEERIP NODE ASN
bgppeer-global 192.168.56.103 (global) 61234安裝routereflector
docker run --privileged --net=host -d
--name=calico-rr
-e IP=192.168.56.104
-e ETCD_ENDPOINTS=https://192.168.56.101:2379
-v /etc/etcd/ssl:/etc/calico/ssl
-e ETCD_CA_CERT_FILE=/etc/calico/ssl/etcd-root-ca.pem
-e ETCD_CERT_FILE=/etc/calico/ssl/etcd.pem
-e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem
calico/routereflector:v0.6.1檢視效果：
root@k8s-master:~# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-----------+-------+----------+-------------+
| 192.168.56.103 | global | up | 09:13:23 | Established |
+----------------+-----------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-master:~# netstat -anp | grep ESTABLISH | grep bird
tcp 0 0 192.168.56.101:179 192.168.56.103:54903 ESTABLISHED 26558/bird每臺機器都只會與rr建立一條連線，並且與rr通訊即可拿到所有路由，大大減少了連線數量
至此，本文結束
在下才疏學淺，有撒湯漏水的，請各位不吝賜教…