在上一篇文章《quay.io/coreos/etcd 基於Docker映象的叢集搭建》中，介紹了ETCD叢集的搭建。在此基礎上，我們進一步實踐calico docker的應用。
PaaS 平臺的網路需求：
在使用Docker構建PaaS平臺的過程中，我們首先遇到的問題是需要選擇一個滿足需求的網路模型：
1）讓每個容器擁有自己的網路棧，特別是獨立的 IP 地址；
2）能夠進行跨伺服器的容器間通訊，同時不依賴特定的網路裝置；
3）有訪問控制機制，不同應用之間互相隔離，有呼叫關係的能夠通訊。
調研了幾個主流的網路模型：
1）Docker原生的Bridge模型：NAT機制導致無法使用容器IP進行跨伺服器通訊；
2）Docker原生的Host模型：大家都使用和伺服器相同的IP，埠衝突問題很麻煩；
3）Weave OVS等基於隧道的模型：由於是基於隧道的技術，在使用者態進行封包解包，效能折損比較大，同時出現問題時網路抓包除錯會很不便。
在對上述模型都不怎麼滿意的情況下，發現了一個還不怎麼被大家關注的新專案：Project Calico。Project Calico是純三層的SDN實現，它基於BPG協議和Linux自己的路由轉發機制，不依賴特殊硬體，沒有使用NAT或Tunnel等技術。能夠方便的部署在物理伺服器，虛擬機器（如 OpenStack）或者容器環境下。同時它自帶的基於Iptables的ACL管理元件非常靈活，能夠滿足比較複雜的安全隔離需求。
傳統overlay網路架構
Calico提供的網路解決方案
本次搭建的基礎環境：
底層OS：Centos7 docker版本：1.8.2-el7.centos IP: 伺服器A：192.168.7.168 伺服器B：192.168.7.170 伺服器C：192.168.7.172
三臺機器上搭建基於docker的ETCD叢集——參見《quay.io/coreos/etcd 基於Docker映象的叢集搭建》
具體操作步驟：（注，請仔細觀察命令，[root@AAA ~]# calicoctl node 表示在A主機上執行的命令，同理B、C）
1、下載calicoctl及docker.io/calico/node映象（三臺機器均需要相同操作）
下載calicoctl，地址如下。為下載之後的檔案賦予可執行許可權，並複製到/usr/bin/下 連結：http://pan.baidu.com/s/1nuHn5hB 密碼：7yce 下載calico-node映象 [root@AAA ~]# docker pull docker.io/calico/node
2、啟動calico-node
[root@AAA ~]# calicoctl node No IP provided. Using detected IP: 192.168.7.168 Calico node is running with id: 6e754df308342753b259e89850f51b3e002780958bbc3f7c0803436548666560 [root@AAA ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6e754df30834 calico/node:latest "/sbin/start_runit" About a minute ago Up About a minute calico-node 0b5f487c20ae quay.io/coreos/etcd "/etcd -name qf2200-c" 4 minutes ago Up 4 minutes 4001/tcp, 0.0.0.0:2379-2380->2379-2380/tcp, 7001/tcp etcd
[root@BBB ~]# calicoctl node No IP provided. Using detected IP: 192.168.7.170 Calico node is running with id: 836bb8208dd992333c4ebc81d6312d1c0e53acffeca1b2ab3942a9483744fdf0 [root@BBB ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 836bb8208dd9 calico/node:latest "/sbin/start_runit" 19 seconds ago Up 19 seconds calico-node fa52ef61ccee quay.io/coreos/etcd "/etcd -name qf2200-c" 2 minutes ago Up 2 minutes 4001/tcp, 7001/tcp, 0.0.0.0:2379-2380->2379-2380/tcp etcd
[root@CCC ~]# calicoctl node No IP provided. Using detected IP: 192.168.7.172 Calico node is running with id: ff71c5939b119e724fca59e24039c7bbbc2adba9078f0b6c5ffa89359df92e2d [root@CCC ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ff71c5939b11 calico/node:latest "/sbin/start_runit" 21 seconds ago Up 20 seconds calico-node eb29998e8e92 quay.io/coreos/etcd "/etcd -name qf2200-c" 2 minutes ago Up 2 minutes 4001/tcp, 0.0.0.0:2379-2380->2379-2380/tcp, 7001/tcp etcd
3、處理calico 的IP資源池
[root@AAA ~]# calicoctl pool show +----------------+---------+ | IPv4 CIDR | Options | +----------------+---------+ | 192.168.0.0/16 | | +----------------+---------+ +--------------------------+---------+ | IPv6 CIDR | Options | +--------------------------+---------+ | fd80:24e2:f998:72d6::/64 | | +--------------------------+---------+ [root@AAA ~]# calicoctl pool remove 192.168.0.0/16 [root@AAA ~]# calicoctl pool show +-----------+---------+ | IPv4 CIDR | Options | +-----------+---------+ +-----------+---------+ +--------------------------+---------+ | IPv6 CIDR | Options | +--------------------------+---------+ | fd80:24e2:f998:72d6::/64 | | +--------------------------+---------+ [root@AAA ~]# calicoctl pool add 10.0.238.0/24 --nat-outgoing --ipip (支援跨子網的主機上的Docker間網路互通，需要新增--ipip引數；如果要Docker訪問外網，需要新增--nat-outgoing引數。) [root@AAA ~]# calicoctl pool show +---------------+-------------------+ | IPv4 CIDR | Options | +---------------+-------------------+ | 10.0.238.0/24 | ipip,nat-outgoing | +---------------+-------------------+ +--------------------------+---------+ | IPv6 CIDR | Options | +--------------------------+---------+ | fd80:24e2:f998:72d6::/64 | | +--------------------------+---------+
[root@BBB ~]# calicoctl pool show +---------------+-------------------+ | IPv4 CIDR | Options | +---------------+-------------------+ | 10.0.238.0/24 | ipip,nat-outgoing | +---------------+-------------------+ +--------------------------+---------+ | IPv6 CIDR | Options | +--------------------------+---------+ | fd80:24e2:f998:72d6::/64 | | +--------------------------+---------+
4、處理calico profile（類似於VLAN）
[root@AAA ~]# calicoctl profile add p1 Created profile p1 [root@AAA ~]# calicoctl profile add p2 Created profile p2 [root@AAA ~]# calicoctl profile show +------+ | Name | +------+ | p1 | | p2 | +------+
[root@CCC ~]# calicoctl profile show
+------+
| Name |
+------+
| p1 |
| p2 |
+------+
5、啟動net=none的容器
[root@AAA ~]# docker run -tid --name redis --restart=always --log-driver=none --net=none redis /run.sh b6d894f4cfcf36f5d19f3798447825730c80e95d1a9f98f326b77fae0ed85277 [root@AAA ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b6d894f4cfcf redis "/run.sh" 3 seconds ago Up 2 seconds redis 6e754df30834 calico/node:latest "/sbin/start_runit" 14 minutes ago Up 14 minutes calico-node 0b5f487c20ae quay.io/coreos/etcd "/etcd -name qf2200-c" 16 minutes ago Up 16 minutes 4001/tcp, 0.0.0.0:2379-2380->2379-2380/tcp, 7001/tcp etcd
[root@BBB ~]# docker run -tid --name redis --restart=always --log-driver=none --net=none redis /run.sh 4de1a0e2b2af5ad6c7f33138161105d46a07ce70d0b90b513125b28390a6a185 [root@BBB ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4de1a0e2b2af redis "/run.sh" 2 seconds ago Up 1 seconds redis 836bb8208dd9 calico/node:latest "/sbin/start_runit" 15 minutes ago Up 15 minutes calico-node fa52ef61ccee quay.io/coreos/etcd "/etcd -name qf2200-c" 17 minutes ago Up 17 minutes 4001/tcp, 0.0.0.0:2379-2380->2379-2380/tcp, 7001/tcp etcd
[root@CCC ~]# docker run -tid --name redis --restart=always --log-driver=none --net=none redis /run.sh b6801f99494ada054a8ef00fc5b74ff4aba4e156e506d94c0b781fa20f8b6f50 [root@CCC ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b6801f99494a redis "/run.sh" 2 seconds ago Up 1 seconds redis ff71c5939b11 calico/node:latest "/sbin/start_runit" 15 minutes ago Up 15 minutes calico-node eb29998e8e92 quay.io/coreos/etcd "/etcd -name qf2200-c" 17 minutes ago Up 17 minutes 4001/tcp, 0.0.0.0:2379-2380->2379-2380/tcp, 7001/tcp etcd
6、為容器配置IP及VLAN
[root@AAA ~]# calicoctl container add redis 10.0.238.1 IP 10.0.238.1 added to redis [root@AAA ~]# docker exec -ti redis ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0: <NOARP> mtu 0 qdisc noop state DOWN link/ipip 0.0.0.0 brd 0.0.0.0 39: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 12:b7:51:07:81:10 brd ff:ff:ff:ff:ff:ff inet 10.0.238.1/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::10b7:51ff:fe07:8110/64 scope link valid_lft forever preferred_lft forever [root@AAA ~]# calicoctl container redis profile append p1 Profile(s) p1 appended.
[root@BBB ~]# calicoctl container add redis 10.0.238.2 IP 10.0.238.2 added to redis [root@BBB ~]# docker exec -ti redis ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0: <NOARP> mtu 0 qdisc noop state DOWN link/ipip 0.0.0.0 brd 0.0.0.0 29: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether ca:19:45:10:19:80 brd ff:ff:ff:ff:ff:ff inet 10.0.238.2/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::c819:45ff:fe10:1980/64 scope link valid_lft forever preferred_lft forever [root@BBB ~]# calicoctl container redis profile append p1 Profile(s) p1 appended. [root@BBB ~]# calicoctl container redis profile append p2 Profile(s) p2 appended.
[root@CCC ~]# calicoctl container add redis 10.0.238.3 IP 10.0.238.3 added to redis [root@CCC ~]# docker exec -ti redis ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0: <NOARP> mtu 0 qdisc noop state DOWN link/ipip 0.0.0.0 brd 0.0.0.0 25: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether ca:6b:e2:63:44:28 brd ff:ff:ff:ff:ff:ff inet 10.0.238.3/32 scope global eth1 valid_lft forever preferred_lft forever inet6 fe80::c86b:e2ff:fe63:4428/64 scope link valid_lft forever preferred_lft forever [root@CCC ~]# calicoctl container redis profile append p2 Profile(s) p2 appended.
7、宿主機及容器網路拓撲
8、測試
[root@AAA ~]# docker exec -ti redis /bin/bash [root@b6d894f4cfcf /]# ping 10.0.238.1 （本機，可達） PING 10.0.238.1 (10.0.238.1) 56(84) bytes of data. 64 bytes from 10.0.238.1: icmp_seq=1 ttl=64 time=0.113 ms 64 bytes from 10.0.238.1: icmp_seq=2 ttl=64 time=0.052 ms ^C --- 10.0.238.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.052/0.082/0.113/0.031 ms [root@b6d894f4cfcf /]# ping 10.0.238.2 （同VLAN，可達） PING 10.0.238.2 (10.0.238.2) 56(84) bytes of data. 64 bytes from 10.0.238.2: icmp_seq=1 ttl=62 time=1.02 ms 64 bytes from 10.0.238.2: icmp_seq=2 ttl=62 time=0.533 ms ^C --- 10.0.238.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.533/0.776/1.020/0.245 ms [root@b6d894f4cfcf /]# ping 10.0.238.3 （不同VLAN，不可達） PING 10.0.238.3 (10.0.238.3) 56(84) bytes of data. ^C --- 10.0.238.3 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3022ms
[root@BBB ~]# docker exec -ti redis /bin/bash [root@4de1a0e2b2af /]# ping 10.0.238.1 （同VLAN，可達） PING 10.0.238.1 (10.0.238.1) 56(84) bytes of data. 64 bytes from 10.0.238.1: icmp_seq=1 ttl=62 time=2.08 ms 64 bytes from 10.0.238.1: icmp_seq=2 ttl=62 time=1.02 ms ^C --- 10.0.238.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 1.027/1.555/2.084/0.529 ms [root@4de1a0e2b2af /]# ping 10.0.238.2 （本機，可達） PING 10.0.238.2 (10.0.238.2) 56(84) bytes of data. 64 bytes from 10.0.238.2: icmp_seq=1 ttl=64 time=0.154 ms 64 bytes from 10.0.238.2: icmp_seq=2 ttl=64 time=0.066 ms ^C --- 10.0.238.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.066/0.110/0.154/0.044 ms [root@4de1a0e2b2af /]# ping 10.0.238.3 （同VLAN，可達） PING 10.0.238.3 (10.0.238.3) 56(84) bytes of data. 64 bytes from 10.0.238.3: icmp_seq=1 ttl=62 time=1.06 ms 64 bytes from 10.0.238.3: icmp_seq=2 ttl=62 time=0.442 ms ^C --- 10.0.238.3 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.442/0.752/1.062/0.310 ms
[root@CCC ~]# docker exec -ti redis /bin/bash [root@b6801f99494a /]# ping 10.0.238.1 （不同VLAN，不可達） PING 10.0.238.1 (10.0.238.1) 56(84) bytes of data. ^C --- 10.0.238.1 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2001ms [root@b6801f99494a /]# ping 10.0.238.2 （同VLAN，可達） PING 10.0.238.2 (10.0.238.2) 56(84) bytes of data. 64 bytes from 10.0.238.2: icmp_seq=1 ttl=62 time=0.384 ms 64 bytes from 10.0.238.2: icmp_seq=2 ttl=62 time=0.460 ms ^C --- 10.0.238.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1016ms rtt min/avg/max/mdev = 0.384/0.422/0.460/0.038 ms [root@b6801f99494a /]# ping 10.0.238.3 （本機，可達） PING 10.0.238.3 (10.0.238.3) 56(84) bytes of data. 64 bytes from 10.0.238.3: icmp_seq=1 ttl=64 time=0.055 ms 64 bytes from 10.0.238.3: icmp_seq=2 ttl=64 time=0.054 ms ^C --- 10.0.238.3 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.054/0.054/0.055/0.007 ms