使用KubernetesIngress來實現類似Istio條件路由

微服務 Istio / SpringCloud日益被越來越多的客戶關注，Istio提供了各種酷炫的流量控制功能，但Istio距離生產部署可用仍然還有差距。條件路由是否可以在已有的Kubernetes Ingress架構中實現，以最小的代價實現應用的微服務化遷移。答案是肯定的，通過對ingress自定義location/server塊的定義，以及upsteam自動生產的規則，可以實現複雜條件路由的支援，類似istio, match。




[不滿足] 1. 嘗試一， 修改Ingress的server-snippet/location-snippet屬性來重定向請求到後臺服務, 不能按期望跳轉

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: ip_hash
    nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;


      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }


     if ( $agentflag = 1 ) {
      .  return 301 http://default-cardinfo-homepage-80;
     }
    nginx.ingress.kubernetes.io/upstream-hash-by: ip_hash
    nginx.ingress.kubernetes.io//location-snippet: |
                  if ( $agentflag = 1 ) {
                     proxy_pass http://default-cardinfo-homepage-80;
                  }
  creationTimestamp: 2018-08-27T12:06:32Z
  generation: 2
  name: nginx-test
  namespace: default
  resourceVersion: "43290755"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/nginx-test
  uid: a310ca06-a9f1-11e8-a613-00163e0c87f1
spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /
  - host: mobile.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-homepage
          servicePort: 80
        path: /
status:
  loadBalancer:
    ingress:
    - {}

檢查ingressController自動生成的nginx server定義, 僅僅依靠server-snippet 以及location-snippet是不能完成請求的自動跳轉。server-snippet at ingress level does not work for reverse proxy server-snippet spec

[不滿足] 嘗試二, 使用IngressController級別的定義 location-snippet， location-snippet會傳播到所有的location-block, 不滿足期望。

location-snippet of ingress controller is applied to all location block, it does not work as expected


location-snippet



apiVersion: v1
data:
  location-snippet: |
    if ( $agentflag = 1 ) {
                     proxy_pass http://default-cardinfo-homepage-80;
     }
  proxy-body-size: 20m
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"proxy-body-size":"20m"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"ingress-nginx"},"name":"nginx-configuration","namespace":"kube-system"}}
  creationTimestamp: 2018-02-07T09:06:33Z
  labels:
    app: ingress-nginx
  name: nginx-configuration
  namespace: kube-system
  resourceVersion: "43308775"
  selfLink: /api/v1/namespaces/kube-system/configmaps/nginx-configuration
  uid: 312dd92f-0be6-11e8-856b-00163e0c87f1

[解決] 嘗試三， 使用configuration-snippet + server-snippet + upstream default name 來實現路由跳轉到不同的kubernetes service服務。

configuration-snippet is alternative of location snippet, combination configuration-snippet and server-snippet to parse header, set global variable, then proxy specific backend service with upstream name created by IngressController. e.g. default-cardinfo-homepage-80

metadata:
  annotations:
     nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;

      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ( $agentflag = 1 ) {
         proxy_pass http://default-cardinfo-homepage-80;
      }

Put everything together

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: ip_hash
    nginx.ingress.kubernetes.io/server-snippet: |
      set $agentflag 0;

      if ($http_user_agent ~* "(Mobile)" ){
        set $agentflag 1;
      }
    nginx.ingress.kubernetes.io/upstream-hash-by: ip_hash
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ( $agentflag = 1 ) {
         proxy_pass http://default-cardinfo-homepage-80;
      }
  name: nginx-condition
  namespace: default
spec:
  rules:
  - host: stickyingress.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-recommendation
          servicePort: 80
        path: /
  - host: mobile.example.com
    http:
      paths:
      - backend:
          serviceName: cardinfo-homepage
          servicePort: 80
        path: /
status:
  loadBalancer:
    ingress:
    - {}

測試

使用不同的Header 模擬不同使用者瀏覽器請求，使用者請求根據瀏覽器的型別被重定向到不同的後臺服務和入口。以stickyingress.example.com為入口的使用者使用”user-agent: (Mobile)”的使用者，自動跳轉到後臺服務cardinfo-homepage，其他使用者仍然進入cardinfo-recommendation。

curl stickyingress.example.com:32619 -v
* Rebuilt URL to: stickyingress.example.com:32619/
* Hostname was NOT found in DNS cache
*   Trying 192.168.33.239...
* Connected to stickyingress.example.com (192.168.33.239) port 32619 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.37.0
> Host: stickyingress.example.com:32619
> Accept: */*
>
< HTTP/1.1 200
* Server nginx/1.13.7 is not blacklisted
< Server: nginx/1.13.7
< Date: Wed, 14 Nov 2018 04:46:19 GMT
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 17
< Connection: keep-alive
<
* Connection #0 to host stickyingress.example.com left intact
recommendation v2

dns:~ # curl stickyingress.example.com:32619 -v -H "user-agent: (Mobile)"

* Rebuilt URL to: stickyingress.example.com:32619/
* Hostname was NOT found in DNS cache
*   Trying 192.168.33.239...
* Connected to stickyingress.example.com (192.168.33.239) port 32619 (#0)
> GET / HTTP/1.1
> Host: stickyingress.example.com:32619
> Accept: */*
> user-agent: (Mobile)
>
< HTTP/1.1 200
* Server nginx/1.13.7 is not blacklisted
< Server: nginx/1.13.7
< Date: Wed, 14 Nov 2018 04:46:31 GMT
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 26
< Connection: keep-alive
< Vary: Accept-Encoding
<
* Connection #0 to host stickyingress.example.com left intact
OK. Vist /cardinfo please.

Conclusion 總結

使用Ingress的自定義location/server block，以及upstream的隱含定義，依然可以實現類似Istio的自動流量路由功能，後臺Pod出現更新升級並不會影響到條件跳轉。 使用者可以使用最小的代價通過Kubernetes Ingress實現ABTest，灰度，流量路由，並且享受到Kubernetes服務發現，和滾動更新帶來的優勢。