docker筆記26-ingress資源和ingress controller
Ingress：就是能利用 Nginx（不常用）、Haproxy（不常用）、Traefik（常用）、Envoy（常用） 啥的負載均衡器暴露叢集內服務的工具。
Ingress為您提供七層負載均衡能力，您可以透過 Ingress 配置提供外部可訪問的 URL、負載均衡、SSL、基於名稱的虛擬主機等。作為叢集流量接入層，Ingress 的高可靠性顯得尤為重要。
小知識：我們把k8s裡面的pod服務釋出到叢集外部，可以用ingress，也可以用NodePort。
externalLB：外部的負載均衡器
service site：只是用來給pod分組歸類的。
[root@master manifests]# kubectl explain ingress
建立名稱空間：
[root@master manifests]# kubectl create namespace ingress-nginx namespace/dev created [root@master manifests]# kubectl get ns NAME STATUS AGE default Active 17d ingress-nginx Active 8s kube-public Active 17d kube-system Active 17d
訪問
各檔案的作用： configmap.yaml:提供configmap可以線上更行nginx的配置 default-backend.yaml:提供一個預設的後臺錯誤頁面 404 namespace.yaml:建立一個獨立的名稱空間 ingress-nginx rbac.yaml:建立對應的role rolebinding 用於rbac tcp-services-configmap.yaml:修改L4負載均衡配置的configmap udp-services-configmap.yaml:修改L4負載均衡配置的configmap with-rbac.yaml:有應用rbac的nginx-ingress-controller元件
訪問，裡面是ingress的部署文件
[root@master ~]# mkdir ingress-nginx [root@master ~]# cd ingress-nginx
部署ingress方法一（分步部署）：
下載如下配置檔案：
[root@master ingress-nginx]# for file in namespace.yaml configmap.yaml rbac.yaml tcp-services-configmap.yaml with-rbac.yaml udp-services-configmap.yaml; do wget
[root@master ingress-nginx]# ls configmap.yaml namespace.yaml rbac.yaml tcp-services-configmap.yaml udp-services-configmap.yaml with-rbac.yaml
1、建立名稱空間：
[root@master ingress-nginx]# kubectl apply -f namespace.yaml namespace/ingress-nginx configured
2、把剩下的ymal檔案全應用
[root@master ingress-nginx]# kubectl apply -f ./ configmap/nginx-configuration created namespace/ingress-nginx configured serviceaccount/nginx-ingress-serviceaccount created clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created role.rbac.authorization.k8s.io/nginx-ingress-role created rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created configmap/tcp-services created configmap/udp-services created deployment.extensions/nginx-ingress-controller created
[root@master ingress-nginx]# kubectl get pods -n ingress-nginx -w NAME READY STATUS RESTARTS AGE default-http-backend-6586bc58b6-qd9fk 0/1 running 0 4m nginx-ingress-controller-6bd7c597cb-zcbbz 0/1 running 0 1m
可以看到ingress-nginx名稱空間裡面有兩個pod都處於running狀態
部署ingress方法二（一鍵部署）：
只下載mandatory.yaml檔案，因為這個檔案裡面包含了上面所有yaml檔案裡面的內容。這是一鍵部署。
[root@master ingress-nginx]# wget
[root@master ingress-nginx]#kubectl apply -f
[root@master ~]# kubectl get pods -n ingress-nginx -w -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE default-http-backend-6586bc58b6-qd9fk 1/1 Running 0 11h 10.244.1.95 node1 <none> nginx-ingress-controller-6bd7c597cb-jlqzp 1/1 Running 3 11h 10.244.1.96 node1 <none>
可以看到ingress-nginx名稱空間裡面有兩個pod都處於running狀態
安裝service-nodeport
上面我們把ingress-nginx部署到了1號node上。接下來我們還需要部署一個service-nodeport服務，才能實現把叢集外部流量接入到叢集中來。
[root@master ingress]# wget
我們為了不讓service nodeport自動分配埠，我們自己指定一下nodeport，修改檔案中加兩個nodePort引數，最終如下：
[root@master ingress]# cat service-nodeport.yaml apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP nodePort: 30080 - name: https port: 443 targetPort: 443 protocol: TCP nodePort: 30443 selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx
[root@master ingress]# kubectl apply -f service-nodeport.yaml service/ingress-nginx created
[root@master ingress]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default-http-backend ClusterIP 10.110.74.183 <none> 80/TCP 12h ingress-nginx NodePort 10.102.78.188 <none> 80:30080/TCP,443:30443/TCP 2m
上面我看到80對應30080，,43對應30443
我們直接透過node1節點的ip就可以訪問到應用：
[root@master ingress]# curl default backend - 404
定義myapp service
[root@master manifests]# mkdir /root/manifests/ingress
[root@master ~]# kubectl explain service.spec.ports
[root@master ingress]# cat deploy-demo.yaml apiVersion: v1 kind: Service #必須設定為無頭service metadata: name: myapp namespace: default spec: selector: app: myapp release: canary ports: - name: http targetPort: 80 #這是容器port port: 80 #這是service port --- apiVersion: apps/v1 kind: Deployment metadata: name: myapp-deploy namespace: default spec: replicas: 2 selector: #標籤選擇器 matchLabels: #匹配的標籤為 app: myapp release: canary template: metadata: labels: app: myapp #和上面的myapp要匹配 release: canary spec: containers: - name: myapp image: ikubernetes/myapp:v1 ports: - name: http containerPort: 80
[root@master ingress]# kubectl apply -f deploy-demo.yaml service/myapp created deployment.apps/myapp-deploy unchanged
[root@master ingress]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17d myapp ClusterIP 10.108.177.62 <none> 80/TCP 1m
[root@master ingress]# kubectl get pods NAME READY STATUS RESTARTS AGE myapp-deploy-69b47bc96d-79fqh 1/1 Running 0 1d myapp-deploy-69b47bc96d-tc54k 1/1 Running 0 1d
把myapp service透過ingress釋出出去
下面我們再定義一個清單檔案，把myapp應用透過Ingress（相當於nginx的反向代理功能）釋出出去：
[root@master ingress]# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-myapp namespace: default #要和deployment和要釋出的service處於同一個名稱空間 annotations: #這個註解說明我們要用到的ingress-controller是nginx，而不是traefic，enjoy kubernetes.io/ingress.class: "nginx" spec: rules: - host: myapp.zhixin.com #表示訪問這個域名，就會轉發到後端myapp管理的pod上的服務: http: paths: - path: backend: serviceName: myapp servicePort: 80
[root@master ingress]# kubectl apply -f ingress-myapp.yaml ingress.extensions/ingress-myapp created
[root@master ingress]# kubectl get ingress NAME HOSTS ADDRESS PORTS AGE ingress-myapp myapp.zhixin.com 80 8m
[root@master ingress]# kubectl describe ingress
[root@master ingress]# kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-6586bc58b6-qd9fk 1/1 Running 0 12h nginx-ingress-controller-6bd7c597cb-jlqzp 1/1 Running 3 12h
進入ingress-controller互動式命令列裡面，可以清晰的看到nginx是怎麼反向代理我們myapp.zhixin.com的：
[root@master ingress]# kubectl exec -n ingress-nginx -it nginx-ingress-controller-6bd7c597cb-jlqzp -- /bin/sh
$ cat nginx.conf
## start server myapp.zhixin.com
server {
server_name myapp.zhixin.com ;
listen 80;
set $proxy_upstream_name "-";
location / {
set $namespace "default";
set $ingress_name "ingress-myapp";
set $service_name "myapp";
set $service_port "80";
........
測試，下面我們把myapp.zhixin.com域名解析到node1 ip 172.16.1.101上。
[root@master ingress]# curl myapp.zhixin.com:30080 Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
把tomcat service透過ingress釋出出去（新例子）
[root@master ingress]# cat tomcat-demo.yaml apiVersion: v1 kind: Service #必須設定為無頭service metadata: name: tomcat namespace: default spec: selector: app: tomcat release: canary ports: - name: http targetPort: 8080 #這是容器port port: 8080 #這是service port - name: ajp targetPort: 8009 port: 8009 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-deploy namespace: default spec: replicas: 2 selector: #標籤選擇器 matchLabels: #匹配的標籤為 app: tomcat release: canary template: metadata: labels: app: tomcat #和上面的myapp要匹配 release: canary spec: containers: - name: tomcat image: tomcat:8.5.34-jre8-alpine #在上面找 ports: - name: http containerPort: 8080 - name: ajp containerPort: 8009
[root@master ingress]# kubectl apply -f tomcat-demo.yaml service/tomcat created deployment.apps/tomcat-deploy created
[root@master ingress]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17d tomcat ClusterIP 10.109.76.87 <none> 8080/TCP,8009/TCP 1m
[root@master ingress]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-deploy-64c4d54df4-68sk8 1/1 Running 0 51s tomcat-deploy-64c4d54df4-7b58g 1/1 Running 0 51s
[root@master ingress]# cat ingress-tomcat.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat namespace: default #要和deployment和要釋出的service處於同一個名稱空間 annotations: #這個註解說明我們要用到的ingress-controller是nginx，而不是traefic，enjoy kubernetes.io/ingress.class: "nginx" spec: rules: - host: tomcat.zhixin.com #表示訪問這個域名，就會轉發到後端myapp管理的pod上的服務: http: paths: - path: backend: serviceName: tomcat servicePort: 8080
[root@master ingress]# kubectl apply -f ingress-tomcat.yaml ingress.extensions/ingress-myapp configured
[root@master ingress]# kubectl get ingress NAME HOSTS ADDRESS PORTS AGE ingress-tomcat tomcat.zhixin.com 80 11s
[root@master ingress]# kubectl describe ingress ingress-tomcat Name: ingress-tomcat Namespace: default Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- tomcat.zhixin.com tomcat:8080 (<none>) Annotations: kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 1m nginx-ingress-controller Ingress default/ingress-tomcat
把tomcat.zhixin.com解析到node1上節點物理ip（我的是172.16.1.101）
測試，可以看到tomcat歡迎介面：
[root@master ingress]# curl tomcat.zhixin.com:30080
使用https訪問（新例子）
1、先做個自籤的證書（我們這裡不演示CA的例子）
[root@master ingress]# openssl genrsa -out tls.key 2048
[root@master ingress]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/O=DevOps/CN=tomcat.zhixin.com
2、透過secret把證書注入到pod中。
[root@master ingress]# kubectl create secret tls tomcat-infress-secret --cert=tls.crt --key=tls.key secret/tomcat-infress-secret created
[root@master ingress]# kubectl get secret NAME TYPE DATA AGE default-token-5r85r kubernetes.io/service-account-token 3 17d tomcat-ingress-secret kubernetes.io/tls 2 41s
[root@master ingress]# kubectl describe secret tomcat-ingress-secret Name: tomcat-ingress-secret Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 1245 bytes tls.key: 1679 bytes
3、配置ingress為tls方式
[root@master ingress]# cat ingress-tomcat-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat-tls namespace: default #要和deployment和要釋出的service處於同一個名稱空間 annotations: #這個註解說明我們要用到的ingress-controller是nginx，而不是traefic，enjoy kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - tomcat.zhixin.com secretName: tomcat-ingress-secret #kubectl get secret命令查到的名字 rules: - host: tomcat.zhixin.com #表示訪問這個域名，就會轉發到後端myapp管理的pod上的服務: http: paths: - path: backend: serviceName: tomcat servicePort: 8080
[root@master ingress]# kubectl get ingress NAME HOSTS ADDRESS PORTS AGE ingress-tomcat tomcat.zhixin.com 80 2h ingress-tomcat-tls tomcat.zhixin.com 80, 443 3m
[root@master ingress]# kubectl describe ingress ingress-tomcat-tls
Name: ingress-tomcat-tls
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
tomcat-ingress-secret terminates tomcat.zhixin.com
Rules:
Host Path Backends
---- ---- --------
tomcat.zhixin.com
tomcat:8080 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-tomcat-tls","namespace":"default"},"spec":{"rules":[{"host":"tomcat.zhixin.com","http":{"paths":[{"backend":{"serviceName":"tomcat","servicePort":8080},"path":null}]}}],"tls":[{"hosts":["tomcat.zhixin.com"],"secretName":"tomcat-ingress-secret"}]}}
kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 4m nginx-ingress-controller Ingress default/ingress-tomcat-tls
4、連如ingress-controller檢視nginx.conf的配置
[root@master ingress]# kubectl get pods -n ingress-nginx NAME READY STATUS RESTARTS AGE default-http-backend-6586bc58b6-qd9fk 1/1 Running 0 16h nginx-ingress-controller-6bd7c597cb-jlqzp 1/1 Running 3 16h
[root@master ingress]# kubectl exec -n ingress-nginx -it nginx-ingress-controller-6bd7c597cb-jlqzp -- /bin/sh
$ $ cat nginx.conf
## start server tomcat.zhixin.com
server {
server_name tomcat.zhixin.com ;
listen 80;
set $proxy_upstream_name "-";
listen 443 ssl http2;
看到有listen 443了。
5、測試https
[root@master ingress]# curl
部署方法三（利用ingress 的80埠）
前面兩種部署方法，是用node ip + 非80埠，訪問k8s叢集內部的服務。可是，我們實際生產中更希望的是node ip + 80埠的方式，訪問k8s叢集內的服務。我感覺這個方法最好，下面就就介紹這個方法。
這部分內容參考的是博文http://blog.51cto.com/devingeng/2149377
下載地址
ingress-nginx檔案位於deploy目錄下，各檔案的作用： configmap.yaml:提供configmap可以線上更行nginx的配置 default-backend.yaml:提供一個預設的後臺錯誤頁面 404 namespace.yaml:建立一個獨立的名稱空間 ingress-nginx rbac.yaml:建立對應的role rolebinding 用於rbac tcp-services-configmap.yaml:修改L4負載均衡配置的configmap udp-services-configmap.yaml:修改L4負載均衡配置的configmap with-rbac.yaml:有應用rbac的nginx-ingress-controller元件
修改with-rbac.yaml
apiVersion: extensions/v1beta1 kind: Daemonset metadata: name: nginx-ingress-controller namespace: ingress-nginx spec: selector: matchLabels: app: ingress-nginx template: metadata: labels: app: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount hostNetwork: true containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --annotations-prefix=nginx.ingress.kubernetes.io env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 nodeSelector: custom/ingress-controller-ready: "true"
需要修改的地方：
kind: DaemonSet：官方原始檔案使用的是deployment，replicate 為 1，這樣將會在某一臺節點上啟動對應的nginx-ingress-controller pod。外部流量訪問至該節點，由該節點負載分擔至內部的service。測試環境考慮防止單點故障，改為DaemonSet然後刪掉replicate ，配合親和性部署在制定節點上啟動nginx-ingress-controller pod，確保有多個節點啟動nginx-ingress-controller pod，後續將這些節點加入到外部硬體負載均衡組實現高可用性。
hostNetwork: true：新增該欄位，暴露nginx-ingress-controller pod的服務埠（80）
nodeSelector: 增加親和性部署，有custom/ingress-controller-ready 標籤的節點才會部署該DaemonSet
為需要部署nginx-ingress-controller的節點設定lable
kubectl label nodes vmnode2 custom/ingress-controller-ready=true kubectl label nodes vmnode3 custom/ingress-controller-ready=true kubectl label nodes vmnode4 custom/ingress-controller-ready=true
載入yaml檔案
kubectl apply -f namespace.yaml kubectl apply -f default-backend.yaml kubectl apply -f configmap.yaml kubectl apply -f tcp-services-configmap.yaml kubectl apply -f udp-services-configmap.yaml kubectl apply -f rbac.yaml kubectl apply -f with-rbac.yaml
檢視pod是否正常建立
##下載映象可能會比較慢，等待一會所有pod都是Running狀態，按Ctrl + c 退出
[root@vmnode1 deploy]# kubectl get pods --namespace=ingress-nginx --watch NAME READY STATUS RESTARTS AGEdefault- http-backend-6c59748b9b-hc8q9 1/1 Running 0 6m nginx-ingress-controller-7fmlp 1/1 Running 1 13d nginx-ingress-controller-j95fb 1/1 Running 1 13d nginx-ingress-controller-ld2jw 1/1 Running 1 13d
測試ingress
建立一個tomcat的Service
[root@k8s-master1 test]# cat mytomcat.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: mytomcat spec: replicas: 2 template: metadata: labels: run: mytomcat spec: containers: - name: mytomcat image: tomcat ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: mytomcat labels: run: mytomcat spec: type: NodePort ports: - port: 8080 targetPort: 8080 selector: run: mytomcat
[root@k8s-master1 test]# kubectl apply -f mytomcat.yaml
配置ingress轉發檔案：
[root@k8s-master1 test]# cat test-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress namespace: default spec: rules: - host: test.zhixin.com http: paths: - path: / backend: serviceName: mytomcat servicePort: 8080
host: 對應的域名
path: url上下文
backend:後向轉發 到對應的 serviceName: servicePort:
[root@k8s-master1 test]# kubectl apply -f test-ingress.yaml ingress.extensions/test-ingress created
nginx-ingress-controller執行在node1和nod2兩個節點上。如果網路中有dns伺服器，在dns中把這兩個域名對映到nginx-ingress-controller執行的任意一個節點上，如果沒有dns伺服器只能修改host檔案了。
正規的做法是在node1和node2這兩個節點上安裝keepalive，生成一個vip。在dns上把域名和vip做對映。
我這裡直接在node1節點上操作了：
我這裡node1節點的ip是172.16.22.201；node2節點的ip是172.16.22.202
[root@k8s-master1 test]# echo " 172.16.22.201 test.zhixin.com" >> /etc/hosts [root@k8s-master1 test]# echo "172.16.22.202 test.zhixin.com" >> /etc/hosts
然後訪問測試：
看到，我們把域名test.zhixin.com繫結到Node節點的ip補上，然後我們直接訪問，就能訪問到k8s叢集裡面的pod服務。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28916011/viewspace-2214747/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- ingress-nginx-controller部署NginxController
- docker筆記34-容器資源需求、資源限制及HeapSterDocker筆記
- docker筆記25-k8s-service資源Docker筆記K8S
- jMeter Transaction Controller 學習筆記JMeterController筆記
- k8s階段05 Operator, DaemonSet, Job, CronJob, Ingress和Ingress Controller(藍綠/恢復釋出), helmK8SController
- 筆記資源整理筆記
- helm 安裝 nginx-ingress-controller v1.10.0NginxController
- 乾貨|Kubernetes叢集部署 Nginx-ingress ControllerNginxController
- k8s安裝nginx-ingress-controllerK8SNginxController
- 使用 Nocalhost 開發 Kubernetes 中的 APISIX Ingress ControllerAPIController
- Docker筆記（八）：資料管理Docker筆記
- 【Kubernetes系列】第6篇 Ingress controller - nginx元件介紹ControllerNginx元件
- 【Kubernetes系列】第5篇 Ingress controller - traefik元件介紹Controller元件
- docker 筆記Docker筆記
- 筆記：Docker筆記Docker
- Docker筆記Docker筆記
- docker筆記35-資源指標API及自定義指標APIDocker筆記指標API
- docker筆記22-k8s資源清單定義入門Docker筆記K8S
- 如何使用 Docker 來限制 CPU、記憶體和 IO等資源？Docker記憶體
- k8s 新版本 部署 Ingress-nginx controllerK8SNginxController
- docker 筆記2Docker筆記
- docker 筆記4Docker筆記
- Docker Stack 筆記Docker筆記
- docker筆記（一）Docker筆記
- docker使用筆記Docker筆記
- CORS（跨域資源共享）筆記CORS跨域筆記
- docker的資源控制（CPU、記憶體、IO）Docker記憶體
- Docker筆記之DockerfileDocker筆記
- docker簡單筆記Docker筆記
- Docker學習筆記Docker筆記
- Docker的使用筆記Docker筆記
- Docker 學習筆記Docker筆記
- Docker筆記之Docker初體驗Docker筆記
- Docker筆記（四）：Docker映象管理Docker筆記
- Docker筆記（一）：什麼是DockerDocker筆記
- docker筆記12-容器資料卷volumesDocker筆記
- Docker學習筆記：映象、容器、資料卷Docker筆記
- Effective C++ 筆記（3）資源管理C++筆記