搭建平臺目的:
k8s中搭建jenkins master/slave架構,解決單jenkins執行效率低,資源不足等問題(jenkins master 排程任務到 slave上,併發執行任務,提升任務執行的效率)
CI/CD環境特點:
Slave彈性伸縮
基於映象隔離構建環境
流水線釋出,易維護
一、環境準備
服務名 | 地址 | 版本 |
k8s-master | 10.48.14.100 | v1.22.3 |
k8s-node1 | 10.48.14.50 | v1.22.3 |
k8s-node2 | 10.48.14.51 | v1.22.3 |
gogs程式碼倉庫 | 10.48.14.50:30080 | |
harbor映象倉庫 | 10.48.14.50:8888 | v1.8.1 |
使用gogs作為程式碼倉庫,harbor作為映象倉庫:搭建參考(https://www.cnblogs.com/cfzy/p/16049885.html)
二、瞭解釋出流程
1.藍綠髮布 專案邏輯上分為AB組,在專案升級時,首先把A組從負 載均衡中摘除,進行新版本的部署。 B組仍然繼續提供 服務。A組升級完成上線,B組從負載均衡中摘除。 特點: 策略簡單 升級/回滾速度快 使用者無感知,平滑過渡 缺點: 需要兩倍以上伺服器資源 短時間內浪費一定資源成本
2.灰度釋出 灰度釋出:
只升級部分服務,即讓一部分使用者繼續用 老版本,一部分使用者開始用新版本,如果使用者對新版 本沒有什麼意見,那麼逐步擴大範圍,把所有使用者都 遷移到新版本上面來。 特點: 保證整體系統穩定性 使用者無感知,平滑過渡 缺點: 自動化要求高
k8s中的落地方式
3.滾動釋出 滾動釋出: 每次只升級一個或多個服務,升級完成 後加入生產環境,不斷執行這個過程,直到叢集中 的全部舊版升級新版本。 特點: 使用者無感知,平滑過渡 缺點: 部署週期長 釋出策略較複雜 不易回滾
三、在Kubernetes中部署Jenkins
3.1 部署jenkins
建立動態PVC:為Jenkins提供持久化儲存(因為之前建立了NFS作為後端儲存的PVC"managed-nfs-storage",所以直接拿來用了)
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins spec: storageClassName: "managed-nfs-storage" accessModes: - ReadWriteMany resources: requests: storage: 5Gi
kubectl apply -f pvc.yml
建立deploy資源執行Jenkins服務:
apiVersion: apps/v1 kind: Deployment metadata: name: jenkins labels: name: jenkins spec: replicas: 1 selector: matchLabels: name: jenkins template: metadata: name: jenkins labels: name: jenkins spec: serviceAccountName: jenkins containers: - name: jenkins image: jenkins/jenkins:lts-jdk11 ports: - containerPort: 8080 - containerPort: 50000 resources: limits: cpu: 2 memory: 2Gi requests: cpu: 1 memory: 1Gi env: - name: TZ value: Asia/Shanghai - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home securityContext: fsGroup: 1000 volumes: - name: jenkins-home persistentVolumeClaim: claimName: jenkins
kubectl apply -f deployment.yml
建立名為Jenkins的SA,並授權:
# 建立名為jenkins的ServiceAccount apiVersion: v1 kind: ServiceAccount metadata: name: jenkins --- # 建立名為jenkins的Role,授予允許管理API組的資源Pod kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins rules: - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- # 將名為jenkins的Role繫結到名為jenkins的ServiceAccount apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: jenkins subjects: - kind: ServiceAccount name: jenkins
kubectl apply -f rbac.yaml
暴露Jenkins服務埠:
apiVersion: v1 kind: Service metadata: name: jenkins spec: selector: name: jenkins type: NodePort ports: - name: http port: 80 targetPort: 8080 protocol: TCP nodePort: 30006 - name: agent port: 50000 protocol: TCP targetPort: 50000
kubectl apply -f service.yml
為Jenkins的URL設定域名訪問:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: jenkins annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/ssl-redirect: "true" kubernetes.io/tls-acme: "true" nginx.ingress.kubernetes.io/client_max_body_size: 100m nginx.ingress.kubernetes.io/proxy-body-size: 50m nginx.ingress.kubernetes.io/proxy-request-buffering: "off" spec: rules: - host: jenkins.test.com http: paths: - path: / pathType: Prefix backend: service: name: jenkins port: number: 80
kubectl apply -f ingress.yml
在host檔案新增dns記錄
訪問jenkins:jenkins.test.com
登入頁面,安裝推薦外掛,如果安裝失敗就更換成國內源地址
3.2 配置jenkins下載外掛地址,並安裝必要外掛
cd $jenkins_home/ sed -i 's#https://updates.jenkins.io/update-center.json#http://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json#g' hudson.model.UpdateCenter.xml cd $jenkins_home/updates 替換外掛源地址: sed -i 's#https://updates.jenkins.io/download#http://mirrors.aliyun.com/jenkins#g' default.json 替換谷歌地址: sed -i 's#http://www.google.com#http://www.baidu.com#g' default.json 安裝外掛:Git/Git Parameter/Pipeline/Kubernetes/Kubernetes Continuous Deploy/Config File Provider Kubernetes Continuous Deploy: 用於將資源配置部署到Kubernetes(該外掛在構建過程中報錯,所以被我棄用了) Config File Provider:用於儲存kubectl用於連線k8s叢集的kubeconfig配置檔案
3.3 Jenkins在K8S中動態建立代理
3.3.1 配置Kubernetes plugin Jenkins頁面配置k8s叢集資訊:系統管理——系統配置——Cloud——配置叢集
注意:這個是最重要的一個配置,決定整個安裝的成敗,"kubernetes地址" 用"https://kubernetes.default"或者"https://k8s叢集主節點的ip+埠", 然後點選"連線測試",連線成功會出現k8s版本號。 為什麼連k8s不需要憑證:jenkins是在k8s內部搭建的,所以不需要k8s憑證,如果是在外部搭建的就需要新增k8s憑證 jenkins地址: kubectl get svc #檢視jenkins的埠 jenkins通道:這個引數是Jenkins Master和Jenkins Slave之間通訊必須配置的,kubectl get svc #檢視ip和埠
3.3.2 構建Jenkins—slave映象(Dockerfile) #jenkins 官方有jenkins-slave 製作好的映象,可以直接 docker pull jenkins/jnlp-slave 下載到本地並上傳本地私有映象廠庫。官方的映象好處就是不需要再單獨安裝maven,kubectl 這樣的命令了,可以直接使用。 可能還需要安裝gradle,jdk等,所以我們自己製作映象,構建映象所需要的檔案: #在https://github.com/fxkjnj/kubernetes/tree/main/jenkins-for_kubernetes/jenkins-slave 目錄下
Dockerfile檔案如下:
FROM centos:7 MAINTAINER liang ENV JAVA_HOME=/usr/local/java
ENV PATH=$JAVA_HOME/bin:/usr/local/gradle/bin:$PATH
RUN yum install -y maven curl git libtool-ltdl-devel && \ yum clean all && \ rm -rf /var/cache/yum/* && \ mkdir -p /usr/share/jenkins COPY jdk-11.0.9 /usr/local/java
COPY gradle6.4 /usr/local/gradle COPY slave.jar /usr/share/jenkins/slave.jar COPY jenkins-slave /usr/bin/jenkins-slave COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave COPY kubectl /usr/bin ENTRYPOINT ["jenkins-slave"]
jenkins-slave:shell指令碼,用於啟動slave.jar
settings.xml: 修改maven官方源為阿里雲源
slave.jar: agent程式,接收master下發的任務
kubectl: 讓jenkins-slave可以執行kubectl命令,cp /usr/bin/kubectl ./
構建dockerfile,生成slave-agent映象
docker build -t jenkins-slave-jdk:11 .
上傳到harbor倉庫
docker tag jenkins-slave-jdk:11 10.48.14.50:8888/library/jenkins-slave-jdk:11
docker login -u admin 10.48.14.50:8888
docker push 10.48.14.50:8888/library/jenkins-slave-jdk:11
3.3.3 建立一個流水線任務,測試jenkins-slave功能(在k8s中動態建立代理)
建立流水線任務 "test"
編寫pipeline測試指令碼(宣告式指令碼)
需要注意的是,spec中定義containers名字一定要寫jnlp
pipeline { agent { kubernetes { label "jenkins-slave" yaml """ kind: Pod metadata: name: jenkins-slave spec: containers: - name: jnlp image: "10.48.14.50:8888/library/jenkins-slave-jdk:11" """ } } stages { stage('測試'){ steps { sh """ echo hello """ } } } }
儲存,然後構建任務,檢視日誌資訊
測試完成,功能正常。
在構建的時候,k8s叢集default名稱空間下,會臨時啟動一個pod(jenkins-slave-dr835-dh9dz),這個pod就是jenkins動態建立的代理,
用於執行jenkins-master下發的構建任務,當jenkins構建完成後,這個pod自動銷燬
四、Jenkins在K8S中持續部署(完整流程)
4.1 持續整合和持續部署流程
持續整合CI:提交程式碼——程式碼構建——可部署的包——打包映象——推送映象倉庫 持續部署CD:kubectl命令列/yaml檔案——建立資源——暴露應用——更新映象/回滾/擴容——刪除資源 jenkins在k8s中持續整合部署流程 拉取程式碼:git checkout 程式碼編譯:mvn clean
構建映象並推送遠端倉庫
部署到K8S
開發測試
用kubectl命令列持續部署 1、建立資源(deployment) kubectl create deployment tomcat --image=tomcat:v1 kubectl get pods,deploy 2、釋出服務(service) kubectl expose deployment tomcat --port=80 --target-port=8080 --name=tomcat-service --type=NodePort --port 叢集內部訪問的service埠,即通過clusterIP:port可以訪問到某個service --target-port 是pod的埠,從port和nodeport來的流量經過kube-proxy流入到後端pod的targetport上,最後進入容器 nodeport:外部訪問k8s叢集中service的埠,如果不定義埠號會預設分配一個 containerport:是pod內部容器的埠,targetport對映到containerport(一般在deployment中設定) kubectl get service 3、升級 kubectl set image deployment tomcat 容器名稱=tomcat:v2 --record=true #檢視升級狀態 kubectl rollout status deployment/tomcat 4、擴容縮容 kubectl scale deployment tomcat --replicas=10 5、回滾 kubectl rollout history deployment/tomcat #檢視版本釋出歷史 kubectl rollout undo deployment/tomcat #回滾到上一版本 kubectl rollout undo deployment/tomcat --to-revision=2 #回滾到指定版本 6、刪除 kubectl delete deployment/tomcat #刪除deployment資源 kubectl delete service/tomcat-service #刪除service資源
4.2 分步生成CI/CDpipeline語法
4.2.1 拉取程式碼(git checkout)
新增憑證(git倉庫、harbor倉庫):系統管理——憑據配置——新增harbor、git倉庫的使用者名稱/密碼
用pipeline語法生成器,生成拉取程式碼步驟的pipeline語法:
根據自己的程式碼倉庫資訊填寫,然後點選生成流水線指令碼,就有了拉取程式碼的指令碼語法:(credentialsID、URL等資訊可以定義成變數傳輸)
checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[credentialsId: 'a5ec87ae-87a1-418e-aa49-53c4aedcd261', url: 'http://10.48.14.100:30080/001/java-demo.git']]])
4.2.2 程式碼編譯
mvn clean package -Dmaven.test.skip=true
4.2.3 構建映象
製作一個tomcat映象,因為java服務要跑著tomcat中:下載安裝包apache-tomcat-8.5.34.tar.gz,並編寫Dockerfile
FROM centos:7 MAINTAINER liang ENV VERSION=8.5.34 RUN yum install -y java-1.8.0-openjdk wget curl unzip iproute net-tools && \ yum clean all && \ rm -rf /var/cache/yum/* COPY apache-tomcat-${VERSION}.tar.gz / RUN tar -zxf apache-tomcat-${VERSION}.tar.gz && \ mv apache-tomcat-${VERSION} /usr/local/tomcat && \ rm -rf apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/* && \ mkdir /usr/local/tomcat/webapps/test && \ echo "ok" > /usr/local/tomcat/webapps/test/status.html && \ sed -i '1a JAVA_OPTS="-Djava.security.edg=file:/dev/./urandom"' /usr/local/tomcat/bin/catalina.sh && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime ENV PATH $PATH:/usr/local/tomcat/bin WORKDIR /usr/local/tomcat EXPOSE 8080 CMD ["catalina.sh","run"]
構建映象,並上傳映象到harbor倉庫
docker build -t tomcat:v1 .
docker tag tomcat:v1 10.48.14.50:8888/library/tomcat:v1
docker login -u admin 10.48.14.50:8888
docker push 10.48.14.50:8888/library/tomcat:v1
以tomcat:v1為基礎映象,構建專案映象,並上傳到harbor倉庫
FROM 10.48.14.50:8888/library/tomcat:v1 LABEL maitainer lizhenliang RUN rm -rf /usr/local/tomcat/webapps/* ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
docker build -t 10.48.14.50:8888/dev/java-demo:v1 .
docker login -u admin 10.48.14.50:8888
docker push 10.48.14.50:8888/dev/java-demo:v1
通過credential外掛生成ID號來隱藏harbor使用者名稱密碼,生成pipeline語法:
pipeline中構建映象並上傳到harbor倉庫:
#其中涉及的變數可以pipeline中定義 stage('構建映象'){ steps { withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) { sh """ echo ' FROM ${registry}/library/tomcat:v1 MAINTAINER liang RUN rm -rf /usr/local/tomcat/webapps/* ADD target/*.war /usr/local/tomcat/webapps/ROOT.war ' > Dockerfile docker build -t ${image_name} . docker login -u ${username} -p '${password}' ${registry} docker push ${image_name} """ } } }
4.2.4 部署服務到k8s
在k8s中為使用者admin授權,生成kubeconfig檔案。或者直接複製/root/.kube/config(這是一個kubeconfig檔案)
Jenkins-slave映象已經有kubectl命令,只需要kubeconfig就可以連線k8s叢集
把生成的kubeconfig檔案放到Jenkins中:需要安裝Config File Provider外掛,在Mansged files中配置
Manage Jenkins -> Managed files -> Add a new Config -> Custom file(自定義檔案)
將生成的kubeconfig檔案內容複製進去,複製ID號,在pipeline指令碼定義變數:def k8s_auth = "ID號"
用pipeline語法生成器,生成部署資源到k8s的pipeline語法:其中Target引數可以自定義
pipeline中部署資源到k8s:
stage('部署到K8S平臺'){ steps { configFileProvider([configFile(fileId: "${k8s_auth}", targetLocation: 'admin.kubeconfig')]) { sh """ kubectl apply -f deploy.yaml -n ${Namespace} --kubeconfig=admin.kubeconfig sleep 10 kubectl get pod -n ${Namespace} --kubeconfig=admin.kubeconfig """ } } }
4.3 編寫建立專案資源的deploy檔案
專案是使用Jenkins在Kubernetes中持續部署一個無狀態的tomcat pod應用;涉及到deployment控制器 以及採用NodePort 的方式去訪問pod
deploy.yaml檔案必須和專案程式碼在同一個路徑下(否則kubectl無法指定yaml檔案就無法建立pod),所以編寫完yaml後,上傳到專案倉庫中
apiVersion: apps/v1 kind: Deployment metadata: labels: name: java-demo name: java-demo namespace: NS spec: replicas: RSCOUNT selector: matchLabels: name: java-demo template: metadata: labels: name: java-demo spec: imagePullSecrets: - name: SECRET_NAME containers: - image: IMAGE_NAME name: java-demo --- apiVersion: v1 kind: Service metadata: namespace: NS labels: name: java-demo name: java-demo spec: type: NodePort ports: - port: 80 protocol: TCP targetPort: 8080 selector: name: java-demo
4.4 定義環境變數,進行引數化構建,以及一些指令碼優化
4.4.1 對jenkins-slave建立pod進行優化 每次maven 打包會產生依賴的庫檔案,為了加快每次編譯打包的速度,我們可以建立一個pvc或掛載目錄,用來儲存maven每次打包產生的依賴檔案。 以及我們需要將 k8s 叢集 node 主機上的docker 命令掛載到Pod 中,用於映象的打包 ,推送,修改後的jenkins-salve如下:
kubernetes { label "jenkins-slave" yaml """ kind: Pod metadata: name: jenkins-slave spec: containers: - name: jnlp image: "10.48.14.50:8888/library/jenkins-slave-jdk:11" imagePullPolicy: Always env: - name: TZ value: Asia/Shanghai volumeMounts: - name: docker-cmd mountPath: /usr/bin/docker - name: docker-sock mountPath: /var/run/docker.sock - name: maven-cache mountPath: /root/.m2 - name: gradle-cache mountPath: /root/.gradle volumes: - name: docker-cmd hostPath: path: /usr/bin/docker - name: docker-sock hostPath: path: /var/run/docker.sock - name: maven-cache hostPath: path: /tmp/m2 - name: gradle-cache hostPath: path: /tmp/gradle """
4.4.2 建立一個登入harbor倉庫的secret憑證(部署專案的yaml檔案要從harbor拉取映象需要認證) kubectl create secret docker-registry registrypullauth --docker-username=admin --docker-password=Harbor12345 --docker-server=10.48.14.50:8888
4.4.3 定義環境變數,修改專案的deploy.yaml檔案,進行引數化構建
定義環境變數,在pipeline語法中引用變數: def registry = "10.48.14.50:8888" #harbor倉庫地址 def project = "dev" #harbor存放映象的倉庫名 def app_name = "java-demo" #專案名 def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}" #編譯打包成的映象名 def git_address = "http://10.48.14.100:30080/001/java-demo.git" // 認證 def secret_name = "registrypullauth" #harbor使用者名稱密碼生成的secret def docker_registry_auth = "b07ed5ba-e191-4688-9ed2-623f4753781c" #harbor使用者密碼生成的id def git_auth = "a5ec87ae-87a1-418e-aa49-53c4aedcd261" def k8s_auth = "3cd3f414-a0e2-4bc0-8808-78c64e6ad7d2" def JAVA_OPTS = "-Xms128m -Xmx256m -Dfile.encoding=UTF8 -Duser.timezone=GMT+08 -Dspring.profiles.active=test"
引數化構建過程中,互動內容:
程式碼分支(prod,dev,test)
副本數(1,3,5,7)
名稱空間(prod,dev,test)
修改專案的deploy.yaml檔案,替換成引數變數:
sed -i 's#IMAGE_NAME#${image_name}#' deploy.yaml
sed -i 's#SECRET_NAME#${secret_name}#' deploy.yaml
sed -i 's#RSCOUNT#${ReplicaCount}#' deploy.yaml
sed -i 's#NS#${Namespace}#' deploy.yaml
指定kubeconfig,執行專案pod
kubectl apply -f deploy.yaml -n ${Namespace} --kubeconfig=admin.kubeconfig
4.5 完整的pipeline指令碼
def registry = "10.48.14.50:8888" // 專案 def project = "dev" def app_name = "java-demo" def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}" def git_address = "http://10.48.14.100:30080/001/java-demo.git" // 認證 def secret_name = "registrypullauth" def docker_registry_auth = "b07ed5ba-e191-4688-9ed2-623f4753781c" def git_auth = "a5ec87ae-87a1-418e-aa49-53c4aedcd261" def k8s_auth = "3cd3f414-a0e2-4bc0-8808-78c64e6ad7d2" def JAVA_OPTS = "-Xms128m -Xmx256m -Dfile.encoding=UTF8 -Duser.timezone=GMT+08 -Dspring.profiles.active=test" pipeline { agent { kubernetes { label "jenkins-slave" yaml """ kind: Pod metadata: name: jenkins-slave spec: containers: - name: jnlp image: "${registry}/library/jenkins-slave-jdk:11" imagePullPolicy: Always env: - name: TZ value: Asia/Shanghai volumeMounts: - name: docker-cmd mountPath: /usr/bin/docker - name: docker-sock mountPath: /var/run/docker.sock - name: gradle-cache mountPath: /root/.gradle - name: maven-cache mountPath: /root/.m2 volumes: - name: docker-cmd hostPath: path: /usr/bin/docker - name: docker-sock hostPath: path: /var/run/docker.sock - name: gradle-cache hostPath: path: /tmp/gradle - name: maven-cache hostPath: path: /tmp/m2 """ } } parameters { choice (choices: ['1', '3', '5', '7'], description: '副本數', name: 'ReplicaCount') choice (choices: ['dev','test','prod','default'], description: '名稱空間', name: 'Namespace') } stages { stage('拉取程式碼'){ steps { checkout([$class: 'GitSCM', branches: [[name: "${params.Branch}"]], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]] ]) } } stage('程式碼編譯'){ steps { sh """ pwd mvn clean package -Dmaven.test.skip=true """ } } stage('構建映象'){ steps { withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) { sh """ echo ' FROM ${registry}/library/tomcat:v1 LABEL maitainer lizhenliang RUN rm -rf /usr/local/tomcat/webapps/* ADD target/*.war /usr/local/tomcat/webapps/ROOT.war ' > Dockerfile docker build -t ${image_name} . docker login -u ${username} -p '${password}' ${registry} docker push ${image_name} """ } } } stage('部署到K8S平臺'){ steps { configFileProvider([configFile(fileId: "${k8s_auth}", targetLocation: 'admin.kubeconfig')]) { sh """ pwd ls sed -i 's#IMAGE_NAME#${image_name}#' deploy.yaml sed -i 's#SECRET_NAME#${secret_name}#' deploy.yaml sed -i 's#RSCOUNT#${ReplicaCount}#' deploy.yaml sed -i 's#NS#${Namespace}#' deploy.yaml kubectl apply -f deploy.yaml -n ${Namespace} --kubeconfig=admin.kubeconfig sleep 10 kubectl get pod -n ${Namespace} --kubeconfig=admin.kubeconfig """ } } } } }
4.6 構建專案,查詢日誌
檢視構建過程和日誌