本文目標
由於 OpenShift 官方提供的映象與模板(OpenJDK8)不完全滿足業務需要:
-
不包含飛行記錄功能。只有 OpenJDK11 以上才被 Oracle 開源
-
生成堆 dump 很大很慢(公司專案 JVM 堆最小16G,最大32G),
匯出-壓縮-傳輸
較長(半小時以上),對業務停頓時間過長 -
映象內建的獲取效能指標的命令很少,常規網路狀態查詢命令如
netstat
、ping
等都沒有 -
映象內部s2i流程編寫得過於複雜,映象構建過程快取多且未移除,導致最終映象普遍體積過大(350M的業務應用,打出映象有 1.1G…)
為了方便定位效能問題,筆者仔細閱讀了 OpenShift v3.11 版本的 s2i 構建流程、映象自定義等相關文件,定製出了自己的模板與映象,目前已經在生產環境上驗證通過,在這裡記錄一二。如果有幫到你,點個關注和贊再走吧~
本文主體思路是使用 s2i工具建立基礎的構建除錯環境,修改Dockerfile與構建指令碼生成映象,再推送映象和模板到上篇文章OpenShift 本地開發環境配置(基於 Minishift)的環境中,用一個demo測試下最終效果。
自定義 s2i 映象部分
安裝 s2i 工具
s2i 工具官方Release Page:https://github.com/openshift/source-to-image/releases
#下載s2i工具
wget https://github.com/openshift/source-to-image/releases/download/v1.3.1/source-to-image-v1.3.1-a5a77147-linux-amd64.tar.gz
#解壓s2i工具到/usr/bin
sudo tar zxf source-to-image-v1.3.1-a5a77147-linux-amd64.tar.gz -C /usr/bin
#檢視s2i版本
s2i version
初始化 s2i 目錄結構
筆者是在
~/WorkSpace/openshift
目錄下開始操作的,提前說下免得讀者操作有誤。
#生成s2i指令碼目錄結構,create引數為建立,第3引數為映象名,第3引數為生成目錄名
s2i create oracle-jdk-8 oracle-jdk-8-catelog
#進入生成的目錄中
cd oracle-jdk-8-catelog
#檢視基本目錄結構
tree -L 3
.
├── Dockerfile #映象Dockerfile
├── Makefile #Makefile構建指令碼
├── README.md
├── s2i
│ └── bin #s2i指令碼目錄
│ ├── assemble #裝配指令碼
│ ├── run #執行指令碼
│ ├── save-artifacts #可選,儲存製品指令碼,增量構建時用
│ └── usage #使用說明指令碼
└── test
├── run #測試執行指令碼
└── test-app #構建的原始碼(本例中可以是jar包)
└── index.html
這裡筆者用不到 s2i/bin/save-artifacts
,因為不需要增量構建,刪除之; test/test-app/index.html
也用不到,刪除之。
rm -rf s2i/bin/save-artifacts
rm -rf test/test-app/index.html
下載並解壓 OracleJDK8安裝包,適度精簡
由於筆者已經將jdk8安裝包下載好了,下載 OracleJDK 付費版本以前的版本,參考我的部落格,這裡就直接從 ~/Downloads
解壓到 ~/WorkSpace/openshift/oracle-jdk-8-catelog
下
#解壓jdk安裝包
tar zxvf ~/Downloads/jdk-8u181-linux-x64.tar.gz -C ~/WorkSpace/openshift/oracle-jdk-8-catelog
#進入jdk解壓目錄
cd /home/hellxz/WorkSpace/openshift/oracle-jdk-8-catelog/jdk1.8.0_181
#適度精簡,刪除原始碼包,能小26M
rm -rf javafx-src.zip src.zip
#回到模板初始化的目錄
cd ..
修改 Dockerfile
vim Dockerfile
,把預設的都刪除掉,新增下邊的內容:
FROM debian:buster
ENV BUILDER_VERSION 1.0
# Set labels used in OpenShift to describe the builder image
LABEL description="Source To Image (S2I) image for Hellxz Providing Oracle JDK 8" \
maintainer="Hellxz Zhang <hellxz@foxmail.com>" \
io.k8s.description="Platform for building and running Java applications with Oracle JDK 8" \
io.k8s.display-name="Java Applications" \
io.openshift.expose-services="8080:http" \
io.openshift.tags="builder,java" \
io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
io.openshift.s2i.destination="/tmp"
# Copy Oracle JDK8
COPY ./jdk1.8.0_181 /usr/lib/jvm
# Copy scripts to s2i build path, must same as `io.openshift.s2i.scripts-url` label specified.
COPY ./s2i/bin/ /usr/libexec/s2i
# Environments
ENV JAVA_HOME="/usr/lib/jvm" \
PATH="/usr/lib/jvm/bin:${PATH}" \
TZ="Asia/Shanghai"
# Create oraclejdk user with home dir /deployments
# Grant user exec privilege
# Set apt mirror and install some utils then clean
# Generate UTF-8 locales
RUN mkdir /deployments && \
useradd -M -d /deployments -u 1001 -s /bin/bash oraclejdk8 && \
chown oraclejdk8 /deployments && chmod 777 /deployments && \
bash -c "echo -e \"deb http://mirrors.163.com/debian/ buster main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-updates main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-backports main non-free contrib\ndeb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\" > /etc/apt/sources.list" && \
apt-get update -y && apt-get install -y net-tools locales procps && \
apt-get clean all -y && rm -rf /var/lib/apt/lists/* && \
sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
# To avoid build image error tips set LC env after locale.gen
ENV LC_ALL="en_US.UTF-8"
# Switch user
USER 1001
# Switch WorkDir
WORKDIR /deployments
EXPOSE 8080 8443
# Set the default CMD for the image
CMD ["/usr/libexec/s2i/usage"]
上述的Dockerfile做了什麼呢?
- 基於非常穩定的 Debian10
- 為映象打Label標籤,s2i 構建時會讀取這些標籤
- 複製OracleJDK目錄及s2i指令碼
- 配置JDK環境變數、UTF-8字符集、指定中國時區
- 建立部署目錄、建立執行使用者、更換國內源、安裝必要收集效能指標工具(netstat和ps)
- 指定工作目錄、執行使用者、預設暴露的埠、預設的
CMD
入口
完成Dockerfile的修改還沒完,我們還需要修改 s2i 構建與執行指令碼,以適應剛剛修改的映象配置
修改 s2i 指令碼
s2i指令碼在 s2i/bin
目錄下,我們需要修改 assemble
指令碼來改變組裝流程,修改 run
指令碼以更換啟動命令。
修改assemble
vim s2i/bin/assemble
#!/bin/bash -e
#
# S2I assemble script for the 'oracle-jdk-8' image.
# The 'assemble' script builds your application source so that it is ready to run.
#
# For more information refer to the documentation:
# https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md
#
# Prevent some horror bugs. example: rm -rf ${SOME_UNEXISTS_VARIABLE}/*
set -e
# If the 'oracle-jdk-8' assemble script is executed with the '-h' flag, print the usage.
if [[ "$1" == "-h" ]]; then
exec /usr/libexec/s2i/usage
fi
echo "---> Moving Artifact from source..."
mv /tmp/src/*.jar /deployments/
echo "---> Clearing tmp dir..."
rm /tmp/src/.git -rf
echo "<--- Build Success..."
以上指令碼主要做了兩件事,一個是從 /tmp/src
下將 jar 包複製到部署目錄,在之前的文章中介紹過 s2i 的工作流程,在構建時會從 Git 上拉取製品或原始碼到 /tmp/src
下;另一個是刪除 /tmp/src/.git
目錄,原因是 Git 的 .git
目錄會同樣儲存壓縮後的製品或原始碼檔案,刪除以減小映象體積。
修改run
vim s2i/bin/run
#!/bin/bash -e
#
# S2I run script for the 'oracle-jdk-8' image.
# The run script executes the server that runs your application.
#
# For more information see the documentation:
# https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md
#
exec java ${JVM_OPTS} -Djava.security.egd=file:/dev/./urandom -jar /deployments/*.jar
run 指令碼只作啟動功能,為了方便傳 JVM 調優引數,定製了 JVM_OPTS
環境變數供使用。引數中的隨機數配置能提高服務生成隨機數的能力,提高啟動速度。
修改 Makefile
修改Makefile的作用是簡化構建映象的命令,筆者這裡修改了映象名以及新增了推送私有映象倉庫的引數,這點上 npm
有點類似於 Makefile
。
IMAGE_NAME = 192.168.99.1:5000/oracle-jdk-8
.PHONY: test
test:
docker build -t $(IMAGE_NAME)-candidate .
IMAGE_NAME=$(IMAGE_NAME)-candidate test/run
.PHONY: push
push:
docker build -t $(IMAGE_NAME) .
docker push $(IMAGE_NAME)
想了解更多 Makefile 寫法,可以參考
左耳朵耗子
陳皓 寫的《跟我一起寫Makefile》
編寫測試指令碼
test/run
這個測試指令碼相當於完成了一次 s2i 到部署的過程,先準備測試映象,再執行s2i的構建過程增量構建(將測試製品程式與映象進行組裝,容器內部呼叫了 assemble
指令碼),然後啟動容器,執行定時測試指令碼訪問容器埠地址,通過就算成功了。
由於預設測試指令碼中測試服務是否可用是訪問地址判斷返回值是否200狀態碼,我這邊提供個springboot的demo 部署後訪問 /test
端點返回結果 ,首先構建下這個demo:
#克隆測試demo
git clone https://github.com/hellxz/cicd-demo.git test/test-app
#手動構建
cd test/test-app
mvn package -DskipTests
#複製jar包並刪除本文無關的內容
mv target/*.jar .
rm -rf *.sh *file *.xml *.md src target .git *.yaml .gitignore .mvn
#回到Makefile所在目錄
cd ../..
修改 test/run
指令碼,調整探測間隔與訪問的介面地址,本demo中需修改116行為5,以及120和121行訪問路徑新增 /test
執行測試
make test
我們能看到 springboot 的 banner 下有行字 Starting CicdDemoApplication v0.0.1 using Java 1.8.0_181 on f770c60456f4 with PID 1 (/deployments/cicd-demo-0.0.1.jar started by oraclejdk8 in /deployments)
,這可以證明測試的demo的確是以 oraclejdk8 使用者執行成功的,而且 pid 為 1。再往下看也沒有檢測報錯的資訊,說明測試通過,可以使用 Minishift測試了。
推送映象到倉庫中
#執行Makefile中定義的構建推送命令
make push
hellxz@debian:~/WorkSpace/openshift/oracle-jdk-8-catelog$ make push
docker build -t 192.168.99.1:5000/oracle-jdk-8 .
Sending build context to Docker daemon 389.2MB
Step 1/12 : FROM debian:buster
---> 2b6f409b1d24
Step 2/12 : ENV BUILDER_VERSION 1.0
---> Using cache
---> 46de72742080
Step 3/12 : LABEL description="Source To Image (S2I) image for Hellxz Providing Oracle JDK 8" maintainer="Hellxz Zhang <hellxz@foxmail.com>" io.k8s.description="Platform for building and running Java applications with Oracle JDK 8" io.k8s.display-name="Java Applications" io.openshift.expose-services="8080:http" io.openshift.tags="builder,java" io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" io.openshift.s2i.destination="/tmp"
---> Using cache
---> fcbadb0d0d39
Step 4/12 : COPY ./jdk1.8.0_181 /usr/lib/jvm
---> Using cache
---> ebb551cc9d4c
Step 5/12 : COPY ./s2i/bin/ /usr/libexec/s2i
---> Using cache
---> 5f08421b9527
Step 6/12 : ENV JAVA_HOME="/usr/lib/jvm" PATH="/usr/lib/jvm/bin:${PATH}" TZ="Asia/Shanghai"
---> Using cache
---> 3c663a5370c8
Step 7/12 : RUN mkdir /deployments && useradd -M -d /deployments -u 1001 -s /bin/bash oraclejdk8 && chown oraclejdk8 /deployments && chmod 777 /deployments && bash -c "echo -e \"deb http://mirrors.163.com/debian/ buster main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-updates main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-backports main non-free contrib\ndeb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\" > /etc/apt/sources.list" && apt-get update -y && apt-get install -y net-tools locales procps && apt-get clean all -y && rm -rf /var/lib/apt/lists/* && sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
---> Using cache
---> 5e9c181bb69b
Step 8/12 : ENV LC_ALL="en_US.UTF-8"
---> Using cache
---> 60091bb63284
Step 9/12 : USER 1001
---> Using cache
---> f8e6a2e96a8a
Step 10/12 : WORKDIR /deployments
---> Using cache
---> 66f6925e3ab6
Step 11/12 : EXPOSE 8080 8443
---> Using cache
---> bda814465820
Step 12/12 : CMD ["/usr/libexec/s2i/usage"]
---> Using cache
---> f30388d3181c
Successfully built f30388d3181c
Successfully tagged 192.168.99.1:5000/oracle-jdk-8:latest
docker push 192.168.99.1:5000/oracle-jdk-8
Using default tag: latest
The push refers to repository [192.168.99.1:5000/oracle-jdk-8]
16970671df83: Pushed
97a51654d23f: Pushed
16fd63c1b14c: Pushed
d6a325d281f2: Pushed
latest: digest: sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79 size: 1160
匯入映象到 OpenShift 內部映象倉庫中
這裡Minishift匯入映象到內部倉庫和OpenShift是一致的,說是OpenShift也可以。
#匯入需要oc命令列工具,minishift預設已經提供了,但未加入到Path中,可以執行如下命令臨時把oc加入Path
eval (minishift oc-env) #如果報錯,則只執行括號內命令,根據提示執行相應命令
oc import-image 192.168.99.1:5000/oracle-jdk-8:latest --confirm --insecure
hellxz@debian:~/WorkSpace/openshift$ oc import-image 192.168.99.1:5000/oracle-jdk-8:latest --confirm --insecure
imagestream.image.openshift.io/oracle-jdk-8 imported
Name: oracle-jdk-8
Namespace: myproject
Created: 1 second ago
Labels: <none>
Annotations: openshift.io/image.dockerRepositoryCheck=2021-11-21T14:24:57Z
Docker Pull Spec: 172.30.1.1:5000/myproject/oracle-jdk-8
Image Lookup: local=false
Unique Images: 1
Tags: 1
latest
tagged from 192.168.99.1:5000/oracle-jdk-8:latest
will use insecure HTTPS or HTTP connections
* 192.168.99.1:5000/oracle-jdk-8@sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
1 second ago
Image Name: oracle-jdk-8:latest
Docker Image: 192.168.99.1:5000/oracle-jdk-8@sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
Name: sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
Created: 1 second ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
Image Size: 221.8MB in 4 layers
Layers: 50.44MB sha256:07471e81507f7cf1100827f10c60c3c0422d1222430e34e527d97ec72b14a193
161.8MB sha256:98b418031cdbc51e52d88a2632516069e753d5293ec440dae46163a45880492a
899B sha256:61addb0f8207e85f3a734299eda8b8afae987076de0cf67b60f33c9c9846f6b6
9.598MB sha256:09daea12eb7b9cd76b4629e22d566a6646eb6e19ea3c92706c7bc41bf0285384
Image Created: 30 minutes ago
Author: <none>
Arch: amd64
Command: /usr/libexec/s2i/usage
Working Dir: /deployments
User: 1001
Exposes Ports: 8080/tcp, 8443/tcp
Docker Labels: description=Source To Image (S2I) image for Hellxz Providing Oracle JDK 8
io.k8s.description=Platform for building and running Java applications with Oracle JDK 8
io.k8s.display-name=Java Applications
io.openshift.expose-services=8080:http
io.openshift.s2i.destination=/tmp
io.openshift.s2i.scripts-url=image:///usr/libexec/s2i
io.openshift.tags=builder,java
maintainer=Hellxz Zhang <hellxz@foxmail.com>
Environment: PATH=/usr/lib/jvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
BUILDER_VERSION=1.0
JAVA_HOME=/usr/lib/jvm
TZ=Asia/Shanghai
LC_ALL=en_US.UTF-8
如果報錯,優先考慮是否登入了oc,
oc login -u developer
,執行後再嘗試匯入。
自定義s2i模板部分
定義模板檔案
這裡直接提供模板內容,基於 openjdk8的模板修改而來的:
oraclejdk-8-basic-s2i.yaml
apiVersion: template.openshift.io/v1
kind: Template
message: A new java application has been created in your project.
metadata:
annotations:
description: An Java application using Oracle JDK 8.
iconClass: icon-rh-openjdk
openshift.io/display-name: OracleJDK 8 by Hellxz
openshift.io/provider-display-name: Hellxz Zhang <hellxz001@foxmail.com>.
tags: java
template.openshift.io/long-description: This template defines resources needed to develop Oracle JDK 8 Java based application.
template.openshift.io/support-url: hellxz001@foxmail.com
version: 0.0.1
labels:
template: oraclejdk-8-basic-s2i
xpaas: 0.0.1
name: oraclejdk-8-basic-s2i
objects:
- apiVersion: v1
kind: Service
metadata:
annotations:
description: The application's http port.
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
ports:
- port: 8080
targetPort: 8080
selector:
deploymentConfig: ${APPLICATION_NAME}
- apiVersion: v1
id: ${APPLICATION_NAME}-http
kind: Route
metadata:
annotations:
description: Route for application's http service.
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
host: ${HOSTNAME_HTTP}
to:
name: ${APPLICATION_NAME}
- apiVersion: v1
kind: ImageStream
metadata:
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
- apiVersion: v1
kind: BuildConfig
metadata:
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
output:
to:
kind: ImageStreamTag
name: ${APPLICATION_NAME}:latest
source:
contextDir: ""
git:
ref: ${SOURCE_REPOSITORY_REF}
uri: ${SOURCE_REPOSITORY_URL}
type: Git
strategy:
sourceStrategy:
forcePull: true
from:
kind: ImageStreamTag
name: oracle-jdk-8:latest
namespace: ${IMAGE_STREAM_NAMESPACE}
type: Source
triggers:
- imageChange: {}
type: ImageChange
- type: ConfigChange
- apiVersion: v1
kind: DeploymentConfig
metadata:
labels:
application: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
replicas: 1
selector:
deploymentConfig: ${APPLICATION_NAME}
strategy:
type: Recreate
template:
metadata:
labels:
application: ${APPLICATION_NAME}
deploymentConfig: ${APPLICATION_NAME}
name: ${APPLICATION_NAME}
spec:
containers:
- env: []
image: ${APPLICATION_NAME}
imagePullPolicy: Always
name: ${APPLICATION_NAME}
ports:
- containerPort: 8778
name: jolokia
protocol: TCP
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 8443
name: https
protocol: TCP
terminationGracePeriodSeconds: 75
triggers:
- imageChangeParams:
automatic: true
containerNames:
- ${APPLICATION_NAME}
from:
kind: ImageStreamTag
name: ${APPLICATION_NAME}:latest
type: ImageChange
- type: ConfigChange
parameters:
- description: The name for the application.
displayName: Application Name
name: APPLICATION_NAME
required: true
value: oraclejdk8-app
- description: Git source URI for application
displayName: Git Repository URL
name: SOURCE_REPOSITORY_URL
required: true
- description: Git branch/tag reference
displayName: Git Reference
name: SOURCE_REPOSITORY_REF
value: master
- description: 'Custom hostname for http service route. Leave blank for default hostname,
e.g.: <application-name>-<project>.<default-domain-suffix>'
displayName: Custom http Route Hostname
name: HOSTNAME_HTTP
- description: Namespace in which the ImageStreams for Red Hat Middleware images are
installed. These ImageStreams are normally installed in the openshift namespace.
You should only need to modify this if you've installed the ImageStreams in a
different namespace/project.
displayName: ImageStream Namespace
name: IMAGE_STREAM_NAMESPACE
required: true
value: myproject
模板的結構簡單說明:
- 定義格式基於k8s,是紅帽擴充套件的 Template API型別
- 源資料 - 記錄模板的內容,會在開啟模板的UI上顯示
- objects - 各種構建到部署過程的所有物件定義
- Service - k8s原生有的服務物件
- Route - OpenShift獨有的路由物件
- ImageStream - OpenShift獨有的映象流物件
- BuildConfig - OpenShift獨有的構建配置物件
- DeploymentConfig - OpenShift獨有的部署配置物件,生成k8s中的Deployment物件
- parameters - 可修改引數,或者變數,供objects中各種物件引用,從而生成相關聯的一系列物件
匯入模板到 OpenShift
oc apply -f oraclejdk-8-basic-s2i.yaml
hellxz@debian:~/WorkSpace/openshift$ oc apply -f=oraclejdk-8-basic-s2i.yaml
template.template.openshift.io/oraclejdk-8-basic-s2i created
使用自定義模板和映象部署 Java 程式
預備工作
由於OpenShift是由Git倉庫拉取製品或原始碼進行構建的,所以需要把測試程式(上文中用到的cicd-demo-0.0.1.jar)上傳到 Git 版本控制中,這裡延用上篇文章中 Minishift 開發環境中配置的 Gitea(需要預先建立個賬號和倉庫,這裡建立的倉庫名為demo,使用者名稱為hellxz)。
#進入測試程式jar所在目錄
cd ~/WorkSpace/openshift/oracle-jdk-8-catelog/test/test-app
#初始化當前目錄為git倉庫
git init
#新增遠端倉庫地址(按實際操作來,變通一些)
git remote add origin git remote add origin http://localhost:3000/hellxz/demo.git
#設定使用者名稱和email
git config --local user.name hellxz
git config --local user.email hellxz001@foxmail.com
#提交
git add .
git commit -m "init"
#推送遠端倉庫
git push --set-upstream origin master
如下圖,已經推送完畢。
使用自定義模板部署程式
模板中引用的是自定義映象的名稱,名稱空間也是myproject(Minishift預設的,生產環境可以改成需要的名稱再上傳,調整可多次匯入)
首次部署流程比較麻煩,這裡分多個動圖錄制
建立應用
Gitea 配置的對外地址配置得有點問題,不過問題不大……
建立拉取Git的secret
需要告知s2i構建指令碼 Git 倉庫的使用者和密碼,這樣才能拉得下來製品,僅首次建立應用時需要建立。
修改Build配置yaml,新增 Git 倉庫 secret 引用
執行構建,檢視構建日誌
檢視測試程式執行狀態
如圖,容器執行已經是 Ready: true 狀態,說明啟動狀態正常。
使用域名訪問服務看看
由於沒有外部的dns,域名可以修改 /etc/hosts
檔案代替
測試通過,自定義映象和模板都能正常工作。
寫作不易,如果本文對您有所幫助,就點個關注點個贊再走唄~ 我是 Hellxz,我們下次再見。
本文同步釋出以下兩個地址,未經許可禁止轉載。