極狐GitLab Runner 信任域名證書

极狐GitLab發表於2024-03-04
Gitlab

本文作者 徐曉偉

極狐Gitlab Runner 信任例項域名證書，用於註冊註冊極狐 GitLab Runner。

問題

參見 極狐gitlab-runner-host.md

說明

  • 解決方案是使用頒發給域名 gitlab.test.helm.xuxiaowei.cn 的證書，可以使用自己的域名去各大雲廠商免費申請，或者使用自己根據域名 gitlab.test.helm.xuxiaowei.cn 生成的證書

    1. 阿里雲SSL(https)證書免費申請
    2. 騰訊雲SSL(https)證書免費申請
    3. 華為雲SSL(https)證書免費申請
    4. 百度雲SSL(https)證書免費申請

域名證書解決方案

方案1：重新配置極狐 GitLab，自動生成對應證書並自動配置，然後在極狐 GitLab runner 中信任證書

  1. 問題1：tls: failed to verify certificate: x509: certificate signed by unknown authority

    1. 文件

      1. runners 部分
      2. 自簽名證書或自定義證書頒發機構
      3. ConfigMap
      4. 配置 Pod 使用 ConfigMap

    2. 根據上述文件可知，有三種方式可以解決證書信任問題：

      1. 在 極狐GitLab Runner 註冊時配置 --tls-ca-file 引數並指向證書檔案
      2. 在配置檔案中增加 tls-ca-file 並指向配置檔案， 此方式本文不做介紹，可參考 gitlab-runner-kubernetes-host.md ，原理相同
      3. 將證書放在 /etc/gitlab-runner/certs/ 資料夾下，使用域名命名證書，證書字尾名是 .crt， 由於篇幅限制，這裡只介紹這一種方式

    3. 由於 k8s 的 pod 可能隨時會被刪除（故障轉移、主動刪除等），所以直接在 pod 內部配置不是上策

    4. k8s 中的 ConfigMap 可以掛載到 pod 內部，所以可以考慮在 ConfigMap 中配置證書，然後 pod 內部使用 ConfigMap 中的證書

    5. 下載證書

      1. 方式1

        1. 瀏覽器訪問域名 https://gitlab.test.helm.xuxiaowei.cn
        2. 按照圖中操作，下載證書，上傳之伺服器，用於後期操作

      2. 方式2

        # 下載證書
openssl s_client -showcerts -connect gitlab.test.helm.xuxiaowei.cn:443 -servername gitlab.test.helm.xuxiaowei.cn < /dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab.test.helm.xuxiaowei.cn.crt

    6. 將證書匯入到 k8s 中

      # -n=gitlab-test：指定名稱空間
# create configmap etc-gitlab-runner-certs：建立 ConfigMap 名稱是 etc-gitlab-runner-certs
# --from-file=gitlab.test.helm.xuxiaowei.cn.crt=gitlab.test.helm.xuxiaowei.cn.crt：配置來自檔案，檔名 gitlab.test.helm.xuxiaowei.cn.crt，放入 ConfigMap 中的鍵也是 gitlab.test.helm.xuxiaowei.cn.crt
kubectl -n=gitlab-test create configmap etc-gitlab-runner-certs --from-file=gitlab.test.helm.xuxiaowei.cn.crt=gitlab.test.helm.xuxiaowei.cn.crt

# 檢視
# kubectl -n=gitlab-test get configmap etc-gitlab-runner-certs -o yaml

    7. 匯出 helm 極狐GitLab 配置

      helm -n gitlab-test get values my-gitlab > my-gitlab.yaml

    8. 修改 helm 極狐GitLab 配置

      gitlab-runner:
  # 掛載卷名稱
  volumeMounts:
    # 掛載到 pod 路徑（檔案/資料夾）
    # 此處是在容器內執行的 gitlab runner，由於許可權限制等原因，
    # 所以 配置檔案不是在 /etc/gitlab-runner/ 目錄下，而是 /home/gitlab-runner/.gitlab-runner/
    - mountPath: /home/gitlab-runner/.gitlab-runner/certs
      name: etc-gitlab-runner-certs-volumes
  # 卷
  volumes:
    # 卷型別
    - configMap:
        items:
        # configMap 中的鍵
        - key: gitlab.test.helm.xuxiaowei.cn.crt
          # 掛載的路徑
          path: gitlab.test.helm.xuxiaowei.cn.crt
        name: etc-gitlab-runner-certs
      # 配置 ConfigMap 名稱
      name: etc-gitlab-runner-certs-volumes

    9. 更新 helm 極狐GitLab 配置

      helm upgrade -n gitlab-test --install my-gitlab gitlab/gitlab -f my-gitlab.yaml --timeout 600s --version 7.7.0

    10. 檢視修改結果

      kubectl -n gitlab-test get deployments.apps my-gitlab-gitlab-runner -o yaml

      apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "5"
    meta.helm.sh/release-name: my-gitlab
    meta.helm.sh/release-namespace: gitlab-test
  creationTimestamp: "2023-12-22T05:03:46Z"
  generation: 7
  labels:
    app: my-gitlab-gitlab-runner
    app.kubernetes.io/managed-by: Helm
    chart: gitlab-runner-0.59.2
    heritage: Helm
    release: my-gitlab
  name: my-gitlab-gitlab-runner
  namespace: gitlab-test
  resourceVersion: "30086"
  uid: 8c46c44a-5b67-44ae-90d0-008daa3fa388
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: my-gitlab-gitlab-runner
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/configmap: f35865354f043583d0903b0a8350830a486eb0e289d18271cf3f533e7d89c5f7
        checksum/secrets: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
        gitlab.com/prometheus_port: "9252"
        gitlab.com/prometheus_scrape: "true"
      creationTimestamp: null
      labels:
        app: my-gitlab-gitlab-runner
        chart: gitlab-runner-0.59.2
        heritage: Helm
        release: my-gitlab
    spec:
      containers:
        - command:
            - /usr/bin/dumb-init
            - --
            - /bin/bash
            - /configmaps/entrypoint
          env:
            - name: CI_SERVER_URL
              value: https://gitlab.test.helm.xuxiaowei.cn
            - name: RUNNER_EXECUTOR
              value: kubernetes
            - name: REGISTER_LOCKED
              value: "false"
            - name: RUNNER_TAG_LIST
            - name: KUBERNETES_NAMESPACE
              value: gitlab-test
          image: registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v16.6.1
          imagePullPolicy: IfNotPresent
          livenessProbe:
            exec:
              command:
                - /bin/bash
                - /configmaps/check-live
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          name: my-gitlab-gitlab-runner
          ports:
            - containerPort: 9252
              name: metrics
              protocol: TCP
          readinessProbe:
            exec:
              command:
                - /usr/bin/pgrep
                - gitlab.*runner
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 3
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: false
            runAsNonRoot: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /secrets
              name: projected-secrets
            - mountPath: /home/gitlab-runner/.gitlab-runner
              name: etc-gitlab-runner
            - mountPath: /configmaps
              name: configmaps
            - mountPath: /home/gitlab-runner/.gitlab-runner/certs
              name: etc-gitlab-runner-certs-volumes
      dnsPolicy: ClusterFirst
      hostAliases:
        - hostnames:
            - gitlab.test.helm.xuxiaowei.cn
          ip: 172.25.25.32
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65533
        runAsUser: 100
      serviceAccount: my-gitlab-gitlab-runner
      serviceAccountName: my-gitlab-gitlab-runner
      terminationGracePeriodSeconds: 3600
      volumes:
        - emptyDir:
            medium: Memory
          name: runner-secrets
        - emptyDir:
            medium: Memory
          name: etc-gitlab-runner
        - name: projected-secrets
          projected:
            defaultMode: 420
            sources:
              - secret:
                  name: my-gitlab-minio-secret
              - secret:
                  items:
                    - key: runner-registration-token
                      path: runner-registration-token
                    - key: runner-token
                      path: runner-token
                  name: my-gitlab-gitlab-runner-secret
        - configMap:
            defaultMode: 420
            name: my-gitlab-gitlab-runner
          name: configmaps
        - configMap:
            defaultMode: 420
            items:
              - key: gitlab.test.helm.xuxiaowei.cn.crt
                path: gitlab.test.helm.xuxiaowei.cn.crt
            name: etc-gitlab-runner-certs
          name: etc-gitlab-runner-certs-volumes
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: "2023-12-22T07:43:25Z"
      lastUpdateTime: "2023-12-22T07:43:25Z"
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: "True"
      type: Available
    - lastTransitionTime: "2023-12-22T05:03:46Z"
      lastUpdateTime: "2023-12-22T07:43:25Z"
      message: ReplicaSet "my-gitlab-gitlab-runner-597d6d8f7c" has successfully progressed.
      reason: NewReplicaSetAvailable
      status: "True"
      type: Progressing
  observedGeneration: 7
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

    11. 檢視 gitlab runner 日誌

      [root@anolis-7-9 ~]# kubectl -n gitlab-test get pod | grep gitlab-runner
my-gitlab-gitlab-runner-597d6d8f7c-8v466             1/1     Running     0              5m52s
[root@anolis-7-9 ~]#

      根據日誌判斷，已經註冊成功了

      [root@anolis-7-9 ~]# kubectl -n gitlab-test logs -f my-gitlab-gitlab-runner-597d6d8f7c-8v466
Registration attempt 1 of 30
Runtime platform                                    arch=amd64 os=linux pid=16 revision=f5da3c5a version=16.6.1
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...

Created missing unique system ID                    system_id=r_3WsywNzJqRud
Merging configuration from template file "/configmaps/config.template.toml"
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://docs.gitlab.com/ee/ci/runners/new_creation_workflow
Registering runner... succeeded                     runner=BtGwLEwc
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "/home/gitlab-runner/.gitlab-runner/config.toml"
Runtime platform                                    arch=amd64 os=linux pid=7 revision=f5da3c5a version=16.6.1
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0 max_builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...

There might be a problem with your config based on jsonschema annotations in common/config.go (experimental feature):
jsonschema: '/runners/0/kubernetes/pull_policy' does not validate with https://gitlab.com/gitlab-org/gitlab-runner/common/config#/$ref/properties/runners/items/$ref/properties/kubernetes/$ref/properties/pull_policy/$ref/type: expected array, but got null

Configuration loaded                                builds=0 max_builds=10
listen_address not defined, metrics & debug endpoints disabled  builds=0 max_builds=10
[session_server].listen_address not defined, session endpoints disabled  builds=0 max_builds=10
Initializing executor providers                     builds=0 max_builds=10
^C
[root@anolis-7-9 ~]#

    12. 管理員訪問 https://gitlab.test.helm.xuxiaowei.cn/admin/runners 可以看到 pod my-gitlab-gitlab-runner-597d6d8f7c-8v466 已經註冊成了

      1. 如果存在極狐 GitLab Runner Pod 被刪除（故障轉譯、手動刪除等等），這裡可能會存在不可用的 Runner

  2. 問題2： tls: failed to verify certificate: x509: certificate is valid for ingress.local, not gitlab.test.helm.xuxiaowei.cn

    1. 訪問的域名與使用的證書補匹配，可透過更換證書、域名來解決此問題（一般情況是更換證書）

    2. 如果要更換證書：

      1. 證書屬於正規機構頒發的（如：上述各大雲廠商的證書），請看下方方案2
      2. 如果是自己生成的證書（不受信任），請看下方方案2，並結合上方問題1來解決此問題

方案2：配置正規機構頒發的證書（如：上述各大雲廠商的證書），一般無需在 gitlab 配置信任證書，即可正常使用

  1. 正常情況

    1. 要求證書與域名對應
    2. 將證書上傳至伺服器，解壓，使用證書檔案建立 k8s TLS Secret
    3. 修改 極狐GitLab 配置，使用 上述步驟建立的 k8s TLS Secret

  2. 異常情況：正規機構頒發的證書，在極狐 GitLab Runner 中依然無法正常使用，但是在瀏覽器中可以正常使用

    1. 原因是頒發機構的根證書不在極狐 GitLab Runner 的信任列表中，需要手動新增信任（例如：常見的 Windows 較低版本執行一些軟體時，也是無法執行，顯示證書無法識別，需要在 Windows 匯入軟體簽名所使用的證書鏈）

相關文章