k8s 1.28.2 叢集部署 docker registry 接入 MinIO 儲存

月巴左耳东發表於2024-11-14
K8SDocker

目錄
  • docker registry 部署
    • 生成 htpasswd 檔案
      • 生成 secret 檔案
    • 生成 registry 配置檔案
    • 建立 service
    • 建立 statefulset
    • 建立 ingress
    • 驗證 docker registry
  • docker registry 監控
  • docker registry ui

docker registry dockerfile

docker registry 配置檔案

S3 storage driver

registry:2.8.3 Image hierarchy

docker registry 部署

生成 htpasswd 檔案

<username> <password> 改成自己想配置的，如果密碼有特殊字元，要用單引號包起來

docker run --rm \
  docker.m.daocloud.io/httpd:latest \
  htpasswd -Bbn <username> <password> > htpasswd

生成 secret 檔案

kubectl create secret generic docker-registry-auth \
  -n registry \
  --from-file=htpasswd

生成 registry 配置檔案

因為涉及到 MinIO 的 accesskey 和 secretkey，這裡採用 secret 的方式來生成配置檔案

---
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-cm
  namespace: registry
stringData:
  config.yml: |-
    version: 0.1
    log:
      level: info
      fields:
        service: registry
    storage:
      delete:
        enabled: true
      cache:
          blobdescriptor: inmemory
      s3:
        accesskey: wJpkHB8rznvZBRLfKmBz
        secretkey: ZHIyklv5tktYvGR0iFqBiL9NKh7JKbhyDR9SNAYp
        region: default
        regionendpoint: http://minio.api.devops.icu
        forcepathstyle: true
        accelerate: false
        bucket: docker-registry
        encrypt: false
        secure: false
        v4auth: true
        chunksize: 5242880
        multipartcopymaxconcurrency: 10
    http:
      addr: :5000
      debug:
        addr: :5001
        prometheus:
            enabled: true
            path: /metrics
      headers:
        X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
        threshold: 3
    auth:
      htpasswd:
        realm: basic-realm
        path: /auth/htpasswd
type: Opaque

建立 service

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: docker-registry
  name: docker-registry-svc
  namespace: registry
spec:
  ports:
  - name: http
    port: 5000
    targetPort: http
  - name: http-metrics
    port: 5001
    targetPort: http-metrics
  selector:
    app.kubernetes.io/name: docker-registry
  type: ClusterIP

建立 statefulset

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: docker-registry
  name: docker-registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: docker-registry
  serviceName: docker-registry-svc
  template:
    metadata:
      labels:
        app.kubernetes.io/name: docker-registry
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: docker-registry
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - image: docker.m.daocloud.io/registry:2.8.3
        livenessProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        name: docker-registry
        ports:
        - containerPort: 5000
          name: http
        - containerPort: 5001
          name: http-metrics
        readinessProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 2000m
            memory: 2.5Gi
          requests:
            cpu: 100m
            memory: 100Mi
        startupProbe:
          failureThreshold: 60
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 1
        volumeMounts:
        - mountPath: /etc/docker/registry
          name: config
        - mountPath: /auth
          name: auth
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        secret:
          secretName: docker-registry-cm
      - name: auth
        secret:
          secretName: docker-registry-auth

建立 ingress

沒有 ingress 可以開 nodeport 來實現

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 5G
  name: docker-registry
  namespace: registry
spec:
  ingressClassName: nginx
  rules:
  - host: registry.devops.icu
    http:
      paths:
      - backend:
          service:
            name: docker-registry-svc
            port:
              number: 5000
        path: /
        pathType: Prefix

驗證 docker registry

/etc/docker/daemon.json 增加 registry 地址

"insecure-registries": ["ip:埠"]
# 或者
"insecure-registries": ["域名"]

登入 docker registry

docker login http://registry.devops.icu

修改 tag

docker tag docker.m.daocloud.io/registry:2.8.3 registry.devops.icu/registry:2.8.3

上傳映象

docker push registry.devops.icu/registry:2.8.3

docker registry 監控

grafana id：9621

prometheus 配置檔案參考

    - job_name: docker-registry
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        regex: registry;docker-registry-svc
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:5001
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace

docker registry ui

Github 專案地址：Joxit/docker-registry-ui-2.5.7

相關的變數和引數詳見：available-options

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: docker-registry-ui
  name: docker-registry-ui-svc
  namespace: registry
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: docker-registry-ui
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: docker-registry-ui
  name: docker-registry-ui
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: docker-registry-ui
  template:
    metadata:
      labels:
        app.kubernetes.io/name: docker-registry-ui
    spec:
      containers:
      - env:
        - name: SINGLE_REGISTRY
          value: "true"
        - name: SHOW_CATALOG_NB_TAGS
          value: "true"
        - name: REGISTRY_SECURED
          value: "true"
        - name: NGINX_PROXY_PASS_URL
          value: http://docker-registry-svc.registry.svc.cluster.local:5000
        - name: NGINX_PROXY_HEADER_Authorization
          value: $http_authorization
        image: joxit/docker-registry-ui:2.5.7
        imagePullPolicy: IfNotPresent
        name: docker-registry-ui
      securityContext:
        fsGroup: 101
        runAsUser: 101
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: docker-registry-ui
  namespace: registry
spec:
  ingressClassName: nginx
  rules:
  - host: registry.ui.devops.icu
    http:
      paths:
      - backend:
          service:
            name: docker-registry-ui-svc
            port:
              number: 8080
        path: /
        pathType: Prefix

相關文章