kubernetes實踐之二十七：Harbor

百聯達發表於2018-04-29
一：簡介

Harbor是一個用於儲存和分發Docker映象的企業級Registry伺服器。

映象的儲存harbor使用的是官方的docker registry(v2命名是distribution)服務去完成。harbor在docker distribution的基礎上增加了一些安全、訪問控制、管理的功能以滿足企業對於映象倉庫的需求。harbor以docker-compose的規範形式組織各個元件，並透過docker-compose工具進行啟停。

docker的registry是用本地儲存或者s3都是可以的，harbor的功能是在此之上提供使用者許可權管理、映象複製等功能，提高使用的registry的效率。Harbor的映象複製功能是透過docker registry的API去複製，這種做法遮蔽了繁瑣的底層檔案操作、不僅可以利用現有docker registry功能不必重複造輪子，而且可以解決衝突和一致性的問題。

二：Harbor架構


三：主要元件
  • Proxy：對應啟動元件nginx。它是一個nginx反向代理，代理Notary client（映象認證）、Docker client（映象上傳下載等）和瀏覽器的訪問請求（Core Service）給後端的各服務；
  • UI（Core Service）：對應啟動元件harbor-ui。底層資料儲存使用mysql資料庫，主要提供了四個子功能： 
    • UI：一個web管理頁面ui；
    • API：Harbor暴露的API服務；
    • Auth：使用者認證服務，decode後的token中的使用者資訊在這裡進行認證；auth後端可以接db、ldap、uaa三種認證實現；
    • Token服務（上圖中未體現）：負責根據使用者在每個project中的role來為每一個docker push/pull命令issuing一個token，如果從docker client傳送給registry的請求沒有帶token，registry會重定向請求到token服務建立token。
  • Registry：對應啟動元件registry。負責儲存映象檔案，和處理映象的pull/push命令。Harbor對映象進行強制的訪問控制，Registry會將客戶端的每個pull、push請求轉發到token服務來獲取有效的token。
  • Admin Service：對應啟動元件harbor-adminserver。是系統的配置管理中心附帶檢查儲存用量，ui和jobserver啟動時候需要載入adminserver的配置；
  • Job Sevice：對應啟動元件harbor-jobservice。負責映象複製工作的，他和registry通訊，從一個registry pull映象然後push到另一個registry，並記錄job_log；
  • Log Collector：對應啟動元件harbor-log。日誌彙總元件，透過docker的log-driver把日誌彙總到一起；
  • Volnerability Scanning：對應啟動元件clair。負責映象掃描
  • Notary：對應啟動元件notary。負責映象認證
  • DB：對應啟動元件harbor-db，負責儲存project、 user、 role、replication、image_scan、access等的metadata資料。
四：安裝

1、安裝python-pip
yum -y install epel-release
yum -y install python-pip

2、安裝docker-compose
pip install docker-compose
待安裝完成後，執行查詢版本的命令，即可安裝docker-compose
docker-compose version

3.安裝Harbor
wget
tar -zxvf harbor-offline-installer-v1.4.0.tgz

配置檔案 harbor.cfg

點選(此處)摺疊或開啟

  1. ## Configuration file of Harbor

  3. #The IP address or hostname to access admin UI and registry service.
  4. #DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
  5. hostname = 120.79.156.135

  7. #The protocol for accessing the UI and token/notification service, by default it is http.
  8. #It can be set to https if ssl is enabled on nginx.
  9. ui_url_protocol = http

  11. #Maximum number of job workers in job service
  12. max_job_workers = 3

  14. #Determine whether or not to generate certificate for the registry's token.
  15. #If the value is on, the prepare script creates new root cert and private key
  16. #for generating token to access the registry. If the value is off the default key/cert will be used.
  17. #This flag also controls the creation of the notary signer's cert.
  18. customize_crt = on

  20. #The path of cert and key files for nginx, they are applied only the protocol is set to https
  21. ssl_cert = /mnt/harbor/cert/server.crt
  22. ssl_cert_key = /mnt/harbor/cert/server.key

  24. #The path of secretkey storage
  25. secretkey_path = /mnt/harbor

  27. #Admiral's url, comment this attribute, or set its value to NA when Harbor is standalone
  28. admiral_url = NA

  30. #Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
  31. log_rotate_count = 50
  32. #Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
  33. #If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
  34. #are all valid.
  35. log_rotate_size = 200M

  37. #NOTES: The properties between BEGIN INITIAL PROPERTIES and END INITIAL PROPERTIES
  38. #only take effect in the first boot, the subsequent changes of these properties
  39. #should be performed on web ui

  41. #************************BEGIN INITIAL PROPERTIES************************

  43. #Email account settings for sending out password resetting emails.

  45. #Email server uses the given username and password to authenticate on TLS connections to host and act as identity.
  46. #Identity left blank to act as username.
  47. email_identity =

  49. email_server = smtp.mydomain.com
  50. email_server_port = 25
  51. email_username = sample_admin@mydomain.com
  52. email_password = abc
  53. email_from = admin <sample_admin@mydomain.com>
  54. email_ssl = false
  55. email_insecure = false

  57. ##The initial password of Harbor admin, only works for the first time when Harbor starts.
  58. #It has no effect after the first launch of Harbor.
  59. #Change the admin password from UI after launching Harbor.
  60. harbor_admin_password = Weinong$2017

  62. ##By default the auth mode is db_auth, i.e. the credentials are stored in a local database.
  63. #Set it to ldap_auth if you want to verify a user's credentials against an LDAP server.
  64. auth_mode = db_auth

  66. #The url for an ldap endpoint.
  67. ldap_url = ldaps://ldap.mydomain.com

  69. #A user's DN who has the permission to search the LDAP/AD server.
  70. #If your LDAP/AD server does not support anonymous search, you should configure this DN and ldap_search_pwd.
  71. #ldap_searchdn = uid=searchuser,ou=people,dc=mydomain,dc=com

  73. #the password of the ldap_searchdn
  74. #ldap_search_pwd = password

  76. #The base DN from which to look up a user in LDAP/AD
  77. ldap_basedn = ou=people,dc=mydomain,dc=com

  79. #Search filter for LDAP/AD, make sure the syntax of the filter is correct.
  80. #ldap_filter = (objectClass=person)

  82. # The attribute used in a search to match a user, it could be uid, cn, email, sAMAccountName or other attributes depending on your LDAP/AD
  83. ldap_uid = uid

  85. #the scope to search for users, 0-LDAP_SCOPE_BASE, 1-LDAP_SCOPE_ONELEVEL, 2-LDAP_SCOPE_SUBTREE
  86. ldap_scope = 2

  88. #Timeout (in seconds) when connecting to an LDAP Server. The default value (and most reasonable) is 5 seconds.
  89. ldap_timeout = 5

  91. #Verify certificate from LDAP server
  92. ldap_verify_cert = true

  94. #Turn on or off the self-registration feature
  95. self_registration = on

  97. #The expiration time (in minute) of token created by token service, default is 30 minutes
  98. token_expiration = 30

  100. #The flag to control what users have permission to create projects
  101. #The default value "everyone" allows everyone to creates a project.
  102. #Set to "adminonly" so that only admin user can create project.
  103. project_creation_restriction = everyone

  105. #************************END INITIAL PROPERTIES************************

  107. #######Harbor DB configuration section#######

  109. #The address of the Harbor database. Only need to change when using external db.
  110. db_host = mysql

  112. #The password for the root user of Harbor DB. Change this before any production use.
  113. db_password = Weinong$2017

  115. #The port of Harbor database host
  116. db_port = 3306

  118. #The user name of Harbor database
  119. db_user = root

  121. ##### End of Harbor DB configuration#######

  123. #The redis server address. Only needed in HA installation.
  124. redis_url =

  126. ##########Clair DB configuration############

  128. #Clair DB host address. Only change it when using an exteral DB.
  129. clair_db_host = postgres

  131. #The password of the Clair's postgres database. Only effective when Harbor is deployed with Clair.
  132. #Please update it before deployment. Subsequent update will cause Clair's API server and Harbor unable to access Clair's database.
  133. clair_db_password = password

  135. #Clair DB connect port
  136. clair_db_port = 5432

  138. #Clair DB username
  139. clair_db_username = postgres

  141. #Clair default database
  142. clair_db = postgres

  144. ##########End of Clair DB configuration############

  146. #The following attributes only need to be set when auth mode is uaa_auth
  147. uaa_endpoint = uaa.mydomain.org
  148. uaa_clientid = id
  149. uaa_clientsecret = secret
  150. uaa_verify_cert = true
  151. uaa_ca_cert = /path/to/ca.pem


  154. ### Docker Registry setting ###
  155. #registry_storage_provider can be: filesystem, s3, gcs, azure, etc.
  156. registry_storage_provider_name = filesystem
  157. #registry_storage_provider_config is a comma separated "key: value" pairs, e.g. "key1: value, key2: value2".
  158. #Refer to https://docs.docker.com/registry/configuration/#storage for all available configuration.
  159. registry_storage_provider_config =
配置檔案docker-compose.yml

點選(此處)摺疊或開啟

  1. version: '2'
  2. services:
  3.   log:
  4.     image: vmware/harbor-log:v1.4.0
  5.     container_name: harbor-log
  6.     restart: always
  7.     volumes:
  8.       - /mnt/harbor/log/:/var/log/docker/:z
  9.       - ./common/config/log/:/etc/logrotate.d/:z
  10.     ports:
  11.       - 127.0.0.1:1514:10514
  12.     networks:
  13.       - harbor
  14.   registry:
  15.     image: vmware/registry-photon:v2.6.2-v1.4.0
  16.     container_name: registry
  17.     restart: always
  18.     volumes:
  19.       - /mnt/harbor/registry:/storage:z
  20.       - ./common/config/registry/:/etc/registry/:z
  21.     networks:
  22.       - harbor
  23.     environment:
  24.       - GODEBUG=netdns=cgo
  25.     command:
  26.       ["serve", "/etc/registry/config.yml"]
  27.     depends_on:
  28.       - log
  29.     logging:
  30.       driver: "syslog"
  31.       options:
  32.         syslog-address: "tcp://127.0.0.1:1514"
  33.         tag: "registry"
  34.   mysql:
  35.     image: vmware/harbor-db:v1.4.0
  36.     container_name: harbor-db
  37.     restart: always
  38.     volumes:
  39.       - /mnt/harbor/database:/var/lib/mysql:z
  40.     networks:
  41.       - harbor
  42.     env_file:
  43.       - ./common/config/db/env
  44.     depends_on:
  45.       - log
  46.     logging:
  47.       driver: "syslog"
  48.       options:
  49.         syslog-address: "tcp://127.0.0.1:1514"
  50.         tag: "mysql"
  51.   adminserver:
  52.     image: vmware/harbor-adminserver:v1.4.0
  53.     container_name: harbor-adminserver
  54.     env_file:
  55.       - ./common/config/adminserver/env
  56.     restart: always
  57.     volumes:
  58.       - /mnt/harbor/config/:/etc/adminserver/config/:z
  59.       - /mnt/harbor/secretkey:/etc/adminserver/key:z
  60.       - /mnt/harbor/:/data/:z
  61.     networks:
  62.       - harbor
  63.     depends_on:
  64.       - log
  65.     logging:
  66.       driver: "syslog"
  67.       options:
  68.         syslog-address: "tcp://127.0.0.1:1514"
  69.         tag: "adminserver"
  70.   ui:
  71.     image: vmware/harbor-ui:v1.4.0
  72.     container_name: harbor-ui
  73.     env_file:
  74.       - ./common/config/ui/env
  75.     restart: always
  76.     volumes:
  77.       - ./common/config/ui/app.conf:/etc/ui/app.conf:z
  78.       - ./common/config/ui/private_key.pem:/etc/ui/private_key.pem:z
  79.       - ./common/config/ui/certificates/:/etc/ui/certificates/:z
  80.       - /mnt/harbor/secretkey:/etc/ui/key:z
  81.       - /mnt/harbor/ca_download/:/etc/ui/ca/:z
  82.       - /mnt/harbor/psc/:/etc/ui/token/:z
  83.     networks:
  84.       - harbor
  85.     depends_on:
  86.       - log
  87.       - adminserver
  88.       - registry
  89.     logging:
  90.       driver: "syslog"
  91.       options:
  92.         syslog-address: "tcp://127.0.0.1:1514"
  93.         tag: "ui"
  94.   jobservice:
  95.     image: vmware/harbor-jobservice:v1.4.0
  96.     container_name: harbor-jobservice
  97.     env_file:
  98.       - ./common/config/jobservice/env
  99.     restart: always
  100.     volumes:
  101.       - /mnt/harbor/job_logs:/var/log/jobs:z
  102.       - ./common/config/jobservice/app.conf:/etc/jobservice/app.conf:z
  103.       - /mnt/harbor/secretkey:/etc/jobservice/key:z
  104.     networks:
  105.       - harbor
  106.     depends_on:
  107.       - ui
  108.       - adminserver
  109.     logging:
  110.       driver: "syslog"
  111.       options:
  112.         syslog-address: "tcp://127.0.0.1:1514"
  113.         tag: "jobservice"
  114.   proxy:
  115.     image: vmware/nginx-photon:v1.4.0
  116.     container_name: nginx
  117.     restart: always
  118.     volumes:
  119.       - ./common/config/nginx:/etc/nginx:z
  120.     networks:
  121.       - harbor
  122.     ports:
  123.       - 80:80
  124.       - 443:443
  125.       - 4443:4443
  126.     depends_on:
  127.       - mysql
  128.       - registry
  129.       - ui
  130.       - log
  131.     logging:
  132.       driver: "syslog"
  133.       options:
  134.         syslog-address: "tcp://127.0.0.1:1514"
  135.         tag: "proxy"
  136. networks:
  137.   harbor:
  138.     external: false
安裝Harbor
./install.sh

可以使用docker-compose來管理Harbor的生命週期。 一些有用的命令列出如下:
docker-compose ps  檢視


docker-compose stop  停止
docker-compose start 啟動
docker-compose down  刪除，利用./install.sh可以重新安裝

五：驗證

1.碰到的問題

a. 錯誤提示

點選(此處)摺疊或開啟

  1. /usr/lib/python2.7/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.21.1) or chardet (2.2.1) doesn
原因：python庫中urllib3 (1.21.1) or chardet (2.2.1) 的版本不相容
解決辦法：
pip uninstall urllib3
pip uninstall  chardet
pip install requests

b.錯誤提示：“harbor failed to initialize the system: read /etc/adminserver/key: is a directory”
原因：harbor.cfg中的secretkey_path和docker-compose.yml中的設定不一致

2.ui 介面


來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28624388/viewspace-2153546/，如需轉載，請註明出處，否則將追究法律責任。

相關文章