前文我們瞭解了k8s節點汙點和pod的對節點汙點容忍度相關話題，回顧請參考：https://www.cnblogs.com/qiuhom-1874/p/14255486.html；今天我們來聊一下擴充套件k8s相關話題；
k8s上建立資源物件的過程
我們知道在k8s上，資源是有型別的，不同型別的資源，其定義方式和使用的欄位各不相同；使用者建立一個資源，實際上就是把k8s抽象的資源做例項化，即把k8s抽象出來的資源，通過資源清單賦值，建立出來的物件就是我們例項化對應型別資源的結果；使用者建立一個資源，首先會把請求傳送給apiserver，通過apiserver的認證、授權、准入控制以後，對應建立資源的定義就存放在etcd中，控制器通過watch機制監視apiserver上的資源變動，通過對應資源變動事件觸發對應型別資源的控制器將對應資源建立出來，並通過控制器內部的和解迴圈監控著對應資源狀態是否和使用者定義的期望狀態一樣，如果發現不一樣，內部和解迴圈就會被觸發，對應控制器會向apiserver發起建立資源的請求，將對應資源重建，讓對應資源的狀態始終滿足使用者期望的狀態；從上述的過程來看，使用者建立一個資源分兩個步驟，第一步是將對應請求傳送給apiserver，通過apiserver把對應資源定義的資訊存放在etcd中；第二個步驟是對應資源型別的控制器通過apiserver從etcd中讀取對應資源的定義，將其建立出來；對於etcd來說，它本是就一個kv資料庫，可以儲存任意型別的kv資料，但在在k8s上，apiserver將不同型別的資源定義抽象成不同的資源，使得使用者建立對應資源，必須是滿足對應型別資源定義的規範，然後將規範的定義存放在etcd中；簡單講apiserver就是把使用者存入etcd中的資料做了一層抽象，使得使用者不能夠隨意將任意資料儲存到etcd中，存入etcd中的資料必須是滿足對應apiserver介面定義的規範；這就好比我們在使用mysql資料庫時，必須遵守對應庫中的對應表的定義；
在k8s上建立自定義資源型別
在k8s上，資源的型別有很多，比如pod，service，PersistentVolume，PersistentVolumeClaim等等，這些都是一些基礎的資源型別；我們要建立某種資源，直接使用對應的資源型別，例項化一個物件即可；假如我們要在k8s上建立一個叢集，我們是不是可以直接使用某種型別的資源，例項化一個叢集物件呢？理論上是可以的，但是前提是對應k8s上有對應型別的資源；有對應型別的資源，使用者就可以把對應建立資源的定義存放在etcd中；除了有對應型別的資源，我們還需要有對應的控制器將對應資源建立出來；這樣一來對於不同叢集或應用來說，其組織方式和邏輯都不同，使用的資源型別和控制器也有所不同；使用者要想例項化更高階的資源來，就必須自己手動實現定義其資源型別，將對應資源型別例項化為物件；除此之外必要時還需要自己實現對應資源的控制器；簡單講使用者想要實現更高階的資源型別，就必須擴充套件現有k8s的資源型別和控制器；
在k8s上擴充套件資源型別的方式有三種，第一種是crd，crd是k8s內建的資源型別，該型別資源主要用來建立使用者自定義資源型別的資源；即通過crd資源，可以將使用者自定義資源型別轉換為k8s上資源型別；第二種是自定義apiserver；這種方式要比第一種方式要複雜一點，需要使用者手動開發程式實現對應功能的apiserver，讓其使用者建立自定義型別資源能夠通過自定義apiserver實現；第三種方式就是修改現有k8sapiserver，讓其支援對應使用者自定義資源型別；
自定義控制器
自定義資源型別我們可以使用crd資源實現，也可以使用自定義apiserver或修改原有apiserver程式碼實現，但是隻有資源型別是不能夠讓對應自定義型別資源例項化為一個自定義資源物件，只有自定義資源型別，使用者建立對應資源型別的資源物件時，只能把對應資源型別的定義資訊寫入到etcd中，它不能真正的跑起來，要想真正的跑起來，我們還需要一個自定義控制器，專門負責監聽對應的資源型別的資源變化，將對應資源例項化為對應k8s上的資源物件；當然不是所有的自定義型別的資源都需要自定義控制器，如果對應自定義型別資源呼叫了底層基礎控制器來管控對應自定義資源，那麼對應自定義型別資源就不需要使用自定義控制器；我們知道控制器是k8s上的一個重要元件，它的工作邏輯是註冊監聽在apiserver上對應型別的資源變動，如果對應資源狀態不滿足使用者期望狀態，它就會根據內部的和解迴圈來請求apiserver將對應型別資源的定義傳送給它，然後根據資源定義來重建對應的資源，讓其狀態始終和使用者期望的狀態保持一致；自定義控制器也是同樣的邏輯，使用自定義控制器的目的也是讓對應自定義型別資源能夠被自定義控制器監聽，一旦對應資源發生變動，它能夠將其在k8s上建立出來，並一直保持和使用者期望的狀態吻合；自定義控制器和自定義資源型別可以分開實現，也可以合併在一起實現，即自定義控制器程式能夠自動建立crd資源，讓其對應自定義型別資源能夠被k8s識別並將其建立出來；具體是分開實現還是合併在一起實現，取決開發自定義控制器程式設計師；
crd資源定義幫助
[root@master01 ~]# kubectl explain crd
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
DESCRIPTION:
CustomResourceDefinition represents a resource that should be exposed on
the API server. Its name MUST be in the format <.spec.name>.<.spec.group>.
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
spec <Object> -required-
spec describes how the user wants the resources to appear
status <Object>
status indicates the actual state of the CustomResourceDefinition
[root@master01 ~]#
提示：crd資源是k8s上的標準資源之一，它的定義主要有apiVersion，kind，metadata，spec和status；其中kind型別為CustomResourceDefinition，apiVersion是apiextensions.k8s.io/v1；這兩個是固定格式；spec欄位是用來定義對應指定資源型別資源的相關屬性；
crd.spec欄位說明
[root@master01 ~]# kubectl explain crd.spec
KIND: CustomResourceDefinition
VERSION: apiextensions.k8s.io/v1
RESOURCE: spec <Object>
DESCRIPTION:
spec describes how the user wants the resources to appear
CustomResourceDefinitionSpec describes how a user wants their resource to
appear
FIELDS:
conversion <Object>
conversion defines conversion settings for the CRD.
group <string> -required-
group is the API group of the defined custom resource. The custom resources
are served under `/apis/<group>/...`. Must match the name of the
CustomResourceDefinition (in the form `<names.plural>.<group>`).
names <Object> -required-
names specify the resource and kind names for the custom resource.
preserveUnknownFields <boolean>
preserveUnknownFields indicates that object fields which are not specified
in the OpenAPI schema should be preserved when persisting to storage.
apiVersion, kind, metadata and known fields inside metadata are always
preserved. This field is deprecated in favor of setting
`x-preserve-unknown-fields` to true in
`spec.versions[*].schema.openAPIV3Schema`. See
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields
for details.
scope <string> -required-
scope indicates whether the defined custom resource is cluster- or
namespace-scoped. Allowed values are `Cluster` and `Namespaced`.
versions <[]Object> -required-
versions is the list of all API versions of the defined custom resource.
Version names are used to compute the order in which served versions are
listed in API discovery. If the version string is "kube-like", it will sort
above non "kube-like" version strings, which are ordered lexicographically.
"Kube-like" versions start with a "v", then are followed by a number (the
major version), then optionally the string "alpha" or "beta" and another
number (the minor version). These are sorted first by GA > beta > alpha
(where GA is a version with no suffix such as beta or alpha), and then by
comparing major version, then minor version. An example sorted list of
versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2,
foo1, foo10.
[root@master01 ~]#
提示：crd.spec中group欄位使用來描述對應自定義型別資源的群組名稱，其值為字串；names欄位是用來描述自定義型別資源對應型別，名稱等等，其值為一個物件；scope欄位用來定義對應自定義資源是那個級別的資源；該欄位的值只能為Cluster或Namespaced；versions欄位是用來指定對應自定義資源的版本，以及對應型別資源的屬性欄位等資訊，該欄位為一個列表物件；
示例：定義一個自定義型別資源
[root@master01 ~]# cat crontab-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# 名字必需與下面的 spec 欄位匹配，並且格式為 '<名稱的複數形式>.<組名>'
name: crontabs.stable.example.com
spec:
# 組名稱，用於 REST API: /apis/<組>/<版本>
group: stable.example.com
# 列舉此 CustomResourceDefinition 所支援的版本
versions:
- name: v1
# 每個版本都可以通過 served 標誌來獨立啟用或禁止
served: true
# 其中一個且只有一個版本必需被標記為儲存版本
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
cronSpec:
type: string
image:
type: string
replicas:
type: integer
# 可以是 Namespaced 或 Cluster
scope: Namespaced
names:
# 名稱的複數形式，用於 URL：/apis/<組>/<版本>/<名稱的複數形式>
plural: crontabs
# 名稱的單數形式，作為命令列使用時和顯示時的別名
singular: crontab
# kind 通常是單數形式的駝峰編碼（CamelCased）形式。你的資源清單會使用這一形式。
kind: CronTab
# shortNames 允許你在命令列使用較短的字串來匹配資源
shortNames:
- ct
[root@master01 ~]#
應用資源清單前使用kubectl get crontab
[root@master01 ~]# kubectl get crontab error: the server doesn't have a resource type "crontab" [root@master01 ~]#
提示：在沒有應用資源清單前使用kubectl get crontab，它會提示我們所沒有型別為crontab的資源
應用資源清單
[root@master01 ~]# kubectl apply -f crontab-crd.yaml customresourcedefinition.apiextensions.k8s.io/crontabs.stable.example.com created [root@master01 ~]# kubectl get crontab No resources found in default namespace. [root@master01 ~]#
提示：應用資源清單後，再次使用kubectl get crontab就沒有報錯了，只是提示在default名稱空間沒有對應型別資源；
檢視crd
[root@master01 ~]# kubectl get crd NAME CREATED AT bgpconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z bgppeers.crd.projectcalico.org 2021-01-03T15:49:21Z blockaffinities.crd.projectcalico.org 2021-01-03T15:49:21Z clusterinformations.crd.projectcalico.org 2021-01-03T15:49:21Z crontabs.stable.example.com 2021-01-12T12:39:00Z felixconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworksets.crd.projectcalico.org 2021-01-03T15:49:21Z hostendpoints.crd.projectcalico.org 2021-01-03T15:49:21Z ipamblocks.crd.projectcalico.org 2021-01-03T15:49:21Z ipamconfigs.crd.projectcalico.org 2021-01-03T15:49:21Z ipamhandles.crd.projectcalico.org 2021-01-03T15:49:21Z ippools.crd.projectcalico.org 2021-01-03T15:49:21Z kubecontrollersconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z networkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z networksets.crd.projectcalico.org 2021-01-03T15:49:22Z [root@master01 ~]# kubectl get crd/crontabs.stable.example.com NAME CREATED AT crontabs.stable.example.com 2021-01-12T12:39:00Z [root@master01 ~]#
檢視詳情
[root@master01 ~]# kubectl get crd/crontabs.stable.example.com
NAME CREATED AT
crontabs.stable.example.com 2021-01-12T12:39:00Z
[root@master01 ~]# kubectl describe crd/crontabs.stable.example.com
Name: crontabs.stable.example.com
Namespace:
Labels: <none>
Annotations: <none>
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2021-01-12T12:39:00Z
Generation: 1
Managed Fields:
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:acceptedNames:
f:kind:
f:listKind:
f:plural:
f:shortNames:
f:singular:
f:conditions:
Manager: kube-apiserver
Operation: Update
Time: 2021-01-12T12:39:00Z
API Version: apiextensions.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
f:conversion:
.:
f:strategy:
f:group:
f:names:
f:kind:
f:listKind:
f:plural:
f:shortNames:
f:singular:
f:scope:
f:versions:
f:status:
f:storedVersions:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-01-12T12:39:00Z
Resource Version: 805506
UID: b92a90f4-c953-4876-a496-030c9ba023fd
Spec:
Conversion:
Strategy: None
Group: stable.example.com
Names:
Kind: CronTab
List Kind: CronTabList
Plural: crontabs
Short Names:
ct
Singular: crontab
Scope: Namespaced
Versions:
Name: v1
Schema:
openAPIV3Schema:
Properties:
Spec:
Properties:
Cron Spec:
Type: string
Image:
Type: string
Replicas:
Type: integer
Type: object
Type: object
Served: true
Storage: true
Status:
Accepted Names:
Kind: CronTab
List Kind: CronTabList
Plural: crontabs
Short Names:
ct
Singular: crontab
Conditions:
Last Transition Time: 2021-01-12T12:39:00Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: 2021-01-12T12:39:00Z
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Stored Versions:
v1
Events: <none>
[root@master01 ~]#
使用自定義資源型別crontab建立資源
[root@master01 ~]# cat my-crontab.yaml apiVersion: "stable.example.com/v1" kind: CronTab metadata: name: my-new-cron-object spec: cronSpec: "* * * * */5" image: my-awesome-cron-image [root@master01 ~]#
提示：上述資源清單表示建立一個型別為Crontab的資源，該資源的群組版本為stable.example.com/v1;
應用資源清單
[root@master01 ~]# kubectl apply -f my-crontab.yaml
crontab.stable.example.com/my-new-cron-object created
[root@master01 ~]# kubectl get ct
NAME AGE
my-new-cron-object 5s
[root@master01 ~]# kubectl describe ct/my-new-cron-object
Name: my-new-cron-object
Namespace: default
Labels: <none>
Annotations: <none>
API Version: stable.example.com/v1
Kind: CronTab
Metadata:
Creation Timestamp: 2021-01-12T12:45:29Z
Generation: 1
Managed Fields:
API Version: stable.example.com/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:cronSpec:
f:image:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-01-12T12:45:29Z
Resource Version: 806182
UID: 31a88a3d-fa99-42b8-80f6-3e4559efdc40
Spec:
Cron Spec: * * * * */5
Image: my-awesome-cron-image
Events: <none>
[root@master01 ~]#
提示：可以看到對應型別資源已經建立成功；以上示例只是單純的crd的使用示例，沒有任何實質的作用；
部署自定義控制器
示例：部署mongodb-aperator
1、克隆專案
[root@master01 ~]# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git Cloning into 'mongodb-kubernetes-operator'... remote: Enumerating objects: 95, done. remote: Counting objects: 100% (95/95), done. remote: Compressing objects: 100% (74/74), done. remote: Total 4506 (delta 30), reused 60 (delta 15), pack-reused 4411 Receiving objects: 100% (4506/4506), 18.04 MiB | 183.00 KiB/s, done. Resolving deltas: 100% (2621/2621), done. [root@master01 ~]#
2、建立名稱空間mongodb，並進入到mongodb-kubernetes-operator目錄應用crd資源，建立自定義資源型別
[root@master01 mongodb-kubernetes-operator]# kubectl create ns mongodb namespace/mongodb created [root@master01 mongodb-kubernetes-operator]# kubectl get ns NAME STATUS AGE default Active 35d ingress-nginx Active 22d kube-node-lease Active 35d kube-public Active 35d kube-system Active 35d kubernetes-dashboard Active 11d mongodb Active 4s [root@master01 mongodb-kubernetes-operator]# ls agent build deploy docs go.sum pkg release.json scripts testdata version APACHE2 cmd dev_notes go.mod LICENSE.md README.md requirements.txt test tools.go [root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_mongodb_crd.yaml -n mongodb Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/mongodb.mongodb.com created [root@master01 mongodb-kubernetes-operator]#
驗證：檢視mongodb型別資源是否已經建立成功？
[root@master01 mongodb-kubernetes-operator]# kubectl get crd NAME CREATED AT bgpconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z bgppeers.crd.projectcalico.org 2021-01-03T15:49:21Z blockaffinities.crd.projectcalico.org 2021-01-03T15:49:21Z clusterinformations.crd.projectcalico.org 2021-01-03T15:49:21Z crontabs.stable.example.com 2021-01-12T12:39:00Z felixconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z globalnetworksets.crd.projectcalico.org 2021-01-03T15:49:21Z hostendpoints.crd.projectcalico.org 2021-01-03T15:49:21Z ipamblocks.crd.projectcalico.org 2021-01-03T15:49:21Z ipamconfigs.crd.projectcalico.org 2021-01-03T15:49:21Z ipamhandles.crd.projectcalico.org 2021-01-03T15:49:21Z ippools.crd.projectcalico.org 2021-01-03T15:49:21Z kubecontrollersconfigurations.crd.projectcalico.org 2021-01-03T15:49:21Z mongodb.mongodb.com 2021-01-13T06:38:22Z networkpolicies.crd.projectcalico.org 2021-01-03T15:49:21Z networksets.crd.projectcalico.org 2021-01-03T15:49:22Z [root@master01 mongodb-kubernetes-operator]# kubectl get crd/mongodb.mongodb.com NAME CREATED AT mongodb.mongodb.com 2021-01-13T06:38:22Z [root@master01 mongodb-kubernetes-operator]#
3、安裝operator
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/operator/ -n mongodb deployment.apps/mongodb-kubernetes-operator created role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created serviceaccount/mongodb-kubernetes-operator created [root@master01 mongodb-kubernetes-operator]#
提示：mongodb-kubernetes-operator這個專案是將自定義控制器和自定義資源型別分開實現的；其operator只負責建立和監聽對應資源型別的變化，在資源有變化時，例項化為對應資源物件，並保持對應資源物件狀態吻合使用者期望狀態；上述四個清單中主要是建立了一個sa賬戶，並對對應的sa使用者授權；
operator.yaml內容
[root@master01 mongodb-kubernetes-operator]# cat deploy/operator/operator.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongodb-kubernetes-operator
spec:
replicas: 1
selector:
matchLabels:
name: mongodb-kubernetes-operator
template:
metadata:
labels:
name: mongodb-kubernetes-operator
spec:
serviceAccountName: mongodb-kubernetes-operator
containers:
- name: mongodb-kubernetes-operator
image: quay.io/mongodb/mongodb-kubernetes-operator:0.3.0
command:
- mongodb-kubernetes-operator
imagePullPolicy: Always
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "mongodb-kubernetes-operator"
- name: AGENT_IMAGE # The MongoDB Agent the operator will deploy to manage MongoDB deployments
value: quay.io/mongodb/mongodb-agent:10.19.0.6562-1
- name: VERSION_UPGRADE_HOOK_IMAGE
value: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
- name: MONGODB_IMAGE
value: "library/mongo"
- name: MONGODB_REPO_URL
value: "registry.hub.docker.com"
[root@master01 mongodb-kubernetes-operator]#
提示：上述資源主要用deploy控制器執行對應自定義控制器為一個pod；
驗證：檢視operator是否正常執行
[root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 26s [root@master01 mongodb-kubernetes-operator]#
提示：能夠看到operator正常執行，就表示operator已經安裝成功；
驗證：使用自定義資源型別建立一個mongodb 副本集叢集
[root@master01 mongodb-kubernetes-operator]# cat deploy/crds/mongodb.com_v1_mongodb_cr.yaml
---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
name: example-mongodb
spec:
members: 3
type: ReplicaSet
version: "4.2.6"
security:
authentication:
modes: ["SCRAM"]
users:
- name: my-user
db: admin
passwordSecretRef: # a reference to the secret that will be used to generate the user's password
name: my-user-password
roles:
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: my-scram
# the user credentials will be generated from this secret
# once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
name: my-user-password
type: Opaque
stringData:
password: 58LObjiMpxcjP1sMDW
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml
mongodb.mongodb.com/example-mongodb created
secret/my-user-password created
[root@master01 mongodb-kubernetes-operator]#
應用清單
[root@master01 mongodb-kubernetes-operator]# kubectl apply -f deploy/crds/mongodb.com_v1_mongodb_cr.yaml -n mongodb mongodb.mongodb.com/example-mongodb created secret/my-user-password created [root@master01 mongodb-kubernetes-operator]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 0/2 Pending 0 9s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 88s [root@master01 mongodb-kubernetes-operator]#
提示：這裡可以看到對應pod處於pending狀態；
檢視pod詳細資訊
[root@master01 mongodb-kubernetes-operator]# kubectl describe pod/example-mongodb-0 -n mongodb|grep -A 10 "Events" Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 66s (x2 over 66s) default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. [root@master01 mongodb-kubernetes-operator]#
提示：這裡提示沒有可以用的pvc；
刪除mongodb名稱空間下pvc
[root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-volume-example-mongodb-0 Pending 92s [root@master01 mongodb-kubernetes-operator]# kubectl delete pvc --all -n mongodb persistentvolumeclaim "data-volume-example-mongodb-0" deleted [root@master01 mongodb-kubernetes-operator]# kubectl get pvc -n mongodb No resources found in mongodb namespace. [root@master01 mongodb-kubernetes-operator]#
建立pv和pvc
[root@master01 mongodb-kubernetes-operator]# cd
[root@master01 ~]# cat pv-demo.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-v1
labels:
app: example-mongodb-svc
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /data/v1
server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-v2
labels:
app: example-mongodb-svc
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /data/v2
server: 192.168.0.99
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-pv-v3
labels:
app: example-mongodb-svc
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes: ["ReadWriteOnce","ReadWriteMany","ReadOnlyMany"]
persistentVolumeReclaimPolicy: Retain
mountOptions:
- hard
- nfsvers=4.1
nfs:
path: /data/v3
server: 192.168.0.99
[root@master01 ~]#
應用清單建立pv
[root@master01 ~]# kubectl apply -f pv-demo.yaml persistentvolume/nfs-pv-v1 created persistentvolume/nfs-pv-v2 created persistentvolume/nfs-pv-v3 created [root@master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv-v1 1Gi RWO,ROX,RWX Retain Available 3s nfs-pv-v2 1Gi RWO,ROX,RWX Retain Available 3s nfs-pv-v3 1Gi RWO,ROX,RWX Retain Available 3s [root@master01 ~]#
建立pvc清單
[root@master01 ~]# cat pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-example-mongodb-0
namespace: mongodb
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-example-mongodb-1
namespace: mongodb
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 500Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-volume-example-mongodb-2
namespace: mongodb
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 500Mi
[root@master01 ~]#
應用清單建立pvc
[root@master01 ~]# kubectl get pvc -n mongodb No resources found in mongodb namespace. [root@master01 ~]# kubectl apply -f pvc-demo.yaml persistentvolumeclaim/data-volume-example-mongodb-0 created persistentvolumeclaim/data-volume-example-mongodb-1 created persistentvolumeclaim/data-volume-example-mongodb-2 created [root@master01 ~]# kubectl get pvc -n mongodb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-volume-example-mongodb-0 Bound nfs-pv-v1 1Gi RWO,ROX,RWX 6s data-volume-example-mongodb-1 Bound nfs-pv-v2 1Gi RWO,ROX,RWX 6s data-volume-example-mongodb-2 Bound nfs-pv-v3 1Gi RWO,ROX,RWX 6s [root@master01 ~]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE nfs-pv-v1 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-0 102s nfs-pv-v2 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-1 102s nfs-pv-v3 1Gi RWO,ROX,RWX Retain Bound mongodb/data-volume-example-mongodb-2 102s [root@master01 ~]#
提示：可以看到對應pvc和pv已經繫結好了；
驗證：檢視mongodb副本集叢集是否跑起來了？
[root@master01 ~]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 6m19s example-mongodb-1 0/2 PodInitializing 0 10s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 7m38s [root@master01 ~]# kubectl get pods -n mongodb -w NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 6m35s example-mongodb-1 1/2 Running 0 26s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 7m54s example-mongodb-1 2/2 Running 0 43s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Init:0/1 0 0s example-mongodb-2 0/2 Init:0/1 0 1s example-mongodb-2 0/2 Terminating 0 4s example-mongodb-2 0/2 Terminating 0 6s example-mongodb-2 0/2 Terminating 0 20s example-mongodb-2 0/2 Terminating 0 20s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Pending 0 0s example-mongodb-2 0/2 Init:0/1 0 0s example-mongodb-2 0/2 Init:0/1 0 1s example-mongodb-2 0/2 PodInitializing 0 7s example-mongodb-2 1/2 Running 0 14s example-mongodb-2 2/2 Running 0 36s ^C[root@master01 ~]# kubectl get pods -n mongodb NAME READY STATUS RESTARTS AGE example-mongodb-0 2/2 Running 0 8m example-mongodb-1 2/2 Running 0 111s example-mongodb-2 2/2 Running 0 48s mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 9m19s [root@master01 ~]#
提示：可以看到對應的pod已經正常跑起來了；
驗證：使用mongo 連線mongodbpod，看看對應副本集叢集是否工作正常？
[root@master01 ~]# kubectl get pods -n mongodb -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-mongodb-0 2/2 Running 0 9m12s 10.244.4.101 node04.k8s.org <none> <none>
example-mongodb-1 2/2 Running 0 3m3s 10.244.2.130 node02.k8s.org <none> <none>
example-mongodb-2 2/2 Running 0 2m 10.244.1.130 node01.k8s.org <none> <none>
mongodb-kubernetes-operator-7d557bcc95-th8js 1/1 Running 0 10m 10.244.3.116 node03.k8s.org <none> <none>
[root@master01 ~]# mongo 10.244.4.101
MongoDB shell version v4.4.3
connecting to: mongodb://10.244.4.101:27017/test?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b9d16fe9-6a74-4638-96e6-70aaf3c83bfa") }
MongoDB server version: 4.2.6
WARNING: shell and server versions do not match
example-mongodb:PRIMARY> show dbs
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
Error: Authentication failed.
0
example-mongodb:PRIMARY> use admin
switched to db admin
example-mongodb:PRIMARY> db.auth('my-user','58LObjiMpxcjP1sMDW')
1
example-mongodb:PRIMARY> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
example-mongodb:PRIMARY> db.isMaster()
{
"hosts" : [
"example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"example-mongodb-1.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"example-mongodb-2.example-mongodb-svc.mongodb.svc.cluster.local:27017"
],
"setName" : "example-mongodb",
"setVersion" : 1,
"ismaster" : true,
"secondary" : false,
"primary" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"me" : "example-mongodb-0.example-mongodb-svc.mongodb.svc.cluster.local:27017",
"electionId" : ObjectId("7fffffff0000000000000003"),
"lastWrite" : {
"opTime" : {
"ts" : Timestamp(1610520741, 1),
"t" : NumberLong(3)
},
"lastWriteDate" : ISODate("2021-01-13T06:52:21Z"),
"majorityOpTime" : {
"ts" : Timestamp(1610520741, 1),
"t" : NumberLong(3)
},
"majorityWriteDate" : ISODate("2021-01-13T06:52:21Z")
},
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 100000,
"localTime" : ISODate("2021-01-13T06:52:27.873Z"),
"logicalSessionTimeoutMinutes" : 30,
"connectionId" : 153,
"minWireVersion" : 0,
"maxWireVersion" : 8,
"readOnly" : false,
"ok" : 1,
"$clusterTime" : {
"clusterTime" : Timestamp(1610520741, 1),
"signature" : {
"hash" : BinData(0,"EcWzL7O9Ue9kmm6cQ4FumkcIP6g="),
"keyId" : NumberLong("6917119940596072451")
}
},
"operationTime" : Timestamp(1610520741, 1)
}
example-mongodb:PRIMARY>
提示：可以看到三個mongodb pod是副本集關係；其中example-mongodb-0是主節點，其他兩個是從節點；
最後還是說一下，我在做上面的實驗中，雖然mongodb operator已經正常工作，但是用mongo這個客戶端工具連線到主節點沒法寫資料，提示沒有許可權；但對應的使用者在對應庫是有讀寫許可權的；在admin庫下建立使用者能提示使用者新增成功，但是過幾秒查詢使用者資訊，發現使用者不存在，不知道什麼原因，有知道的朋友麻煩告知下（博主郵箱：linux-1874@qq.com），博主將感激不盡。。