# 詳解MeterSphere 配置外部Mysql5.7的全過程

我從cnaaa.com購買了雲伺服器。

## 搭建一個主從複製（Master-Slave）的MySQL叢集

> 
>
> ![img](https:////upload-images.jianshu.io/upload_images/27470379-e5d16b88ff6795eb.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1080/format/webp)

## 本地儲存原理

> 這裡使用了本地儲存，也就是說，使用者希望Kubernetes能夠直接使用宿主機上的本地磁碟目錄，而不依賴於遠端儲存服務，來提供持久化的容器Volume
>  1、我們把儲存固定在一個節點上，但是Pod在排程的時候，是飄來飄去的，怎麼能讓Pod透過PVC也能固定在PVC上？
>  2、給這個Pod加上一個nodeAffinity行不行？

> 當然行，但是這變相破壞了開發人員對資源物件的定義規範了，開發人員應該不需要時刻考慮排程的細節。排程的改動應該交給運維就行。所以我們為了實現本地儲存，我們採用了**延遲繫結**的方法。方法很簡單，我們都知道StorageClass一般由運維人員設計，我們只需要在StorageClass指定no-provisioner。這是因為Local Persistent Volume目前尚不支援Dynamic Provisioning，所以它沒辦法在使用者建立PVC的時候，就自動建立出對應的PV。與此同時，這個StorageClass還定義了一個volumeBindingMode=WaitForFirstConsumer的屬性。它是Local Persistent Volume裡一個非常重要的特性，即：延遲繫結。

## 建立PV

> 先在Node （實驗用的Node節點IP是node1）節點上，預先分配幾個PV（不建議在生產上這樣操作），要保證在node1上有對應的目錄

```css
ssh root@node1
mkdir -pv /data/svr/projects/{mysql,mysql2,mysql3}
```

###### vim 01-persistentVolume-1.yaml

```bash
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-mysql-pv
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/svr/projects/mysql
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1
```

###### vim 01-persistentVolume-2.yaml

```bash
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-mysql-pv-2
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/svr/projects/mysql2
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1
```

###### vim 01-persistentVolume-3.yaml

```bash
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-mysql-pv-3
spec:
  capacity:
    storage: 15Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /data/svr/projects/mysql3
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1
```

> 記住，這是在生產上不推薦的做法，我只是實驗用途才這樣手動預先建立，**正規的做法應該透過StorageClass採用Dynamic Provisioning， 而不是Static Provisioning機制生產PV

```csharp
[root@master mysql]# kubectl apply -f 01-persistentVolume-1.yaml 
persistentvolume/example-mysql-pv created
[root@master mysql]# kubectl apply -f 01-persistentVolume-2.yaml 
persistentvolume/example-mysql-pv-2 created
[root@master mysql]# kubectl apply -f 01-persistentVolume-3.yaml 
persistentvolume/example-mysql-pv-3 created
[root@master mysql]# kubectl get pv
NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS    REASON   AGE
example-mysql-pv     15Gi       RWO            Delete           Available           local-storage            8s
example-mysql-pv-2   15Gi       RWO            Delete           Available           local-storage            6s
example-mysql-pv-3   15Gi       RWO            Delete           Available           local-storage            3s
```

###### 02-storageclass.yaml

```bash
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
```

```bash
[root@master mysql]# kubectl apply -f 02-storageclass.yaml 
storageclass.storage.k8s.io/local-storage created
[root@master mysql]# kubectl get sc
NAME            PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  5s
```

## 建立Namespace

###### 03-mysql-namespace.yaml

```php
apiVersion: v1
kind: Namespace
metadata:
  name: mysql
  labels:
    app: mysql
```

```tsx
[root@master mysql]# kubectl apply -f 03-mysql-namespace.yaml 
namespace/mysql unchanged
[root@master mysql]# kubectl get ns -n mysql
NAME              STATUS   AGE
mysql             Active   3d
```

## 建立資料庫的配置檔案configmap

###### 使用ConfigMap為Master/Slave節點分配不同的配置檔案

###### vim 04-mysql-configmap.yaml

```bash
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql
  namespace: mysql
  labels:
    app: mysql
data:
  master.cnf: |
    # Master配置
    [mysqld]
    log-bin=mysqllog
    skip-name-resolve
  slave.cnf: |
    # Slave配置
    [mysqld]
    super-read-only
    skip-name-resolve
    log-bin=mysql-bin
    replicate-ignore-db=mysql
```

```csharp
[root@master mysql]# kubectl apply -f 04-mysql-configmap.yaml 
configmap/mysql created
[root@master mysql]# kubectl get cm -n mysql
NAME               DATA   AGE
mysql              2      5s
```

## 建立MySQL密碼Secret

###### 05-mysql-secret.yaml

```cpp
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
  namespace: mysql
  labels:
    app: mysql
type: Opaque
data:
  password: MTIzNDU2 # echo -n "123456" | base64
```

```csharp
[root@master mysql]# kubectl apply -f 05-mysql-secret.yaml 
secret/mysql-secret created
[root@master mysql]# kubectl get secret -n mysql
NAME                  TYPE                                  DATA   AGE
mysql-secret          Opaque                                1      18s
```

## 使用Service為MySQL提供讀寫分離

###### 06-mysql-services.yaml

```cpp
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: mysql
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
  clusterIP: None
  selector:
    app: mysql
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-read
  namespace: mysql
  labels:
    app: mysql
spec:
  ports:
  - name: mysql
    port: 3306
  selector:
    app: mysql
```

> 使用者所有寫請求，必須以DNS記錄的方式直接訪問到Master節點，也就是mysql-0.mysql這條DNS記錄。

> 使用者所有讀請求，必須訪問自動分配的DNS記錄可以被轉發到任意一個Master或Slave節點上，也就是mysql-read這條DNS記錄。

```csharp
[root@master mysql]# kubectl apply -f 06-mysql-services.yaml 
service/mysql created
service/mysql-read created
[root@master mysql]# kubectl get svc -n mysql
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
mysql        ClusterIP   None           <none>        3306/TCP   3s
mysql-read   ClusterIP   10.107.90.19   <none>        3306/TCP   3s
```

## 建立MySQL叢集例項

###### 使用StatefulSet搭建MySQL主從叢集

###### 07-mysql-statefulset.yaml

```bash
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  serviceName: mysql
  replicas: 2
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: init-mysql
        image: mysql:5.7
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        command:
        - bash
        - "-c"
        - |
          set -ex
          # 從Pod的序號，生成server-id
          [[ $(hostname) =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          echo [mysqld] > /mnt/conf.d/server-id.cnf
          # 由於server-id不能為0，因此給ID加100來避開它
          echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
          # 如果Pod的序號為0，說明它是Master節點，從ConfigMap裡把Master的配置檔案複製到/mnt/conf.d目錄下
          # 否則，複製ConfigMap裡的Slave的配置檔案
          if [[ ${ordinal} -eq 0 ]]; then
            cp /mnt/config-map/master.cnf /mnt/conf.d
          else
            cp /mnt/config-map/slave.cnf /mnt/conf.d
          fi
        volumeMounts:
        - name: conf
          mountPath: /mnt/conf.d
        - name: config-map
          mountPath: /mnt/config-map
      - name: clone-mysql
        image: gcr.io/google-samples/xtrabackup:1.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        command:
        - bash
        - "-c"
        - |
          set -ex
          # 複製操作只需要在第一次啟動時進行，所以資料已經存在則跳過
          [[ -d /var/lib/mysql/mysql ]] && exit 0
          # Master 節點（序號為 0）不需要這個操作
          [[ $(hostname) =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          [[ $ordinal == 0 ]] && exit 0
          # 使用ncat指令，遠端地從前一個節點複製資料到本地
          ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
          # 執行 --prepare，這樣複製來的資料就可以用作恢復了
          xtrabackup --prepare --target-dir=/var/lib/mysql
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      containers:
      - name: mysql
        image: mysql:5.7
        env:
 #        - name: MYSQL_ALLOW_EMPTY_PASSWORD
 #          value: "1"
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        ports:
        - name: mysql
          containerPort: 3306
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
        livenessProbe:
          exec:
            command: ["mysqladmin", "ping", "-uroot", "-p${MYSQL_ROOT_PASSWORD}"]
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["mysqladmin", "ping", "-uroot", "-p${MYSQL_ROOT_PASSWORD}"]
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
      - name: xtrabackup
        image: gcr.io/google-samples/xtrabackup:1.0
        ports:
        - name: xtrabackup
          containerPort: 3307
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        command:
        - bash
        - "-c"
        - |
          set -ex
          cd /var/lib/mysql
          # 從備份資訊檔案裡讀取MASTER_LOG_FILE和MASTER_LOG_POS這2個欄位的值，用來拼裝叢集初始化SQL
          if [[ -f xtrabackup_slave_info ]]; then
            # 如果xtrabackup_slave_info檔案存在，說明這個備份資料來自於另一個Slave節點
            # 這種情況下，XtraBackup工具在備份的時候，就已經在這個檔案裡自動生成了“CHANGE MASTER TO”SQL語句
            # 所以，只需要把這個檔案重新命名為change_master_to.sql.in，後面直接使用即可
            mv xtrabackup_slave_info change_master_to.sql.in
            # 所以，也就用不著xtrabackup_binlog_info了
            rm -f xtrabackup_binlog_info
          elif [[ -f xtrabackup_binlog_info ]]; then
            # 如果只是存在xtrabackup_binlog_info檔案，說明備份來自於Master節點，就需要解析這個備份資訊檔案，讀取所需的兩個欄位的值
            [[ $(cat xtrabackup_binlog_info) =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
            rm xtrabackup_binlog_info
            # 把兩個欄位的值拼裝成SQL，寫入change_master_to.sql.in檔案
            echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                  MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
          fi
          # 如果存在change_master_to.sql.in，就意味著需要做叢集初始化工作
          if [[ -f change_master_to.sql.in ]]; then
            # 但一定要先等MySQL容器啟動之後才能進行下一步連線MySQL的操作
            echo "Waiting for mysqld to be ready（accepting connections）"
            until mysql -h 127.0.0.1 -uroot -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do sleep 1; done
            echo "Initializing replication from clone position"
            # 將檔案change_master_to.sql.in改個名字
            # 防止這個Container重啟的時候，因為又找到了change_master_to.sql.in，從而重複執行一遍初始化流程
            mv change_master_to.sql.in change_master_to.sql.orig
            # 使用change_master_to.sql.orig的內容，也就是前面拼裝的SQL，組成一個完整的初始化和啟動Slave的SQL語句
            mysql -h 127.0.0.1 -uroot -p${MYSQL_ROOT_PASSWORD} << EOF
          $(< change_master_to.sql.orig),
            MASTER_HOST='mysql-0.mysql.mysql',
            MASTER_USER='root',
            MASTER_PASSWORD='${MYSQL_ROOT_PASSWORD}',
            MASTER_CONNECT_RETRY=10;
          START SLAVE;
          EOF
          fi
          # 使用ncat監聽3307埠。
          # 它的作用是，在收到傳輸請求的時候，直接執行xtrabackup --backup命令，備份MySQL的資料併傳送給請求者
          exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
            "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=${MYSQL_ROOT_PASSWORD}"
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
          subPath: mysql
        - name: conf
          mountPath: /etc/mysql/conf.d
      volumes:
      - name: conf
        emptyDir: {}
      - name: config-map
        configMap:
          name: mysql
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - "ReadWriteOnce"
      storageClassName: local-storage
      resources:
        requests:
          storage: 3Gi
```

> 整體的StatefulSet有兩個Replicas，一個Master，一個Slave，然後使用init-mysql這個initContainers進行配置檔案的初始化。接著使用clone-mysql這個initContainers進行資料的傳輸；同時使用xtrabackup這個**sidecar容器**進行SQL初始化和資料傳輸功能。
>  建立 StatefulSet

```css
kubectl apply -f 07-mysql-statefulset.yaml
```

> 可以看到，StatefulSet啟動成功後，會有兩個Pod執行。接下來，我們可以嘗試向這個MySQL叢集 發起請求，執行一些SQL操作來驗證它是否正常。整個過程因為拉取mysql和一個gcr.io/google-samples/xtrabackup:1.0國外的映象會很慢,但是在建立mysql-0拉取一次之後，後續建立m>ysql-1就相對很快了。
>  最後，容器檢查pod的執行狀態

```csharp
[root@master mysql]# kubectl get po -n mysql
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          2m7s
mysql-1   2/2     Running   0          116s
```

## 服務驗證

###### 驗證主從狀態

```dart
[root@master mysql]# kubectl -n mysql exec mysql-1 -c mysql -- bash -c "mysql -uroot -p123456 -e 'show slave status \G'"
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: mysql-0.mysql.mysql
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 10
              Master_Log_File: mysqllog.000003
          Read_Master_Log_Pos: 154
               Relay_Log_File: mysql-1-relay-bin.000002
                Relay_Log_Pos: 319
        Relay_Master_Log_File: mysqllog.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: mysql0
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 154
              Relay_Log_Space: 528
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 100
                  Master_UUID: f8c7768a-6702-11ec-a648-c204e42fb62e
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
mysql: [Warning] Using a password on the command line interface can be insecure.
```

###### 接下來，我們透過Master容器建立資料庫和表、插入資料庫

```swift
[root@master mysql]# kubectl -n mysql exec mysql-0 -c mysql -- bash -c "mysql -uroot -p123456 -e 'create database test'"

[root@master mysql]# kubectl -n mysql exec mysql-0 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;create table counter(c int);'"

[root@master mysql]# kubectl -n mysql exec mysql-0 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;insert into counter values(123)'"

[root@master mysql]# kubectl -n mysql exec mysql-1 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;select * from counter'"  
c
123
```

###### 當看到輸出結果，主從同步正常了

###### 擴充套件節點

```csharp
[root@master mysql]# kubectl -n mysql scale statefulset mysql -—replicas=3
[root@master mysql]# kubectl get po -n mysql
NAME      READY   STATUS    RESTARTS   AGE
mysql-0   2/2     Running   0          11m
mysql-1   2/2     Running   0          11m
mysql-2   2/2     Running   0          20s
```

###### 檢視擴容節點資料是否已同步

```swift
kubectl -n mysql exec mysql-2 -c mysql -- bash -c "mysql -uroot -p123456 -e 'use test;select * from counter’"  
c
123
```</none></none>

詳解MeterSphere 配置外部Mysql5.7的全過程

相關文章