ceph叢集

MacoPlus發表於2024-08-25

環境

3臺centos 8機器，每臺機器上邊3個磁碟

機器名：ceph1、ceph2、ceph3

ceph-ansible叢集部署

在ceph1上邊準備好ceph-ansible

git clone https://github.com/ceph/ceph-ansible.git
cd ceph-ansible
git checkout stable-5.0 #centos 7用4.0
pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
echo "PATH=\$PATH:/usr/local/bin" >>~/.bashrc
source ~/.bashrc
ansible --version #正常看到版本號，說明部署成功

## 注意點
## 1. 配置ceph1免密登陸ceph2、ceph3(ceph1本身也需免密)
## 2. 防火牆需關閉、時間記得檢視是否同步

修改ceph-ansible的環境變數檔案和hosts

cp group_vars/all.yml.sample  group_vars/all.yml

cat all.yml|grep -v ^#|grep -v ^$
---
dummy:
ceph_release_num: 15
cluster: ceph
mon_group_name: mons
osd_group_name: osds
rgw_group_name: rgws
mds_group_name: mdss
mgr_group_name: mgrs
ntp_service_enabled: true
ntp_daemon_type: chronyd
ceph_origin: repository
ceph_repository: community
ceph_repository_type: cdn
ceph_stable_release: octopus
monitor_interface: eno3
journal_size: 10240 # OSD journal size in MB
public_network: 0.0.0.0/0
radosgw_interface: eno3
dashboard_admin_user: admin
dashboard_admin_password: xxxxxxxxxx
grafana_admin_user: admin
grafana_admin_password: xxxxxxxxxx
注意：
  1. 需修改 monitor_interface/radosgw_interface 為目標主機預設網路卡名，如 bond0
  2. 目標主機需要安裝元件 yum -y install ca-certificates
  3. python3 -m pip install six pyyaml
  4. ceph_release_num 根據系統版本修改，CentOS 7 為 14，跟 stable-4.0 對應為 ceph nautilus 版本
  5. ceph_stable_release 根據系統版本修改，CentOS 7 為 14，跟 stable-4.0 對應為 ceph nautilus 版本
  6. public_network 根據系統所在 IP 地址段修改，例如：192.168.0.0/16
cp group_vars/osds.yml.sample group_vars/osds.yml

cat osds.yml|grep -v ^#|grep -v ^$
---
dummy:
copy_admin_key: true
devices:
  - /dev/sdb
  - /dev/sdc
  - /dev/sdd

hosts.yml

# Ceph admin user for SSH and Sudo
[all:vars]
ansible_ssh_user=root
ansible_become=true
ansible_become_method=sudo
ansible_become_user=root

# Ceph Monitor Nodes
[mons]
ceph1
ceph2
ceph3

[mdss]
ceph1
ceph2
ceph3

[rgws]
ceph1
ceph2
ceph3

[osds]
ceph1
ceph2
ceph3

[mgrs]
ceph1
ceph2
ceph3

[grafana-server]
ceph1

site.yml

- hosts:
  - mons
  - osds
  - mdss
  - rgws
  #- nfss
  #- rbdmirrors
  #- clients
  - mgrs
  #- iscsigws
  #- iscsi-gws # for backward compatibility only!
  - grafana-server
  #- rgwloadbalancers

部署ceph叢集

ansible-playbook -i hosts site.yml

執行成功輸出如下：

解除安裝ceph叢集

cd /usr/local/ceph-ansible

ansible-playbook -i hosts infrastructure-playbooks/purge-cluster.yml

yum list installed | grep ceph

部署完成後檢查

ceph df

ceph osd df

新增 osds node

將新新增的 osds node 新增到 hosts 檔案 [osds] 區域，然後執行

ansible-playbook -vv -i hosts site-container.yml --limit {new osds node}

kubernetes+ceph

使用rbd儲存

配置storageclass

# 在k8s叢集中需要用到ceph的節點上安裝好ceph-common（有核心要求，這個需要注意）
# 需要使用kubelet使用rdb命令map附加rbd建立的image
yum install -y ceph-common

# 建立osd pool（在ceph的mon節點）
ceph osd pool create kube 128
ceph osd pool ls

# 建立k8s訪問ceph的使用者（在ceph的mon節點）
cd /etc/ceph
ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring

# 檢視key 在ceph的mon或者admin節點
ceph auth get-key client.admin
ceph auth get-key client.kube

# 建立admin的secret
CEPH_ADMIN_SECRET='xxxxxxxxxxxxxxxxxxxx=='
kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" \
--from-literal=key=$CEPH_ADMIN_SECRET \
--namespace=kube-system

# 在xxx-system名稱空間建立pvc用於訪問ceph的secret
CEPH_KUBE_SECRET='xxxxxxxxxxxxxxxxxxxxxx=='
kubectl create secret generic ceph-user-secret --type="kubernetes.io/rbd" \
--from-literal=key=$CEPH_KUBE_SECRET \
--namespace=xxx-system

# 檢視secret
kubectl get secret ceph-user-secret -nxxx-system -o yaml
kubectl get secret ceph-secret -nkube-system -o yaml

# 配置StorageClass
cat storageclass-ceph-rdb.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: xxx-ceph-rdb
provisioner: kubernetes.io/rbd
parameters:
  monitors: xxx.xxx.xxx.xxx:6789,xxx.xxx.xxx.xxx:6789,xxx.xxx.xxx.xxx:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: kube
  userId: kube
  userSecretName: ceph-user-secret
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"

# 建立
kubectl apply -f storageclass-ceph-rdb.yaml

# 檢視
kubectl get sc

使用cephFS儲存

部署cephfs-provisioner

# 官方沒有cephfs動態卷支援
# 使用社群提供的cephfs-provisioner
cat external-storage-cephfs-provisioner.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-provisioner
  namespace: xxx-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
subjects:
  - kind: ServiceAccount
    name: cephfs-provisioner
    namespace: xxx-system
roleRef:
  kind: ClusterRole
  name: cephfs-provisioner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-provisioner
  namespace: xxx-system
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-provisioner
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-provisioner
subjects:
- kind: ServiceAccount
  name: cephfs-provisioner
  namespace: xxx-system

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cephfs-provisioner
  namespace: xxx-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: cephfs-provisioner
    spec:
      containers:
      - name: cephfs-provisioner
        image: "quay.io/external_storage/cephfs-provisioner:v2.0.0-k8s1.11"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/cephfs
        command:
        - "/usr/local/bin/cephfs-provisioner"
        args:
        - "-id=cephfs-provisioner-1"
      serviceAccount: cephfs-provisioner

kubectl apply -f external-storage-cephfs-provisioner.yaml

# 檢視狀態，等待running之後，再進行後續的操作
kubectl get pod -n kube-system

配置storageclass

more storageclass-cephfs.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: xxx-cephfs
provisioner: ceph.com/cephfs
parameters:
  monitors: xxx.xxx.xxx.xxx:6789,xxx.xxx.xxx.xxx:6789,xxx.xxx.xxx.xxx:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  claimRoot: /volumes/kubernetes

# 建立
kubectl apply -f storageclass-cephfs.yaml

# 檢視
kubectl get sc

問題注意

  • ceph-ansible部署ceph過程中可能會出現安裝版本問題（centos8有出現），這個時候檢查一下yum源，是不是對應的centos的版本，centos7就用ceph對應centos7的yum源，centos8就用ceph對應centos8的yum源
  • 由於uat部署的ceph叢集是2.15比較高的版本，用的核心版本比較高，所以對ceph-common部署機器的核心版本要些要求，需要注意一下。如果之後生產部署全部用的centos7，那核心版本問題就不需要擔心
  • 如果使用ansible部署，記得注意osd加入後，防火牆是否開啟了。如果不希望防火牆開啟記得關閉或者改一下ansible指令碼

相關文章