helm 部署Prometheus

環境資訊：
kubernetes版本：v1.18.2（kubeasz部署）
helm 版本：v3.2.1
系統版本：centos7.7
helm倉庫：http://mirror.azure.cn/kubernetes/charts

一、部署Prometheus-operator

1、安裝helm3並新增倉庫

wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
tar -xf helm-v3.2.1-linux-amd64.tar.gz
mv ./linux-amd64/helm /usr/bin
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm search repo prometheus-community

2、映象下載

1>prometheus-operator 涉及到的映象

quay.io/prometheus/alertmanager:v0.21.0
docker.io/jimmidyson/configmap-reload:v0.3.0
kiwigrid/k8s-sidecar:0.1.151
grafana/grafana:7.0.5
quay.io/coreos/kube-state-metrics:v1.9.7
quay.io/coreos/prometheus-operator:v0.38.1
squareup/ghostunnel:v1.5.2
quay.io/prometheus/node-exporter:v1.0.0
quay.io/prometheus/prometheus:v2.18.2
quay.io/coreos/prometheus-config-reloader:v0.38.1
docker.io/jimmidyson/configmap-reload:v0.3.0

2>上述映象已經匯出，可以通過docker run 方式copy一份，或者通過百度網盤下載(605M)

docker run -itd --name=download registry.baidubce.com/tools/download-prometheus-operator:v0.38.1
docker cp download:/opt/prometheus-operator-0381.tar .
或者百度網盤下載
連結：https://pan.baidu.com/s/1jAKS7fHeuz8xy3alraBDJA 
提取碼：f219 

最後再各個節點載入映象
docker load -i prometheus-operator-0381.tar

3、部署Prometheus

1>部署prometheus-operator

想要提前自定義配置可以把chart包下載下來
#helm pull prometheus-community/prometheus-operator（可選）
kubectl create ns monitoring
helm install prometheus-operator --set rbacEnable=true --namespace=monitoring  --wait prometheus-community/prometheus-operator

2>、安裝失敗清理（如果安裝失敗無法使用helm解除安裝）

 kubectl -n monitoring get deployments.apps|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete deployments.apps
 kubectl -n monitoring get svc|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete svc
 kubectl -n monitoring get sts|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete sts
 kubectl -n monitoring get crd|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete crd
 kubectl -n monitoring get servicemonitorings.monitoringing.coreos.com|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete servicemonitorings.monitoringing.coreos.com
 kubectl -n monitoring get ds|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete ds
 kubectl -n monitoring get job|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete job
 kubectl -n monitoring get pod|tail -n +2|awk '{print $1}'|xargs kubectl -n monitoring delete pod
 kubectl get PodSecurityPolicy|grep prome|awk '{print $1}'|xargs kubectl delete podsecuritypolicies.policy
 kubectl get clusterrole|grep prometheus|awk '{print $1}'|xargs kubectl delete clusterrole
 kubectl get clusterrolebindings.rbac.authorization.k8s.io|grep prometheus|awk '{print $1}'|xargs kubectl delete clusterrolebindings.rbac.authorization.k8s.io
 kubectl  -n kube-system get svc|grep prometheus|awk '{print $1}'|xargs kubectl -n kube-system delete svc
 kubectl  -n kube-system get mutatingwebhookconfigurations.admissionregistration.k8s.io|grep prometheus|awk '{print $1}'|xargs kubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io
 kubectl  -n kube-system get validatingwebhookconfigurations.admissionregistration.k8s.io|grep prometheus|awk '{print $1}'|xargs kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io
 kubectl delete ns monitoring

3>通過nodeport或者ingress暴露grafana和prometheus

kubectl -n monitoring patch svc prometheus-operator-grafana -p '{"spec":{"type":"NodePort"}}'
kubectl -n monitoring patch svc prometheus-operator-prometheus -p '{"spec":{"type":"NodePort"}}'
獲取grafana admin使用者密碼
kubectl  -n monitoring get secrets |grep Opaque|grep grafana|awk '{print $1}'|xargs kubectl -n monitoring get secrets -o yaml|grep admin-password|grep -v f:|awk -F "admin-password: " '{print $2}'|base64 -d

二、告警

1、自定義告警規則
2、配置smtp
3、ldap管理grafana使用者