在之前的文章中，我們在k8s中部署了consul 生產叢集。今天我繼續在k8s中部署一個vault的生產叢集。
Vault可以在高可用性（HA）模式下執行，以透過執行多個Vault伺服器來防止中斷。Vault通常受儲存後端的IO限制的約束，而不是受計算要求的約束。某些儲存後端（例如Consul）提供了附加的協調功能，使Vault可以在HA配置中執行，而其他一些則提供了更強大的備份和還原過程。
在高可用性模式下執行時，Vault伺服器具有兩個附加狀態：備用和活動狀態。在Vault群集中，只有一個例項將處於活動狀態並處理所有請求（讀取和寫入），並且所有備用節點都將請求重定向到活動節點。
部署
我們的consul 叢集複用之前文章中部署的consul叢集。
vault配置檔案server.hcl如下：
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "POD_IP:8201"
tls_disable = "true"
}
storage "consul" {
address = "127.0.0.1:8500"
path = "vault/"
}
api_addr = "http://POD_IP:8200"
cluster_addr = "https://POD_IP:8201"接下我們建立configmap：
kubectl create configmap vault --from-file=server.hcl大家可以注意到配置檔案中的POD_IP，我們將會在容器啟動的時候，sed替換成真實的pod的IP。
我們採用StatefulSet方式部署一個兩個節點的vault叢集。透過sidecar的方式將consul client agent和vault部署到一個Pod中。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: vault
labels:
app: vault
spec:
serviceName: vault
podManagementPolicy: Parallel
replicas: 3
updateStrategy:
type: OnDelete
selector:
matchLabels:
app: vault
template:
metadata:
labels:
app: vault
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- consul
topologyKey: kubernetes.io/hostname
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- vault
topologyKey: kubernetes.io/hostname
containers:
- name: vault
command:
- "/bin/sh"
- "-ec"
args:
- |
sed -E "s/POD_IP/${POD_IP?}/g" /vault/config/server.hcl > /tmp/server.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/server.hcl
image: "vault:1.4.2"
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
add:
- IPC_LOCK
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: VAULT_ADDR
value: "http://127.0.0.1:8200"
- name: VAULT_API_ADDR
value: "http://$(POD_IP):8200"
- name: SKIP_CHOWN
value: "true"
volumeMounts:
- name: vault-config
mountPath: /vault/config/server.hcl
subPath: server.hcl
ports:
- containerPort: 8200
name: vault-port
protocol: TCP
- containerPort: 8201
name: cluster-port
protocol: TCP
readinessProbe:
# Check status; unsealed vault servers return 0
# The exit code reflects the seal status:
# 0 - unsealed
# 1 - error
# 2 - sealed
exec:
command: ["/bin/sh", "-ec", "vault status -tls-skip-verify"]
failureThreshold: 2
initialDelaySeconds: 5
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 5
lifecycle:
# Vault container doesn't receive SIGTERM from Kubernetes
# and after the grace period ends, Kube sends SIGKILL. This
# causes issues with graceful shutdowns such as deregistering itself
# from Consul (zombie services).
preStop:
exec:
command: [
"/bin/sh", "-c",
# Adding a sleep here to give the pod eviction a
# chance to propagate, so requests will not be made
# to this pod while it's terminating
"sleep 5 && kill -SIGTERM $(pidof vault)",
]
- name: consul-client
image: consul:1.7.4
env:
- name: GOSSIP_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: consul
key: gossip-encryption-key
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
args:
- "agent"
- "-advertise=$(POD_IP)"
- "-config-file=/etc/consul/config/client.json"
- "-encrypt=$(GOSSIP_ENCRYPTION_KEY)"
volumeMounts:
- name: consul-config
mountPath: /etc/consul/config
- name: consul-tls
mountPath: /etc/tls
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
volumes:
- name: vault-config
configMap:
defaultMode: 420
name: vault
- name: consul-config
configMap:
defaultMode: 420
name: consul-client
- name: consul-tls
secret:
secretName: consul
如果你的k8s叢集pod網段flat，可以和vpc當中的主機互相訪問。那麼按照以上的配置即可。否則需要設定pod的hostNetwork: true。
檢視部署情況：
kubectl get pods -l app=vault
NAME READY STATUS RESTARTS AGE
vault-0 2/2 Running 0 3m3s
vault-1 2/2 Running 0 3m3s此時補充一下consul client agent 的配置檔案：
{
"bind_addr": "0.0.0.0",
"client_addr": "0.0.0.0",
"ca_file": "/etc/tls/ca.pem",
"cert_file": "/etc/tls/consul.pem",
"key_file": "/etc/tls/consul-key.pem",
"data_dir": "/consul/data",
"datacenter": "dc1",
"domain": "cluster.consul",
"server": false,
"verify_incoming": true,
"verify_outgoing": true,
"verify_server_hostname": true,
"retry_join": [
"prod.discovery-01.xx.sg2.consul",
"prod.discovery-02.xx.sg2.consul",
"prod.discovery-03.xx.sg2.consul"
]
}prod.discovery-01.xx.sg2.consul 是我們私有域名，分別解析到之前部署的三個consul例項。
現在需要初始化和啟動每個Vault例項
首先exec到其中一個vault例項：
kubectl exec -it vault-68bcdf8dbc-7gf29 -c vault sh執行
vault operator init
Unseal Key 1: 4uyvFnGT8WxM7OXXvFJh0ich8W/4yDh27MBBj
Unseal Key 2: RzbrhGbV4hA+MlxkzwtPRP7aGXA3UaK95+5eb
Unseal Key 3: hBIv4GiVkMvrWMDnxoW7m4MAYZqgX/xvwF1KS
Unseal Key 4: +KyBJREqU+1p4qao1red/i7EX0ASmzWP2Ch79
Unseal Key 5: 8v0Q3ZHvMi7QwsJxmH3ay8h7KrJAE3ESgh+qK
Initial Root Token: s.mbHbP3WOWGEpaCT8zaoVl
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.接著使用上面生成的Unseal Key 去 Unseal 三次：
vault operator unseal <unseal_key_1>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 3b5933b9-4120-5dcb-40df-afc8ab9e6563
Version 1.4.2
HA Enabled true
vault operator unseal <unseal_key_2>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 3b5933b9-4120-5dcb-40df-afc8ab9e6563
Version 1.4.2
HA Enabled true
vault operator unseal <unseal_key_3>
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.4.2
Cluster Name vault-cluster-b9554129
Cluster ID e6cedfdd-07d2-520a-9a7c-c4e857803c7e
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>此時檢視status：
vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.4.2
Cluster Name vault-cluster-b9554129
Cluster ID e6cedfdd-07d2-520a-9a7c-c4e857803c7e
HA Enabled true
HA Cluster https://10.xx.xx.229:8201
HA Mode active接下來操作另外一個例項，用同樣的key Unseal 三次。
最後檢視狀態：
vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.4.2
Cluster Name vault-cluster-b9554129
Cluster ID e6cedfdd-07d2-520a-9a7c-c4e857803c7e
HA Enabled true
HA Cluster https://10.xx.3.229:8201
HA Mode standby
Active Node Address http://10.xx.3.229:8200最後建立svc:
apiVersion: v1
kind: Service
metadata:
name: vault
labels:
app: vault
spec:
type: ClusterIP
ports:
- port: 8200
targetPort: 8200
protocol: TCP
name: vault
selector:
app: vault總結
- 對於一些高可用的部署，我們需要加一些反親和性的設定，比如我們設定了vault之間的反親和性，以及和consul的反親和性。
- 由於我們執行的1號程式是sh，所以我們必須自己透過preStop實現優雅退出。