nvidia-docker2 在 Kubernetes 上實踐

nvida-docker2 可以幫助我們將舊的加速計算應用程式容器化，將特定的 GPU 資源分配給容器，並可以輕鬆地跨不同的環境共享應用程式、協同工作和測試應用程式。今天帶來的分享是有關 nvidia-docker2 在大規模 Kubernetes 叢集上的實踐記錄。本文首發於 OpsDev.cn，轉載已獲取作者授權。

# If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers

docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f

sudo yum remove nvidia-docker

# Add the package repositories

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \

  sudo tee /etc/yum.repos.d/nvidia-docker.repo

# Install nvidia-docker2 and reload the Docker daemon configuration

sudo yum install -y nvidia-docker2

sudo pkill -SIGHUP dockerd

# Test nvidia-smi with the latest official CUDA image

docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi

{

    "default-runtime":"nvidia",

    "runtimes": {

        "nvidia": {

          "path": "/usr/bin/nvidia-container-runtime",

          "runtimeArgs": []

        }

    },

}

systemctl restart docker

Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  name: nvidia-device-plugin-daemonset

  namespace: kube-system

spec:

  template:

    metadata:

      # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler

      # reserves resources for critical add-on pods so that they can be rescheduled after

      # a failure.  This annotation works in tandem with the toleration below.

      annotations:

        scheduler.alpha.kubernetes.io/critical-pod: ""

      labels:

        name: nvidia-device-plugin-ds

    spec:

      tolerations:

      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.

      # This, along with the annotation above marks this pod as a critical add-on.

      - key: CriticalAddonsOnly

        operator: Exists

      containers:

      - image: nvidia/k8s-device-plugin:1.9

        name: nvidia-device-plugin-ctr

        securityContext:

          allowPrivilegeEscalation: false

          capabilities:

            drop: ["ALL"]

        volumeMounts:

          - name: device-plugin

            mountPath: /var/lib/kubelet/device-plugins

      volumes:

        - name: device-plugin

          hostPath:

            path: /var/lib/kubelet/device-plugins

kubectl create -f nvidia-docker-plugin.yml

apiVersion: v1

kind: Pod

metadata:

  name: cuda-vector-add

spec:

  restartPolicy: OnFailure

  containers:

    - name: cuda-vector-add

      image: "k8s.gcr.io/cuda-vector-add:v0.1"

      resources:

        limits:

          nvidia.com/gpu: 1

  nodeSelector:

    accelerator: nvidia-tesla-k80 # or nvidia-tesla-k80 etc.

kubectl create -f nvidia-docker2-gpu-pod.yml

由360雲平臺團隊打造的技術分享公眾號，內容涉及雲端計算、資料庫、大資料、監控、泛前端、自動化測試等眾多技術領域，通過夯實的技術積累和豐富的一線實戰經驗，為你帶來最有料的技術分享