3

我在 COS 上使用带有 T4 GPU 的 Compute Engine VM 已经有一段时间了,它一直运行良好,直到最近它cos-extensions install gpu不像以前那样运行。

I0830 07:32:58.419130     987 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0830 07:32:58.427417     987 install.go:74] Running on COS build id 16108.470.16
I0830 07:32:58.427566     987 installer.go:187] Getting the default GPU driver version
I0830 07:32:58.427911     987 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-tools/16108.470.16/gpu_default_version
I0830 07:32:58.548403     987 utils.go:120] Successfully downloaded gpu_default_version from https://storage.googleapis.com/cos-tools/16108.470.16/gpu_default_version
I0830 07:32:58.548594     987 install.go:85] Installing GPU driver version 450.119.04
I0830 07:32:58.549646     987 cache.go:72] map[BUILD_ID:16108.470.11 DRIVER_VERSION:450.119.04]
I0830 07:32:58.549674     987 install.go:120] Did not find cached version, installing the drivers...
I0830 07:32:58.549681     987 installer.go:82] Configuring driver installation directories
I0830 07:32:58.563327     987 installer.go:196] Updating container's ld cache
I0830 07:32:58.793692     987 signature.go:30] Downloading driver signature for version 450.119.04
I0830 07:32:58.793721     987 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/16108.470.16/extensions/gpu/450.119.04.signature.tar.gz
E0830 07:32:58.828902     987 artifacts.go:106] Failed to download extensions/gpu/450.119.04.signature.tar.gz from public GCS: failed to download 450.119.04.signature.tar.gz, status: 404 Not Found
E0830 07:32:58.829401     987 install.go:175] failed to download driver signature: failed to download driver signature for version 450.119.04: failed to download extensions/gpu/450.119.04.signature.tar.gz

安装程序似乎找不到驱动程序签名。我已经对此进行了调查并遵循了解决方法

/usr/bin/docker run --rm \
    --privileged \
    --net=host \
    --pid=host \
    --volume /dev:/dev \
    --volume /:/root \
    --volume /var/lib/toolbox/nvidia:/usr/local/nvidia \
    --env NVIDIA_DRIVER_VERSION=450.119.04 \
    gcr.io/cos-cloud/cos-gpu-installer:latest

但得到了这个

+ COS_KERNEL_INFO_FILENAME=kernel_info
+ COS_KERNEL_SRC_HEADER=kernel-headers.tgz
+ TOOLCHAIN_URL_FILENAME=toolchain_url
+ TOOLCHAIN_ENV_FILENAME=toolchain_env
+ TOOLCHAIN_PKG_DIR=/build/cos-tools
+ CHROMIUMOS_SDK_GCS=https://storage.googleapis.com/chromiumos-sdk
+ ROOT_OS_RELEASE=/root/etc/os-release
+ KERNEL_SRC_HEADER=/build/usr/src/linux
+ NVIDIA_DRIVER_VERSION=450.119.04
+ NVIDIA_DRIVER_MD5SUM=
+ NVIDIA_INSTALL_DIR_HOST=/var/lib/nvidia
+ NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia
+ ROOT_MOUNT_DIR=/root
+ CACHE_FILE=/usr/local/nvidia/.cache
+ LOCK_FILE=/root/tmp/cos_gpu_installer_lock
+ LOCK_FILE_FD=20
+ set +x
[INFO    2021-08-30 07:36:38 UTC] PRELOAD: false
[INFO    2021-08-30 07:36:38 UTC] Running on COS build id 16108.470.16
[INFO    2021-08-30 07:36:38 UTC] Data dependencies (e.g. kernel source) will be fetched from https://storage.googleapis.com/cos-tools/16108.470.16
[INFO    2021-08-30 07:36:38 UTC] Checking if this is the only cos-gpu-installer that is running.
[INFO    2021-08-30 07:36:38 UTC] Checking if third party kernel modules can be installed
/tmp/esp /
/
[INFO    2021-08-30 07:36:38 UTC] Checking cached version
/entrypoint.sh: line 172: CACHE_BUILD_ID: unbound variable

似乎 COS 和 COS GPU 驱动程序(也许?)发生了一些变化,但只是想知道除了等待 GCP 解决问题之外是否有解决此问题的方法。

4

1 回答 1

1

这与Jan Vansteenlandt相关的案例相同。

这发生在某些版本的 COS 中;

例如现在可用的最新稳定 COS 版本 - 89-16108:

vm-16108 ~ # cos-extensions list Available extensions for COS version
89-16108.470.16:

[gpu]

没有以与您的情况相同的方式列出[gpu]和运行的驱动程序。cos-extensions install gpu在尝试运行您提到的 docker 容器时,也产生了相同的结果。

这是一个已知问题,已在IssueTracker上提出。您可以关闭链接并单击+1按钮,也可以评论并在线程中发布您自己的发现。

线程中还有一个解决方法,所以你可以试一试。

如果您可以使用一些旧版本的 COS(例如 85-13310) - 列出了驱动程序:

vm-13310 ~ # cos-extensions list
Available extensions for COS version 85-13310.1308.10:

[gpu]
450.119.04 [default]

当您运行cos-extensions install gpu它时,它将成功安装 NVIDIA 驱动程序:


vm-13310 ~ # cos-extensions install gpu
I0831 14:25:11.405591    1168 main.go:21] Checking if this is the only cos_gpu_installer that is running.
I0831 14:25:11.407510    1168 install.go:74] Running on COS build id 13310.1308.10
I0831 14:25:11.407519    1168 installer.go:187] Getting the default GPU driver version
I0831 14:25:11.407581    1168 utils.go:72] Downloading gpu_default_version from https://storage.googleapis.com/cos-tools/13310.1308.10/gpu_default_version
I0831 14:25:11.448046    1168 utils.go:120] Successfully downloaded gpu_default_version from https://storage.googleapis.com/cos-tools/13310.1308.10/gpu_default_version
I0831 14:25:11.448539    1168 install.go:85] Installing GPU driver version 450.119.04
I0831 14:25:11.448751    1168 cache.go:69] error: failed to read file /root/var/lib/nvidia/.cache: open /root/var/lib/nvidia/.cache: no such file or directory
I0831 14:25:11.448942    1168 install.go:120] Did not find cached version, installing the drivers...
I0831 14:25:11.449084    1168 installer.go:82] Configuring driver installation directories
I0831 14:25:11.469718    1168 installer.go:196] Updating container's ld cache
I0831 14:25:11.480682    1168 signature.go:30] Downloading driver signature for version 450.119.04
I0831 14:25:11.481007    1168 utils.go:72] Downloading 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/13310.1308.10/extensions/gpu/450.119.04.signature.tar.gz
I0831 14:25:11.506186    1168 utils.go:120] Successfully downloaded 450.119.04.signature.tar.gz from https://storage.googleapis.com/cos-tools/13310.1308.10/extensions/gpu/450.119.04.signature.tar.gz
I0831 14:25:11.506541    1168 signature.go:37] Decompressing signature /build/sign-gpu-driver/450.119.04.signature.tar.gz
I0831 14:25:11.510104    1168 installer.go:68] Downloading GPU driver installer version 450.119.04
I0831 14:25:11.511637    1168 utils.go:72] Downloading GPU driver installer from https://storage.googleapis.com/nvidia-drivers-us-public/nvidia-cos-project/85/tesla/450_00/450.119.04/NVIDIA-Linux-x86_64-450.119.04_85-13310-1308-10.cos
I0831 14:25:12.885856    1168 utils.go:120] Successfully downloaded GPU driver installer from https://storage.googleapis.com/nvidia-drivers-us-public/nvidia-cos-project/85/tesla/450_00/450.119.04/NVIDIA-Linux-x86_64-450.119.04_85-13310-1308-10.cos

-----  removed some lines for better readibility  -----

I0831 14:28:49.433597    1168 cache.go:58] Updated cached version as
I0831 14:28:49.498379    1168 cache.go:60] BUILD_ID=13310.1308.10
I0831 14:28:49.498560    1168 cache.go:60] DRIVER_VERSION=450.119.04
I0831 14:28:49.498694    1168 installer.go:32] Verifying GPU driver installation
I0831 14:28:50.309502    1168 utils.go:334] Tue Aug 31 14:28:50 2021       
I0831 14:28:50.309879    1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.311093    1168 utils.go:334] | NVIDIA-SMI 450.119.04   Driver Version: 450.119.04   CUDA Version: 11.0     |
I0831 14:28:50.311300    1168 utils.go:334] |-------------------------------+----------------------+----------------------+
I0831 14:28:50.311497    1168 utils.go:334] | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
I0831 14:28:50.311640    1168 utils.go:334] | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
I0831 14:28:50.311784    1168 utils.go:334] |                               |                      |               MIG M. |
I0831 14:28:50.311949    1168 utils.go:334] |===============================+======================+======================|
I0831 14:28:50.322257    1168 utils.go:334] |   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
I0831 14:28:50.322566    1168 utils.go:334] | N/A   76C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
I0831 14:28:50.322708    1168 utils.go:334] |                               |                      |                  N/A |
I0831 14:28:50.322878    1168 utils.go:334] +-------------------------------+----------------------+----------------------+
I0831 14:28:50.323119    1168 utils.go:334]                                                                                
I0831 14:28:50.323293    1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.323431    1168 utils.go:334] | Processes:                                                                  |
I0831 14:28:50.323597    1168 utils.go:334] |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
I0831 14:28:50.323715    1168 utils.go:334] |        ID   ID                                                   Usage      |
I0831 14:28:50.323863    1168 utils.go:334] |=============================================================================|
I0831 14:28:50.324222    1168 utils.go:334] |  No running processes found                                                 |
I0831 14:28:50.324439    1168 utils.go:334] +-----------------------------------------------------------------------------+
I0831 14:28:50.465730    1168 modules.go:48] Updating host's ld cache
I0831 14:28:52.305122    1168 install.go:167] Finished installing the drivers.

于 2021-08-31T14:34:24.540 回答