fix(gpu): tear down mismatched VHD-prebaked driver before managed install by ganeshkumarashok · Pull Request #8916 · Azure/AgentBaker

ganeshkumarashok · 2026-07-13T20:14:54Z

Summary

Fixes nvidia-smi failing with "Failed to initialize NVML: Driver/library version mismatch" on NAP GPU nodes (all SKUs) and agentpool A10/GRID nodes.

Root cause: the shared Ubuntu VHD now pre-bakes the cuda-lts driver (#8786/#8803) and drops a DKMS marker. The existing cleanUpPrebakedGPUDriver only runs on nodes that do not install the managed driver (GPU_NODE != true OR skip). A managed GPU node whose driver differs from the prebake keeps the stale prebaked module + /usr/bin/lib64 libs, which then collide with the driver it installs:

NAP cuda nodes — same kind (cuda) as the cuda-lts prebake but a different version (e.g. 580.126.09 vs 580.159.04).
GRID / converged A10 nodes — different kind (grid).

The consume/skip-build path that would reconcile this is not in this release branch, so every managed GPU node reinstalls its own driver on top of the prebaked one → NVML mismatch.

Fix

In configGPUDrivers (Ubuntu path), before the managed install, tear down the prebaked driver unless it is an exact match for what this node installs:

same kind — marker driver_kind vs mapped NVIDIA_GPU_DRIVER_TYPE, and
same version — on-disk DKMS version (/var/lib/dkms/nvidia/<version>) vs GPU_DV.

Kind-only matching is insufficient — it misses the NAP same-kind/different-version collision (the largest impact class). Any unprovable field (empty marker kind, missing DKMS version, unset GPU_DV) falls through to teardown. No-op when there is no prebake marker. On an exact match (the intended consume fast path) the prebake is kept.

Why this is safe on `official/v20260702`

Consume/skip-build is not in this branch, so keeping a prebake buys no provisioning benefit today; the only observable effect of the guard is removing a driver that would otherwise break the node. On an exact match, behavior is unchanged.

Test plan

go build ./pkg/agent/...
make generate-testdata (no snapshot drift — CSE scripts aren't stored verbatim)
Isolated logic harness: keep on exact match; teardown on NAP version mismatch, GRID kind mismatch, missing marker kind, missing DKMS version, unset GPU_DV; no-op without a marker
shellcheck clean on the new functions
Added ShellSpec coverage in cse_config_spec.sh (cleanUpMismatchedPrebakedGPUDriver)
e2e on a NAP cuda node + agentpool GRID/A10 node booting the prebaked VHD

🤖 Generated with Claude Code

…tall Mitigates ICM 833717717: nvidia-smi "Failed to initialize NVML: Driver/library version mismatch" on NAP GPU nodes (all SKUs) and agentpool A10/GRID nodes. Root cause: the shared Ubuntu VHD now pre-bakes the cuda-lts driver (commits #8786/#8803) and drops a DKMS marker. The existing teardown (cleanUpPrebakedGPUDriver) only runs on nodes that do NOT install the managed driver (GPU_NODE!=true OR skip). A managed GPU node whose driver differs from the prebake keeps the stale prebaked module + /usr/bin/lib64 libs, which then collide with the driver it installs: - NAP cuda nodes: same kind (cuda) as the cuda-lts prebake but a different version (e.g. 580.126.09 vs 580.159.04) - GRID / converged A10 nodes: different kind entirely (grid) The consume/skip-build path that would reconcile this is not in this release, so every managed node reinstalls its own driver on top of the prebaked one -> NVML mismatch. Fix: in configGPUDrivers (Ubuntu path), before installing the managed driver, tear down the prebaked driver unless it is an EXACT match for what this node installs -- same kind (marker driver_kind vs mapped NVIDIA_GPU_DRIVER_TYPE) AND same version (on-disk DKMS version vs GPU_DV). Kind-only matching is insufficient because it misses the NAP same-kind/different-version collision. Any unprovable field (empty marker kind, missing DKMS version, unset GPU_DV) falls through to teardown. No-op when there is no prebake marker. Adds ShellSpec coverage for keep (exact match), teardown (NAP version mismatch, GRID kind mismatch, missing kind, missing version). Co-Authored-By: Claude <noreply@anthropic.com>

Copilot

Pull request overview

This PR mitigates GPU node NVML “Driver/library version mismatch” failures on Ubuntu VHDs that now ship with a prebaked NVIDIA driver, by ensuring any prebaked driver is torn down before the managed driver install unless it is an exact kind+version match.

Changes:

Add a pre-install guard in configGPUDrivers (Ubuntu) to remove a prebaked NVIDIA driver when the marker kind and on-disk DKMS version don’t exactly match the node’s intended driver.
Refactor driver-kind mapping into a reusable helper and add DKMS-based prebaked version detection.
Add ShellSpec coverage for the new teardown decision logic.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
parts/linux/cloud-init/artifacts/cse_config.sh	Adds “mismatched prebake” detection/teardown (kind+version) before managed GPU driver install; refactors kind mapping helper.
spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh	Adds ShellSpec tests validating keep-vs-teardown behavior for prebaked GPU drivers.

+        It 'tears down when the prebaked on-disk version cannot be determined (empty)'
+            marker="$(mktemp)"
+            printf 'driver_kind=cuda\n' > "$marker"
+            GPU_DKMS_MARKER_FILE="$marker"
+            NVIDIA_GPU_DRIVER_TYPE="cuda"; GPU_DV="580.159.04"; STUB_PREBAKED_VER=""
+            When call cleanUpMismatchedPrebakedGPUDriver
+            The output should include "action=teardown"
+            The output should include "STUB_TEARDOWN_CALLED"
+            rm -f "$marker"
+        End
+    End


github-actions · 2026-07-13T20:21:49Z

Changes cached containers or packages on windows VHDs

Please get a Windows SIG member to approve.

The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.

Additions are new things cached.
Deletions are things no longer cached.

diff --git a/vhd_files/2022-containerd-gen2.txt b/vhd_files/2022-containerd-gen2.txt
index e187816..db10c9e 100644
--- a/vhd_files/2022-containerd-gen2.txt
+++ b/vhd_files/2022-containerd-gen2.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2022-containerd.txt b/vhd_files/2022-containerd.txt
index 760db3c..94de353 100644
--- a/vhd_files/2022-containerd.txt
+++ b/vhd_files/2022-containerd.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index a901175..4f3c531 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index 4d7aa19..d0ea692 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 1441011..ab44d8b 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

+prebakedGPUDriverVersion() {
+    local d
+    for d in /var/lib/dkms/nvidia/*/; do
+        [ -d "${d}" ] || continue
+        basename "${d}"
+        return 0
+    done
+}


+        It 'tears down when the prebaked on-disk version cannot be determined (empty)'
+            marker="$(mktemp)"
+            printf 'driver_kind=cuda\n' > "$marker"
+            GPU_DKMS_MARKER_FILE="$marker"
+            NVIDIA_GPU_DRIVER_TYPE="cuda"; GPU_DV="580.159.04"; STUB_PREBAKED_VER=""
+            When call cleanUpMismatchedPrebakedGPUDriver
+            The output should include "action=teardown"
+            The output should include "STUB_TEARDOWN_CALLED"
+            rm -f "$marker"
+        End
+    End


+# ---- hotfix: auto-generated by hotfix-generate GH Action ----
+- path: {{GetCSEConfigScriptFilepath}}
+  permissions: "0744"
+  encoding: gzip
+  owner: root
+  content: !!binary |
+    {{GetVariableProperty "cloudInitData" "provisionConfigs"}}
+
+# ---- end hotfix ----


aks-node-assistant · 2026-07-13T22:05:27Z

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172048134
Failed jobs/stages/tasks: build2204arm64gen2containerd, build2204gen2containerd, and build2404gen2containerd / Test, Scan, and Cleanup, exit code 2.

Detective summary: All three VHD Test/Scan jobs reached CIS-CAT Pro Assessor and failed before completing assessment: the selected assessment content was limited by the associated CIS license key, Assessment 1 Exit Value: 122, surfaced as script exit 2.

Likely cause / signature: Compliance scanner infrastructure/license entitlement issue; wiki signature $sig.

Confidence: High; the scanner reported license/content limitation rather than a CIS rule pass/fail regression.

Recommended owner/action: VHD gate/compliance scanner infra owner should fix the CIS-CAT license/content bundle. No PR author action unless reruns show product-specific failures after scanner entitlement is restored.

Strongest alternative: The GPU hotfix/template changes caused VHD content regressions; less likely because VHD content tests passed before CIS-CAT failed on assessment license/content entitlement.

Evidence: ADO build summary, Test/Scan logs 431/456/506, source commit 2bacfb3a7cd8f0a84142bdf376ff5799620ac18a, associated commits 0a716e813fecf1914f91cfaa6a81f614895f2df6 and de4c9a071977f9aecbb00ae514a31b2cef8b26b6.

Wiki signature: $sig

Copilot AI review requested due to automatic review settings July 13, 2026 20:14

ganeshkumarashok requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, karenychen, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright, xuexu6666 and zachary-bailey as code owners July 13, 2026 20:14

ganeshkumarashok temporarily deployed to test July 13, 2026 20:14 — with GitHub Actions Inactive

Copilot started reviewing on behalf of ganeshkumarashok July 13, 2026 20:17 View session

chore: auto-generate hotfix template entries

0a716e8

aks-node-assistant Bot temporarily deployed to test July 13, 2026 20:18 Inactive

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 13, 2026 20:21

Copilot started reviewing on behalf of ganeshkumarashok July 13, 2026 20:22 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

ganeshkumarashok changed the title ~~fix(gpu): tear down mismatched VHD-prebaked driver before managed install (ICM 833717717)~~ fix(gpu): tear down mismatched VHD-prebaked driver before managed install Jul 13, 2026

aks-node-assistant Bot deleted a comment from djsly Jul 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(gpu): tear down mismatched VHD-prebaked driver before managed install#8916

fix(gpu): tear down mismatched VHD-prebaked driver before managed install#8916
ganeshkumarashok wants to merge 2 commits into
official/v20260702from
ganesh/prebake-mismatch-teardown-0702

ganeshkumarashok commented Jul 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

aks-node-assistant Bot commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ganeshkumarashok commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fix

Why this is safe on official/v20260702

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jul 13, 2026

Changes cached containers or packages on windows VHDs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

aks-node-assistant Bot commented Jul 13, 2026

AgentBaker Linux gate detective

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ganeshkumarashok commented Jul 13, 2026 •

edited

Loading

Why this is safe on `official/v20260702`