fix(ubuntu): don't treat benign ESM apt-hook stderr as apt-get failure (CSE exit 99)

[xuexu6666]

Problem

Node bootstrap can fail with CSE exit 99 (ERR_APT_UPDATE_TIMEOUT / VMSS AptUpdateTimeoutVMExtensionError) even though apt-get update actually succeeded, so the node never joins the cluster.

_apt_get_update() / apt_get_dist_upgrade() in parts/linux/cloud-init/artifacts/ubuntu/cse_helpers_ubuntu.sh decide success/failure by grepping the merged stdout+stderr (2>&1) for ^W:, ^E:, or the over-broad ^Error.*:

! (apt-get ${apt_opts} update 2>&1 | tee $out | grep -E "^([WE]:.*)|^([Ee][Rr][Rr][Oo][Rr].*)$") && ...

The Ubuntu Pro / ESM apt hook (apt-news / esm-cache) logs this to stderr when it can't reach its snapd/dbus socket during early boot:

Error connecting: Error sending credentials: Error sending message: Broken pipe

That line starts with Error, so the grep flags the whole run as failed, retries 10×, return 1 → caller apt_get_update || exit $ERR_APT_UPDATE_TIMEOUT → exit 99. apt itself succeeded (Hit:1 …/ubuntu/24.04/prod noble InRelease, exit 0). It's intermittent because it depends on the early-boot dbus timing race.

Observed live on a real GB300 node (k8s 1.36.1, 2404gen2arm64containerd/202606.19.0): aks-node-controller exitCode=99, apt-get …/microsoft-prod.list update shows Hit:1 … noble InRelease with the benign broken-pipe line as the only "error", and KubeletStartTime empty.

Fix

Strip only the known-benign ESM/D-Bus line before the existing error grep, at both apt call sites. Intentionally minimal — the existing error regex is unchanged, so real apt failures (E: / W: / Err: prefixed) are still detected. Avoids regressing error detection on VHDs that live in production for 6 months.

• New file-scope APT_BENIGN_STDERR_REGEX (documented; not readonly so re-sourcing stays safe).
• | grep -vE "${APT_BENIGN_STDERR_REGEX}" inserted ahead of the error grep in _apt_get_update and apt_get_dist_upgrade.

Tests

Adds a _apt_get_update error detection ShellSpec block:

• benign ESM line only → succeeds (regression test for this bug)
• clean update → succeeds
• real E:/Err: output → still fails after retries

shellspec spec/parts/linux/cloud-init/artifacts/cse_helpers_ubuntu_spec.sh → 12 examples, 0 failures. TDD: the benign-line test failed before the fix, passes after.

Notes

• make generate produces no diff (scripts are go:embed'd; no committed testdata copy of this script).

[Copilot]

Pull request overview

This PR fixes an intermittent Ubuntu node bootstrap failure where apt-get update succeeds but CSE exits with 99 due to benign Ubuntu Pro/ESM apt-hook stderr lines being misclassified as fatal errors by the existing stdout+stderr grep-based detection.

Changes:

• Introduces a narrowly scoped APT_BENIGN_STDERR_REGEX and filters matching lines out before the existing apt error grep in _apt_get_update() and apt_get_dist_upgrade().
• Adds ShellSpec coverage for _apt_get_update to ensure benign ESM hook output no longer triggers retries/failure while real E:/Err: errors still fail.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
parts/linux/cloud-init/artifacts/ubuntu/cse_helpers_ubuntu.sh	Filters known-benign ESM/D-Bus “Broken pipe” stderr lines before existing apt error detection in update and dist-upgrade wrappers.
spec/parts/linux/cloud-init/artifacts/cse_helpers_ubuntu_spec.sh	Adds regression tests for _apt_get_update to validate benign stderr no longer causes exit-99 false positives.

[cameronmeissner]

wouldn't this be dangerous in the sense that we'd be silently ignoring potentially legitimate issues with being able to reach PMC/Canonical repos remotely?

[xuexu6666]

The line we strip is a local D-Bus/IPC error, not a remote one.

Error sending credentials … Broken pipe is the ubuntu-pro-client hook (20apt-esm-hook.conf) doing systemctl start apt-news.service esm-cache.service and failing the local systemd handshake in early boot — nothing to do with HTTP/DNS to PMC. A real PMC/Canonical outage shows up as apt's own Err: / E: Failed to fetch / Could not connect, which don't match the filter and still fail. Added a test (still FAILS when PMC/Canonical is genuinely unreachable) to lock that in.

Verified on live GB300 hardware (same vanilla image 202606.19.0): on non-baked 1.36.1 the CSE hit exit 99 while apt-get update had actually succeeded (Hit:1 … noble InRelease, only the broken-pipe line present); on baked 1.35.5 the nodes joined fine. Also confirmed the hook fires unattached (is_attached=False, ubuntu-pro-client installed).

[djsly]

I thought I removed the ESM service in a previous PR

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[cameronmeissner]

after offline discussion we don't want to move forward with this without further investigation into what's actually causing apt to produce these errors to stderr