Skip to content

fix: start at boot and wait for network online target in cse_cmd.sh#8911

Open
awesomenix wants to merge 1 commit into
mainfrom
nishp/anc/dontwaitnetwork
Open

fix: start at boot and wait for network online target in cse_cmd.sh#8911
awesomenix wants to merge 1 commit into
mainfrom
nishp/anc/dontwaitnetwork

Conversation

@awesomenix

Copy link
Copy Markdown
Contributor

dont wait for network-online.target start in parallel, this saves about 4s

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to reduce node boot/provisioning latency by removing the aks-node-controller.service dependency on network-online.target (so the unit can be started earlier/in parallel) and instead waiting for network readiness within the provisioning command script (cse_cmd.sh) and (selectively) in the ANC wrapper when hotfix download paths are used.

Changes:

  • Make aks-node-controller.service a static unit (no [Install] section) and remove After=/Wants= for network-online.target.
  • Add network-online.target wait logic to cse_cmd.sh (and the corresponding ANC template), plus conditional waits in aks-node-controller-wrapper.sh before hotfix network operations.
  • Update VHD build and validation scripts to keep the ANC unit disabled/static and validate the new state.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
vhdbuilder/packer/test/linux-vhd-content-test.sh Updates VHD validation to require ANC unit to be static.
vhdbuilder/packer/pre-install-dependencies.sh Disables ANC service during VHD build to keep it from running before provisioning artifacts exist.
parts/linux/cloud-init/artifacts/cse_cmd.sh Adds an explicit wait for network-online.target before running provisioning.
parts/linux/cloud-init/artifacts/aks-node-controller.service Removes network-online.target ordering and makes the unit static.
parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh Adds conditional network-online waits for hotfix-related operations.
aks-node-controller/parser/templates/cse_cmd.sh.gtpl Keeps the generated cse_cmd.sh template in sync with the new network-online wait behavior.

Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh
Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl
Comment thread parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh
Comment thread parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh Outdated
RemainAfterExit=yes

[Install]
WantedBy=basic.target

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you probably were aware of this too

🔴 High Risk — 🏗️ Architecture / Backward Compatibility
parts/linux/cloud-init/artifacts/aks-node-controller.service removes [Install] and WantedBy, and vhdbuilder/packer/pre-install-dependencies.sh now disables the unit. That drops the long-standing boot fallback path (service auto-start) and makes provisioning depend entirely on boothook start timing.
This aligns with the failing PR E2E signal: multiple scenarios are stuck in provision-wait loop ([ -f /opt/azure/containers/provision.complete ]) and never reach expected CSE output (ADO build 172002687, Run AgentBaker E2E).
Mitigation: restore a fallback start path (or equivalent robust trigger) so ANC still starts when boothook path misses/races.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that we drop in the cse-cmd.sh or the aks node config only as part of boothook.

So we want to wait for it to be written before we can startup aks node controller otherwise we will have an issue where the service starts up before that and misses the file completely.

In future when we exclusively move to aks node controller we can add an explicity wait for aks node config before service starts that way everything is synchronized.

Copilot AI review requested due to automatic review settings July 14, 2026 01:19
@awesomenix awesomenix force-pushed the nishp/anc/dontwaitnetwork branch from ff8ad6a to 0a50baa Compare July 14, 2026 01:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh
Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl
Comment thread parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh
@aks-node-assistant

Copy link
Copy Markdown
Contributor

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172097539
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E, exit code 1.

Detective summary: E2E failed with the recurring broad gotestsum multi-leaf pattern: many ACL/ARM64/FIPS/RCV1P leaves failed, and the task/test-results output is not enough to isolate a single root assertion for the selected failed pair.

Likely cause / signature: $aggSig; this pushed the signature above the recurring threshold, so repair item #38796920 was created.

Confidence: Medium for grouping; PR author should inspect only targeted subgroups if scenario artifacts show a consistent PR-related assertion.

Recommended owner/action: Node Lifecycle E2E owner should inspect scenario artifacts/test-log JSON and split to narrower signatures when concrete assertions recur.

Strongest alternative: PR-specific CSE/RCV1P regression; less likely as the top-level gate signature because this run matches the broad recurring aggregate, but targeted artifact review is warranted.

Evidence: ADO timeline marks E2E task failed; test run 528965996 contains many failed IDs; gotestsum summary starts with broad ACL failures and no single dominant assertion in the available output.

Wiki signature: $aggSig

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants