fix: start at boot and wait for network online target in cse_cmd.sh#8911
fix: start at boot and wait for network online target in cse_cmd.sh#8911awesomenix wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to reduce node boot/provisioning latency by removing the aks-node-controller.service dependency on network-online.target (so the unit can be started earlier/in parallel) and instead waiting for network readiness within the provisioning command script (cse_cmd.sh) and (selectively) in the ANC wrapper when hotfix download paths are used.
Changes:
- Make
aks-node-controller.servicea static unit (no[Install]section) and removeAfter=/Wants=fornetwork-online.target. - Add
network-online.targetwait logic tocse_cmd.sh(and the corresponding ANC template), plus conditional waits inaks-node-controller-wrapper.shbefore hotfix network operations. - Update VHD build and validation scripts to keep the ANC unit disabled/static and validate the new state.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
vhdbuilder/packer/test/linux-vhd-content-test.sh |
Updates VHD validation to require ANC unit to be static. |
vhdbuilder/packer/pre-install-dependencies.sh |
Disables ANC service during VHD build to keep it from running before provisioning artifacts exist. |
parts/linux/cloud-init/artifacts/cse_cmd.sh |
Adds an explicit wait for network-online.target before running provisioning. |
parts/linux/cloud-init/artifacts/aks-node-controller.service |
Removes network-online.target ordering and makes the unit static. |
parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh |
Adds conditional network-online waits for hotfix-related operations. |
aks-node-controller/parser/templates/cse_cmd.sh.gtpl |
Keeps the generated cse_cmd.sh template in sync with the new network-online wait behavior. |
| RemainAfterExit=yes | ||
|
|
||
| [Install] | ||
| WantedBy=basic.target |
There was a problem hiding this comment.
you probably were aware of this too
🔴 High Risk — 🏗️ Architecture / Backward Compatibility
parts/linux/cloud-init/artifacts/aks-node-controller.service removes [Install] and WantedBy, and vhdbuilder/packer/pre-install-dependencies.sh now disables the unit. That drops the long-standing boot fallback path (service auto-start) and makes provisioning depend entirely on boothook start timing.
This aligns with the failing PR E2E signal: multiple scenarios are stuck in provision-wait loop ([ -f /opt/azure/containers/provision.complete ]) and never reach expected CSE output (ADO build 172002687, Run AgentBaker E2E).
Mitigation: restore a fallback start path (or equivalent robust trigger) so ANC still starts when boothook path misses/races.
There was a problem hiding this comment.
The problem is that we drop in the cse-cmd.sh or the aks node config only as part of boothook.
So we want to wait for it to be written before we can startup aks node controller otherwise we will have an issue where the service starts up before that and misses the file completely.
In future when we exclusively move to aks node controller we can add an explicity wait for aks node config before service starts that way everything is synchronized.
ff8ad6a to
0a50baa
Compare
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172097539 Detective summary: E2E failed with the recurring broad gotestsum multi-leaf pattern: many ACL/ARM64/FIPS/RCV1P leaves failed, and the task/test-results output is not enough to isolate a single root assertion for the selected failed pair. Likely cause / signature: $aggSig; this pushed the signature above the recurring threshold, so repair item #38796920 was created. Confidence: Medium for grouping; PR author should inspect only targeted subgroups if scenario artifacts show a consistent PR-related assertion. Recommended owner/action: Node Lifecycle E2E owner should inspect scenario artifacts/test-log JSON and split to narrower signatures when concrete assertions recur. Strongest alternative: PR-specific CSE/RCV1P regression; less likely as the top-level gate signature because this run matches the broad recurring aggregate, but targeted artifact review is warranted. Evidence: ADO timeline marks E2E task failed; test run 528965996 contains many failed IDs; gotestsum summary starts with broad ACL failures and no single dominant assertion in the available output. Wiki signature: $aggSig |
dont wait for network-online.target start in parallel, this saves about 4s