Skip to content

fix: make anc even faster by 7s#8902

Open
awesomenix wants to merge 1 commit into
mainfrom
nishp/rapid/anc
Open

fix: make anc even faster by 7s#8902
awesomenix wants to merge 1 commit into
mainfrom
nishp/rapid/anc

Conversation

@awesomenix

@awesomenix awesomenix commented Jul 11, 2026

Copy link
Copy Markdown
Contributor
  • warm up aks node controller saves about 3s.
  • warm up coredns and stls bootstrap binaries saving about 13s.
  • disable pub key auth only when requested, so regular path remains fast 4s.
  • disable ig service because we have already imported the gadgets we dont need to do it again at startup, its unnecessarily competing for resources.
  • add cse_preload.sh to prefetch
 ig service only imports OCI artifacts.

 ig image import :

1. Opens the gadget tar bundle.
2. Copies its blobs using ORAS.
3. Stores them under  /var/lib/ig/oci-store .
4. Updates the OCI index and tracking file.

It does not run gadgets, load eBPF programs, call  modprobe , or load kernel modules. eBPF programs are loaded later by commands such as  ig run .

Disabling the boot service is safe provided  ig image list  works directly from the newly booted VHD. It also avoids  ExecStop  removing the baked images during shutdown.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to reduce Linux node provisioning time by warming the aks-node-controller binary early and by removing an unnecessary systemd ordering dependency on SSH, so ANC can start sooner.

Changes:

  • Preload aks-node-controller by invoking its version command during early boot cache warming.
  • Validate SSHD candidate config with sshd -T instead of sshd -t in configureSSHPubkeyAuth.
  • Remove ssh.service/sshd.service from aks-node-controller.service After= ordering.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
parts/linux/cloud-init/artifacts/cse_preload.sh Adds ANC binary invocation to warm the page cache early.
parts/linux/cloud-init/artifacts/cse_config.sh Changes how the generated sshd_config candidate is validated.
parts/linux/cloud-init/artifacts/aks-node-controller.service Removes SSH-related unit ordering so ANC can start earlier.

Comment thread parts/linux/cloud-init/artifacts/cse_config.sh Outdated
Copilot AI review requested due to automatic review settings July 11, 2026 10:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

Comment thread parts/linux/cloud-init/artifacts/cse_preload.sh Outdated
Comment thread parts/linux/cloud-init/artifacts/cse_preload.sh Outdated
Comment thread vhdbuilder/packer/install-ig.sh
Comment thread vhdbuilder/packer/install-ig.sh
Comment thread parts/linux/cloud-init/artifacts/cse_main.sh Outdated
Copilot AI review requested due to automatic review settings July 11, 2026 10:23
Copilot AI review requested due to automatic review settings July 12, 2026 00:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comment thread e2e/validators.go Outdated
Copilot AI review requested due to automatic review settings July 12, 2026 00:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Comment thread parts/linux/cloud-init/artifacts/cse_cmd.sh Outdated
Comment thread aks-node-controller/parser/templates/cse_cmd.sh.gtpl Outdated
Comment thread parts/linux/cloud-init/artifacts/aks-node-controller.service
Comment thread e2e/validators.go Outdated
Copilot AI review requested due to automatic review settings July 12, 2026 14:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

vhdbuilder/packer/install-ig.sh:251

  • Now that the IG import systemd unit is being disabled, ignoring an ig_import_gadgets failure and still writing the /etc/ig.d/skip_vhd_ig sentinel can leave the VHD in a broken state (no gadgets imported, and no boot-time import fallback because the unit is disabled). This can also make validations think IG is present even when import failed.

Make gadget import a hard failure (or only create the sentinel when import succeeds) so the VHD build can’t silently bake a VHD without the required assets.

    # disable the systemd service (baseline files copied by packer_source.sh)
    ig_disable_service_unit || echo "[ig] Failed to disable ${IG_SERVICE_NAME}"
    ig_import_gadgets || echo "[ig] Gadget import failed during build"

    # Create skip sentinel file to indicate IG was installed from VHD
    mkdir -p /etc/ig.d
    touch "${IG_SKIP_FILE}"
    chmod 644 "${IG_SKIP_FILE}"

Comment thread e2e/validators.go Outdated
Copilot AI review requested due to automatic review settings July 12, 2026 14:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comment thread vhdbuilder/packer/install-ig.sh
Comment thread vhdbuilder/packer/install-ig.sh
Copilot AI review requested due to automatic review settings July 12, 2026 15:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comment thread parts/linux/cloud-init/artifacts/cse_preload.sh Outdated
Comment thread vhdbuilder/prefetch/internal/containerimage/testdata/prefetch.sh Outdated
Comment thread vhdbuilder/packer/install-ig.sh
Copilot AI review requested due to automatic review settings July 14, 2026 01:30
@awesomenix awesomenix force-pushed the nishp/rapid/anc branch 2 times, most recently from 36fe37b to 4d1705f Compare July 14, 2026 01:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Comment thread vhdbuilder/prefetch/internal/containerimage/testdata/prefetch.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Comment thread vhdbuilder/prefetch/internal/containerimage/testdata/prefetch.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Comment on lines +124 to +127
if ! systemctl disable --now "${IG_SERVICE_NAME}"; then
echo "[ig] Failed to disable ${IG_SERVICE_NAME}"
return 1
fi
Comment on lines 49 to 53
list, err := components.ParseList(opts.componentsPath)
if err != nil {
fmt.Printf("parsing components json: %s", err)
os.Exit(1)
}
Comment on lines +74 to +79
func resolvePostfixPath(t *testing.T) string {
repoBasePath, err := exec.Command("git", "rev-parse", "--show-toplevel").Output()
assert.NoError(t, err, "unable to determine repo root with git rev-parse")
basePath := strings.TrimSpace(string(repoBasePath))
return filepath.Join(basePath, artifactsRelPath, "cse_preload.sh")
}
@aks-node-assistant

Copy link
Copy Markdown
Contributor

AgentBaker Linux gate detective

Run: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172147607
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E, exit code 1.

Detective summary: Narrow E2E failure in Test_ACLGen2FIPSTL/scriptless_nbc: VM and pod readiness completed, then VM validation reported a critical kernel issue/call trace. The default ACL Gen2 FIPS TL variant passed, so this is not the broad aggregate no-body flake.

Likely cause / signature: $aclSig.

Confidence: Medium-high; PR author action recommended.

Recommended owner/action: Inspect scenario kernel-log call trace and compare scriptless NBC vs default ACL Gen2 FIPS TL provisioning path.

Strongest alternative: Generic E2E reporting flake; less likely because this is a narrow failed leaf with a concrete kernel-call-trace validation signature.

Evidence: ADO timeline marks E2E task failed; test results show two failed IDs; gotestsum leaf summary isolates ACL Gen2 FIPS TL scriptless NBC while default passed.

Wiki signature: $aclSig

@djsly

djsly commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

@awesomenix how can we ensure that we don't have regression on the timing improvements ? can we augment the cse perf suites to ensure we don't regress ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants