fix(codeapi-chart): de-hook package-init Job so it runs once, not per-sync#12
Merged
Merged
Conversation
…-sync The package-init Job was an Argo CD Sync hook (hook + hook-delete-policy), so every sync recreated and re-ran it. Its nodeSelector targets the scale-to-zero `sandboxspot` spot pool, so each sync woke a spot node just to re-check an already-populated PVC. Remove the Sync-hook annotations and drop ttlSecondsAfterFinished so the Job becomes a plain managed resource at sync-wave -5 (matching pvc.yaml). It runs once; an unchanged sync is an apply no-op that neither re-runs the Job nor wakes the pool. Keeping it a separate pod preserves internet access for cold builds (the sandbox-runner pod's NetworkPolicy denies egress). The completed Job now persists with an immutable spec, so forcing a rebuild or bumping runtime versions requires deleting the Job first; document this at the forceRebuild value and in the README (which also no longer mislabels the Job as a Helm pre-install hook).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
De-hook the
codeapipackage-init Job in the Helm chart so it runs once and persists instead of re-running on every ArgoCD sync.argocd.argoproj.io/hook: Sync+argocd.argoproj.io/hook-delete-policyannotations.ttlSecondsAfterFinished.argocd.argoproj.io/sync-wave: -5(same pattern as the chart's PVC).Files:
helm/codeapi/templates/package-init-job.yaml,helm/codeapi/values.yaml,helm/codeapi/README.md.Why
The package-init Job compiles the language runtimes into the shared
codeapi-packagesPVC. It targets thesandboxspotnode pool — the only x86 pool, and scale-to-zero. As an ArgoCD Sync hook, the Job was re-created and re-run on every sync of the codeapi Application, and each run scheduled a pod ontosandboxspot, waking the spot pool from zero even when the runtimes were already built. That is wasted spot-node churn on every reconcile.De-hooking makes the Job ordinary desired state: ArgoCD creates it once (wave -5, before the app), it builds against the marker (fast no-op when already initialized), Completes, and persists. Subsequent syncs apply an unchanged, completed Job → no new pod → the spot pool is not re-woken.
Behavior change to be aware of
A completed Kubernetes Job spec is immutable, and (de-hooked) the Job now persists. Because of that,
helm upgrade/ArgoCD sync alone cannot patch it. To force a rebuild or bump runtime versions you must delete the Job first:…then set
forceRebuild=true(or bump the version values) and sync. The README documents this at all threeforceRebuildsites. If skipped,forceRebuild=truesilently no-ops (stale runtimes stay in the PVC) — ArgoCD surfaces this as anOutOfSyncdiff rather than an error.No change to pod identity, serviceAccount, NetworkPolicy selection (
component: package-initlabel unchanged — the Job stays outside the sandbox-runner deny-internet policy, as required to fetch runtimes), nodeSelector/tolerations (stillsandboxspot), capabilities, or mounted secrets. Job env is version strings +FORCE_REBUILDonly — no secrets.Verification
Deployed via df-cloud's per-app Pulumi stack (ArgoCD Application
targetRevisionrepinned to this branch's HEAD) and empirically verified on both stg and prd:sandboxspotonce, pod Completed, Job persists1/1with no TTL. Live annotations =sync-wave: -5only (no hook, no hook-delete-policy).sandboxspotscaled back to 0 on the autoscaler idle timer.Security review: no security-boundary, credential, or privilege change.
Manual check before merge
mainand keeps this commit permanently reachable — the stg + prd codeapi ArgoCD Applications pin this exact SHA, so the branch must not be deleted before it is onmain.