feat(m13): webhook HMAC, secrets/config injection, reliability, promote

DO-THIS-LOCAL.md

@@ -0,0 +1,162 @@
+# Do this local — M13 cluster / E2E follow-up
+
+Cloud agents for this PR ran **`--local-only`** (no `kubectl` / Docker / Kind). Unit, schema, manifest, and Newman-collection coverage for the new features ships in the PR; **cluster-backed verification must be done on a machine with a Tekton-capable cluster**.
+
+**Coverage already in CI/local-only:** pytest for webhook HMAC, injection helpers, reliability classifier, resource profiles, promote builder/API, JSON Schema fixture validation, and static checks that PR/promote YAML include `retries` / `max-retries` / `validate-secrets`. Newman requests for promote + injection-status are in `tests/postman/orchestrator-tests.json` (run when orchestrator is live).
+
+## Prerequisites
+
+- Kind (or equivalent) cluster with Tekton Pipelines installed
+- This repo checked out on branch `cursor/m13-production-hardening-features-b923` (or `main` after merge)
+- Python venv + tools from [docs/AGENT-REGRESSION.md](docs/AGENT-REGRESSION.md)
+
+```bash
+python3 -m venv .venv && . .venv/bin/activate
+pip install -r orchestrator/requirements.txt -r management-gui/backend/requirements-dev.txt
+pip install -e 'libs/tekton-dag-common[test]' -e 'libs/baggage-python[test]'
+# optional, for schema fixture tests:
+pip install 'jsonschema>=4.0'
+```
+
+## 1. Apply new Tekton definitions
+
+```bash
+kubectl apply -f tasks/promote-images.yaml -n tekton-pipelines
+kubectl apply -f tasks/deploy-full-stack.yaml -n tekton-pipelines
+kubectl apply -f pipeline/stack-promote-pipeline.yaml -n tekton-pipelines
+kubectl apply -f pipeline/stack-pr-pipeline.yaml -n tekton-pipelines   # retries + max-retries param
+```
+
+Rebuild/republish images so in-cluster pods pick up webhook HMAC, promote mode, and injection-status:
+
+```bash
+./scripts/publish-orchestrator-image.sh
+# optional: GUI BFF (injection-status proxy)
+./scripts/publish-management-gui-image.sh
+# then restart the orchestrator / management-gui Deployments / Helm release
+```
+
+## 2. Platform regression (required)
+
+```bash
+bash scripts/run-regression-agent.sh
+# expect: regression exit code: 0
+# with kubectl: should NOT fall back to --local-only
+```
+
+Stricter (Newman + required `stack-dag-verify`):
+
+```bash
+bash scripts/run-regression-stream.sh --cluster --require-dag-verify
+```
+
+Newman collection now includes **promote** and **injection-status** requests (`tests/postman/orchestrator-tests.json`).
+
+## 3. Feature smoke (manual / scripted)
+
+### 3a. Secrets/config injection
+
+1. Create a Secret + ConfigMap in the app namespace (e.g. `staging`):
+
+```bash
+kubectl -n staging create secret generic demo-fe-db --from-literal=PASSWORD=test
+kubectl -n staging create configmap demo-fe-config --from-literal=LOG_LEVEL=debug
+```
+
+2. Temporarily add to an app in a stack YAML (local-only edit is fine):
+
+```yaml
+secrets:
+  env-from: [demo-fe-db]
+config:
+  env-from: [demo-fe-config]
+```
+
+3. Call injection-status (port-forward orchestrator first):
+
+```bash
+curl -s "http://localhost:9091/api/apps/demo-fe/injection-status?namespace=staging" | jq .
+# expect: ok=true, secrets.demo-fe-db=present, configmaps.demo-fe-config=present
+```
+
+4. Run bootstrap (or a deploy that uses `deploy-full-stack`) and confirm the Deployment has `envFrom` for those refs. Delete the Secret and re-deploy to confirm **fail-fast** when `validate-secrets=true`.
+
+### 3b. Webhook HMAC
+
+```bash
+# Create webhook secret (key: secret)
+kubectl -n tekton-pipelines create secret generic github-webhook-secret \
+  --from-literal=secret='local-test-secret'
+
+# With WEBHOOK_SECRET / WEBHOOK_SECRET_NAME configured on the orchestrator:
+# - request WITHOUT X-Hub-Signature-256 → 401
+# - request WITH valid sha256 HMAC → 200 and PipelineRun created
+```
+
+### 3c. Promote pipeline (dry-run)
+
+`target_environment: staging` resolves URL/creds from `stacks/registries.yaml` when
+`target_registry` / `credentials_secret` are omitted. Multi-app: `"changed_app": "demo-fe,demo-api"`.
+
+```bash
+curl -s -X POST http://localhost:9091/api/run \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "mode": "promote",
+    "release_version": "0.0.1",
+    "target_environment": "staging",
+    "changed_app": "demo-fe",
+    "target_registry": ""
+  }' | jq .
+# Empty target_registry → promote-images dry-run (no crane copy)
+kubectl get pipelinerun -n tekton-pipelines -l tekton.dev/pipeline=stack-promote
+```
+
+Management GUI BFF: `GET /api/teams/<team>/apps/<app>/injection-status` (same shape as orchestrator).
+
+Approval gate:
+
+```bash
+curl -s -X POST http://localhost:9091/api/run \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "mode": "promote",
+    "release_version": "0.0.1",
+    "target_environment": "production",
+    "changed_app": "demo-fe",
+    "require_approval": true,
+    "approved_by": "you@example.com"
+  }' | jq .
+```
+
+### 3d. Reliability
+
+- Confirm PR PipelineRuns include `spec.timeouts.pipeline` and param `max-retries`.
+- Confirm `stack-pr-test` compile/containerize tasks show `retries: 2` in the applied Pipeline.
+- Optional: set `PIPELINE_TIMEOUT=45m` / `MAX_RETRIES=0` on the orchestrator Deployment and trigger a run.
+
+## 4. Optional heavy E2E
+
+```bash
+bash scripts/run-regression.sh --kind-e2e
+```
+
+Use when changing bootstrap, intercepts, or full-stack deploy behavior. Long-running.
+
+## 5. Still open after local smoke (product follow-ups)
+
+These are **not** blocked on this checklist; track in [milestones/milestone-13.md](milestones/milestone-13.md):
+
+- Intercept deploy (`deploy-intercept*`) secret/config wiring
+- Management GUI panel for injection-status
+- Helm `appConfig` / ESO templates
+- Cross-cluster deploy task
+- `pytest-cov` CI gate
+
+## Done when
+
+- [ ] `bash scripts/run-regression-agent.sh` → **`regression exit code: 0`** with cluster tiers (not `--local-only` only)
+- [ ] Newman includes promote + injection-status green against live orchestrator
+- [ ] At least one promote dry-run PipelineRun **Succeeded**
+- [ ] injection-status shows present/missing correctly for a test Secret
+- [ ] Webhook rejects bad HMAC when secret is configured

README.md

@@ -2,6 +2,19 @@
 
 Standalone Tekton pipeline system for **local development and proof-of-concept**. Stack-aware CI/CD with header-based traffic interception, multi-framework support, and an in-cluster orchestration service.
 
+## What's new (M13 foundations)
+
+Production-hardening building blocks are in the tree (see [milestones/milestone-13.md](milestones/milestone-13.md) and [DO-THIS-LOCAL.md](DO-THIS-LOCAL.md) for cluster smoke):
+
+| Feature | What you get |
+|---------|----------------|
+| **Webhook HMAC** | GitHub `X-Hub-Signature-256` verification (`WEBHOOK_SECRET` or K8s Secret); unsigned allowed only when no secret is configured; named Secret missing → fail closed |
+| **Secrets & config injection** | Stack YAML `secrets` / `config` blocks → `deploy-full-stack` `envFrom` + volume mounts, pre-deploy validation, `GET /api/apps/<app>/injection-status` (orchestrator + Management GUI BFF) |
+| **Pipeline reliability** | PipelineRun timeouts + `max-retries` audit param, compile/containerize `retries: 2`, transient-failure classifier, per-tool resource profiles in `tekton-dag-common` |
+| **Promote pipeline** | `stack-promote` + `promote-images` (crane), `stacks/registries.yaml` target resolution, `POST /api/run` `mode=promote` with optional approval gate and dockerconfig credentials |
+
+Still open on M13: intercept deploy wiring, Helm `appConfig` / ESO templates, GUI status panels, Prometheus/observability, cross-cluster deploy, script-level retries driven by `max-retries`.
+
 ## Demo Videos
 
 🎬 **GitHub Pages (all segments + players):** [jmjava.github.io/tekton-dag/](https://jmjava.github.io/tekton-dag/)  
@@ -72,19 +85,19 @@ Each row links to the **in-browser player** on Pages (`#seg-…`) and to the **c
 | [M12](milestones/milestone-12.md) | **Completed** | Architecture customization: shared Python package, Helm ConfigMap/PVC templates, parameterized pipelines (no hardcoded `localhost:5000`), `scripts/common.sh`, build image variants (Java 11/17/21, Node 18/20/22, Python 3.10–3.12, PHP 8.1–8.3), custom pipeline hook tasks (pre/post build/test), stack JSON schema, 62 orchestrator pytest tests, 14 shared-package tests. Full docs: [CUSTOMIZATION.md](docs/CUSTOMIZATION.md), [TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md](docs/TEAM-ONBOARDING-STACKS-AND-BAGGAGE.md), MAINTENANCE.md, Helm README. |
 | [M12.2](milestones/milestone-12.2.md) | **Partial** | **Part A done:** doc sync + archive. **Part B open:** regression + Management GUI [docs & demo plan](docs/TESTING-AND-REGRESSION-OVERVIEW.md) / [GUI extension](docs/MANAGEMENT-GUI-EXTENSION.md) / [video segments](docs/demos/segments-m12-2-regression-gui.md) |
 | [doc-generator](milestones/milestone-doc-generator.md) | **Completed** | Reusable Python library ([`docgen`](https://github.com/jmjava/documentation-generator)) extracting the demo pipeline (TTS, Manim, VHS, ffmpeg, validation, Pages). OCR validation, A/V sync, narration linting, auto-generated GitHub Pages. All 18 demo segments regenerated via `docgen`. |
-| [M13](milestones/milestone-13.md) | **Planned** | Production hardening: retry on transient failures, precise build image sizing, multi-cluster push, operational reliability, observability, secrets injection (ESO/Sealed Secrets), per-app config per environment. See [segment 18](https://jmjava.github.io/tekton-dag/#seg-18) for video walkthrough. |
+| [M13](milestones/milestone-13.md) | **Partial** | Production hardening foundations shipped: webhook HMAC, stack secrets/config + deploy wiring + injection-status APIs, PipelineRun timeouts / task retries / failure classifier / resource profiles, `stack-promote` + registries + approval gate. Open: intercept secret/config wiring, Helm `appConfig` / ESO, GUI panels, observability, cross-cluster deploy. Roadmap video: [segment 18](https://jmjava.github.io/tekton-dag/#seg-18). Local cluster checklist: [DO-THIS-LOCAL.md](DO-THIS-LOCAL.md). |
 
 Older milestones (M2, M3) are in [milestones/completed/](milestones/completed/).
 
-**Next up — [Milestone 13: Production Hardening](milestones/milestone-13.md):**
+**M13 remaining — [Milestone 13: Production Hardening](milestones/milestone-13.md):**
 
-1. **Retry on transient failures** — task-level retries for build/deploy (not tests), spot eviction handling, registry throttle backoff, configurable retry counts, structured retry annotations
-2. **Precise build image sizing** — per-tool resource profiles (Maven ≠ npm ≠ Kaniko), Helm-configurable, stack-level overrides, monitoring baseline
-3. **Multi-cluster push** — remote registry push, promotion pipeline, cross-cluster deploy task, environment gates (manual approval), promotion audit trail in Tekton Results
-4. **Operational reliability** — pipeline timeouts, graceful cleanup on timeout (`finally` block), health-check gates before tests, Results DB backup, Neo4j persistence
-5. **Observability** — Prometheus metrics (build duration, test pass rate, retry count, queue time), alerting rules, cost attribution labels (team/stack/app)
-6. **Secrets injection** — External Secrets Operator (ESO) integration, stack YAML `secrets` block (`env-from` + `volume-mounts`), deploy task wiring, ESO SecretStore per team, Sealed Secrets fallback, pre-deploy secret validation, Management GUI secret status panel
-7. **Per-app config per environment** — stack YAML `config` block, Helm-templated ConfigMaps from `appConfig` values, environment overlay pattern (`values-local.yaml` / `values-staging.yaml` / `values-prod.yaml`), `.env.<app>` support for local dev, config validation hook, Management GUI config view
+1. **Retry depth** — script-level / param-driven retries beyond fixed Tekton `retries: 2`; richer spot/registry backoff in task scripts
+2. **Build image sizing in Tasks** — apply `resource_profiles` / stack `build.resources` to compile & Kaniko pods; Helm `resources.compile-*` values
+3. **Multi-cluster deploy** — cross-cluster apply task, ArgoCD/Helm promotion hooks beyond registry copy
+4. **Operational reliability** — graceful cleanup on timeout, Results DB backup, Neo4j PVC/backup docs
+5. **Observability** — Prometheus metrics, alerting rules, cost attribution labels
+6. **Secrets provider** — ESO SecretStore templates, Sealed Secrets fallback docs, intercept deploy secret wiring, GUI secret status panel
+7. **Config per environment** — Helm `appConfig` ConfigMap templates, env overlay docs, `.env.<app>` for local, GUI config view
 
 **Regression (humans & Cursor agents):** run **`scripts/run-regression-agent.sh`** and iterate with fixes until green — see [AGENTS.md](AGENTS.md) and [docs/AGENT-REGRESSION.md](docs/AGENT-REGRESSION.md). Full tier list: [docs/REGRESSION.md](docs/REGRESSION.md).
 
@@ -103,6 +116,7 @@ flowchart LR
     PR[stack-pr-test]
     Bootstrap[stack-bootstrap]
     Merge[stack-merge-release]
+    Promote[stack-promote]
   end
   subgraph runtime [Runtime]
     Intercept[Intercept: Telepresence or mirrord]
@@ -113,18 +127,20 @@ flowchart LR
   Orchestrator --> PR
   Orchestrator --> Bootstrap
   Orchestrator --> Merge
+  Orchestrator --> Promote
   CLI --> PR
   CLI --> Bootstrap
   PR --> Intercept --> Validate --> Test
 ```
 
-**Three pipelines, one stack:**
+**Four pipelines, one stack:**
 
 | Pipeline | Purpose |
 |----------|---------|
-| **Bootstrap** (`stack-bootstrap`) | Deploy full stack once; prerequisite for PR runs. |
-| **PR** (`stack-pr-test`) | Build changed app with snapshot tag, deploy intercepts, validate, test, post PR comment. No version bump. |
+| **Bootstrap** (`stack-bootstrap`) | Deploy full stack once; prerequisite for PR runs. Injects stack `secrets` / `config` when declared. |
+| **PR** (`stack-pr-test`) | Build changed app with snapshot tag, deploy intercepts, validate, test, post PR comment. No version bump. Compile/containerize retry on infra flakes. |
 | **Merge** (`stack-merge-release`) | Promote RC to release, build, tag release images, push next dev cycle version commit. |
+| **Promote** (`stack-promote`) | Copy release-tagged images to a target registry/environment (`registries.yaml` or API overrides); optional approval gate. |
 
 **Intercept backends:** Telepresence (default) or mirrord, selected via pipeline param `intercept-backend`. Both E2E-verified.
 

docs/AGENT-REGRESSION.md

@@ -54,3 +54,4 @@ Do **not** stop after only:
 
 - [REGRESSION.md](REGRESSION.md) — tiers, flags, env vars  
 - [GITHUB-PAGES.md](GITHUB-PAGES.md) — if work touches Pages  
+- [DO-THIS-LOCAL.md](../DO-THIS-LOCAL.md) — M13 cluster/E2E checklist when the cloud agent only ran `--local-only`

helm/tekton-dag/templates/orchestration-deployment.yaml

@@ -46,6 +46,18 @@ spec:
               value: {{ .Values.stackFile }}
             - name: WEBHOOK_SECRET_NAME
               value: {{ .Values.triggers.webhookSecretName }}
+            - name: WEBHOOK_VERIFY_SIGNATURE
+              value: {{ .Values.orchestrationService.webhookVerifySignature | quote }}
+            - name: PIPELINE_TIMEOUT
+              value: {{ .Values.orchestrationService.pipelineTimeout | quote }}
+            - name: MAX_RETRIES
+              value: {{ .Values.orchestrationService.maxRetries | quote }}
+            - name: REGISTRIES_FILE
+              value: {{ .Values.orchestrationService.registriesFile | quote }}
+            - name: STACKS_DIR
+              value: /stacks
+            - name: TEAMS_DIR
+              value: /teams
           resources:
             {{- toYaml .Values.orchestrationService.resources | nindent 12 }}
           livenessProbe:
@@ -72,10 +84,24 @@ spec:
           configMap:
             name: tekton-dag-stacks
             optional: true
+        # Project per-team ConfigMaps (tekton-dag-teams-<name>) as /teams/<name>/team.yaml
         - name: teams
-          configMap:
-            name: tekton-dag-teams
-            optional: true
+{{- $teamFiles := .Files.Glob "raw/teams/**/team.yaml" }}
+{{- if $teamFiles }}
+          projected:
+            sources:
+{{- range $path, $_ := $teamFiles }}
+{{- $parts := splitList "/" $path }}
+{{- $teamName := index $parts (sub (len $parts) 2) }}
+              - configMap:
+                  name: tekton-dag-teams-{{ $teamName }}
+                  items:
+                    - key: team.yaml
+                      path: {{ $teamName }}/team.yaml
+{{- end }}
+{{- else }}
+          emptyDir: {}
+{{- end }}
 ---
 apiVersion: v1
 kind: Service

helm/tekton-dag/values.yaml

@@ -61,6 +61,12 @@ orchestrationService:
   image: "localhost:5000/tekton-dag-orchestrator:latest"
   replicas: 1
   port: 8080
+  # M13 reliability defaults (PipelineRun timeouts / max-retries audit param)
+  pipelineTimeout: "2h"
+  maxRetries: 2
+  # Verify GitHub webhooks when the named Secret exists (key: secret|value|webhook-secret)
+  webhookVerifySignature: true
+  registriesFile: "/stacks/registries.yaml"
   resources:
     requests:
       cpu: "100m"

libs/tekton-dag-common/pyproject.toml

@@ -10,7 +10,7 @@ requires-python = ">=3.10"
 dependencies = ["pyyaml>=6.0,<7.0"]
 
 [project.optional-dependencies]
-test = ["pytest>=7.0"]
+test = ["pytest>=7.0", "jsonschema>=4.0,<5.0"]
 
 [tool.setuptools.packages.find]
 include = ["tekton_dag_common*"]

libs/tekton-dag-common/tekton_dag_common/__init__.py

@@ -1,5 +1,14 @@
 """Shared utilities for tekton-dag orchestrator and management GUI."""
 
+from tekton_dag_common.deploy_injection import (
+    build_env_from,
+    build_volume_mounts_and_volumes,
+    injection_summary,
+    referenced_configmap_names,
+    referenced_secret_names,
+    sanitize_volume_name,
+    validate_injection_refs,
+)
 from tekton_dag_common.k8s_constants import (
     PIPELINERUN_PLURAL,
     TASKRUN_PLURAL,
@@ -13,6 +22,16 @@
     default_workspaces,
     random_suffix,
 )
+from tekton_dag_common.reliability import (
+    apply_reliability,
+    classify_failure,
+    should_retry,
+)
+from tekton_dag_common.resource_profiles import (
+    get_profile,
+    kaniko_resources,
+    resources_for_app,
+)
 from tekton_dag_common.stack_resolver_base import (
     extract_repo_map,
     get_build_apps,
@@ -27,11 +46,24 @@
     "TEKTON_API_VERSION",
     "TEKTON_GROUP",
     "TEKTON_VERSION",
+    "apply_reliability",
     "base_pipelinerun",
+    "build_env_from",
+    "build_volume_mounts_and_volumes",
+    "classify_failure",
     "default_workspaces",
     "extract_repo_map",
     "get_build_apps",
+    "get_profile",
+    "injection_summary",
+    "kaniko_resources",
     "load_stack_yaml",
     "parse_apps",
     "random_suffix",
+    "referenced_configmap_names",
+    "referenced_secret_names",
+    "resources_for_app",
+    "sanitize_volume_name",
+    "should_retry",
+    "validate_injection_refs",
 ]