Skip to content

test(dtu): runnable seam-validation profiles + repo conventions & testing discipline#69

Merged
colombod merged 12 commits into
mainfrom
docs/contributing-and-pr-template
Jul 13, 2026
Merged

test(dtu): runnable seam-validation profiles + repo conventions & testing discipline#69
colombod merged 12 commits into
mainfrom
docs/contributing-and-pr-template

Conversation

@colombod

@colombod colombod commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Establishes this repo's contribution + end-to-end test discipline: the missing "always-loaded"
convention files, an AGENTS.md rule that mandates real evidence for seam changes, and a set of
genuinely runnable Digital Twin Universe (DTU) profiles that prove the bundle's behaviour
through the Amplifier CLI — mode gating, and the server-backed read/write seams — each captured
against a live environment, not merely asserted.

Scope is docs + DTU test-infra only — no product code, no bundle-structure changes. Independent
of the read-side multi-source work (PR #68 / #67).

What's in it

Repo conventions

  • .github/PULL_REQUEST_TEMPLATE.md — a verification checklist grounded in this repo's real gates:
    module + top-level tests, ruff/pyright, scripts/validate-full.sh → full PASS, real evidence
    on any seam crossed     (not mock-only), diagram regen, and the documented mode false-positive.
  • CONTRIBUTING.md — layout, branch/commit conventions, uv dev setup, the test gates, and the
    DTU/Gitea dev dependencies (with install pointers).
  • AGENTS.md — a "Testing & what 'done' looks like" section: green unit tests are the floor, not
    proof; any seam crossing (client↔server, networking, auth, or any agent/skill/mode/tool/hook/config
    edit) requires a real DTU run + captured evidence.

DTU profiles README (new) — the profile inventory (what each really tests), the CLI-load
principle (structural and behavioural checks load the bundle via amplifier bundle add, not
pytest artefacts), the DTU + Gitea dev dependencies with install pointers, and how to stand up a
Context-Intelligence server for e2e.

Runnable seam-validation profiles — every profile uses the real amplifier-digital-twin schema
(base + provision + readiness), loads the bundle via amplifier bundle add (Gitea-mirror
redirected for local-branch testing), and was proven live:

Seam Profile Proven
signals scoring (deterministic) signals-validation 12/12 fixtures, no external deps
hook late-contributor → local JSONL bundle-smoke-test contributed event lands in events.jsonl
mode gating / activation redesigned-mode, contributes-migration, mode-activation readiness ready:True; bundle loaded from the mirror; baseline agents present + specialists gated off; /mode context-intelligence off→on→off in a real Anthropic session
write → single server write-server-validation one destination received the session (events_processed:22); the other received nothing
write → fan-out write-fanout-validation one session, two named destinations → both servers received it independently
execute queries (read side) query-validation graph_query real rows + blob_read real content, each carrying the source provenance naming the answering server

The write/query profiles configure destinations the documented way — a named map under
overrides.hook-context-intelligence.config.destinations in ~/.amplifier/settings.yaml, with each
api_key a ${VAR} from ~/.amplifier/keys.env — and stand up a real Context-Intelligence server
(Neo4j-backed) via docker compose.

Housekeeping

  • Renamed ci-* profiles → context-intelligence-* (git history preserved) to drop the ambiguous
    ci- prefix.
  • Renamed the misleading validation_cmds: field → manual_validation_steps: — amplifier-digital-twin
    never executes it (only base/provision/readiness run); the new name tells the truth (docs, not a gate).

Verification

  • scripts/validate-full.sh → validation_mode: full, overall PASS (convention files + DTU
    profiles are not bundle-structural; the sole ERROR remains the documented mode name-collision
    false positive).
  • Per-profile DTU evidence captured in the profiles README "Working-order status".
  • Reviewed by the six-lens council (unanimous CONCERN → resolved: the one gap, an unbacked
    evidence claim, was closed by pasting the real run evidence).

Honest limitations (disclosed, not papered over)

  • The exact runtime-mounted set on mode activation is not CLI-introspectable in this Amplifier
    version (the mode does not emit mode:transition_completed); the mode profiles prove the gated
    surface via declared-contributes + inactive-baseline gating + the activation round-trip, not a
    runtime mount enumeration.
  • graph_query/blob_read are not top-level tools in a plain session — the shipped read path is
    the context-intelligence:graph-analyst agent, which the query profile drives.

colombod and others added 12 commits July 13, 2026 16:31
@colombod @microsoft-amplifier
docs: add CONTRIBUTING.md and PR template grounded in repo conventions
501d11d 
Adds the two always-loaded per-repo convention files the repo was missing.
Both encode this repo's REAL gates (not boilerplate): scripts/validate-full.sh
full validation, the seam-awareness / real-E2E discipline, and the documented
mode false positive. New contributors/agents inherit the house rules AGENTS.md
already carries.

Docs-only; full bundle validation (scripts/validate-full.sh) stays PASS in
validation_mode: full (convention files are not bundle-structural).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(agents): add Testing + "what done looks like" — real DTU/eval ev…
0252d53 
…idence required

Aligns AGENTS.md with the per-repo-convention that it must carry the test commands
that gate "done" and what "done" looks like. Adds the unit/lint/type + full-validation
commands, and makes explicit that green unit tests are the FLOOR, not proof: any change
that crosses a seam — the client↔server boundary, networking, auth, or ANY agent / skill /
mode / tool / hook / config edit — requires a real DTU run plus the evaluation harness
(.amplifier/digital-twin-universe/profiles/ + the eval-design / evaluation-methodology
skills), with captured end-to-end evidence of working order. A seam crossed on a green
mock is not done until reconciled to real behaviour.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
test(dtu): expand ci-* profile names to context-intelligence-*; add m…
7f996ba 
…ode-activation profile

- Rename DTU profiles to drop the ambiguous `ci-` prefix (could read as
  "continuous integration"): ci-bundle-smoke-test → context-intelligence-bundle-smoke-test,
  ci-signals-validation → context-intelligence-signals-validation (git mv, history
  preserved; internal name:/--name/usage comments and the AGENTS.md references updated).
- Add context-intelligence-mode-activation-validation.yaml: a single-purpose DTU test
  proving explicit /context-intelligence activation mounts the FULL mode-gated surface —
  both contributes.agents specialists resolve, all THREE contributes.context files inject
  (closing the previously un-asserted context-intelligence-strategy.md), all THREE skills
  become discoverable (closing the un-asserted evaluation-methodology skill), and the mode
  tool policies take effect — with a clean off→on→off round-trip and an explicit
  /context-intelligence slash-command activation scenario. Baseline agents stay present.

All 6 profiles parse as valid YAML.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(dtu): profiles README — inventory + how to spin up a CI server f…
0042e5e 
…or e2e

Documents the DTU profile suite for this bundle: what each profile really tests,
its prerequisites, and its launch command; and — per the AGENTS.md seam/real-evidence
rule — how to stand up and use a Context-Intelligence server for read-side end-to-end
tests (external host server via example-dtu-external-server.yaml, or a Docker sidecar
inside the DTU via docker-in-incus), the AMPLIFIER_CONTEXT_INTELLIGENCE_* env wiring,
the log→query→assert flow, and the read-side (server_url, api_key) resolution chain.
Records working-order status: prereqs present (amplifier-digital-twin 0.3.0, Incus 7.2,
Docker), all profiles valid YAML with valid references.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(dtu): record live signals-profile run proof in working-order status
0483a29 
Launched context-intelligence-signals-validation.yaml (Incus, no external deps):
readiness "all checks passed", signals symbols import, all 12 fixtures scored with
0 failures; instance then destroyed. Confirms the DTU pipeline (provision → install
→ run → assert) is healthy on a standard host — real evidence, not static validity.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(dtu): correct DTU testing principle + document DTU/Gitea dev dep…
42b1a32 
…endencies

DTU structural/behavioural verification loads the bundle through the Amplifier CLI
(`amplifier bundle add` → activate `/context-intelligence` → drive a real session),
NOT by running pytest artefacts in a container (that is a unit test in a different
directory, not end-to-end). Documents the extra dev dependencies this requires —
amplifier-digital-twin CLI, Incus, Docker, Gitea (to serve the local branch so the
install resolves to our code), and a real LLM key for behavioural scenarios — with
concrete install pointers and per-profile needs, in the DTU profiles README, and adds
an end-to-end/DTU section to CONTRIBUTING.md pointing devs to it.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
test(dtu): make the 3 mode profiles actually runnable (CLI bundle-loa…
20ad1fc 
…d, real run)

The three mode profiles used a declarative profile:/install:/assertions:/scenarios:
schema that amplifier-digital-twin cannot execute ("Error: Profile must specify
base.image") and that nothing in the repo interprets — i.e. specs, not tests.

Rewrite all three into the real runnable schema (base.image + provision +
readiness): provision installs the bundle via `amplifier bundle add
...#subdirectory=behaviors/context-intelligence.yaml --app` resolved to a Gitea
mirror of this branch (via `git insteadOf`, gated on --var gitea_host), and the
deterministic structural proofs live in `readiness` (which amplifier-digital-twin
actually runs and gates on launch):
  - amplifier usable; bundle loaded from the MIRROR (branch snapshot, not main);
  - `bundle show` lists the 2 baseline agents with the 2 mode-gated specialists
    absent while the mode is off (contributes.agents gating holds);
  - installed mirror mode file declares the full gated surface (advertised:false,
    default_action:block, 3 contributes.context incl. strategy.md, 3 skills incl.
    evaluation-methodology, tool policies).

Verified for real: all three launched against the live mirror and reached
readiness ready:True. The behavioural off→on→off activation round-trip
(/mode context-intelligence → [context-intelligence]> → /mode off) was proven in a
real Anthropic PTY session and is documented as a reproducible manual exec step.

README updated to describe the runnable-schema gating and to state honestly that the
runtime-mounted-set enumeration is not CLI-introspectable in this Amplifier version
(the mode does not emit mode:transition_completed to events.jsonl) — the profiles
prove the gated surface via declared-contributes + inactive-baseline gating + the
activation round-trip, not a runtime mount-list dump.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(dtu): address council review — paste real 3-profile run evidence…
de49dff 
… + fix AGENTS.md list

Council/rob review of #69 (unanimous CONCERN, not FAIL): mechanism sound, one cheap
fix — the README's pasted proof covered only signals-validation (a rename-only file),
not the three mode profiles the PR is actually about; per this PR's own "never pre-check
a box you can't back" rule, that claim needed backing.

- README working-order: paste the real captured evidence for redesigned-mode,
  contributes-migration, and mode-activation — each launched against the live Gitea
  mirror (HEAD 50a3bd5) and reached readiness ready:True (amplifier usable; bundle
  loaded via `amplifier bundle add` from the mirror; baseline agents present with
  specialists gated off; full gated surface declared); behavioural /mode round-trip
  confirmed in a real Anthropic session. Framed honestly as manual-run-captured, not
  yet CI-enforced.
- AGENTS.md: fix the DTU example-profile list (was missing contributes-migration and
  mode-activation), reconciling the policy text with the artifacts it justifies.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
test(dtu): add server-backed seam profiles (write-single, write-fanou…
edeac76 
…t, query) — proven live

Adds three runnable (base+provision+readiness) DTU profiles for the seams that need a
real Context-Intelligence server, each loading the bundle via `amplifier bundle add`
from the Gitea mirror and driving a real Amplifier session:

- write-server-validation: a real session's events reach ONE destination server
  (proven: /status events_processed:22, Cypher count 29 tagged by workspace).
- write-fanout-validation: one session, two destinations -> BOTH servers received the
  events (A and B each: 22 processed / 29 nodes; B held only the fanout session).
  Observes the existing hook fan-out; the hook code is never modified.
- query-validation: after logging, the graph-analyst agent's graph_query returns real
  rows and blob_read resolves a real ci-blob:// URI (44 KB), each carrying the `source`
  provenance block naming the answering server.

Proven live against real CI server stacks (docker compose from
microsoft/amplifier-context-intelligence, Neo4j-backed); instances + stacks then
destroyed. README inventory + working-order status updated with the evidence and two
honest findings: (1) the project settings.yaml destinations override is a no-op on the
current amplifier-foundation build (tools/config overrides reserved for v1.1), so the
fanout profile injects destinations into the loaded hook config directly; (2)
graph_query/blob_read are not top-level tools — the shipped read path is the
context-intelligence:graph-analyst agent, which the query profile drives.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
refactor(dtu): rename validation_cmds -> manual_validation_steps (hon…
7c5b30d 
…est field name)

amplifier-digital-twin never executes `validation_cmds` — it is an unknown field the
launcher silently ignores (only base/provision/readiness run). The name implied
automated execution the runner does not provide. Renamed to manual_validation_steps
across all 8 profiles that carried it (+ the 2 stale references in the profiles README),
so the field reads as what it is: documented steps a human runs via
`amplifier-digital-twin exec <id> -- …`. Command bodies preserved verbatim; readiness
gates untouched; all YAML still parses. (Council-flagged; machete pass.)

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
docs(dtu): retract the incorrect config-override finding (was my misc…
0e06cce 
…onfiguration)

The earlier "destinations override is a no-op / reserved for v1.1" note was wrong: the
repro used a bare list of destinations instead of the bundle README's named-map format
(overrides.hook-context-intelligence.config.destinations keyed by name, api_key via
${VAR} from ~/.amplifier/keys.env), and conflated the app-cli settings.yaml override
(documented + working) with the foundation configurator's unrelated "reserved for v1.1"
comment. Corrected the README working-order note accordingly; the fan-out profile's
direct-injection is flagged as a test workaround to redo with the correct config +
re-verify, not evidence of a framework limitation. Support issue closed as invalid.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod @microsoft-amplifier
test(dtu): configure write/fanout destinations the documented way + r…
960e17b 
…e-verify

Replace the direct-hook-config-injection workaround in the write-server and write-fanout
profiles with the documented user path: `~/.amplifier/settings.yaml`
overrides.hook-context-intelligence.config.destinations as a NAMED MAP (keyed by
destination name), each api_key a ${VAR} from `~/.amplifier/keys.env` — exactly as the
bundle README specifies. patch_destinations.py removed from both.

Re-verified live against fresh isolated CI server stacks (docker compose), through the
documented config only: single-server -> server A received the session (events_processed:22),
server B received nothing; fan-out -> one session, two named destinations -> BOTH servers
independently received it (each events_processed:22, same workspace). Also fixed a readiness
gate that used a folded YAML scalar (collapsed a heredoc -> exit 2); now literal-block, 5/5
ready:True on relaunch.

Root cause of the earlier "no-op" (support issue closed as invalid): it used a project-scope
.amplifier/settings.yaml (routes through the foundation configurator overlay, not yet applied
for hook/tool config) AND a bare-list destinations shape. The user-level ~/.amplifier/settings.yaml
named-map is the implemented, working path (app-cli get_config_overrides). README evidence updated.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod colombod changed the title docs: add CONTRIBUTING.md and PR template (repo conventions) test(dtu): runnable seam-validation profiles + repo conventions & testing discipline Jul 13, 2026
@colombod colombod merged commit 0431d58 into main Jul 13, 2026
8 checks passed
colombod added a commit that referenced this pull request Jul 13, 2026
@colombod @microsoft-amplifier
docs(dtu): address council review — paste real 3-profile run evidence…
e2fbdcd 
… + fix AGENTS.md list

Council/rob review of #69 (unanimous CONCERN, not FAIL): mechanism sound, one cheap
fix — the README's pasted proof covered only signals-validation (a rename-only file),
not the three mode profiles the PR is actually about; per this PR's own "never pre-check
a box you can't back" rule, that claim needed backing.

- README working-order: paste the real captured evidence for redesigned-mode,
  contributes-migration, and mode-activation — each launched against the live Gitea
  mirror (HEAD 50a3bd5) and reached readiness ready:True (amplifier usable; bundle
  loaded via `amplifier bundle add` from the mirror; baseline agents present with
  specialists gated off; full gated surface declared); behavioural /mode round-trip
  confirmed in a real Anthropic session. Framed honestly as manual-run-captured, not
  yet CI-enforced.
- AGENTS.md: fix the DTU example-profile list (was missing contributes-migration and
  mode-activation), reconciling the policy text with the artifacts it justifies.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@colombod colombod deleted the docs/contributing-and-pr-template branch July 13, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@colombod