[SUPERSEDED by PR #107 — please close] K3 f_θ/S5 (folded into the #107 GPU beta)#106

Closed

FluffyAIcode wants to merge 46 commits into

AgentMemory/v04-pr-k3-block-c-f-theta-design-and-skeleton-8e7ffrom

AgentMemory/v04-pr-k3-block-c-f-theta-v2-trainer-fix-recall-8e7f

Commits on Jun 10, 2026

K3 f_θ trainer v2 — fix recall=0 (cosine+mag loss + NIAH data + cosine LR + 5× longer)

cursoragent
and
FluffyAIcode
committed
K3 f_θ trainer v3 — one-shot attention-output distillation (skip v2 intermediate)

cursoragent
and
FluffyAIcode
committed
K3 S6: --mix-alpha-sweep fidelity->recall diagnostic (interpolate evicted K/V between f_theta and true; map recall vs residual rel_mse)

cursoragent
and
FluffyAIcode
committed
K3 attn_distill v3 evidence: train reduction 21.47x (attn-output rel-err 1.0->~0.20), but integrated NIAH recall still 0/10 both rungs (arch gate PASS)

cursoragent
and
FluffyAIcode
committed
K3 S6 alpha-sweep on attn_distill v3: recall 0 for all alpha<1.0 (degenerate — attn_distill K/V are ~135x off-scale; k_norm/v_norm normalize scale away, so raw-space mix is confounded)

cursoragent
and
FluffyAIcode
committed
K3 S6 alpha-sweep on scale-matched relmse v3: recall knee in (0,0.5]; full-attn rel_mse 0.36 -> recall 1.0, 1.44 -> 0; eval-domain err (1.44) >> in-domain (0.58) = distribution shift

cursoragent
and
FluffyAIcode
committed
K3 f_θ trainer v4: attn_distill_hybrid loss — fix the f_θ collapse exposed by alpha-sweep

cursoragent
and
FluffyAIcode
committed
K3 S6 knee refinement (relmse v3): recall transition alpha 0.3->0.4->0.5 = full-attn rel_mse 0.71(0/10)->0.52(6/10)->0.36(10/10)

cursoragent
and
FluffyAIcode
committed
K3 trainer aid: forward NIAH_MIN_LINES/NIAH_MAX_LINES env to --niah-{min,max}-lines (was ignored)

cursoragent
and
FluffyAIcode
committed
K3 fix: import apply_rotary_pos_emb for attn_distill_hybrid too (was only attn_distill -> hybrid crashed)

cursoragent
and
FluffyAIcode
committed
K3 v4a warm-start hybrid checkpoint (rank256, init relmse v3, attn_distill_hybrid, gen1024, niah140, 10k): reduction 3.42x, attn-output ratio ~0.24

cursoragent
and
FluffyAIcode
committed
K3 v4b fresh hybrid checkpoint (rank768, 128 NIAH, gen1024, niah140, 20k): reduction 8.01x, attn-output ratio ~0.21

cursoragent
and
FluffyAIcode
committed
K3 v4a/v4b hybrid integrated NIAH evidence: both recall 0/10 both rungs (arch PASS) despite scale-matched hybrid + NIAH data + bigger/longer/warm-start

cursoragent
and
FluffyAIcode
committed
K3 fidelity probe v4a/v4b: eval full-attn rel_mse 1.42/1.52 (== relmse v3's 1.44) — full-attn K/V fidelity floor independent of loss/rank/data; blend to 0.36 -> recall 1.0 (threshold confirmed)

cursoragent
and
FluffyAIcode
committed
K3 v4a/v4b canonical NIAH + alpha-sweep artifacts: NIAH 0/10 both; sweep recall flips 0->1 between alpha 0.25 (full-attn ~0.8) and 0.5 (~0.37), identical for both

cursoragent
and
FluffyAIcode
committed

Commits on Jun 11, 2026