e2e eval harness silently ignores `*_fp32_config.json` recipes (fp32 not in KNOWN_PRECISIONS) — affects #1084, #1093

[DingmaomaoBJTU]

Summary

The e2e eval harness discovers authored recipes by a precision token parsed from the filename. In scripts/e2e_eval/utils/recipes.py:

KNOWN_PRECISIONS: tuple[str, ...] = ("fp16", "w8a16", "w8a8")

discover_recipe_variants() → split_task_precision() only strips those suffixes; any other trailing token (e.g. fp32) yields precision=None, and the file is dropped in discover_recipe_variants (if cfg_task != task or precision is None: continue).

Consequence: a recipe named *_fp32_config.json is never recognized as a precision variant. If a model dir contains only fp32-named recipes, discover_recipe_variants returns empty and the harness silently falls back to winml config auto-generation — the authored recipe is never built or evaluated.

Why this is a naming trap

The catalog convention is that the unquantized bucket is named *_fp16_config.json with quant: null:

• 75 / 75 *_fp16_config.json recipes under examples/recipes/ have quant: null; 0 carry quant.mode=fp16.
• fp16 is realized per-EP at eval time via --precision fp16 (which resolves to quant.mode=fp16), not baked into the recipe. _run_recipe_build in run_eval.py builds authored recipes recipe-driven (winml build -c <recipe>), with no --precision/--device flag.
• On CPU the unquantized recipe deliberately stays fp32 — see tests/e2e/test_config_e2e.py: "device=cpu + precision=auto must NOT trigger FP16 conversion."

So a quant: null recipe named _fp16_ is correct and intended. The same content renamed _fp32_ falls outside the harness's recognized precisions and is dropped.

Impact (two open PRs)

PR	Model (in models_all.json)	Recipes shipped	Harness result
#1084	audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim	audio-classification_fp32_config.json (only)	no variant discovered → authored recipe ignored, evals via auto-config
#1093	impira/layoutlm-document-qa	question-answering_fp32_config.json, question-answering_w8a16_config.json	fp32 skipped → only w8a16 discovered, no unquantized/fp16-bucket eval

Contrast (correct): usyd-community/vitpose-plus-base ships keypoint-detection_fp16_config.json + keypoint-detection_w8a8_config.json — both discovered.

Suggested fix

Rename the *_fp32_config.json recipes in #1084 and #1093 back to *_fp16_config.json. Their quant: null content already matches the fp16-bucket convention, so the rename makes them discoverable and consistent with the other 75 recipes — no content change needed.

Alternatively, if fp32-named recipes are meant to be first-class, add fp32 to KNOWN_PRECISIONS and the harness precision-resolution. The rename is the lower-risk, convention-aligned option.

References

• scripts/e2e_eval/utils/recipes.py — KNOWN_PRECISIONS, split_task_precision, discover_recipe_variants
• scripts/e2e_eval/run_eval.py — _run_recipe_build (recipe-driven build, no --precision)
• tests/e2e/test_config_e2e.py — CPU stays fp32 by design

[DingmaomaoBJTU]

Triage Summary

Priority: P1 — silent data loss; authored recipes silently dropped from eval
Component: e2e eval harness (scripts/e2e_eval/utils/recipes.py)
Root Cause: KNOWN_PRECISIONS is ("fp16", "w8a16", "w8a8") — missing fp32. split_task_precision() returns None for *_fp32_config.json recipes, causing discover_recipe_variants() to silently skip them.

Impact:

• Add audeering wav2vec2 dimensional emotion (speech regression) support #1084 (wav2vec2): authored fp32 recipe completely ignored → evals via auto-config
• recipe(layoutlm): add document QA support for impira/layoutlm-document-qa #1093 (layoutlm-document-qa): fp32 variant skipped → only w8a16 discovered

Recommended Fix (low-risk): Rename *_fp32_config.json recipes in #1084 and #1093 to *_fp16_config.json (content is quant: null, matching the 75 existing fp16-bucket recipes). No code change needed.

Alternative: Add "fp32" to KNOWN_PRECISIONS if fp32-named recipes should be first-class.

Triaged: 2026-07-14