Add audeering wav2vec2 dimensional emotion (speech regression) support#1084
Add audeering wav2vec2 dimensional emotion (speech regression) support#1084DingmaomaoBJTU wants to merge 3 commits into
Conversation
REVIEWER verdict: REQUEST_CHANGESI independently re-ran the reviewer checklist from PR head Blocking items (producer action required)
Checked evidence
Reachable verified EP: |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
da4d514 to
f3b531a
Compare
|
Addressed all 4 items in f3b531a: rebased onto origin/main 130acfe, renamed recipe fp16→fp32 (0 FLOAT16 initializers confirmed), added baseline-FAIL gate evidence to item 7, re-ran CPU perf (representative median avg 381.46 ms across runs 377.25, 380.48, 382.43, 401.01 ms; variance noted). Ready for re-review. |
|
APPROVE Verified PR #1084 from fresh PR head Checkout / scope
Engineering/code review
PR body / report audit
Independent rebuild and artifact verification
Goal ladder re-verification
Baseline gate
Prior REQUEST_CHANGES items
EP coverage annotation
Sign-off
|
nn.Linear.__call__ is typed to return Any, so
eturn x tripped the strict
mypy [no-any-return] gate (CI lint). Return the projection via
cast("torch.Tensor", ...) to match the repo house style (blip.py/mu2.py) and
drop the now-unnecessary RET504 assign-then-return. Runtime behavior unchanged.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
xieofxie
left a comment
There was a problem hiding this comment.
Nice, clean addition — the routing follows house style well: hyphenated MODEL_CLASS_MAPPING keys that rely on the resolver's _→- normalization (consistent with the qwen3-* variants), register_specialization(..., "WinMLModelForGenericTask"), and the model_type_override plumbing via the recipe loader block. Lint passes and the RegressionHead/mean-pool forward match the audeering reference, which lines up with the L2 cosine ≈1.0 you reported.
My main ask is test coverage — see the inline note on wav2vec2.py. Every comparable model-class/ONNX-config addition ships a small unit test, and this one doesn't. The rest are minor nits.
| } | ||
|
|
||
|
|
||
| MODEL_CLASS_MAPPING: dict[tuple[str, str], type] = { |
There was a problem hiding this comment.
No unit test ships with this module. Every comparable MODEL_CLASS_MAPPING / OnnxConfig addition has one under tests/unit/models/ — e.g. test_vitpose_mapping.py, segformer/test_onnx_config.py, sam2/test_onnx_config.py, and the qwen3-* modeling tests. A tiny test here would lock in the behavior that actually matters:
("wav2vec2-emotion-regression", "audio-classification")is present in the aggregatedMODEL_CLASS_MAPPINGand resolves toEmotionModel;resolve_task(cfg, task="audio-classification", model_class="EmotionModel", model_type_override="wav2vec2_emotion_regression")returnsEmotionModel(guards the_→-normalization contract this whole file depends on);- the
OnnxConfigexposes the expectedinputs/outputsaxes.
Without it, a rename of EMOTION_REGRESSION_MODEL_TYPE or a resolver change to the normalization could silently break routing with nothing failing in CI.
There was a problem hiding this comment.
Added tests/unit/models/wav2vec2/test_onnx_config.py (commit 7110135). It locks in all three behaviors:
("wav2vec2-emotion-regression", "audio-classification")is present in the aggregatedMODEL_CLASS_MAPPINGand resolves toEmotionModel(plus the module mapping ⊆ aggregate check);resolve_task(cfg, task="audio-classification", model_class="EmotionModel", model_type_override="wav2vec2_emotion_regression")returnsEmotionModel, guarding the_→-normalization contract;Wav2Vec2EmotionRegressionIOConfigis registered withTasksManagerforaudio-classificationand exposes the expectedinputs/outputsaxes.
8 tests, all passing.
| MODEL_CLASS_MAPPING as _WAV2VEC2_CLASS_MAPPING, | ||
| ) | ||
| from .wav2vec2 import ( | ||
| Wav2Vec2EmotionRegressionIOConfig as _Wav2Vec2EmotionRegressionIOConfig, |
There was a problem hiding this comment.
This side-effect import is missing the # triggers registration trailing comment that every other *IOConfig import in this file carries (bart, blip, segformer, qwen3-*, zoedepth, …). Worth adding for consistency so it's clear the import exists for the @register_onnx_overwrite side effect, not for use. It can also be folded into the same from .wav2vec2 import (...) block as the mapping import above rather than a second from .wav2vec2 statement.
There was a problem hiding this comment.
Added the # triggers registration comment and collapsed the mapping import to a single statement (commit 7110135).
I kept it as two from .wav2vec2 statements rather than one block, though: lint.isort.combine-as-imports is false in pyproject.toml, so ruff splits aliased imports back onto separate statements (same reason bart/t5/siglip each get their own line). The comment sits on its own line inside the parens, matching the existing qwen3 long-name imports.
| ## Models | ||
|
|
||
| Total: **75** (model, task) tuples that pass fp16 eval on all 10 (EP, device) buckets. | ||
| Total: **76** (model, task) tuples with curated model recipes. |
There was a problem hiding this comment.
This rewords the shared header from "pass fp16 eval on all 10 (EP, device) buckets" to "with curated model recipes" for all 76 rows. This model is legitimately fp32/CPU-only so it can't meet the old bar — but relaxing the sentence for the whole table quietly drops the fp16/all-buckets guarantee that the other 75 entries do satisfy. Consider keeping the stronger wording and noting the fp32/CPU exception, rather than weakening the guarantee for every row.
There was a problem hiding this comment.
Good catch — not by design. Reworded in commit 7110135 to keep the stronger guarantee instead of relaxing it for every row:
Total: 76 (model, task) tuples with curated recipes. 75 pass fp16 eval on all 10 (EP, device) buckets; audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim (audio-classification) is fp32/CPU-only.
So the fp16/all-buckets bar still stands for the other 75 and the fp32/CPU exception is called out explicitly.
- Add tests/unit/models/wav2vec2/test_onnx_config.py covering the MODEL_CLASS_MAPPING entry -> EmotionModel, the resolve_task underscore (_ -> -) normalization contract, and the IOConfig registration plus input/output axes. - Add the '# triggers registration' comment to the IOConfig side-effect import and collapse the mapping import to a single statement. - Restore the fp16/all-buckets guarantee wording in the recipes README and note the new fp32/CPU-only exception instead of weakening every row. - Drop the unused **kwargs from RegressionHead.forward.
This PR adds support for
audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim, routed asaudio-classificationwith an emotion regression head (wav2vec2_emotion_regression). The shipped outcome is L1 on the CPU target EP, and the produced ONNX artifact is fp32 with a matching fp32 recipe name. The highest Goal verdict honestly reached by the tester is L2 PASS, with cosine 0.9999998807907104 and max_abs 1.6689300537109375e-06.Recipe path(s)
examples/recipes/audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim/audio-classification_fp32_config.jsonREADME row
trueBuild output dir
temp\fix_build\Build log
✅ Build complete in 256.5sAppended findings
model_knowledge/wav2vec2.jsonfindingswav2vec2-001..wav2vec2-005were appended on the skill repo Lane A, not in this model PR.optim.gelu_fusion=true,optim.matmul_add_fusion=true, quant stayed null.Optimum-coverage probe
VENDOR-ONLYadded_by_winml=[]Claimed (Effort, Goal, Outcome)
["cpu"]130acfe42523b8aa553b1dd10eecd7a1328b832e(currentorigin/mainat rebase time)winml --version:winml, version 0.2.0winml build -c examples\recipes\audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim\audio-classification_fp32_config.json -m audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim -o temp\fix_build\; build exit 0;✅ Build complete in 256.5s; ONNX inputinput_values[1,16000], outputshidden_states[1,1024]andlogits[1,3]; initializer dtypes{FLOAT:233, INT64:10}withFLOAT16_COUNT 0, correctly fp32.Goal-ladder verdict table
winml build -c examples\recipes\audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim\audio-classification_fp32_config.json -m audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim -o temp\fix_build\winml perf -m temp\fix_build\model.onnx --device cpu --ep cpupython temp\audeering_l2_parity.pyMethodology-evolution declaration
_meta-056effort-mis-estimate (planner Optimum-probe edit),_meta-057goal-ceiling mis-estimate (planner Goal-axis edit), and_meta-058doc-code-driftuser_skill→dev_skill(explainer/reviewer path fix) were filed on the SKILL repo Lane A branch.9d380baeand are intentionally not included in this model PR to keep lanes unpolluted.Perf & eval data
winml analyze --model temp\fix_build\model.onnx --ep all --format jsontemp\fix_analyze_output.txt