You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a recipe under examples/recipes/ that lets users reproduce the Qwen3 (Qwen/Qwen3-0.6B) onnxruntime-genai bundle build at the config level, mirroring the existing per-(model, task, precision) recipe convention.
Deferred from the genai-bundle PR (#1081) to keep that PR focused; capturing the design notes here so the follow-up can execute directly.
Background
winml build -m Qwen/Qwen3-0.6B --ep qnn --device npu|auto -o out/bundle emits a genai bundle. It is composed of 4 ONNX components plus a code-driven assembly step:
The 4 components each have a WinMLBuildConfig (config-expressible). The final assembly (genai_config.json, QNN stage session options, ONNX passes) is driven by the registered GenaiBundleRecipe in src/winml/modelkit/models/hf/qwen3/genai.py — it is not a WinMLBuildConfig, so it cannot be captured as a recipe JSON.
Per-component WinMLBuildConfig JSONs following the existing role-suffix convention (cf. microsoft_trocr-*/image-to-text_fp16_config_{decoder,encoder}.json):
text-generation_w8a16_config_context.json
text-generation_w8a16_config_iterator.json
feature-extraction_fp32_config_embeddings.json
feature-extraction_w4a32_config_lm_head.json
A README.md that maps components → bundle, gives the one-command bundle build (assembly), and shows how to rebuild/inspect a single component with winml build -c <config>.json.
Add the Qwen3 rows to the top-level examples/recipes/README.md model table.
Also cross-link docs/samples/qwen3-genai-bundle.md.
Design notes / gotchas for the implementer
Faithful generation: generate the component configs from code (e.g. winml config --model-type <component> --task <t> --device <d> --ep <ep> --precision <p>) rather than hand-authoring — the transformer configs are large (~28 layers of past/present KV tensors).
Transformer max_cache_len override: the composite defaults max_cache_len to max_position_embeddings (40960 for Qwen3-0.6B), but the bundle recipe pins max_cache_len=2048 / prefill_seq_len=64 (see QWEN3_GENAI_BUNDLE_RECIPE in genai.py). To faithfully reproduce the default bundle, the two transformer configs must reflect max_cache_len=2048 (and prefill seq=64 / decode seq=1). Decide whether to bake 2048 in or document the override — winml config uses a single --shape-config for both composite sub-models, so per-sub-model seq differentiation comes from the model classes' _default_seq_len (64/1) automatically.
lm_head precision detail: the w4a32 lm_head is finalized to block_size=32, accuracy_level=4 by Qwen3LMHeadOnlyQuantFinalizer (int8 MLAS GEMM). The regression where this fell back to 128/0 was fixed in feat(build): one-command Qwen3 genai bundle via winml build (npu + qnn) #1081 (commit preserving model_type through the weight-only precision policy). The recipe README is a good place to note this so users can verify their rebuilt lm_head.onnx matches.
Assembly is not config-expressible: make the README explicit that the per-component configs reproduce the 4 ONNX graphs, while the bundle directory (genai_config.json + layout) is produced by the one-command winml build (or scripts/qwen3.py export).
Acceptance criteria
examples/recipes/Qwen_Qwen3-0.6B/ with the 4 component configs + README.
Configs load via winml build -c and are faithful to the default bundle (transformer max_cache_len=2048; lm_head w4a32/block_size=32).
Qwen3 rows added to examples/recipes/README.md.
README documents component→bundle mapping, the one-command assembly, and per-component rebuild.
Summary
Add a recipe under
examples/recipes/that lets users reproduce the Qwen3 (Qwen/Qwen3-0.6B) onnxruntime-genai bundle build at the config level, mirroring the existing per-(model, task, precision)recipe convention.Deferred from the genai-bundle PR (#1081) to keep that PR focused; capturing the design notes here so the follow-up can execute directly.
Background
winml build -m Qwen/Qwen3-0.6B --ep qnn --device npu|auto -o out/bundleemits a genai bundle. It is composed of 4 ONNX components plus a code-driven assembly step:ctx.onnx(transformer prefill)qwen3_transformer_only(decoder_prefillsub-model, seq=64)w8a16iter.onnx(transformer decode)qwen3_transformer_only(decoder_gensub-model, seq=1)w8a16embeddings.onnxqwen3_embeddings_onlyfp32lm_head.onnxqwen3_lm_head_onlyw4a32genai_config.json+ tokenizerassemblestep)The 4 components each have a
WinMLBuildConfig(config-expressible). The final assembly (genai_config.json, QNN stage session options, ONNX passes) is driven by the registeredGenaiBundleRecipeinsrc/winml/modelkit/models/hf/qwen3/genai.py— it is not aWinMLBuildConfig, so it cannot be captured as a recipe JSON.Proposed deliverable
Create
examples/recipes/Qwen_Qwen3-0.6B/containing:WinMLBuildConfigJSONs following the existing role-suffix convention (cf.microsoft_trocr-*/image-to-text_fp16_config_{decoder,encoder}.json):text-generation_w8a16_config_context.jsontext-generation_w8a16_config_iterator.jsonfeature-extraction_fp32_config_embeddings.jsonfeature-extraction_w4a32_config_lm_head.jsonREADME.mdthat maps components → bundle, gives the one-command bundle build (assembly), and shows how to rebuild/inspect a single component withwinml build -c <config>.json.examples/recipes/README.mdmodel table.Also cross-link
docs/samples/qwen3-genai-bundle.md.Design notes / gotchas for the implementer
winml config --model-type <component> --task <t> --device <d> --ep <ep> --precision <p>) rather than hand-authoring — the transformer configs are large (~28 layers of past/present KV tensors).max_cache_lenoverride: the composite defaultsmax_cache_lentomax_position_embeddings(40960 for Qwen3-0.6B), but the bundle recipe pinsmax_cache_len=2048/prefill_seq_len=64(seeQWEN3_GENAI_BUNDLE_RECIPEingenai.py). To faithfully reproduce the default bundle, the two transformer configs must reflectmax_cache_len=2048(and prefill seq=64 / decode seq=1). Decide whether to bake 2048 in or document the override —winml configuses a single--shape-configfor both composite sub-models, so per-sub-model seq differentiation comes from the model classes'_default_seq_len(64/1) automatically.w4a32lm_head is finalized toblock_size=32, accuracy_level=4byQwen3LMHeadOnlyQuantFinalizer(int8 MLAS GEMM). The regression where this fell back to128/0was fixed in feat(build): one-command Qwen3 genai bundle via winml build (npu + qnn) #1081 (commit preservingmodel_typethrough the weight-only precision policy). The recipe README is a good place to note this so users can verify their rebuiltlm_head.onnxmatches.genai_config.json+ layout) is produced by the one-commandwinml build(orscripts/qwen3.py export).Acceptance criteria
examples/recipes/Qwen_Qwen3-0.6B/with the 4 component configs + README.winml build -cand are faithful to the default bundle (transformermax_cache_len=2048; lm_headw4a32/block_size=32).examples/recipes/README.md.