Skip to content

Add a config-level reproduction recipe for the Qwen3 genai bundle under examples/recipes/ #1091

Description

@DingmaomaoBJTU

Summary

Add a recipe under examples/recipes/ that lets users reproduce the Qwen3 (Qwen/Qwen3-0.6B) onnxruntime-genai bundle build at the config level, mirroring the existing per-(model, task, precision) recipe convention.

Deferred from the genai-bundle PR (#1081) to keep that PR focused; capturing the design notes here so the follow-up can execute directly.

Background

winml build -m Qwen/Qwen3-0.6B --ep qnn --device npu|auto -o out/bundle emits a genai bundle. It is composed of 4 ONNX components plus a code-driven assembly step:

Component model_type task device precision
ctx.onnx (transformer prefill) qwen3_transformer_only (decoder_prefill sub-model, seq=64) text-generation NPU/QNN w8a16
iter.onnx (transformer decode) qwen3_transformer_only (decoder_gen sub-model, seq=1) text-generation NPU/QNN w8a16
embeddings.onnx qwen3_embeddings_only feature-extraction CPU fp32
lm_head.onnx qwen3_lm_head_only feature-extraction CPU w4a32
genai_config.json + tokenizer assembly (recipe assemble step)

The 4 components each have a WinMLBuildConfig (config-expressible). The final assembly (genai_config.json, QNN stage session options, ONNX passes) is driven by the registered GenaiBundleRecipe in src/winml/modelkit/models/hf/qwen3/genai.py — it is not a WinMLBuildConfig, so it cannot be captured as a recipe JSON.

Proposed deliverable

Create examples/recipes/Qwen_Qwen3-0.6B/ containing:

  • Per-component WinMLBuildConfig JSONs following the existing role-suffix convention (cf. microsoft_trocr-*/image-to-text_fp16_config_{decoder,encoder}.json):
    • text-generation_w8a16_config_context.json
    • text-generation_w8a16_config_iterator.json
    • feature-extraction_fp32_config_embeddings.json
    • feature-extraction_w4a32_config_lm_head.json
  • A README.md that maps components → bundle, gives the one-command bundle build (assembly), and shows how to rebuild/inspect a single component with winml build -c <config>.json.
  • Add the Qwen3 rows to the top-level examples/recipes/README.md model table.

Also cross-link docs/samples/qwen3-genai-bundle.md.

Design notes / gotchas for the implementer

  • Faithful generation: generate the component configs from code (e.g. winml config --model-type <component> --task <t> --device <d> --ep <ep> --precision <p>) rather than hand-authoring — the transformer configs are large (~28 layers of past/present KV tensors).
  • Transformer max_cache_len override: the composite defaults max_cache_len to max_position_embeddings (40960 for Qwen3-0.6B), but the bundle recipe pins max_cache_len=2048 / prefill_seq_len=64 (see QWEN3_GENAI_BUNDLE_RECIPE in genai.py). To faithfully reproduce the default bundle, the two transformer configs must reflect max_cache_len=2048 (and prefill seq=64 / decode seq=1). Decide whether to bake 2048 in or document the override — winml config uses a single --shape-config for both composite sub-models, so per-sub-model seq differentiation comes from the model classes' _default_seq_len (64/1) automatically.
  • lm_head precision detail: the w4a32 lm_head is finalized to block_size=32, accuracy_level=4 by Qwen3LMHeadOnlyQuantFinalizer (int8 MLAS GEMM). The regression where this fell back to 128/0 was fixed in feat(build): one-command Qwen3 genai bundle via winml build (npu + qnn) #1081 (commit preserving model_type through the weight-only precision policy). The recipe README is a good place to note this so users can verify their rebuilt lm_head.onnx matches.
  • Assembly is not config-expressible: make the README explicit that the per-component configs reproduce the 4 ONNX graphs, while the bundle directory (genai_config.json + layout) is produced by the one-command winml build (or scripts/qwen3.py export).

Acceptance criteria

  • examples/recipes/Qwen_Qwen3-0.6B/ with the 4 component configs + README.
  • Configs load via winml build -c and are faithful to the default bundle (transformer max_cache_len=2048; lm_head w4a32/block_size=32).
  • Qwen3 rows added to examples/recipes/README.md.
  • README documents component→bundle mapping, the one-command assembly, and per-component rebuild.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions