diff --git a/docs/commands/build.md b/docs/commands/build.md index a588d0671..4935f129c 100644 --- a/docs/commands/build.md +++ b/docs/commands/build.md @@ -22,6 +22,7 @@ $ winml build [options] |---|---|---|---|---| | `--config` | `-c` | path | `None` | `WinMLBuildConfig` JSON file, generated by `winml config`. If omitted, config is auto-generated from `-m`. | | `--model` | `-m` | string | `None` | Hugging Face model ID or path to an existing `.onnx` file. | +| `--export-type` | | choice | `generic` | Output selector: `generic` builds the stock single/composite ONNX model; `optimized` builds the family's registered runtime-optimized recipe (today the onnxruntime-genai NPU bundle), inferring its target. `optimized` fails fast if the family has no recipe or `--ep`/`--device` contradicts it. | | `--output-dir` | `-o` | path | `None` | Directory for all build artifacts. Mutually exclusive with `--use-cache`. | | `--use-cache/--no-use-cache` | | flag | `false` | Store artifacts in the winml-cli global cache (`~/.cache/winml/`). Mutually exclusive with `--output-dir`. | | `--rebuild/--no-rebuild` | | flag | `false` | Overwrite existing artifacts and re-run the full pipeline. | @@ -58,11 +59,10 @@ single-pass build. Individual stages can be suppressed with `--no-quant`, ## Genai bundles for decoder LLMs (NPU + QNN) -For a registered decoder-LLM family (currently **Qwen3**), targeting the NPU HTP -with an explicit `--ep qnn` switches `winml build` to produce a complete -[onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai) **bundle** -directory instead of the stock per-model ONNX output. Pair `--ep qnn` with -`--device npu`, or with `--device auto` when auto-detection resolves to the NPU: +For a registered decoder-LLM family (currently **Qwen3**), `--export-type +optimized` switches `winml build` from the stock per-model ONNX output to a +complete [onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai) +**bundle** directory: | File | Role | Precision | |---|---|---| @@ -73,20 +73,36 @@ directory instead of the stock per-model ONNX output. Pair `--ep qnn` with | `genai_config.json` + tokenizer | onnxruntime-genai runtime metadata | — | ```bash -# One command: HF decoder LLM → full onnxruntime-genai bundle on the NPU -winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn -# or let device auto-detection pick the NPU: -winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device auto --ep qnn +# One command: HF decoder LLM → full onnxruntime-genai bundle +winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --export-type optimized ``` -The trigger is data-driven and opt-in: **`--ep qnn` must be passed explicitly**, -alongside an NPU target (`--device npu`, or `--device auto` resolving to the NPU). -Every other target — including Qwen3 on CPU/GPU, or an auto-detected NPU target -without an explicit `--ep qnn` — keeps the existing behavior (the stock composite -build). `--output-dir` is required (the bundle is a directory) and `--use-cache` -is not supported. Use `--precision` to override the transformer precision; the CPU -companions always use their bundle-standard precisions. See -[Qwen3 — Genai Bundle](../samples/qwen3-genai-bundle.md) for the full walkthrough. +`--export-type optimized` **infers** the recipe's target (Qwen3 builds for the +NPU HTP via QNN), so `--ep`/`--device` are optional — and, unlike the shortcut +below, the optimized path does not gate on detecting the target device locally. +If you do pin `--ep`/`--device`, they must match the recipe: a contradicting +value, an unregistered family, a pre-exported `.onnx` input, or module mode all +fail fast rather than silently falling back to a generic build. `--output-dir` is +required (the bundle is a directory) and `--use-cache` is not supported. Use +`--precision` to override the transformer precision; the CPU companions always +use their bundle-standard precisions. + +!!! note "Backward-compatible shortcut" + When `--export-type` is omitted, a registered family with an explicit + `--ep qnn` on an NPU target still routes to its optimized bundle. The NPU + target may be explicit (`--device npu`) or resolved from `auto` — whether + `--device auto` is typed or `--device` is left off entirely: + + ```bash + winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn + # or let device auto-detection pick the NPU (--device may be omitted): + winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --ep qnn + ``` + + `--export-type generic` always forces the stock composite build, even for a + registered family on the NPU. + +See [Qwen3 — Genai Bundle](../samples/qwen3-genai-bundle.md) for the full walkthrough. ## Examples @@ -150,9 +166,9 @@ winml build -m microsoft/resnet-50 -o output/ \ pre-exported ONNX inputs because the export step has already happened. - **Existing artifacts are reused by default.** Pass `--rebuild` to force a fresh run after changing the config. -- **Genai bundles require `--output-dir`, not `--use-cache`.** A decoder-LLM - bundle (`--device npu --ep qnn`) writes a directory of components and rejects - `--use-cache`. +- **Genai bundles require `--output-dir`, not `--use-cache`.** An optimized + decoder-LLM bundle (`--export-type optimized`, or the `--device npu --ep qnn` + shortcut) writes a directory of components and rejects `--use-cache`. ## See also diff --git a/docs/samples/qwen3-genai-bundle.md b/docs/samples/qwen3-genai-bundle.md index accf848b4..098f0f94a 100644 --- a/docs/samples/qwen3-genai-bundle.md +++ b/docs/samples/qwen3-genai-bundle.md @@ -29,7 +29,7 @@ memory-bound companions stay on CPU. ```mermaid graph LR - A["winml build -m Qwen/Qwen3-0.6B --device npu --ep qnn"] --> B[Genai bundle recipe] + A["winml build -m Qwen/Qwen3-0.6B --export-type optimized"] --> B[Genai bundle recipe] B --> C[ctx.onnx / iter.onnx — NPU] B --> D[embeddings.onnx — CPU] B --> E[lm_head.onnx — CPU] @@ -41,14 +41,12 @@ graph LR ## Step 1: Build the bundle (one command) -Targeting the NPU HTP with an explicit `--ep qnn` switches `winml build` from the -stock per-model ONNX output to the full genai bundle. Pair it with `--device npu`, -or `--device auto` when auto-detection resolves to the NPU: +`--export-type optimized` switches `winml build` from the stock per-model ONNX +output to the full genai bundle. It **infers** the recipe's target (Qwen3 builds +for the NPU HTP via QNN), so `--ep`/`--device` are optional: ```bash -winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn -# or, letting device auto-detection pick the NPU: -winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device auto --ep qnn +winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --export-type optimized ``` This builds (or reuses from cache) all four components and assembles them, writing @@ -63,11 +61,23 @@ The CPU companions likewise keep their bundle-standard precisions. Force a clean rebuild of every component with `--rebuild`. -!!! note "Opt-in, non-destructive" - The bundle path only triggers on an **explicit `--ep qnn`** together with an - NPU target — either `--device npu` or a `--device auto` that resolves to the - NPU. Qwen3 on any other target — CPU, GPU, or an auto-detected NPU *without* - an explicit `--ep qnn` — still produces the stock composite build, unchanged. +!!! note "Backward-compatible shortcut" + When `--export-type` is omitted, an **explicit `--ep qnn`** targeting the NPU + still routes a registered family to its bundle. The NPU target may be + explicit (`--device npu`) or resolved from `auto` — whether `--device auto` + is typed or `--device` is omitted: + + ```bash + winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --device npu --ep qnn + # or, letting device auto-detection pick the NPU (--device may be omitted): + winml build -m Qwen/Qwen3-0.6B -o out/qwen3-bundle --ep qnn + ``` + + Qwen3 on any other target — CPU, GPU, or an auto-detected NPU *without* an + explicit `--ep qnn` — still produces the stock composite build. `--export-type + generic` always forces the stock build, even on the NPU. A pinned + `--ep`/`--device` that contradicts the recipe fails fast rather than silently + reverting. ## Step 2: Tune context and prefill lengths (optional) diff --git a/src/winml/modelkit/commands/build.py b/src/winml/modelkit/commands/build.py index 83ea3da53..a601a70c5 100644 --- a/src/winml/modelkit/commands/build.py +++ b/src/winml/modelkit/commands/build.py @@ -442,9 +442,88 @@ def _validate_loader_tasks_for_model( return hf_config +def _genai_model_type(config_or_configs: Any, preloaded_hf_config: Any | None) -> str | None: + """Best-effort ``model_type`` for genai-recipe resolution and messaging.""" + model_type = getattr(preloaded_hf_config, "model_type", None) + if ( + not model_type + and not isinstance(config_or_configs, list) + and config_or_configs.loader is not None + ): + model_type = config_or_configs.loader.model_type + return model_type + + +def _resolve_genai_recipe( + *, + model: str | None, + model_is_onnx: bool, + config_or_configs: Any, + preloaded_hf_config: Any | None, +) -> Any | None: + """Return the genai-bundle recipe for this build, or ``None`` if inapplicable. + + ``None`` covers every input an optimized bundle cannot be built from: no + model, a pre-exported ``.onnx`` file, module mode (array config), or a + ``model_type`` with no registered recipe. + """ + if not model or model_is_onnx or isinstance(config_or_configs, list): + return None + from ..models.winml import resolve_genai_bundle + + return resolve_genai_bundle(_genai_model_type(config_or_configs, preloaded_hf_config)) + + +def _resolve_optimized_target( + ctx: click.Context, + recipe: Any, + *, + device: str, + ep: EPNameOrAlias | None, +) -> tuple[str, str]: + """Resolve the ``(ep, device)`` an optimized bundle is built for. + + The recipe's ``supported_targets`` are the source of truth. An explicit + ``--ep``/``--device`` narrows the candidates; a value the recipe cannot serve + is a fail-fast error (an explicit contradiction). When the user pins neither + (or passes ``--device auto``), the recipe's first target is inferred -- + building a bundle is hardware-independent, so no device probing happens here. + """ + from ..utils.constants import normalize_ep_name + + def _fmt(targets: Any) -> str: + return ", ".join(f"--ep {t.ep} --device {t.device}" for t in targets) + + candidates = list(recipe.supported_targets) + + if cli_utils.is_cli_provided(ctx, "ep") and ep is not None: + want = normalize_ep_name(ep) + narrowed = [t for t in candidates if normalize_ep_name(t.ep) == want] + if not narrowed: + raise click.UsageError( + f"--export-type optimized: --ep {ep} is not supported by the " + f"'{recipe.family}' recipe (supported: {_fmt(recipe.supported_targets)})." + ) + candidates = narrowed + + if cli_utils.is_cli_provided(ctx, "device") and device.lower() != "auto": + want_device = device.lower() + narrowed = [t for t in candidates if t.device.lower() == want_device] + if not narrowed: + raise click.UsageError( + f"--export-type optimized: --device {device} is not supported by the " + f"'{recipe.family}' recipe (supported: {_fmt(recipe.supported_targets)})." + ) + candidates = narrowed + + target = candidates[0] + return target.ep, target.device + + def _maybe_build_genai_bundle( ctx: click.Context, *, + export_type: str, model: str | None, model_is_onnx: bool, config_or_configs: Any, @@ -456,83 +535,112 @@ def _maybe_build_genai_bundle( precision: str | None, rebuild: bool, ) -> bool: - """Route a registered decoder-LLM to the genai-bundle builder, if applicable. - - Returns ``True`` when this call fully handled the build (an - onnxruntime-genai bundle was produced) and the caller should stop; ``False`` - to fall through to the normal single/composite pipeline. - - The trigger is intentionally narrow and explicit (see the call site): a - single HuggingFace model whose ``model_type`` has a registered - :class:`~winml.modelkit.models.winml.genai_bundle.GenaiBundleRecipe`, built - with an explicit ``--ep qnn`` and a ``--device`` that targets the NPU -- - either ``--device npu`` or ``--device auto`` that resolves to the NPU on - this system. Nothing here is architecture-specific; the recipe carries every - model detail. + """Build an optimized (onnxruntime-genai) bundle when selected, else fall through. + + Returns ``True`` when this call fully handled the build (an optimized bundle + was produced) and the caller should stop; ``False`` to fall through to the + normal single/composite (``generic``) pipeline. + + Selection: + * ``--export-type optimized`` builds the family's registered recipe; + every unmet precondition (no model, ``.onnx`` input, module mode, no + recipe, or a contradicting ``--ep``/``--device``) is a fail-fast error. + * ``--export-type generic`` never builds a bundle (returns ``False``). + * omitted -> backward-compatible shortcut: a registered family with an + explicit ``--ep qnn`` and an NPU target still routes to its optimized + bundle. The NPU target may be explicit (``--device npu``) or resolved + from ``auto`` -- whether ``--device auto`` is typed or left at its + default; anything else falls through. + + Nothing here is architecture-specific; the recipe carries every model detail. """ - if ( - not model - or model_is_onnx - or isinstance(config_or_configs, list) - or not cli_utils.is_cli_provided(ctx, "device") - or not cli_utils.is_cli_provided(ctx, "ep") - ): + explicit_type = cli_utils.is_cli_provided(ctx, "export_type") + want_optimized = explicit_type and export_type.lower() == "optimized" + + # ``--export-type generic`` forces the stock build even for a registered + # family on the NPU. + if explicit_type and not want_optimized: return False - from ..utils.constants import normalize_ep_name + recipe = _resolve_genai_recipe( + model=model, + model_is_onnx=model_is_onnx, + config_or_configs=config_or_configs, + preloaded_hf_config=preloaded_hf_config, + ) - if normalize_ep_name(ep) != "QNNExecutionProvider": - return False + if want_optimized: + # Explicit request: surface every unmet precondition as an error rather + # than silently falling back to the generic build. + if not model: + raise click.UsageError("--export-type optimized requires -m/--model.") + if model_is_onnx: + raise click.UsageError( + "--export-type optimized is not supported for a pre-exported .onnx " + "input; pass a HuggingFace model id so the recipe can build every component." + ) + if isinstance(config_or_configs, list): + raise click.UsageError( + "--export-type optimized is not supported for module mode (array config)." + ) + if recipe is None: + model_type = _genai_model_type(config_or_configs, preloaded_hf_config) + raise click.UsageError( + f"--export-type optimized: no optimized recipe is " + f"registered for model type '{model_type}'." + ) + else: + # Implicit shortcut (backward-compatible with #1081): a registered family + # + an explicit ``--ep qnn`` + an NPU target. ``--device`` may be omitted + # (its ``auto`` default is resolved below just like an explicit + # ``--device auto``). Anything else falls through to the generic pipeline. + if recipe is None or not cli_utils.is_cli_provided(ctx, "ep"): + return False - # Target must be the NPU HTP: an explicit ``--device npu``, or ``--device - # auto`` that resolves to the NPU (QNN) on this system. ``--ep qnn`` paired - # with any other explicit device is a contradiction left to the normal build - # path (which surfaces the error). - device_target = device.lower() - if device_target == "auto": - from .. import sysinfo + from ..utils.constants import normalize_ep_name - try: - device_target, _ = sysinfo.resolve_device(device, ep=ep) - except ValueError: + if normalize_ep_name(ep) != "QNNExecutionProvider": return False - if device_target != "npu": - return False - from ..models.winml import build_genai_bundle, resolve_genai_bundle + device_target = device.lower() + if device_target == "auto": + from .. import sysinfo - single_config = config_or_configs - model_type = getattr(preloaded_hf_config, "model_type", None) - if not model_type and single_config.loader is not None: - model_type = single_config.loader.model_type - recipe = resolve_genai_bundle(model_type) - if recipe is None: - return False + try: + device_target, _ = sysinfo.resolve_device(device, ep=ep) + except ValueError: + return False + if device_target != "npu": + return False + + # An optimized bundle is being built (explicit request or the shortcut). + # Resolve the (ep, device) the recipe is built for -- an explicit + # contradiction raises here. + bundle_ep, bundle_device = _resolve_optimized_target(ctx, recipe, device=device, ep=ep) - # A genai bundle is fully recipe-driven: every component, shape, precision, + # The bundle is fully recipe-driven: every component, shape, precision, # quantization and compile setting comes from the recipe, so a supplied - # ``-c/--config`` file would be silently discarded. Reject it explicitly - # (the bundle is produced directly from ``-m``), matching the fast path's - # other "don't silently ignore a user control" rejections below. + # ``-c/--config`` file would be silently discarded. Reject it explicitly (the + # bundle is produced directly from ``-m``), matching the other "don't silently + # ignore a user control" rejections below. if cli_utils.is_cli_provided(ctx, "config_file"): raise click.UsageError( - "-c/--config is not supported for a genai bundle build: the bundle's " - "components, shapes, quantization and compilation are fixed by its " + "-c/--config is not supported for an optimized (genai bundle) build: the " + "bundle's components, shapes, quantization and compilation are fixed by its " "recipe. Re-run without -c (the bundle is built directly from -m/--model)." ) if use_cache: raise click.UsageError( - "genai bundle output is a directory; pass --output-dir, not --use-cache." + "optimized (genai bundle) output is a directory; pass --output-dir, not --use-cache." ) if not output_dir: - raise click.UsageError("--output-dir is required for a genai bundle build.") + raise click.UsageError("--output-dir is required for an optimized (genai bundle) build.") - # The genai-bundle pipeline is recipe-driven: quantization, optimization, - # analysis and compilation are fixed by the recipe (with ``--precision`` the - # only transformer knob). The normal single/composite pipeline controls are - # bypassed here, so reject any that the user explicitly passed rather than - # letting a supplied flag silently become a no-op. + # The pipeline is recipe-driven: quantization, optimization, analysis and + # compilation are fixed by the recipe (with ``--precision`` the only + # transformer knob). Reject the normal-pipeline controls the user explicitly + # passed rather than letting a supplied flag silently become a no-op. _unsupported_controls = { "quant": "--quant/--no-quant", "optimize": "--optimize/--no-optimize", @@ -546,20 +654,27 @@ def _maybe_build_genai_bundle( ] if rejected: raise click.UsageError( - f"{', '.join(rejected)} not supported for a genai bundle build: the " + f"{', '.join(rejected)} not supported for an optimized (genai bundle) build: the " "bundle's components, quantization, optimization and compilation are " "fixed by its recipe." ) + from ..models.winml import build_genai_bundle + + # Both branches above guarantee a model id here: the explicit request rejects a + # missing --model, and the implicit shortcut only proceeds when a recipe was + # resolved (which requires a model). Narrow str | None -> str for the call. + assert model is not None bundle_dir = Path(output_dir) override_precision = precision if cli_utils.is_cli_provided(ctx, "precision") else None + model_type = _genai_model_type(config_or_configs, preloaded_hf_config) console.print(f"\n[bold blue]Genai bundle[/bold blue] ({model_type}): {model} -> {bundle_dir}") config_path = build_genai_bundle( model, bundle_dir, recipe, - ep="qnn", - device="npu", + ep=bundle_ep, + device=bundle_device, precision=override_precision, force_rebuild=rebuild, emit=lambda msg: console.print(msg, markup=False), @@ -629,6 +744,18 @@ def _maybe_build_genai_bundle( @cli_utils.precision_option( optional_message="With -c, applied only when --device or --precision is passed.", ) +@click.option( + "--export-type", + type=click.Choice(["generic", "optimized"], case_sensitive=False), + default="generic", + show_default=True, + help="Output selector. 'generic' builds the stock single/composite ONNX model. " + "'optimized' builds the registered runtime-optimized recipe for the family " + "(today: the onnxruntime-genai NPU bundle); it infers the recipe's target and " + "errors if the family has no recipe or --ep/--device contradicts it. When omitted, " + "an explicit --ep qnn on an NPU target still routes a registered family to its " + "optimized bundle (backward-compatible shortcut).", +) @cli_utils.shape_config_option( help_text="JSON with shape overrides for auto-generated HuggingFace export configs.", ) @@ -663,6 +790,7 @@ def build( ep: EPNameOrAlias | None, device: str, precision: str, + export_type: str, shape_config: Path | None, input_specs: Path | None, export_config: Path | None, @@ -713,6 +841,10 @@ def build( # Build with mixed precision (INT8 weights, INT8 activations) winml build -m microsoft/resnet-50 -o output/ --precision w8a8 + + # Build the optimized onnxruntime-genai bundle for a decoder LLM + # (infers the recipe's NPU/QNN target; no --device/--ep needed) + winml build -m Qwen/Qwen3-0.6B -o out/ --export-type optimized """ # Merge top-level -v/-q with subcommand-level flags so either position works. verbose, quiet = cli_utils.resolve_verbosity(ctx, verbose, quiet) @@ -758,7 +890,16 @@ def build( # (auto-mode walks the priority list, falls back to cpu which is always # valid), or raises ValueError for an explicit device with no compatible EP. # So the following available_eps[0] is safe whenever it returns. - if ep is None: + # + # ``--export-type optimized`` is exempt: its (ep, device) is inferred from + # the registered recipe and the build is hardware-independent by contract + # (see _resolve_optimized_target), so probing the host here would make an + # optimized build fail on a machine without the recipe's accelerator (e.g. + # no NPU) before the recipe target is ever inferred. + optimized_requested = ( + cli_utils.is_cli_provided(ctx, "export_type") and export_type.lower() == "optimized" + ) + if ep is None and not optimized_requested: from ..sysinfo import resolve_check_device_ep try: @@ -942,20 +1083,21 @@ def _patch_device(cfg: WinMLBuildConfig) -> None: # allow_unsupported_nodes explicitly regardless of its value. extra_kwargs["allow_unsupported_nodes"] = allow_unsupported_nodes - # ---- GENAI BUNDLE FAST PATH ---- - # A registered decoder-LLM family targeted at the NPU HTP via an explicit - # ``--ep qnn`` -- together with ``--device npu`` or a ``--device auto`` - # that resolves to the NPU -- emits a full onnxruntime-genai bundle - # (ctx/iter/embeddings/lm_head + genai_config.json + tokenizer) in one - # command, instead of the stock per-model ONNX output. The switch is - # data-driven (the genai-bundle recipe registry) and architecture- - # agnostic here -- every model-specific value lives in the recipe. - # ``--ep qnn`` must be explicit, so an auto-detected NPU *without* an - # explicit ``--ep qnn`` and every other device/ep combination keep - # today's behavior unchanged (including this family's stock composite - # build). + # ---- OPTIMIZED (GENAI BUNDLE) EXPORT ---- + # ``--export-type optimized`` builds the family's registered + # runtime-optimized recipe (today: a full onnxruntime-genai bundle -- + # ctx/iter/embeddings/lm_head + genai_config.json + tokenizer), inferring + # the recipe's target. When ``--export-type`` is omitted, a registered + # family targeted at the NPU HTP via an explicit ``--ep qnn`` (with + # ``--device npu`` or a ``--device auto`` resolving to the NPU) still + # routes here as a backward-compatible shortcut. The switch is data-driven + # (the genai-bundle recipe registry) and architecture-agnostic -- every + # model-specific value lives in the recipe. ``--export-type generic`` (or + # any other device/ep combination without the flag) keeps the stock + # single/composite build. if _maybe_build_genai_bundle( ctx, + export_type=export_type, model=model, model_is_onnx=model_is_onnx, config_or_configs=config_or_configs, diff --git a/src/winml/modelkit/models/hf/qwen3/genai.py b/src/winml/modelkit/models/hf/qwen3/genai.py index aaa820082..735bac21a 100644 --- a/src/winml/modelkit/models/hf/qwen3/genai.py +++ b/src/winml/modelkit/models/hf/qwen3/genai.py @@ -37,6 +37,7 @@ from ...winml.genai_bundle import ( GenaiBundleRecipe, GenaiCompanionSpec, + GenaiTarget, GenaiTransformerSpec, register_genai_bundle, ) @@ -287,6 +288,7 @@ def write_genai_bundle( ), ), assemble=write_genai_bundle, + supported_targets=(GenaiTarget(ep="qnn", device="npu"),), transformer_onnx_passes=(strip_gqa_default_attrs,), max_cache_len=2048, prefill_seq_len=64, diff --git a/src/winml/modelkit/models/winml/__init__.py b/src/winml/modelkit/models/winml/__init__.py index 5257519bd..bd1ac303b 100644 --- a/src/winml/modelkit/models/winml/__init__.py +++ b/src/winml/modelkit/models/winml/__init__.py @@ -192,6 +192,7 @@ def register_specialization(model_type: str, task: str, class_name: str) -> None GENAI_BUNDLE_REGISTRY, GenaiBundleRecipe, GenaiCompanionSpec, + GenaiTarget, GenaiTransformerSpec, build_genai_bundle, register_genai_bundle, @@ -220,6 +221,7 @@ def register_specialization(model_type: str, task: str, class_name: str) -> None "WINML_MODEL_CLASS_MAPPING", "GenaiBundleRecipe", "GenaiCompanionSpec", + "GenaiTarget", "GenaiTransformerSpec", "ImageSegmentationOutput", "WinMLCache", diff --git a/src/winml/modelkit/models/winml/genai_bundle.py b/src/winml/modelkit/models/winml/genai_bundle.py index 826f509ed..af6b3c7ce 100644 --- a/src/winml/modelkit/models/winml/genai_bundle.py +++ b/src/winml/modelkit/models/winml/genai_bundle.py @@ -43,6 +43,25 @@ # ========================================================================= +@dataclass(frozen=True) +class GenaiTarget: + """A build target a genai-bundle recipe supports: an ``(ep, device)`` pair. + + Declared on :class:`GenaiBundleRecipe` so ``winml build`` can validate an + explicit ``--export-type optimized`` request against the recipe and infer + the target when the user does not pin one. Tokens are the short CLI forms + (e.g. ``ep="qnn"``, ``device="npu"``) so they forward straight to + :func:`build_genai_bundle`. + + Attributes: + ep: Short execution-provider token (e.g. ``"qnn"``). + device: Device token (e.g. ``"npu"``). + """ + + ep: str + device: str + + @dataclass(frozen=True) class GenaiTransformerSpec: """How to build the transformer half of a genai bundle. @@ -99,6 +118,10 @@ class GenaiBundleRecipe: assemble: Callable assembling the copied/converted components into a bundle directory and writing ``genai_config.json``; must accept the keywords the orchestrator passes (see :func:`build_genai_bundle`). + supported_targets: The ``(ep, device)`` pairs this recipe can build for. + The first entry is the preferred/default target inferred when the + user selects ``--export-type optimized`` without pinning + ``--ep``/``--device``. Must be non-empty (enforced at registration). transformer_onnx_passes: ONNX graph transforms applied to the transformer graphs during assembly. max_cache_len: Default static KV cache length (context_length). @@ -110,6 +133,7 @@ class GenaiBundleRecipe: transformer: GenaiTransformerSpec companions: tuple[GenaiCompanionSpec, ...] assemble: Callable[..., Path] + supported_targets: tuple[GenaiTarget, ...] transformer_onnx_passes: tuple[Callable[[onnx.ModelProto], onnx.ModelProto], ...] = () max_cache_len: int = 2048 prefill_seq_len: int = 64 @@ -128,7 +152,8 @@ def register_genai_bundle(recipe: GenaiBundleRecipe) -> GenaiBundleRecipe: """Register *recipe* under ``recipe.family`` and return it. Raises: - ValueError: If a recipe is already registered for the family. + ValueError: If a recipe is already registered for the family, or if the + recipe declares no :attr:`~GenaiBundleRecipe.supported_targets`. """ key = recipe.family if key in GENAI_BUNDLE_REGISTRY: @@ -136,6 +161,11 @@ def register_genai_bundle(recipe: GenaiBundleRecipe) -> GenaiBundleRecipe: f"genai bundle recipe already registered for {key!r}: " f"{GENAI_BUNDLE_REGISTRY[key]!r}. Cannot register {recipe!r}." ) + if not recipe.supported_targets: + raise ValueError( + f"genai bundle recipe for {key!r} declares no supported_targets; " + "at least one (ep, device) target is required." + ) GENAI_BUNDLE_REGISTRY[key] = recipe return recipe @@ -344,6 +374,7 @@ def build_genai_bundle( "GENAI_BUNDLE_REGISTRY", "GenaiBundleRecipe", "GenaiCompanionSpec", + "GenaiTarget", "GenaiTransformerSpec", "build_genai_bundle", "register_genai_bundle", diff --git a/tests/unit/commands/test_build_genai_bundle.py b/tests/unit/commands/test_build_genai_bundle.py index a3f07c971..51a8c954f 100644 --- a/tests/unit/commands/test_build_genai_bundle.py +++ b/tests/unit/commands/test_build_genai_bundle.py @@ -15,10 +15,11 @@ inject ``model_type`` by stubbing the auto config generator instead. The trigger under test (locked design): - registered decoder-LLM family AND explicit ``--ep qnn`` AND a ``--device`` - that targets the NPU (explicit ``--device npu`` or ``--device auto`` - resolving to the NPU) -> genai bundle. Every other combination keeps the - stock single/composite behavior. + registered decoder-LLM family AND explicit ``--ep qnn`` AND an NPU target + -> genai bundle. The NPU target may be explicit (``--device npu``) or + resolved from ``auto`` -- whether ``--device auto`` is typed or ``--device`` + is omitted (its default). Every other combination keeps the stock + single/composite behavior. """ from __future__ import annotations @@ -27,6 +28,7 @@ from pathlib import Path from unittest.mock import MagicMock, patch +import click import pytest from click.testing import CliRunner @@ -136,6 +138,22 @@ def _fake(model_id, output_dir, recipe, **kwargs): return _fake +def _fake_ctx(provided: set[str]): + """A click.Context stand-in whose parameter sources report ``provided``. + + ``is_cli_provided`` only inspects ``ctx.get_parameter_source(name)``, so this + lets the recipe-target / precondition helpers be unit-tested without driving + the whole command. + """ + ctx = MagicMock(spec=click.Context) + ctx.get_parameter_source.side_effect = lambda name: ( + click.core.ParameterSource.COMMANDLINE + if name in provided + else click.core.ParameterSource.DEFAULT + ) + return ctx + + def test_registered_family_npu_qnn_routes_to_bundle(tmp_path: Path): out = tmp_path / "bundle" recorded: dict = {} @@ -341,6 +359,48 @@ def test_auto_qnn_resolving_to_non_npu_does_not_route(tmp_path: Path): run_single.assert_called_once() +def test_registered_family_qnn_without_device_routes_to_bundle(tmp_path: Path): + """``--ep qnn`` with ``--device`` omitted must route just like ``--device auto``. + + Omitting ``--device`` falls back to the ``auto`` default; the shortcut must + resolve it (here -> NPU) instead of treating a missing flag as "no NPU + target". Regression test for ``--ep qnn`` alone silently building generic. + """ + out = tmp_path / "bundle" + recorded: dict = {} + + with ( + patch(_GENERATE_TARGET, return_value=_fake_config("qwen3")), + patch(_BUNDLE_TARGET, side_effect=_record_bundle(recorded)) as bundle, + patch(_RUN_SINGLE_TARGET) as run_single, + patch(_COMPOSITE_TARGET, return_value=None), + ): + result = _invoke(["-m", "Qwen/Qwen3-0.6B", "-o", str(out), "--ep", "qnn"]) + + assert result.exit_code == 0, result.output + assert bundle.call_count == 1 + run_single.assert_not_called() + kwargs = recorded["kwargs"] + assert kwargs["ep"] == "qnn" + assert kwargs["device"] == "npu" + + +def test_qnn_without_device_resolving_to_non_npu_does_not_route(tmp_path: Path): + """``--ep qnn`` with ``--device`` omitted still gates on the resolved device.""" + with ( + patch(_GENERATE_TARGET, return_value=_fake_config("qwen3")), + patch(_BUNDLE_TARGET) as bundle, + patch(_RUN_SINGLE_TARGET) as run_single, + patch(_COMPOSITE_TARGET, return_value=None), + patch("winml.modelkit.sysinfo.resolve_device", return_value=("cpu", ["cpu"])), + ): + result = _invoke(["-m", "Qwen/Qwen3-0.6B", "-o", str(tmp_path / "o"), "--ep", "qnn"]) + + assert result.exit_code == 0, result.output + bundle.assert_not_called() + run_single.assert_called_once() + + def test_npu_without_explicit_ep_does_not_route(tmp_path: Path): """Auto-resolved QNN (no explicit --ep) must keep the stock path.""" with ( @@ -419,3 +479,231 @@ def test_use_cache_rejected_for_bundle(tmp_path: Path): assert result.exit_code != 0 assert "output-dir" in result.output.lower() bundle.assert_not_called() + + +# --------------------------------------------------------------------------- +# --export-type GENERIC | OPTIMIZED (issue #1090) +# --------------------------------------------------------------------------- + + +def test_export_type_optimized_infers_target_and_builds(tmp_path: Path): + """``--export-type optimized`` builds the recipe, inferring its target. + + No ``--device``/``--ep`` is pinned, so the recipe's first ``supported_targets`` + entry (qwen3 -> qnn/npu) is inferred. Building a bundle is hardware-independent, + so this path does not probe hardware. + """ + out = tmp_path / "bundle" + recorded: dict = {} + + with ( + patch(_GENERATE_TARGET, return_value=_fake_config("qwen3")), + patch(_BUNDLE_TARGET, side_effect=_record_bundle(recorded)) as bundle, + patch(_RUN_SINGLE_TARGET) as run_single, + patch(_COMPOSITE_TARGET, return_value=None), + ): + result = _invoke(["-m", "Qwen/Qwen3-0.6B", "-o", str(out), "--export-type", "optimized"]) + + assert result.exit_code == 0, result.output + assert bundle.call_count == 1 + run_single.assert_not_called() + kwargs = recorded["kwargs"] + assert kwargs["ep"] == "qnn" + assert kwargs["device"] == "npu" + + +def test_export_type_optimized_skips_hardware_probe(tmp_path: Path): + """``--export-type optimized`` must not probe local hardware (auto-EP). + + Its (ep, device) is inferred from the recipe, so a host without the target + accelerator -- modeled here by ``resolve_check_device_ep`` raising, as it + does when no compatible EP is available -- must still build the bundle + instead of failing in the auto-EP resolution before the recipe target is + inferred. Regression test for the reviewer note on PR #1104. + """ + out = tmp_path / "bundle" + recorded: dict = {} + + with ( + patch(_GENERATE_TARGET, return_value=_fake_config("qwen3")), + patch(_BUNDLE_TARGET, side_effect=_record_bundle(recorded)) as bundle, + patch(_RUN_SINGLE_TARGET) as run_single, + patch(_COMPOSITE_TARGET, return_value=None), + patch( + "winml.modelkit.sysinfo.resolve_check_device_ep", + side_effect=ValueError("no compatible EP for device"), + ) as probe, + ): + result = _invoke(["-m", "Qwen/Qwen3-0.6B", "-o", str(out), "--export-type", "optimized"]) + + assert result.exit_code == 0, result.output + probe.assert_not_called() + assert bundle.call_count == 1 + run_single.assert_not_called() + kwargs = recorded["kwargs"] + assert kwargs["ep"] == "qnn" + assert kwargs["device"] == "npu" + + +def test_export_type_optimized_is_case_insensitive(tmp_path: Path): + out = tmp_path / "bundle" + recorded: dict = {} + + with ( + patch(_GENERATE_TARGET, return_value=_fake_config("qwen3")), + patch(_BUNDLE_TARGET, side_effect=_record_bundle(recorded)) as bundle, + patch(_RUN_SINGLE_TARGET), + patch(_COMPOSITE_TARGET, return_value=None), + ): + result = _invoke(["-m", "Qwen/Qwen3-0.6B", "-o", str(out), "--export-type", "OPTIMIZED"]) + + assert result.exit_code == 0, result.output + assert bundle.call_count == 1 + + +def test_export_type_generic_forces_stock_even_on_npu_qnn(tmp_path: Path): + """``--export-type generic`` keeps the stock build for a registered family. + + Even the (otherwise routing) ``--ep qnn`` + NPU combination must fall through + to the single/composite pipeline when generic is explicitly requested. + """ + with ( + patch(_GENERATE_TARGET, return_value=_fake_config("qwen3")), + patch(_BUNDLE_TARGET) as bundle, + patch(_RUN_SINGLE_TARGET) as run_single, + patch(_COMPOSITE_TARGET, return_value=None), + ): + result = _invoke( + [ + "-m", + "Qwen/Qwen3-0.6B", + "-o", + str(tmp_path / "o"), + "--device", + "npu", + "--ep", + "qnn", + "--export-type", + "generic", + ] + ) + + assert result.exit_code == 0, result.output + bundle.assert_not_called() + run_single.assert_called_once() + + +def test_export_type_optimized_unregistered_family_errors(tmp_path: Path): + """Optimized on a family with no recipe fails fast.""" + with ( + patch( + _GENERATE_TARGET, + return_value=_fake_config("resnet", task="image-classification"), + ), + patch(_BUNDLE_TARGET) as bundle, + patch(_RUN_SINGLE_TARGET) as run_single, + patch(_COMPOSITE_TARGET, return_value=None), + ): + result = _invoke( + [ + "-m", + "microsoft/resnet-50", + "-o", + str(tmp_path / "o"), + "--export-type", + "optimized", + ] + ) + + assert result.exit_code != 0 + assert "no optimized recipe" in result.output + assert "resnet" in result.output + bundle.assert_not_called() + run_single.assert_not_called() + + +def test_resolve_optimized_target_infers_first_when_unpinned(): + from winml.modelkit.commands.build import _resolve_optimized_target + from winml.modelkit.models.winml import resolve_genai_bundle + + recipe = resolve_genai_bundle("qwen3") + ep, device = _resolve_optimized_target(_fake_ctx(set()), recipe, device="auto", ep=None) + assert (ep, device) == ("qnn", "npu") + + +def test_resolve_optimized_target_rejects_unsupported_ep(): + from winml.modelkit.commands.build import _resolve_optimized_target + from winml.modelkit.models.winml import resolve_genai_bundle + + recipe = resolve_genai_bundle("qwen3") + with pytest.raises(click.UsageError, match="--ep dml is not supported"): + _resolve_optimized_target(_fake_ctx({"ep"}), recipe, device="auto", ep="dml") + + +def test_resolve_optimized_target_rejects_unsupported_device(): + from winml.modelkit.commands.build import _resolve_optimized_target + from winml.modelkit.models.winml import resolve_genai_bundle + + recipe = resolve_genai_bundle("qwen3") + with pytest.raises(click.UsageError, match="--device cpu is not supported"): + _resolve_optimized_target(_fake_ctx({"device"}), recipe, device="cpu", ep=None) + + +def test_optimized_rejects_onnx_input(): + from winml.modelkit.commands.build import _maybe_build_genai_bundle + + with pytest.raises(click.UsageError, match=r"pre-exported \.onnx"): + _maybe_build_genai_bundle( + _fake_ctx({"export_type"}), + export_type="optimized", + model="model.onnx", + model_is_onnx=True, + config_or_configs=_fake_config("qwen3"), + preloaded_hf_config=None, + output_dir="out", + use_cache=False, + device="auto", + ep=None, + precision=None, + rebuild=False, + ) + + +def test_optimized_rejects_module_mode(): + from winml.modelkit.commands.build import _maybe_build_genai_bundle + + with pytest.raises(click.UsageError, match="module mode"): + _maybe_build_genai_bundle( + _fake_ctx({"export_type"}), + export_type="optimized", + model="Qwen/Qwen3-0.6B", + model_is_onnx=False, + config_or_configs=[_fake_config("qwen3")], + preloaded_hf_config=None, + output_dir="out", + use_cache=False, + device="auto", + ep=None, + precision=None, + rebuild=False, + ) + + +def test_optimized_requires_model(): + from winml.modelkit.commands.build import _maybe_build_genai_bundle + + with pytest.raises(click.UsageError, match="requires -m/--model"): + _maybe_build_genai_bundle( + _fake_ctx({"export_type"}), + export_type="optimized", + model=None, + model_is_onnx=False, + config_or_configs=_fake_config("qwen3"), + preloaded_hf_config=None, + output_dir="out", + use_cache=False, + device="auto", + ep=None, + precision=None, + rebuild=False, + ) diff --git a/tests/unit/models/winml/test_genai_bundle_orchestrator.py b/tests/unit/models/winml/test_genai_bundle_orchestrator.py index d2866acb8..194a62f79 100644 --- a/tests/unit/models/winml/test_genai_bundle_orchestrator.py +++ b/tests/unit/models/winml/test_genai_bundle_orchestrator.py @@ -21,6 +21,7 @@ from winml.modelkit.models.winml import ( GenaiBundleRecipe, GenaiCompanionSpec, + GenaiTarget, GenaiTransformerSpec, build_genai_bundle, ) @@ -63,6 +64,7 @@ def _make_recipe(assemble) -> GenaiBundleRecipe: ), ), assemble=assemble, + supported_targets=(GenaiTarget(ep="qnn", device="npu"),), transformer_onnx_passes=(_dummy_pass,), max_cache_len=2048, prefill_seq_len=64, diff --git a/tests/unit/models/winml/test_genai_bundle_registry.py b/tests/unit/models/winml/test_genai_bundle_registry.py index 08114cc32..a26ba08a9 100644 --- a/tests/unit/models/winml/test_genai_bundle_registry.py +++ b/tests/unit/models/winml/test_genai_bundle_registry.py @@ -17,6 +17,7 @@ from winml.modelkit.models.winml import ( GENAI_BUNDLE_REGISTRY, GenaiBundleRecipe, + GenaiTarget, GenaiTransformerSpec, register_genai_bundle, resolve_genai_bundle, @@ -64,8 +65,29 @@ def test_duplicate_registration_raises(): ), companions=(), assemble=lambda *_a, **_k: None, + supported_targets=(GenaiTarget(ep="qnn", device="npu"),), ) with pytest.raises(ValueError, match="already registered"): register_genai_bundle(dup) # The guard precedes insertion, so the real recipe is left untouched. assert resolve_genai_bundle("qwen3") is existing + + +def test_register_rejects_empty_supported_targets(): + """A recipe with no supported targets is a registration-time error.""" + recipe = GenaiBundleRecipe( + family="no-targets-fam", + transformer=GenaiTransformerSpec( + model_type="x", + task="text-generation", + precision="w8a16", + context_sub_model="a", + iterator_sub_model="b", + ), + companions=(), + assemble=lambda *_a, **_k: None, + supported_targets=(), + ) + with pytest.raises(ValueError, match="no supported_targets"): + register_genai_bundle(recipe) + assert "no-targets-fam" not in GENAI_BUNDLE_REGISTRY