feat(perf): auto-build a genai bundle from a model id for --runtime winml-genai#1109
Conversation
…inml-genai winml perf -m <hf-model-id> --runtime winml-genai previously required a prebuilt genai bundle directory. Mirror the winml runtime's on-the-fly build: when -m is a model id (not a bundle dir), build a genai bundle into the managed cache (ep=qnn/device=npu) and benchmark it, reusing a cached bundle unless --rebuild/--ignore-cache forces a fresh one. A bundle directory is still used as-is and an .onnx path is still rejected.
The model-id genai auto-build wrote into the managed cache even under --ignore-cache, collapsing --ignore-cache and --rebuild into one behavior -- unlike the winml runtime, where --ignore-cache builds in a throwaway temp dir and leaves the cache untouched (as documented in perf.md). Under --ignore-cache, build the auto-built bundle (and its component build cache) in a throwaway temp dir tied to an ExitStack so it outlives the benchmark and is removed after. --rebuild still overwrites the cached bundle; a plain run still reuses it. Also stop reporting --task/--precision/--rebuild/--ignore-cache as "ignored" when a bundle is auto-built from a model id -- those flags drive the build. A prebuilt bundle still ignores them all.
xieofxie
left a comment
xieofxie
left a comment
There was a problem hiding this comment.
Review Summary
Well-structured PR with clean logic, comprehensive test coverage, and good documentation. The auto-build pattern mirrors the existing winml runtime's on-the-fly behavior nicely. A few observations below (none blocking).
Key observations
-
Input routing heuristic (perf.py:1912-1925): When a model string looks like a filesystem path (e.g. user typo'd a directory) but doesn't exist,
is_dir()returns False and it falls into the auto-build branch. The resulting error is serviceable but could confuse users who clearly intended a local path. A cheap heuristic (check for path separators) could surface a friendlier "path does not exist" before attempting the HF lookup. -
Cache-hit integrity (perf.py:1836): The cache check keys solely on
genai_config.jsonexisting. A prior build interrupted mid-write could leave that file but an incomplete bundle. Either a lightweight sanity check (valid JSON + at least one.onnx) or a comment documenting why this is acceptable (GenaiSession will fail fast) would be helpful. -
--epand_GENAI_AUTOBUILD_FLAGS(perf.py:1760): The auto-build pinsep=qnn, but"ep"is not in_GENAI_AUTOBUILD_FLAGS. If a user passes--ep qnnalongside a model id, it gets warned as "ignored" even though it's consistent. Not a bug (correct from the benchmark-routing angle) but could cause minor user confusion. Worth considering. -
Dense table row (perf.md:11): The
--runtimedescription is quite long for a markdown table cell. Consider moving the detailed explanation below the table or to a subsection.
Positives
- ExitStack pattern for temp-dir lifetime management is elegant
- Test matrix is thorough: autobuild, cache-hit, rebuild, ignore-cache, warning suppression, recipe rejection
- Good error messages with actionable hints (pass a prebuilt bundle directory)
- Documentation tip box in the sample page is well-placed
Verdict: Approve — the concerns above are all minor/non-blocking.
On a genai auto-build cache hit, the bundle keyed by model id alone is reused as-is, so --precision/--task are not applied to it. Suppress their ignored-flags warning only when a fresh build actually ran; report them as ignored on a cache hit so the drop is not silent. --rebuild/--ignore-cache always force a build (never reuse), so they stay suppressed on any auto-build. Adds a cache-hit warning test.
3 participants
What
winml perf -m <hf-model-id> --runtime winml-genainow auto-builds a genai bundle from a HuggingFace model id instead of requiring a prebuilt bundle directory.Why
The
winmlruntime already auto-builds an ONNX from a model id on demand. Thewinml-genairuntime forced users to first runwinml build ... --device npu --ep qnnand then pointperfat the output directory. This mirrors the winml behavior so a bare model id "just works".How
_autobuild_genai_bundlehelper: resolves the model family, looks up its genai-bundle recipe, and builds into the managed cache dir (get_model_dir(model)/genai-bundle), pinningep=qnn/device=npu(genai bundles target the NPU HTP via QNN). A prior build is reused unless--rebuild/--ignore-cacheforces a fresh one._run_genai_runtimenow distinguishes inputs: a directory is used as-is (must containgenai_config.json), an.onnxpath is rejected, anything else is treated as a model id and auto-built on demand.perf.mdand theqwen3-genai-bundle.mdsample updated with the one-command flow.Unblocks
winml perf -m Qwen/Qwen3-0.6B --runtime winml-genai.