Skip to content

[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic#29682

Open
mingmingtasd wants to merge 1 commit into
microsoft:mainfrom
mingmingtasd:pr-webgpu-cpu-fallback-fix
Open

[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic#29682
mingmingtasd wants to merge 1 commit into
microsoft:mainfrom
mingmingtasd:pr-webgpu-cpu-fallback-fix

Conversation

@mingmingtasd

@mingmingtasd mingmingtasd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Some nodes in MobileNet and the SD-Turbo pipeline were assigned to the CPU EP even though the WebGPU EP can run them, causing extra GPU<->CPU copies and, in the device-free compile-only flow (where CPU fallback is disabled), aborting session initialization outright. This adds the missing WebGPU kernel coverage and improves the diagnostics that explain why a node is not claimed by an EP. With these fixes all four SD-Turbo sessions (text encoder, UNet, VAE, safety checker) partition fully onto the WebGPU EP with zero CPU fallback.

Kernel coverage fixes, by model

MobileNet — AveragePool (pool.cc, webgpu_execution_provider.cc): register kernels for opsets 11-18, 19-21 and 22, replacing the single opset-11 registration. Root cause: MobileNet (opset 21) uses AveragePool at SinceVersion 19. ORT clamps an open-ended SinceVersion(11) registration to [11, 18] (not [11, INT_MAX]), so the opset-19 node was not claimed and fell back to the CPU EP; the CPU-only NchwcTransformer (Level3) then amplified the single pool into 3 CPU nodes.

SD-Turbo text encoder (CLIP) — Clip int32/uint32 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): register int32 and uint32 Clip for opsets 11-11, 12-12 and 13. Root cause: the CLIP text position-id / index subgraph uses integer Clip, which was previously CPU-only.

SD-Turbo text encoder (CLIP) — Clip int64 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): add a dedicated ClipInt64 kernel/shader for opsets 11-11, 12-12 and 13. Root cause: the WebNN-inserted index Clip before ArgMax (Inserted_Clip_951) is int64. The templated Clip supports only 4-byte element types (sizeof(T)==sizeof(float) static_assert), so int64 previously stayed on the CPU EP and, under compile-only (CPU fallback disabled), stopped the text encoder session from compiling. WebGPU has no native 64-bit integer type; the EP already stores int64 as vec2<u32> and operates on the truncated low 32 bits as i32 (shader_variable.cc GetByOffset/SetByOffset). Because Clip is monotonic, ClipInt64 clamps that i32 value (min/max saturated into i32 range) and sign-extends on write, consistent with the EP's existing int64 semantics.

SD-Turbo safety checker — Cast to uint8 (cast.cc, shader_variable.cc): allow the WebGPU Cast to produce uint8 output. Add uint8 to the Cast output (T2) type constraint, a to-uint8 shader expression, and a Uint8x4 packing path in SetByOffset (4 uint8 packed per u32, low element -> low byte -- the same byte layout already used for bool output). Root cause: WebNN has no bool type and represents a bool graph output as uint8, so the safety checker ends in bool->uint8 casts (Inserted_Cast_972, Inserted_Cast_974). uint8 was not a supported WebGPU Cast output type, so these terminal casts fell back to the CPU EP. Non-terminal bool->uint8->int32 cast chains can be collapsed upstream by CastChainElimination (a Level1 rewrite rule), but a terminal cast has no downstream cast to fold into and must be produced directly, hence this uint8 output support.

Diagnostics

These do not change placement; they make unexpected fallbacks diagnosable.

  • execution_provider.cc (IExecutionProvider::GetCapability): log (VERBOSE) when an EP has no kernel for a still-unassigned node, including op type, opset and domain.
  • kernel_lookup.h (LookUpKernel): capture and log (VERBOSE) the reason TryFindKernel failed (version vs. type-constraint mismatch, or no registration at all). The reason was previously built but discarded by the pointer-only return, leaving only an opaque "kernel not found". This identified Inserted_Clip_951 as int64 (not int32) and the safety checker fallbacks as uint8 casts.
  • session_state.cc (verbose node-placement dump): include the op set version and the input/output types for each node, so a fallback caused by an unsupported element type is obvious from the placement log.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

…tics

Some nodes in MobileNet and the SD-Turbo pipeline were assigned to the CPU EP
even though the WebGPU EP can run them, causing extra GPU<->CPU copies and, in
the device-free compile-only flow (where CPU fallback is disabled), aborting
session initialization outright. This adds the missing WebGPU kernel coverage
and improves the diagnostics that explain why a node is not claimed by an EP.
With these fixes all four SD-Turbo sessions (text encoder, UNet, VAE, safety
checker) partition fully onto the WebGPU EP with zero CPU fallback.

Kernel coverage fixes, by model:

* MobileNet -- AveragePool (pool.cc, webgpu_execution_provider.cc): register
  kernels for opsets 11-18, 19-21 and 22, replacing the single opset-11
  registration.
  Root cause: MobileNet (opset 21) uses AveragePool at SinceVersion 19. ORT
  clamps an open-ended SinceVersion(11) registration to [11, 18] (not
  [11, INT_MAX]), so the opset-19 node was not claimed and fell back to the CPU
  EP; the CPU-only NchwcTransformer (Level3) then amplified the single pool into
  3 CPU nodes.

* SD-Turbo text encoder (CLIP) -- Clip int32/uint32 (unary_elementwise_ops.cc,
  webgpu_execution_provider.cc): register int32 and uint32 Clip for opsets
  11-11, 12-12 and 13.
  Root cause: the CLIP text position-id / index subgraph uses integer Clip,
  which was previously CPU-only.

* SD-Turbo text encoder (CLIP) -- Clip int64 (unary_elementwise_ops.cc,
  webgpu_execution_provider.cc): add a dedicated ClipInt64 kernel/shader for
  opsets 11-11, 12-12 and 13.
  Root cause: the WebNN-inserted index Clip before ArgMax (Inserted_Clip_951)
  is int64. The templated Clip supports only 4-byte element types
  (sizeof(T)==sizeof(float) static_assert), so int64 previously stayed on the
  CPU EP and, under compile-only (CPU fallback disabled), stopped the text
  encoder session from compiling. WebGPU has no native 64-bit integer type; the
  EP already stores int64 as vec2<u32> and operates on the truncated low 32 bits
  as i32 (shader_variable.cc GetByOffset/SetByOffset). Because Clip is monotonic,
  ClipInt64 clamps that i32 value (min/max saturated into i32 range) and
  sign-extends on write, consistent with the EP's existing int64 semantics.

* SD-Turbo safety checker -- Cast to uint8 (cast.cc, shader_variable.cc): allow
  the WebGPU Cast to produce uint8 output. Add uint8 to the Cast output (T2)
  type constraint, a to-uint8 shader expression, and a Uint8x4 packing path in
  SetByOffset (4 uint8 packed per u32, low element -> low byte -- the same byte
  layout already used for bool output).
  Root cause: WebNN has no bool type and represents a bool graph output as
  uint8, so the safety checker ends in bool->uint8 casts (Inserted_Cast_972,
  Inserted_Cast_974). uint8 was not a supported WebGPU Cast output type, so
  these terminal casts fell back to the CPU EP. The upstream bool->uint8->int32
  cast chains are separately collapsed by CastChainElimination (a Level1 rewrite
  rule enabled in the Chromium WebNN session options); the terminal casts have
  no downstream cast to fold into and must be produced directly, hence this
  uint8 output support.

Diagnostics (these do not change placement; they make unexpected fallbacks
diagnosable):
* execution_provider.cc (IExecutionProvider::GetCapability): log (VERBOSE) when
  an EP has no kernel for a still-unassigned node, including op type, opset and
  domain.
* kernel_lookup.h (LookUpKernel): capture and log (VERBOSE) the reason
  TryFindKernel failed (version vs. type-constraint mismatch, or no registration
  at all). The reason was previously built but discarded by the pointer-only
  return, leaving only an opaque "kernel not found". This identified
  Inserted_Clip_951 as int64 (not int32) and the safety checker fallbacks as
  uint8 casts.
* session_state.cc (verbose node-placement dump): include the op set version and
  the input/output types for each node, so a fallback caused by an unsupported
  element type is obvious from the placement log.
@mingmingtasd mingmingtasd force-pushed the pr-webgpu-cpu-fallback-fix branch from b16a4c7 to 6c75b70 Compare July 14, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant