[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic#29682
Open
mingmingtasd wants to merge 1 commit into
Open
[WebGPU EP] Fix unnecessary CPU fallback and improve fallback diagnostic#29682mingmingtasd wants to merge 1 commit into
mingmingtasd wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
…tics Some nodes in MobileNet and the SD-Turbo pipeline were assigned to the CPU EP even though the WebGPU EP can run them, causing extra GPU<->CPU copies and, in the device-free compile-only flow (where CPU fallback is disabled), aborting session initialization outright. This adds the missing WebGPU kernel coverage and improves the diagnostics that explain why a node is not claimed by an EP. With these fixes all four SD-Turbo sessions (text encoder, UNet, VAE, safety checker) partition fully onto the WebGPU EP with zero CPU fallback. Kernel coverage fixes, by model: * MobileNet -- AveragePool (pool.cc, webgpu_execution_provider.cc): register kernels for opsets 11-18, 19-21 and 22, replacing the single opset-11 registration. Root cause: MobileNet (opset 21) uses AveragePool at SinceVersion 19. ORT clamps an open-ended SinceVersion(11) registration to [11, 18] (not [11, INT_MAX]), so the opset-19 node was not claimed and fell back to the CPU EP; the CPU-only NchwcTransformer (Level3) then amplified the single pool into 3 CPU nodes. * SD-Turbo text encoder (CLIP) -- Clip int32/uint32 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): register int32 and uint32 Clip for opsets 11-11, 12-12 and 13. Root cause: the CLIP text position-id / index subgraph uses integer Clip, which was previously CPU-only. * SD-Turbo text encoder (CLIP) -- Clip int64 (unary_elementwise_ops.cc, webgpu_execution_provider.cc): add a dedicated ClipInt64 kernel/shader for opsets 11-11, 12-12 and 13. Root cause: the WebNN-inserted index Clip before ArgMax (Inserted_Clip_951) is int64. The templated Clip supports only 4-byte element types (sizeof(T)==sizeof(float) static_assert), so int64 previously stayed on the CPU EP and, under compile-only (CPU fallback disabled), stopped the text encoder session from compiling. WebGPU has no native 64-bit integer type; the EP already stores int64 as vec2<u32> and operates on the truncated low 32 bits as i32 (shader_variable.cc GetByOffset/SetByOffset). Because Clip is monotonic, ClipInt64 clamps that i32 value (min/max saturated into i32 range) and sign-extends on write, consistent with the EP's existing int64 semantics. * SD-Turbo safety checker -- Cast to uint8 (cast.cc, shader_variable.cc): allow the WebGPU Cast to produce uint8 output. Add uint8 to the Cast output (T2) type constraint, a to-uint8 shader expression, and a Uint8x4 packing path in SetByOffset (4 uint8 packed per u32, low element -> low byte -- the same byte layout already used for bool output). Root cause: WebNN has no bool type and represents a bool graph output as uint8, so the safety checker ends in bool->uint8 casts (Inserted_Cast_972, Inserted_Cast_974). uint8 was not a supported WebGPU Cast output type, so these terminal casts fell back to the CPU EP. The upstream bool->uint8->int32 cast chains are separately collapsed by CastChainElimination (a Level1 rewrite rule enabled in the Chromium WebNN session options); the terminal casts have no downstream cast to fold into and must be produced directly, hence this uint8 output support. Diagnostics (these do not change placement; they make unexpected fallbacks diagnosable): * execution_provider.cc (IExecutionProvider::GetCapability): log (VERBOSE) when an EP has no kernel for a still-unassigned node, including op type, opset and domain. * kernel_lookup.h (LookUpKernel): capture and log (VERBOSE) the reason TryFindKernel failed (version vs. type-constraint mismatch, or no registration at all). The reason was previously built but discarded by the pointer-only return, leaving only an opaque "kernel not found". This identified Inserted_Clip_951 as int64 (not int32) and the safety checker fallbacks as uint8 casts. * session_state.cc (verbose node-placement dump): include the op set version and the input/output types for each node, so a fallback caused by an unsupported element type is obvious from the placement log.
b16a4c7 to
6c75b70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some nodes in MobileNet and the SD-Turbo pipeline were assigned to the CPU EP even though the WebGPU EP can run them, causing extra GPU<->CPU copies and, in the device-free compile-only flow (where CPU fallback is disabled), aborting session initialization outright. This adds the missing WebGPU kernel coverage and improves the diagnostics that explain why a node is not claimed by an EP. With these fixes all four SD-Turbo sessions (text encoder, UNet, VAE, safety checker) partition fully onto the WebGPU EP with zero CPU fallback.
Kernel coverage fixes, by model
MobileNet — AveragePool (
pool.cc,webgpu_execution_provider.cc): register kernels for opsets 11-18, 19-21 and 22, replacing the single opset-11 registration. Root cause: MobileNet (opset 21) uses AveragePool at SinceVersion 19. ORT clamps an open-endedSinceVersion(11)registration to[11, 18](not[11, INT_MAX]), so the opset-19 node was not claimed and fell back to the CPU EP; the CPU-only NchwcTransformer (Level3) then amplified the single pool into 3 CPU nodes.SD-Turbo text encoder (CLIP) — Clip int32/uint32 (
unary_elementwise_ops.cc,webgpu_execution_provider.cc): register int32 and uint32 Clip for opsets 11-11, 12-12 and 13. Root cause: the CLIP text position-id / index subgraph uses integer Clip, which was previously CPU-only.SD-Turbo text encoder (CLIP) — Clip int64 (
unary_elementwise_ops.cc,webgpu_execution_provider.cc): add a dedicatedClipInt64kernel/shader for opsets 11-11, 12-12 and 13. Root cause: the WebNN-inserted index Clip before ArgMax (Inserted_Clip_951) is int64. The templated Clip supports only 4-byte element types (sizeof(T)==sizeof(float)static_assert), so int64 previously stayed on the CPU EP and, under compile-only (CPU fallback disabled), stopped the text encoder session from compiling. WebGPU has no native 64-bit integer type; the EP already stores int64 asvec2<u32>and operates on the truncated low 32 bits as i32 (shader_variable.ccGetByOffset/SetByOffset). Because Clip is monotonic,ClipInt64clamps that i32 value (min/max saturated into i32 range) and sign-extends on write, consistent with the EP's existing int64 semantics.SD-Turbo safety checker — Cast to uint8 (
cast.cc,shader_variable.cc): allow the WebGPU Cast to produce uint8 output. Add uint8 to the Cast output (T2) type constraint, a to-uint8 shader expression, and aUint8x4packing path in SetByOffset (4 uint8 packed per u32, low element -> low byte -- the same byte layout already used for bool output). Root cause: WebNN has no bool type and represents a bool graph output as uint8, so the safety checker ends in bool->uint8 casts (Inserted_Cast_972,Inserted_Cast_974). uint8 was not a supported WebGPU Cast output type, so these terminal casts fell back to the CPU EP. Non-terminal bool->uint8->int32 cast chains can be collapsed upstream by CastChainElimination (a Level1 rewrite rule), but a terminal cast has no downstream cast to fold into and must be produced directly, hence this uint8 output support.Diagnostics
These do not change placement; they make unexpected fallbacks diagnosable.
execution_provider.cc(IExecutionProvider::GetCapability): log (VERBOSE) when an EP has no kernel for a still-unassigned node, including op type, opset and domain.kernel_lookup.h(LookUpKernel): capture and log (VERBOSE) the reasonTryFindKernelfailed (version vs. type-constraint mismatch, or no registration at all). The reason was previously built but discarded by the pointer-only return, leaving only an opaque "kernel not found". This identifiedInserted_Clip_951as int64 (not int32) and the safety checker fallbacks as uint8 casts.session_state.cc(verbose node-placement dump): include the op set version and the input/output types for each node, so a fallback caused by an unsupported element type is obvious from the placement log.