[FLINK-39957][table] Support EXPLAIN of CREATE [OR ALTER] MATERIALIZED TABLE by raminqaf · Pull Request #28510 · apache/flink

raminqaf · 2026-06-22T16:59:56Z

What is the purpose of the change

EXPLAIN previously rejected materialized table DDL, so users had no way to inspect the plan that populates a materialized table without actually running it. This change adds EXPLAIN support for CREATE MATERIALIZED TABLE ... AS, CREATE OR ALTER MATERIALIZED TABLE ... AS (both the alter-on-existing and the in-place
table-to-materialized-table conversion path from FLINK-39847), and ALTER MATERIALIZED TABLE ... AS. The produced plan is the sink that writes the definition query into the materialized table, mirroring how CTAS/RTAS are explained.

EXPLAIN stays side-effect-free: no table is created, altered, or converted, and no refresh job is submitted.

Note on scope: building the sink for a brand-new managed materialized table at EXPLAIN time is not covered, because a managed table's storage options (e.g. path) are only assigned by the catalog at creation time. The alter and conversion paths work because they reuse the existing table's already-resolved connector
options. Resolving managed options at plan time without persisting is left as a follow-up.

Brief change log

Parser: allow CREATE/ALTER MATERIALIZED TABLE statements after EXPLAIN, and reject a non-AS ALTER under EXPLAIN with a clear message.
Make the materialized table create/alter/convert operations translatable as ModifyOperation, carrying the validated AS query as the sink input; ModifyOperationVisitor gains the corresponding visit methods.
Planner: convert the inner materialized table statement when explaining, and translate these operations into the sink RelNode that writes into the resulting table (reusing the existing table's resolved options for alter/convert).
Gateway: guard operation dispatch so directly-executed materialized table statements still route to MaterializedTableManager rather than the generic ModifyOperation translation path (the new ModifyOperation role only enables EXPLAIN, it must not change execution).

Verifying this change

This change added tests and can be verified as follows:

Parser tests for EXPLAIN of CREATE/CREATE OR ALTER/ALTER MATERIALIZED TABLE, including the rejection of a non-AS ALTER.
Planner plan tests (TableSinkTest) asserting the AST / optimized physical / optimized execution plans for the create and alter variants.
Gateway integration tests asserting the full EXPLAIN plan end-to-end (against golden resources) for ALTER ... AS, CREATE OR ALTER on an existing materialized table, and the table-to-materialized-table conversion, plus assertions that EXPLAIN leaves the catalog object unchanged.
Existing materialized table create/alter/convert ITs continue to pass, confirming direct execution dispatch is unchanged.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented (a follow-up should note EXPLAIN support on the materialized table / EXPLAIN statement docs)

AI-tooling disclosure

Generated cleanups by Claude Opus 4.8

…LE in the SQL parser

…translatable as ModifyOperation

…R to a sink RelNode for EXPLAIN

… manager and add EXPLAIN ITs

…-table conversion

flinkbot · 2026-06-22T17:15:08Z

CI report:

fa75dc3 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

snuyanzin · 2026-06-22T17:20:19Z

            throw new SqlExecutionException(
                    "No Statement Set to submit. 'END' statement should be used after 'BEGIN STATEMENT SET'.");
-        } else if (op instanceof ModifyOperation) {
+        } else if (op instanceof ModifyOperation && !(op instanceof MaterializedTableOperation)) {


is there a reason MaterializedTableOperation is not a ModifyOperation

however after that we call callModifyOperations ?

MaterializedTableOperation is a marker shared by SUSPEND, RESUME, REFRESH and DROP too. Those have no definition query, so they can't implement the ModifyOperation contract (getChild() / accept(visitor)) in any meaningful way. More importantly, if the marker were a ModifyOperation, every MT op would fall into the callModifyOperations branch and execute as an INSERT.

That's what the && !(op instanceof MaterializedTableOperation) guard enforces. So only the query-bearing operations implement ModifyOperation, and purely so the planner can translate them into a sink for EXPLAIN. Execution still goes through the manager for all of them.

snuyanzin · 2026-06-22T17:27:41Z

+     * golden resource. The per-test catalog name is normalized to {@code $CATALOG} before
+     * comparison.
+     */
+    static void verifyExplainPlan(


why do we need to create another plan verifier if there is already existing framework for this?
See TableTestBase

Good point. I wanted to do the verification in the IT cases of the Gateway. Now, I removed the custom helper and moved plan-content verification into the planner tests.

Gateway ITs no longer assert plan text. They only check that EXPLAIN returns a single non-empty plan, succeeds, and is is side-effect-free.
Dropped MaterializedTableTestUtils.verifyExplainPlan and the gateway golden files

snuyanzin · 2026-06-22T17:31:22Z

+    public static PlannerQueryOperation validateAndConvertAsQuery(
+            SqlNode asQuery, ConvertContext context) {
+        final FlinkPlannerImpl flinkPlanner = context.getFlinkPlanner();
+        final SqlNode validated = flinkPlanner.validate(asQuery);
+        final Operation operation =
+                SqlNodeToOperationConversion.convert(
+                                flinkPlanner, context.getCatalogManager(), validated)
+                        .orElseThrow(
+                                () ->
+                                        new TableException(


method has pretty generic name, then somewhere in the middle it throws
"Unsupported materialized table definition query: "
either make it generic or may be rename and move into. materializedTableUtils

I will make it more generic. We can even replace the AsQuery validation of RTAS and CTAS with this method. Should I do it in this PR as a separate commit or leave the code there untouched?

…t for CTAS & RTAS

snuyanzin · 2026-06-24T07:23:12Z

+                null,
+                isOverwrite,
+                sink,
+                null); // conflictStrategy


what does this comment mean?

snuyanzin · 2026-06-24T07:23:32Z

+        return convertSinkToRel(
+                relBuilder,
+                input,
+                Collections.emptyMap(),


Suggested change

Collections.emptyMap(),

Map.of(),

here and in other places

raminqaf added 5 commits June 22, 2026 16:11

[FLINK-39957][table] Support EXPLAIN of CREATE/ALTER MATERIALIZED TAB…

7a0b485

…LE in the SQL parser

[FLINK-39957][table] Make materialized table CREATE/ALTER operations …

cf2643a

…translatable as ModifyOperation

[FLINK-39957][table-planner] Translate materialized table CREATE/ALTE…

fd4a3dc

…R to a sink RelNode for EXPLAIN

[FLINK-39957][sql-gateway] Route materialized table operations to the…

34091c4

… manager and add EXPLAIN ITs

[FLINK-39957][table-planner] Support EXPLAIN of table-to-materialized…

a128d52

…-table conversion

snuyanzin reviewed Jun 22, 2026

View reviewed changes

Comment thread .../src/test/java/org/apache/flink/table/gateway/service/MaterializedTableConversionITCase.java Outdated

snuyanzin reviewed Jun 22, 2026

View reviewed changes

raminqaf added 3 commits June 23, 2026 09:27

[FLINK-39957][sql-gateway] Simplify EXPLAIN IT cases assertion

106f471

[FLINK-39957][table-planner] Add EXPLAIN convert table to mt test

75c00a4

[FLINK-39957][table] Make validateAndConvertAsQuery generic and use i…

fa75dc3

…t for CTAS & RTAS

snuyanzin reviewed Jun 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-39957][table] Support EXPLAIN of CREATE [OR ALTER] MATERIALIZED TABLE#28510

[FLINK-39957][table] Support EXPLAIN of CREATE [OR ALTER] MATERIALIZED TABLE#28510
raminqaf wants to merge 8 commits into
apache:masterfrom
raminqaf:os-flink-explain-create-mt

raminqaf commented Jun 22, 2026

Uh oh!

flinkbot commented Jun 22, 2026 •

edited

Loading

Uh oh!

snuyanzin Jun 22, 2026

Uh oh!

raminqaf Jun 23, 2026

Uh oh!

Uh oh!

snuyanzin Jun 22, 2026

Uh oh!

raminqaf Jun 23, 2026

Uh oh!

snuyanzin Jun 22, 2026

Uh oh!

raminqaf Jun 23, 2026 •

edited

Loading

Uh oh!

snuyanzin Jun 24, 2026

Uh oh!

snuyanzin Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

raminqaf commented Jun 22, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

AI-tooling disclosure

Uh oh!

flinkbot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

snuyanzin Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

snuyanzin Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

snuyanzin Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

flinkbot commented Jun 22, 2026 •

edited

Loading

raminqaf Jun 23, 2026 •

edited

Loading