maria: require resync on partial row events by dtunikov · Pull Request #4530 · PeerDB-io/peerdb

dtunikov · 2026-07-02T11:24:05Z

We don't support partial row events. MariaDB 12.3+ emits those when payload size is bigger than binlog_row_event_fragment_threshold (default 1Gb).
This PR handles this event type and requires a resync.
In addition, it introduces UnsupportedBinlogEventCounter(eventType) metric to keep track of skipped events.

claude · 2026-07-02T11:29:32Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

- add metric to count unsupported event types

jgao54 · 2026-07-02T20:02:47Z

+		"unsupported MariaDB PARTIAL_ROW_DATA_EVENT (type %d): fragmented oversized row events cannot be processed; a resync is required",
+		e.EventType)


nit: would be good to also supply schema/table name

I see that generic event doesn't provide this out-of-the-box and implementing our own parser may be out-of-scope, although technically possible according to Claude:

Pull the table ID out of the first fragment. The first fragment's content embeds the original rows event verbatim: its own 19-byte common header, then the rows-event post-header, which begins with a 6-byte little-endian table ID (same layout RowsEvent.Decode reads at row_event.go:1021). So inside ev.Data: // ev.Data layout for PARTIAL_ROW_DATA_EVENT (172): // [0:4] seq_no, [4:8] total_fragments, [8] flags, // [9:17] orig size (only if flags&0x01, i.e. first fragment), then content. pos := 9 if ev.Data[8]&0x01 != 0 { pos += 8 } content := ev.Data[pos:] // content = original rows event: 19-byte header, then post-header tableID := mysql.FixedLengthInt(content[19 : 19+6])

We would also start failing on tables outside of the pipe now without parsing the table name. Could either handle all at once (parsing and skipping if not included) or see if an escalation comes and handle both then

added logic to filter out partial raw failures related to the tables that aren't included in the pipe
the code is not very simple, but the e2e test covers it well
i can rollback if you think that it's too much complexity for such feature (as we actually don't know how often this event happens in production yet)

Trade-off seems ok to me, given the user-friendliness benefits. Thanks for doing the work

jgao54

a few nit comment but lgtm overall

ilidemi · 2026-07-03T00:30:32Z

+		"unsupported MariaDB PARTIAL_ROW_DATA_EVENT (type %d): fragmented oversized row events cannot be processed; a resync is required",
+		e.EventType)


We would also start failing on tables outside of the pipe now without parsing the table name. Could either handle all at once (parsing and skipping if not included) or see if an escalation comes and handle both then

…t error class

codecov · 2026-07-03T13:51:53Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2954	2	2952	343

View the top 1 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent

Stack Traces | 195s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
2026/07/03 13:45:02 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_xsk2ix5z.test_mysql_schema_changes
    clickhouse_mysql_test.go:1957: WaitFor waiting for in-pipe row past out-of-pipe partial row event 2026-07-03 13:45:16.564518412 +0000 UTC m=+485.130611153
2026/07/03 13:45:16 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_1iwesbnb.test_blobs
2026/07/03 13:45:16 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_6zelra5m.test_bit
    clickhouse_mysql_test.go:1957: UNEXPECTED TIMEOUT waiting for in-pipe row past out-of-pipe partial row event 2026-07-03 13:48:17.177314151 +0000 UTC m=+665.743406874
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent (194.78s)

View the full list of 1 ❄️ flaky test(s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH

Flake rate in main: 14.29% (Passed 24 times, Failed 4 times)

Stack Traces | 0.01s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH (0.01s)
2026/07/03 13:49:14 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/03 13:49:14 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_machcl_7ipudwcu.test_composite_pkey_ordering

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-07-03T13:55:51Z

❌ Test Failure

Analysis: The PR's own new test Test_MariaDB_PartialRowEvent fails deterministically across all three matrix jobs (mysql-gtid/pos, MySQL 6.0/7.0/8.0) after ~190s, indicating a real bug in the PR's partial-row-event resync logic rather than a flaky failure.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

ilidemi · 2026-07-07T02:12:57Z

+			attribute.String(otel_metrics.BinlogEventTypeKey, fmt.Sprintf("%s_%d", prefix, int(event.Header.EventType))),
+		)))
+		if _, loaded := c.warnedUnsupportedEventTypes.LoadOrStore(event.Header.EventType, struct{}{}); !loaded {
+			c.logger.Warn("unsupported rows event", slog.Any("type", event.Header.EventType))


As we're now warning on all unhandled events, will need to add a whitelist of benign events. Could be a follow-up once we see logs, or sweep go-mysql code beforehand and add an empty case branch

Also is it "unsupported rows event" anymore

added benign cases:
1546491
wdyt?

ilidemi · 2026-07-07T02:40:10Z

+		"unsupported MariaDB PARTIAL_ROW_DATA_EVENT (type %d): fragmented oversized row events cannot be processed; a resync is required",
+		e.EventType)


Trade-off seems ok to me, given the user-friendliness benefits. Thanks for doing the work

- add AdditionalAttributes to classified error

…ysql-fail-on-maria-partial-row-data-event

fail loudly on partial row maria event

a98a696

dtunikov requested a review from a team as a code owner July 2, 2026 11:24

- classify error

ce8eca6

- add metric to count unsupported event types

dtunikov changed the title ~~fail loudly on partial row maria event~~ maria: require resync on partial row events Jul 2, 2026

dtunikov added 2 commits July 2, 2026 14:11

e2e test for partial row event + test container helper for e2e tests

50ae9f7

cleanup comments

5e96c0f

dtunikov commented Jul 2, 2026

View reviewed changes

Comment thread flow/connectors/mysql/cdc.go Outdated

dtunikov commented Jul 2, 2026

View reviewed changes

Comment thread flow/connectors/mysql/cdc.go Outdated

dtunikov commented Jul 2, 2026

View reviewed changes

Comment thread flow/e2e/clickhouse_mysql_test.go

github-advanced-security AI found potential problems Jul 2, 2026

View reviewed changes

Comment thread flow/e2e/mysql.go Fixed

dtunikov commented Jul 2, 2026

View reviewed changes

Comment thread flow/alerting/classifier.go Outdated

fix lint errs

7ef6ee3

jgao54 reviewed Jul 2, 2026

View reviewed changes

Comment thread flow/e2e/clickhouse_mysql_test.go Outdated

jgao54 approved these changes Jul 2, 2026

View reviewed changes

ilidemi reviewed Jul 3, 2026

View reviewed changes

dtunikov added 3 commits July 3, 2026 14:25

classify "NOTIFY_BINLOG_EVENT_EXCEEDED_MAX_ALLOWED_PACKET" as distinc…

1d13df1

…t error class

event type label as integer

d91b0b5

don't fail if partial row event belong to out of pipe table

110681c

fix parsePartialRowEventTableID

48bf84d

dtunikov commented Jul 3, 2026

View reviewed changes

Comment thread flow/connectors/mysql/cdc_test.go Outdated

common resource flags for mysql/maria test container

730295c

dtunikov commented Jul 6, 2026

View reviewed changes

Comment thread flow/connectors/mysql/cdc.go

dtunikov added 2 commits July 6, 2026 15:51

record other unsupported event types

71770d4

don't warn on the same event type twice

52d8a01

ilidemi approved these changes Jul 7, 2026

View reviewed changes

- fix logs

87758fb

- add AdditionalAttributes to classified error

dtunikov added 4 commits July 7, 2026 13:53

rm partial row event unit test

53126f2

add benign events handling to avoid warnings on those

1546491

simplify additional error attrs

a4afe9d

Merge branch 'main' of github.com:PeerDB-io/peerdb into fix/dbi-815/m…

96a9c7a

…ysql-fail-on-maria-partial-row-data-event

ilidemi approved these changes Jul 8, 2026

View reviewed changes

		"unsupported MariaDB PARTIAL_ROW_DATA_EVENT (type %d): fragmented oversized row events cannot be processed; a resync is required",
		e.EventType)

Uh oh!

Conversation

dtunikov commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Jul 2, 2026

Code review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jgao54 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Jul 3, 2026

❌ Test Failure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dtunikov commented Jul 2, 2026 •

edited

Loading

codecov Bot commented Jul 3, 2026 •

edited

Loading