[Bug] BE write path wedges: a BE-to-BE load-stream brpc socket (8060) goes "Broken" and is never revived (4.0.6, coupled mode)

[ojalberts-itc]

Search before asking

• I had searched in the issues and found no similar issues.

Version

We are on Apache Doris 4.0.6 GA (x86_64, AVX2). The running build string reads doris-4.0.6-rc02,
which is misleading -- it is the GA, not a hand-picked release candidate. The official
apache-doris-4.0.6-bin-x64.tar.gz was cut from the 4.0.6-rc02 tag and the embedded build string
was never bumped to drop the -rc02 suffix. We verified the artifact two independent ways so this
is not dismissed as an RC:

1. Build-commit match. Our binary reports commit 1663f25c16f; the Apache Doris 4.0.6 release is
   commit 1663f25. A binary can only embed that commit if it was built from that exact tree.
2. Cross-mirror sha512 match. apache-doris-4.0.6-bin-x64.tar.gz.sha512 is byte-identical on the
   official release host and the mirror our deploy pulls from:
   8f869c4399088d3dc34e5ade10047495e42c7c0583fb32156adaf0794a56e5942b8c0142c05fc145d58d4148daf0ee8d0dde73c9aab0224f39b2435f406c8ef8.

MySQL-wire version reported: 5.7.99. We have not yet tested 4.1.x.

What's Wrong?

On a coupled-mode 4.0.6 cluster, the BE write path wedges. INSERT / CREATE TABLE AS SELECT /
MV-refresh hang and then fail with failed to write enough replicas N/M ... due to connection errors, while every node still reports Alive=true and reads keep working. Once wedged, only a
full restart of all BE processes recovers it -- a single-BE restart does not.

We chased this for a full day with live instrumentation and found three distinct causes, not one.
Two were ours and are fixed. The third is the reason for this report: a BE-to-BE load-stream brpc
socket on port 8060 goes "Broken" and is never revived, and we cannot fix it from the 4.0.6 config
surface. We believe it is an upstream defect of the Apache brpc
#1168 class.

#	Cause	Trigger	Status
1	Our security group was missing a BE-to-BE self-ingress on port 8040 (webserver_port, clone snapshot download), so clone REPAIR could never complete and the FE ran an unbounded repair-clone storm	replica repair / single-BE restart	Fixed -- our IaC bug, not Doris
2	brpc load-stream-open stall on 8060 under heavy multi-replica load	heavy multi-replica INSERT	Mitigated with experimental_enable_single_replica_insert=true
3	A BE-to-BE load-stream brpc socket (8060) goes "Broken" and is never revived	accumulation of ~6--7 BE-to-BE stream opens	Open -- suspected upstream defect, this report

Cause #1 was our mistake. We include it so it is clear the residual (cause #3) is independent of it:
after we fixed the security group and clones completed cleanly, cause #3 still reproduces.

The bug (cause #3)

On a healthy cluster, repl=3 writes succeed. After roughly 6--7 successful BE-to-BE load-stream-open
operations, a specific brpc socket on the BE-to-BE load-stream path (8060) enters a state where the
next load-stream-open RPC to the affected peer parks until the RPC timeout (~534 s) and then fails,
taking down the whole write path. The socket is never revived. Reads and the Thrift heartbeat (9050,
a separate threadpool) stay healthy the entire time, so SHOW BACKENDS shows Alive=true
throughout.

Signature (verbatim, BE be.WARNING and FE fe.log)

Coordinator-side open failure -- 60s is the tablet_writer_open_rpc_timeout_sec default:

load_stream_stub.cpp:591 open stream failed: [INTERNAL_ERROR]Failed to connect to backend <id>:
  [E1008]Reached timeout=60000ms @10.0.0.105:8060

The long park -- the RPC itself stalls ~534s before failing:

brpc_closure.h RPC meet failed: [E1008]Reached timeout=533999ms @10.0.0.105:8060
brpc_closure.h RPC meet failed: [E1008]Reached timeout=533999ms @10.0.0.118:8060
brpc_closure.h RPC meet failed: [E1008]Reached timeout=533999ms @10.0.0.155:8060

Loopback proof that this is in-process, not network or security group -- a BE times out opening a
tablet-writer to its own 8060, and cancels a load-stream whose source and destination are the same
BE:

load_id=..., txn_id=6, node=10.0.0.118:8060, open failed, err: ... RPC call is timed out,
  error_text=[E1008]Reached timeout=60000ms @10.0.0.118:8060, host: 10.0.0.118

load_stream_stub ... src_id=...499, dst_id=...499, stream_id=1740 is cancelled ...
  write enough replicas 1/3

brpc_client_cache.h:326 open brpc connection to 10.0.0.105:8060 failed:
  [E1008]Reached timeout=60000ms

User-facing FE error:

failed to open DeltaWriter <id>: failed to write enough replicas 1/3 for tablet <id>
  due to connection errors

At the original bring-up wedge the same error read ... 0/1 ....

In-process capture at a live wedge (pstack + bvar)

We captured the in-process BE state during a live wedge (2026-06-22, doris-4.0.6-rc02, commit
1663f25c16f), before the recovery restart. Full dumps are attached.

A write thread (gstack, BE A) is parked in the brpc load-stream OPEN -- the V2 path:

bthread_id_join
 -> brpc::Channel::CallMethod
 -> doris::FailureDetectChannel::CallMethod        be/src/util/brpc_client_cache.h:121
 -> doris::LoadStreamStub::open                    be/src/vec/sink/load_stream_stub.cpp:195   (txn_id=4442, total_streams=2, idle_timeout_ms=30000)
 -> doris::LoadStreamStubs::open                   be/src/vec/sink/load_stream_stub.cpp:574
 -> doris::vectorized::VTabletWriterV2::_open_streams_to_backend   be/src/vec/sink/writer/vtablet_writer_v2.cpp:317
 -> doris::vectorized::VTabletWriterV2::_open_streams              vtablet_writer_v2.cpp:296
 -> doris::vectorized::VTabletWriterV2::open                       vtablet_writer_v2.cpp:272
 -> doris::vectorized::AsyncResultWriter::process_block            be/src/vec/sink/writer/async_result_writer.cpp:119
 -> doris::vectorized::AsyncResultWriter::start_writer             async_result_writer.cpp:105
 -> doris::ThreadPool::dispatch_thread

On another BE the same root appears via the V1 writer path -- VNodeChannel::open_wait
(vtablet_writer.cpp:704) -> bthread_id_join. Both are parked on the brpc load-stream OPEN RPC to
a peer backend.

It is not worker-pool exhaustion, not compaction, not a stub leak -- brpc /vars (8060) and
/metrics (8040) at the wedge:

BE	bthread_worker_usage / count	load_channel_count	tablet_writer_count	brpc_stream_endpoint_stub_count	compaction (base+cumulative)
A	0.20 / 256	2	8	4	0
B	54.6 / 256	3	9	4	0

Workers are nowhere near the 256 ceiling -- the write threads are parked on the RPC, not starved.
Load channels and tablet writers are open and stuck; the stub count is the steady-state 4 (no leak);
compaction is fully idle. rpcz was empty (off by default; :8060/rpcz/enable did not enable it at
runtime on this build), so the parked-RPC evidence is the gstack above.

load_stream_stub cancellations appear across all BEs for the same load.

Trigger: accumulation, not a timer or idle decay

• Across two instrumented runs the wedge fired after 7 OK then wedge, and 6 OK then wedge, repl=3
  write operations. It tracks the number of BE-to-BE load-stream opens, not a wall-clock interval.
• It fires both during a heavy multi-replica load and ~7--11 minutes after a load while the cluster
  is otherwise idle (no further writes issued).
• A restart followed by 60 minutes of pure idle with no load did not wedge. So it is load-induced,
  not idle decay.
• brpc_stream_endpoint_stub_count stayed at 4 across the wedge -- no stub-count leak. It is a
  specific socket going Broken, not stub exhaustion.

Recovery

A full restart of all BE processes clears it. A single-BE restart does not -- the rejoined BE's peers
still hold the broken stub, so it rejoins a wedged mesh.

What You Expected?

When a BE-to-BE load-stream brpc connection breaks, brpc should revive it (or the load-stream-open
RPC should fail fast and the channel reconnect), so the write path recovers on its own. Instead the
open RPC parks ~534s and the entire write path wedges while every node still reports Alive=true, and
only a full BE-fleet restart clears it. A single broken socket should not require dropping all BE
processes to recover.

How to Reproduce?

1. Coupled-mode 4.0.6 cluster, 3 FE + 4 BE, default replication 3, stock be.conf.
2. Create native UNIQUE-KEY merge-on-write tables, replication_num=3,
   DISTRIBUTED BY HASH(...) BUCKETS 16.
3. Run a sequence of multi-replica writes that each open BE-to-BE load streams -- repeated
   INSERT ... SELECT / CREATE TABLE AS SELECT of a few million rows. In our case, four such loads
   plus a handful of UPDATE ... FROM statements per cycle.
4. After ~6--7 such operations -- during the load, or within ~7--11 minutes after -- a write hangs and
   fails write enough replicas N/3 ... connection errors. be.WARNING shows
   [E1008]Reached timeout ... @<be>:8060, including a loopback @<self>:8060.
5. SELECT 1 and SHOW BACKENDS (Alive=true) keep working. Only a full BE restart recovers.

We have not reduced this to a minimal standalone reproducer; it reproduces reliably under our normal
multi-replica load. We will run a targeted reproducer if you suggest one.

Anything Else?

Search / prior art

We searched the issue tracker, the load_stream / move-memtable PR history, the 4.1.x changelogs,
and community forums (English and Chinese) before filing, and found no exact match for the full
signature. The closest structural match is Apache brpc #1168 -- after a downstream node fault the
upstream socket enters a "Broken" state and the health check never revives it; recovery requires
restarting the upstream. Adjacent load-stream lifecycle fixes already in 4.0.6: #34883,
#39231 / #39762, #60148, #60285. Possibly related and unconfirmed for 4.0.x: #56120 ("close brpc
stream after load stream is closed"). If a maintainer recognises this as known or already fixed, a
pointer to the PR is the fastest resolution.

Environment

• Mode: coupled (storage-compute together), FE + BE only. No FoundationDB / Meta Service / Recycler /
  S3 storage vault. Native tablet data lives on BE-local EBS.
• Topology: 3 FE (HA followers) + 4 BE. Each node 8 vCPU / 64 GiB RAM.
• BE storage: one dedicated 500 GB gp3 volume per BE, xfs (noatime,nodiratime), mounted
  /var/lib/doris/storage, gp3 baseline 3000 IOPS / 125 MiB/s.
• OS / JDK: Amazon Linux 2023, Amazon Corretto 17.
• Replication: Doris default 3, across 4 BEs.
• Workload: read Apache Iceberg through a Glue / S3 external catalog, then write the aggregated
  result into native Doris UNIQUE-KEY merge-on-write tables -- CREATE TABLE AS SELECT,
  INSERT ... SELECT, and a few UPDATE ... FROM statements. About 5M rows per table, 4 tables.
• be.conf is effectively stock. The only non-default overrides are mem_limit = 80%,
  storage_root_path, and priority_networks = <self>/32. No brpc / clone / timeout tuning was set
  initially. Full dump at the end.

What we ruled out, with positive evidence

All of the environment-layer suspects below were tested directly, while wedged (raw probes on
2026-06-22).

Hypothesis	Verdict	Evidence
TCP / network / security group / routing on 8060	Ruled out	Raw TCP (/dev/tcp) to :8060 is OPEN to the peer and over loopback to self on all 4 BEs while the brpc RPC on the same port times out; the listener is healthy (LISTEN 0 1024 0.0.0.0:8060). A brpc call failing to its own loopback :8060 while raw TCP to that port succeeds cannot be network/SG/routing.
Host firewall (iptables / nftables / firewalld)	Ruled out	All 4 BEs: iptables 0 non-policy rules (default-ACCEPT), ip6tables 0, nft ruleset empty, firewalld inactive/absent. No host firewall exists.
SELinux	Ruled out	getenforce = Permissive on all 4 (policy targeted, mode permissive) -- it logs but cannot block.
ENA bandwidth throttle	Ruled out	bw_in/out_allowance_exceeded are non-zero cumulative but Δ=0 over a 50s sample during the idle wedge (they moved only during the loads); pps_allowance_exceeded=0, conntrack_allowance_exceeded=0. No active throttle while wedged.
conntrack / ephemeral ports	Ruled out	nf_conntrack module not loaded; ~53--60 of ~28k ephemeral ports used, 3 TIME-WAIT. Neither is exhausted.
Kernel / OOM / packet drops	Ruled out	dmesg / journalctl -k show no drop/deny/reject/oom/conntrack/throttle lines for the window.
Deployment / OS-tuning misconfig	Ruled out	Our install sets all Doris-required kernel tuning (vm.max_map_count=2000000 -- live-confirmed, swap off, nofile 655350, THP madvise) and runs start_be.sh's preflight, which the official apache/doris container deployment skips (SKIP_CHECK_ULIMIT=true). The official FE/BE images add no brpc/network/timeout config we lack -- only priority_networks. So it is not a deployment misconfiguration.
Compaction / merge-on-write delete-bitmap publish	Ruled out	Captured live at the wedge: every compaction metric is 0 on all 4 BEs -- doris_be_compaction_task_state_total{base,cumulative}=0, doris_be_disks_compaction_score=0, doris_be_compaction_used_permits=0, doris_be_compaction_waitting_permits=0, doris_be_load_channel_count=0, doris_be_tablet_writer_count=0.
Resource exhaustion (CPU / memory / IO)	Ruled out	At the wedge the BEs are near-idle: load avg ~0.0--0.09, ~55--60 GB RAM free, doris_be at 2--3% CPU. EBS volumes idle (VolumeReadOps=0, under 1 write IOPS, VolumeQueueLength ~0).
BE soft memory limit / flush back-pressure	Ruled out	Workload-group total_mem_used 0--158 MB against an ~53 GB limit; zero memory-exceed or MemoryGc-cancel lines. Memory would climb if flush stalled.
Crash / auto-restart / kernel OOM	Ruled out	NRestarts=0, single MainPID for the whole window on every BE; dmesg and journalctl -k empty for the window.
Replication factor (repl=3 itself)	Ruled out	A fresh-cluster full 4-table build at repl=3 completed cleanly and sustained, 0 errors. Earlier "repl=3 triggers it" readings were confounded by clusters already degraded by prior single-BE-restart experiments.
BE thread-pool exhaustion	Not the cause	No BE thread pool is pegged at the wedge: EvHttpServer at pool size 128, pipeline schedulers at normal 8/16, no compaction or memtable pool active.

The one mechanism consistent with all of this is a brpc load-stream socket going Broken and never
being revived: raw TCP to :8060 connects (peer and loopback) while every brpc RPC on it times out;
Doris's own health check evicts the stub (remove brpc stub from cache) and recreates it, and the
new stub still times out; the errors are connect/open timeouts (never "Connection refused" or
"reset"); and it clears only when the process is dropped. That is the Apache brpc #1168 class.

Config we tried that did not fix it

Setting	Where	Result
enable_brpc_connection_check = true	be.conf, immutable, rolling restart	No effect. This is the mechanism that should periodically check brpc connections and close/recreate broken ones (brpc_connection_check_timeout_ms = 10s default), but it did not revive the broken load-stream socket. Wedged again at +8 minutes. Kept as general hardening.
experimental_enable_single_replica_insert = true	FE global var	Partial and unreliable. Loads write one replica and clone the rest, so a single load completes instead of hanging, but the idle wedge still fires afterward and a later load still hung despite the setting.

We did not raise tablet_writer_open_rpc_timeout_sec or brpc_socket_max_unwritten_bytes beyond
defaults, because those mask the symptom -- a longer park -- rather than revive the socket. If you
believe a specific brpc knob is the fix, we will test it.

The two causes we fixed ourselves

We are listing these so it is clear the residual is isolated, and because one of them was our
mistake and we would rather name it than route around it.

• Cause 能公开一些公开数据集上的性能测试数据吗? #1, our security-group bug -- fixed. Our BE security group self-referenced 8060 (brpc) and
  9060 (be_port) but not 8040 (webserver_port, the HTTP port used for clone snapshot download
  between BEs). Loads over brpc 8060 worked, but clone REPAIR over
  http://<be>:8040/api/_tablet/_download timed out
  ([HTTP_ERROR]Connection timed out after 15000 milliseconds), so missing replicas never healed
  and the FE ran an unbounded VERY_HIGH repair-clone storm that saturated the BEs. Adding the 8040
  BE-to-BE self-ingress rule fixed it: clones finish, drain to 0, replicas heal. This was our
  infrastructure error, not a Doris defect. We mention it only because, once fixed, cause Support bulk loading from S3 compatible distributed storage #3 still
  reproduces -- which proves Support bulk loading from S3 compatible distributed storage #3 is independent of it.
• Cause 注释英文拼写错误 #2, load-stream-open stall under heavy multi-replica load -- mitigated. Distinct from the
  8040 clone path; this is on the 8060 write path. Mitigated, not cured, by
  experimental_enable_single_replica_insert.

Detection and the workaround we run today

• Detection. The Thrift heartbeat (9050) runs on a separate threadpool from the brpc write path
  (8060), so SHOW BACKENDS ... Alive=true is not a writability signal -- it stayed green for ~2.5
  hours while every write was dead. We added a write-readiness canary, a small bounded INSERT over
  the 8060 path, to our health check, and a wedge now surfaces in seconds instead of hours.
• Recovery. A full BE-fleet restart. A single-BE restart does not clear it.

Code-level analysis (Doris 4.0.6, bundled brpc 1.4.0)

We traced the captured stacks/logs into the 4.0.6 source. The load-bearing finding, from the
target BE's be.INFO during the wedge:

• PInternalService::open_load_stream logs "open load stream, load_id=..." (internal_service.cpp:416)
  as the first line of the handler. During the wedge there were 0 such handler-entry lines on the
  target BEs in the wedge window, versus 1700+ historically -- while the BE worker pools sat
  idle (pstack: threads parked in blocking_get, not saturated; a saturated pool would fail
  try_offer fast, not time out at 60s).
• So the inbound open_load_stream RPC never reaches the Doris service handler. Combined with raw
  TCP to :8060 being OPEN, the stall is between TCP-accept and service-dispatch -- inside brpc
  1.4.0, below Doris's load-stream code. Doris's handler is not the stall point; it is never entered.

We did not pin the exact brpc 1.4.0 line -- it is in the bundled submodule
(thirdparty/vars.sh, apache/brpc tag 1.4.0), and the runtime probe that would pin it (brpc
rpcz / socket bvars) could not be enabled at runtime on this build. One secondary, non-root nuance
we found: FailureDetectChannel invalidates a cached channel only on EHOSTDOWN, not on a timeout
(brpc_client_cache.h:80,125) -- but we captured 249 EHOSTDOWN (Host is down) and channel
rebuilds happened anyway and did not recover the wedge, so that is at most a hardening suggestion, not
the cause.

What we have captured and what else we can provide

We have captured the in-process state at a live wedge. Attached:

• gstack thread dumps of doris_be on the two BEs with parked write threads (full ~1747-thread
  dumps),
• brpc /vars (94 KB) and /metrics from each, showing the worker / load-channel / compaction
  state,
• be.WARNING tails with the [E1008] open failures and the FailureDetectChannel probe failures.

We could not get rpcz -- it is off by default and :8060/rpcz/enable did not enable it at runtime
on this build. If there is a flag or build option to turn rpcz on, tell us and we will capture it. We
can also pull a full gdb -p thread apply all bt, more specific brpc bvars, or FE-side state on
request.

One caveat on timing. This is a dev POC and we are moving on with our implementation, so the cluster
will not stay up indefinitely. The reproducer, the captures above, and any candidate-build testing
are only available while the cluster is still running -- so the sooner we can act on this, the
better.

Questions for the maintainers

1. Our evidence says the open_load_stream RPC never reaches the server handler (0 handler-entry logs,
   idle pools) while raw TCP to :8060 is open -- consistent with a brpc 1.4.0 socket/stream that is
   accepted at TCP but never dispatched, and never revived (the brpc The download link for doris-incubating-thirdparty-20190414 is broken #1168 class). Is this a known
   brpc 1.4.0 defect on the load-stream path, and is there a fixing PR or a brpc version that resolves
   it?
2. Is the load-stream brpc_client_cache expected to revive a Broken socket automatically? In our
   capture it never did, and enable_brpc_connection_check=true did not help. Is that the intended
   recovery path, and should it have recovered the socket?
3. Is there a supported config that makes a Broken load-stream socket fail fast and reconnect, rather
   than park ~534s on the open RPC?
4. Is there evidence that 4.1.x (4.1.2 specifically) contains a relevant brpc / load-stream fix? We
   will run the upgrade test -- restart one BE, drive the load sequence, watch the canary -- and
   report back.

Appendix -- config

be.conf (stock defaults plus these managed overrides only):

JAVA_HOME = /usr/lib/jvm/java-17-amazon-corretto.x86_64
storage_root_path = /var/lib/doris/storage
priority_networks = <node_private_ip>/32
mem_limit = 80%
be_port = 9060            # shipped default
webserver_port = 8040     # shipped default
heartbeat_service_port = 9050   # shipped default
brpc_port = 8060          # shipped default
# added later as hardening; did NOT fix the wedge:
enable_brpc_connection_check = true

fe.conf (stock defaults plus these overrides only):

JAVA_HOME = /usr/lib/jvm/java-17-amazon-corretto.x86_64
meta_dir = /var/lib/doris/fe-meta
priority_networks = <node_private_ip>/32
http_port = 8030          # shipped default
rpc_port = 9020           # shipped default
query_port = 9030         # shipped default
edit_log_port = 9010      # shipped default
# FE global var set at runtime; mitigates cause #2, not cause #3:
experimental_enable_single_replica_insert = true

Table shape (representative):

CREATE TABLE evo_persons (
  identity_hash    varchar(32) NOT NULL,
  id_numbers_hash  varchar(32) NOT NULL,
  ...                          -- aggregated attribute and counter columns
)
UNIQUE KEY(identity_hash, id_numbers_hash)
DISTRIBUTED BY HASH(identity_hash) BUCKETS 16
PROPERTIES ('replication_num'='3', 'enable_unique_key_merge_on_write'='true');

Ports:

Port	Service	At the wedge
8060	brpc (tablet-writer / load-stream OPEN)	timed out, all directions including loopback
8040	webserver (clone snapshot download)	timed out until our security-group fix (cause #1); fine after
9050	Thrift heartbeat (separate threadpool)	stayed responsive, so SHOW BACKENDS showed Alive=true

wedge.10.0.0.105.tar.gz
wedge.10.0.0.118.tar.gz
wedge.10.0.0.155.tar.gz
wedge.10.0.0.229.tar.gz

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_f98169949cb7
This content is generated by AI for reference only.

Initial triage: this report looks credible and should be treated as a BE write-path availability bug, not as a normal backend liveness issue. The strongest evidence in the issue is that writes time out on :8060, including loopback/self cases, while raw TCP is still open and the target-side PInternalService::open_load_stream entry log is absent. If those attached logs match the described window, the request is not reaching the Doris open_load_stream handler and the stall is below the Doris load-stream service logic.

Code check against the 4.0.6 tag/commit reported here also shows a Doris-side recovery gap:

• In 4.0.6, VTabletWriterV2::_open_streams_to_backend() opens V2 load streams through ExecEnv::brpc_streaming_client_cache(), not the ordinary internal brpc cache.
• LoadStreamStub::open() creates a brpc stream, sets open_load_stream_timeout_ms to 60000 ms, gets the cached streaming stub, and calls open_load_stream() synchronously. On cntl.Failed(), it closes the stream and returns the error, but it does not evict the cached streaming stub.
• FailureDetectChannel only marks a cached channel unhealthy when cntl->ErrorCode() == EHOSTDOWN; a plain [E1008] Reached timeout does not poison the cached channel.
• enable_brpc_connection_check does not appear to cover the V2 streaming cache. In 4.0.6 it checks query-context brpc stubs and erases brpc_internal_client_cache(). I did not find a corresponding available() / erase() path for brpc_streaming_client_cache().

That makes the reporter's observation internally consistent: once a cached streaming-channel/socket gets into a state where open_load_stream only times out, later V2 load-stream opens can keep reusing the same cached stub instead of forcing a new channel/socket. It also explains why a full BE-fleet restart clears the mesh and why a single-BE restart may not, because peers can still hold their cached client-side streaming stubs.

I would not yet state that the exact root cause is proven to be Apache brpc #1168. The brpc hypothesis is plausible from the symptom, but the current public evidence pins the first actionable Doris-side issue to missing invalidation/recreation of the streaming brpc client on load-stream open timeout. A maintainer can investigate this without needing to prove the exact brpc internal line first.

I also checked the likely upgrade pointer: 4.1.2 still uses bundled brpc 1.4.0, and the same streaming-cache open path and EHOSTDOWN-only channel invalidation pattern are still present. Current branch-4.0, branch-4.1, and master also still show brpc 1.4.0 and no obvious streaming-cache eviction on this path. So 4.1.2 should still be tested, but I would not present it as a known brpc-version fix based on source inspection alone.

Suggested next maintainer actions:

1. Reproduce or fault-inject an open_load_stream timeout on the V2 path and verify whether the next V2 open reuses the same brpc_streaming_client_cache() entry.
2. Add cache invalidation for brpc_streaming_client_cache() when LoadStreamStub::open() fails with network/open-timeout style errors, or broaden FailureDetectChannel health marking for this open-RPC path beyond only EHOSTDOWN.
3. Add a regression/fault-injection test that forces one load-stream open failure and asserts the following open uses a fresh channel and does not wedge future writes.
4. Keep experimental_enable_single_replica_insert documented only as a mitigation; it reduces multi-replica load-stream exposure but does not fix the cached broken streaming-channel recovery gap.

Information that would make the public diagnosis stronger:

• For one affected load id / txn id, provide the target BE be.INFO window proving there is no open load stream, load_id=... handler-entry line while the coordinator logs the timeout.
• Capture rpcz or equivalent brpc socket-level state with diagnostics enabled before reproducing; runtime enabling apparently did not work on this build.
• Provide a minimal scripted reproducer if possible, or the exact sequence of CTAS / INSERT / UPDATE statements with table DDL, concurrency, and the load ids that correspond to the attached stacks.
• If the cluster is still available, capture gdb -p <doris_be> thread apply all bt plus ss -tanp for the affected BE pairs during the wedge, including loopback.

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_f98169949cb7
This content is generated by AI for reference only.

Small correction to the previous analysis comment: the phrase "Apache brpc #1168" was intended to refer to the Apache brpc prior-art item linked in the issue, not apache/doris#1168. The technical assessment and suggested next steps are unchanged.

[ojalberts-itc]

Update — retested on 4.1.2; could not reproduce on a fresh install with the SG correct from t=0 (on either 4.1.2 or 4.0.6)

TL;DR. We upgraded to Apache Doris 4.1.2 and re-ran our reproduction. 4.1.2 did not wedge. But when we ran the identical test against a fresh 4.0.6 cluster — the exact build this issue was filed against — it also did not wedge. The decisive difference from our original incident is not the Doris version: it is that both new clusters were built from clean infrastructure with our cause-#1 fix (the BE→BE webserver_port/8040 self-ingress) present from t=0. We can no longer reproduce the brpc load-stream wedge on a freshly-provisioned, correctly-configured cluster. We therefore cannot yet attribute the clean 4.1.2 result to an upstream fix — and we want to be honest about that rather than report a false "fixed".

What we did

• Re-deployed the same coupled-mode topology (3 FE + 4 BE, 8 vCPU / 64 GiB nodes, repl=3, local gp3 BE storage) on fresh instances with fresh empty data volumes, first on 4.1.2 (apache-doris-4.1.2-bin-x64.tar.gz, build tag doris-4.1.2-rc01), then a control on 4.0.6 (doris-4.0.6-rc02-…, the build this issue was filed against).
• Held everything but the Doris version constant: same infrastructure, same SG rules (incl. the cause-能公开一些公开数据集上的性能测试数据吗? #1 8040 self-ingress present from first boot), enable_brpc_connection_check=true in be.conf, and experimental_enable_single_replica_insert left unset so the cause-Support bulk loading from S3 compatible distributed storage #3 multi-replica path is fully exercised.

How we tested (identical protocol on both versions)

The workload is the heaviest multi-replica writer we have: build 4 native UNIQUE-KEY merge-on-write tables (replication_num=3, DISTRIBUTED BY HASH(...) BUCKETS 16) by INSERT … SELECT from an Iceberg (S3 Tables) external catalog plus UPDATE … FROM maintenance passes — the same shape of load that originally wedged us.

1. Sustained build — ~26 min of continuous repl=3 writes, ~222M rows across the 4 tables.
2. Idle-watch — a repl=3 write probe every ~40 s for 16 min after the build, covering the 8–11 min post-load window where our original wedge fired.
3. Rapid sequential churn — 60 fast repl=3 multi-replica INSERTs (1–3 s each), the original "fast small load" shape.
4. Concurrent churn — 4 parallel writers (~100 concurrent repl=3 loads), the concurrent shape that wedged us once before.

Per-BE instrumentation each run (the same signals from the original capture):

Signal	Original wedge (4.0.6)	This retest — 4.1.2	This retest — 4.0.6 control
open_load_stream handler entries during window	0 (handler never entered)	~5,100 cluster-wide	~4,000 cluster-wide
[E1008]Reached timeout … @<be>:8060 (incl. loopback)	present	0	0
brpc :8060 "Broken" / never-revived socket	present	0	0
SHOW BACKENDS Alive	true (misleading)	true	true
Write path	wedged (full BE restart to recover)	healthy throughout	healthy throughout

On 4.1.2 the open_load_stream handler was entered ~5,100 times with zero brpc timeouts and zero broken sockets — a clean inversion of the original signature (where the handler was entered zero times during the stall and [E1008] loopback timeouts dominated). The 4.0.6 control behaved the same.

Why we think it didn't reproduce

In our original incident the reliable wedges all occurred on clusters carrying accumulated degradation history: the cause-#1 8040 self-ingress was missing at first, which drove an unbounded clone-repair storm, and we had also cycled single-BE restarts. The brpc load-stream socket went "Broken" against that degraded backdrop. On a fresh cluster with the 8040 rule in place from t=0, the clone storm never happens and we have not been able to drive the BE into the "Broken socket" state — on either 4.0.6 or 4.1.2. This suggests cause #3, as we observed it, may not be reachable independently of the cause-#1 degradation, rather than being a pure function of the steady-state multi-replica write workload.

Honest limitations

• Scale. This retest was at POC scale (one reporting period, 4 of our domains, ~222M output rows, ~26 min). Our production load is ~50× larger and runs for hours. Because the wedge is an accumulation failure, we cannot rule out that it only manifests under the full-scale, many-hour sustained load — which neither cluster has yet seen.
• Non-determinism. Even originally on 4.0.6 the wedge was intermittent (one fresh-cluster build ran clean back then too), so a single clean pass is not proof of a fix.

Where this leaves the issue

We are not claiming 4.1.2 fixes the underlying brpc behaviour — we could not exercise the defect to test that claim. What we can say is: a fresh, correctly-configured cluster did not reproduce the wedge on 4.0.6 or 4.1.2 under heavy, idle, rapid, and concurrent multi-replica load. If it would help triage, our next steps would be (a) deliberately recreating the degraded precondition (8040 ingress absent → clone storm → single-BE restart) to drive a 4.0.6 wedge, then re-testing 4.1.2 under those same conditions, and/or (b) a full production-scale sustained load. Happy to run either and report back, and to pull fresh pstack / brpc /vars / /rpcz dumps if you want them from any of these runs.

[ojalberts-itc]

Correction: the wedge does reproduce — reliably — once enough multi-replica write history accumulates, and here is a fresh in-process capture

My earlier update said we "could not reproduce the wedge on a fresh cluster." That was premature, and I want to correct it with evidence.

On the same fresh-from-Terraform 4.0.6 control cluster (cause-#1 8040 SG fix present from t=0), after a day of accumulated multi-replica write activity, the wedge now reproduces reliably on essentially every stock (multi-replica) write. We had been keeping the cluster usable with experimental_enable_single_replica_insert=true; the moment we set it back to false and issued a single repl=3 INSERT, the write path wedged immediately. Reads and the 9050 heartbeat stayed up; SHOW BACKENDS = all Alive=true throughout. Recovery is still a full BE-fleet restart (a single-BE restart does not clear it).

We captured the in-process state on all 4 BEs at the live wedge (2026-06-23), before any restart.

1. Parked write thread — the brpc load-stream OPEN never returns

gstack on a coordinator BE (full dump = 1746 threads) shows a write thread parked exactly here:

#3  bthread_id_join ()
#4  brpc::Channel::CallMethod(...)
#5  doris::FailureDetectChannel::CallMethod (...)            be/src/util/brpc_client_cache.h:121
#6  doris::LoadStreamStub::open (..., txn_id=554, total_streams=2, idle_timeout_ms=14400000)
                                                             be/src/vec/sink/load_stream_stub.cpp:195
#7  doris::LoadStreamStubs::open (...)                       be/src/vec/sink/load_stream_stub.cpp:574
#8  doris::vectorized::VTabletWriterV2::_open_streams_to_backend (dst_id=1782147896322, ...)
                                                             be/src/vec/sink/writer/vtablet_writer_v2.cpp:317
#9  doris::vectorized::VTabletWriterV2::_open_streams (...)  be/src/vec/sink/writer/vtablet_writer_v2.cpp:296
#10 doris::vectorized::VTabletWriterV2::open (...)           be/src/vec/sink/writer/vtablet_writer_v2.cpp:272
#11 doris::vectorized::AsyncResultWriter::process_block (...) be/src/vec/sink/writer/async_result_writer.cpp:119
#12 doris::vectorized::AsyncResultWriter::start_writer(...)::$_0::operator()()
                                                             be/src/vec/sink/writer/async_result_writer.cpp:105
#16 doris::ThreadPool::dispatch_thread (...)                 be/src/util/threadpool.cpp:616

(Build path /home/zcp/repo_center/doris_release/doris/be/src/... confirms the official 4.0.6 GA build, doris-4.0.6-rc02-1663f25c16f.)

2. [E1008] on :8060 including the in-process loopback

be.WARNING on the wedged target BE (10.0.0.209, backend id 1782147896323) shows it cancelling a load-stream whose source and destination are the same backend — a BE that cannot open a load-stream to its own :8060:

load_stream_stub.cpp:591  open stream failed: [INTERNAL_ERROR]Failed to connect to backend
  1782147896323: [E1008]Reached timeout=60000ms @10.0.0.209:8060
  ; stream: load_id=..., src_id=1782147896323, dst_id=1782147896323, stream_id=...
brpc_closure.h:128  RPC meet failed: [E1008]Reached timeout=533998ms @10.0.0.209:8060

Raw TCP to :8060 (peer and loopback) is OPEN throughout — the stall is in the brpc application layer, not the socket/TCP/SG. (src_id == dst_id loopback proof; the ~534s park is the RPC timeout.)

3. Workers are parked, not saturated

brpc /vars (:8060) at the wedge:

bthread_worker_usage           : 1.14142
bthread_count                  : 5
load_stream_count              : 1–3
rpc_server_8060_connection_count : 13–14

The write threads are blocked on the RPC, not starved — far below any worker ceiling.

4. Zero-clone wedge (isolated from cause #1)

FE-side, all BEs Alive=true and SHOW PROC '/cluster_balance/{running,pending}_tablets' are empty — no clone activity. This is purely the brpc load-stream socket going Broken and never reviving (the brpc #1168 class), not the cause-#1 clone storm.

Full dumps available

Per-BE tar.gz (full pstack × 1746 threads, complete /vars, /rpcz, /metrics, be.WARNING tail) and FE cluster_balance/statistic are ready — I can drag-drop them onto this issue if that's the most useful form (the inline excerpts above are the load-bearing parts).

Next: testing 4.1.2

We are about to switch this cluster to 4.1.2 and run the same test against it. We'll post an update with the 4.1.2 result as soon as that is done.

The workaround that unblocks heavy loads in the meantime is experimental_enable_single_replica_insert=true (writes one replica then clones the rest over the 8040 path, sidestepping the BE-to-BE 8060 load-stream opens — not a fix). Original Question 1 stands: is this a known brpc 1.4.0 load-stream socket defect, and is there a fixing PR or a version that resolves it?

wedge.10.0.0.45.20260623T022225Z.tar.gz
wedge.10.0.0.186.20260623T022225Z.tar.gz
wedge.10.0.0.209.20260623T022225Z.tar.gz
wedge.10.0.0.233.20260623T022225Z.tar.gz

[ojalberts-itc]

Update: the wedge reproduces on Doris 4.1.2 (doris-4.1.2-rc01)

Following the previous comment (where we promised a 4.1.2 test): we redeployed the cluster
fresh on 4.1.2 (doris-4.1.2-rc01-aec169d2025, official x64 GA tarball; 3 FE + 4 BE, repl=3;
the cause-#1 BE→BE :8040 self-ingress SG rule present from t=0; empty data volumes) and ran a
heavy multi-replica write workload with experimental_enable_single_replica_insert OFF.

It wedged — identical signature to 4.0.6. The workload (a ~222M-row repl=3 build + churn) ran
clean during the load, then the write path wedged at idle, ~10 minutes after the last heavy
write. We captured all 4 BEs live before any restart.

The parked write thread — same LoadStreamStub::open as before, now on 4.1.2

Doris's own be.WARNING printed the stack at the failure point:

open stream failed: [INTERNAL_ERROR]Failed to connect to backend <id>: [E1008]Reached timeout=60000ms @10.0.0.227:8060
  0#  doris::LoadStreamStub::open(...)                       be/src/exec/sink/load_stream_stub.cpp:208
  1#  doris::LoadStreamStubs::open(...)                      be/src/exec/sink/load_stream_stub.cpp:574
  2#  doris::VTabletWriterV2::_open_streams_to_backend(...)  be/src/exec/sink/writer/vtablet_writer_v2.cpp:317
  3#  doris::VTabletWriterV2::_open_streams()                be/src/exec/sink/writer/vtablet_writer_v2.cpp:298

(Build path /home/zcp/repo_center/doris_release/doris/be/src/... confirms the official 4.1.2 GA build.)

[E1008] on :8060 across all 4 BEs

be.WARNING E1008/broken-socket counts at the wedge were 23 / 34 / 30 / 24 across the four
BEs. The enable_brpc_connection_check path detects the broken socket and evicts it from cache,
but it never revives:

fragment_mgr.cpp:953   brpc stub: 10.0.0.227:8060 check failed: [E1008]Reached timeout=10000ms @10.0.0.227:8060
fragment_mgr.cpp:974   remove brpc stub from cache: 10.0.0.227:8060, error: [E1008]...
brpc_client_cache.h:326 open brpc connection to 10.0.0.137:8060 failed: [E1008]Reached timeout=2000ms @10.0.0.137:8060
vtablet_writer.cpp:715  failed to open tablet writer may caused by timeout ...

Raw TCP to :8060 stays open throughout; the stall is in the brpc application layer.

Workers parked, not saturated — and zero clone activity

brpc :8060 /vars at the wedge: bthread_worker_usage 0.46–1.76, bthread_count 5, with
rpc_server_8060_connection_count 58–59. The write threads are blocked on the RPC, not starved.
FE SHOW PROC '/cluster_balance/{running,pending}_tablets' was empty — a zero-clone wedge,
purely the brpc load-stream socket going Broken. SHOW BACKENDS showed all 4 BEs Alive=true
throughout (the 9050 heartbeat is a separate threadpool). Both repl=3 and repl=1 writes hung;
only a simultaneous full-BE restart cleared it.

Preconditions — this is why a fresh-cluster smoke test will NOT reproduce it

This is the important part if you try to reproduce. A fresh cluster running a single heavy load
does not wedge. In our run, the identical 85M-row repl=3 load on a freshly-bootstrapped 4.1.2
cluster completed in 73s, clean. The same SQL, same single_replica_insert=OFF, on a cluster
that had accumulated history then failed (125s, then 365s on retry). The discriminating
precondition is the cluster's accumulated/degraded state; heavy multi-replica load is the proximate
trigger on top of that. To reproduce, expect to need sustained accumulation (a full multi-table
build plus a burst of concurrent repl=3 loads), not one load on a clean cluster.

We had two occurrences, with complementary confounds that cancel out:

1. First (fresh bootstrap, no restart): a full ~222M-row build + churn ran clean, then the
   write path wedged at idle ~10 min later. This was on a cluster that had never been
   systemctl restart-ed — so it is not explained by the known clone-storm-after-hard-restart
   path. (It did have single_replica_insert toggled OFF→ON ~2 min before the wedge, so this
   occurrence alone couldn't rule out the toggle.)
2. Replay (no toggle): we re-ran with single_replica_insert OFF the whole time, no toggle;
   the heavy load wedged again (the [E1008] / failed to write enough replicas 1/3 /
   failed to open streams to any BE above). So the toggle is not necessary.

Union of the two: the wedge appears without the toggle and without a hard-restart-degraded
cluster — so neither is the cause. The common factor is an accumulated cluster under sustained
multi-replica load, on 4.1.2.

Net: the brpc BE write-wedge is present on Doris 4.1.2 — it is not fixed. The in-process
signature (the LoadStreamStub::open stack, [E1008] on :8060 across all BEs, zero-clone,
all-Alive=true, full-restart-only recovery) is identical to the 4.0.6 reports — the brpc #1168
class. Full per-BE be.WARNING excerpts, the brpc /vars, and FE cluster-state for both
occurrences are attached as 64708-4.1.2-evidence.tar.gz
(sha256 e52e4c76c2f25dd87d8aab2434dc332530e20ea196eff0c85f67196d7855c2fd). Original Question 1
stands: is this a known brpc 1.4.0 load-stream socket defect, and is there a fixing PR or a
version that resolves it?

64708-4.1.2-evidence.tar.gz

[ojalberts-itc]

Update: the wedge is not OS-related — reproduced on Ubuntu 22.04 (Doris 4.0.6)

To exclude the operating system as a factor, we reproduced this wedge on a second, different OS.
We previously reported it on Amazon Linux 2023; we have now reproduced the same wedge, with
the same in-process signature, on Ubuntu 22.04 LTS, using identical infrastructure — same
Terraform, instance types, topology, security groups (incl. the cause-#1 BE→BE :8040 self-ingress
rule), be.conf/fe.conf, replication factor, and Doris build. The only variable changed is the
host OS. The wedge is present on both. The OS is not the cause — consistent with an upstream
brpc/Doris load-stream defect.

This separates existence (4.0.6 exhibits the wedge — now on two OSes) from trigger (an
accumulated cluster under multi-replica load); the trigger evidence is unchanged from the prior
4.0.6/4.1.2 reports.

Version

doris-4.0.6-rc02-1663f25c16f — official x64 (AVX2) GA tarball. Build path in the stacks is
/home/zcp/repo_center/doris_release/doris/be/src/... (ldb-toolchain-v0.26, gcc 15).

Environment (identical to the Amazon Linux run except the OS)

• 3 FE + 4 BE, AWS r8i.2xlarge (8 vCPU / 64 GiB), coupled (storage-compute) mode, replication = 3.
• experimental_enable_single_replica_insert = false the whole time (no toggle — see confounds).
• OS: Ubuntu 22.04.5 LTS, kernel 6.8.0-1057-aws (vs Amazon Linux 2023, kernel 6.1, in the prior run).

Trigger / preconditions (repro-survivable)

Honest precondition, so a fresh-cluster smoke test does not "disprove" this: native ingest and a
2.8B-row federated read passed at deploy (03:21–03:22 UTC), i.e. the :8060 load path worked
when the cluster was new. The wedge then appeared ~20 min later, after accumulated repl=3
cross-BE activity (native ingest + internal audit_log stream-loads + the test's reads/writes) —
before any heavy bulk build. It is persistent: still wedged ~6 hours after onset (a
repl=3 write canary wedged 4/6 at +6 h; ~6,000 [E1008] lines on :8060 in the first 25 min,
still ~212 per 15 min hours later). It is intermittent at the cluster level (a given repl=3 write
wedges only when its tablet replica set includes a wedged BE) but persistent on the affected BEs,
and it spreads BE→BE over time. This matches the known "accumulated cluster" precondition;
existence on Ubuntu is the new fact here.

The parked write thread — same LoadStreamStub::open, now on Ubuntu

Doris's own be.WARNING prints the stack at the failure point:

open stream failed: [INTERNAL_ERROR]Failed to connect to backend <id>: [E1008]Reached timeout=60000ms @10.0.0.133:8060
  0#  doris::LoadStreamStub::open(...)                       be/src/vec/sink/load_stream_stub.cpp:208
  1#  doris::LoadStreamStubs::open(...)                      be/src/vec/sink/load_stream_stub.cpp:574
  2#  doris::VTabletWriterV2::_open_streams_to_backend(...)  be/src/vec/sink/writer/vtablet_writer_v2.cpp:317
  3#  doris::VTabletWriterV2::_open_streams()                be/src/vec/sink/writer/vtablet_writer_v2.cpp:298
  4#  doris::VTabletWriterV2::open(...)  →  AsyncResultWriter::process_block  →  ThreadPool::dispatch_thread

[E1008] on :8060 across all 4 BEs (parks growing to 538s / 900s)

The broken load-stream socket is never revived; the cancel propagates as
failed to write enough replicas:

load_stream_stub.cpp:369  LoadStreamStub ... src_id=...884, dst_id=...881 is cancelled because of
   [INTERNAL_ERROR]Failed to connect to backend ...881: [E110]... Socket{...addr=10.0.0.43:8060...}: Connection timed out
load_stream_stub.cpp:591  open stream failed: ...[E112]Not connected to 10.0.0.43:8060 yet, server_id=217
vtablet_writer_v2.cpp:321 failed to open stream to backend ...881
brpc_closure.h:128        RPC meet failed: [E1008]Reached timeout=537999ms @10.0.0.133:8060
brpc_closure.h:128        RPC meet failed: [E1008]Reached timeout=900000ms @10.0.0.53:8060
fragment_mgr.cpp:1177/1198 brpc stub: 10.0.0.118:8060 check failed [E1008] → remove brpc stub from cache
stream_load_executor.cpp:107 ... node=10.0.0.133:8060, open failed, failed to open tablet writer,
   error=RPC call is timed out, [E1008]Reached timeout=60000ms @10.0.0.133:8060 ... elapse(s)=64

Workers parked (not saturated), zero-clone, all Alive=true

brpc :8060 /vars at the wedge: bthread_worker_usage ≈ 1.1 (of 256 workers),
bthread_count 5, load_stream_count 1–3 — threads blocked on the RPC, not starved. FE
SHOW PROC '/cluster_balance/{running,pending}_tablets' was empty (zero-clone — isolated from
the cause-#1 tablet-clone path). SHOW BACKENDS showed all 4 BEs Alive=true throughout (the 9050
heartbeat is a separate threadpool). Only a simultaneous full-BE restart clears it; a single-BE
restart does not.

What we ruled out on Ubuntu, with positive evidence (captured while wedged)

Each environment cause excluded by direct test on the wedged Ubuntu cluster — so "it's your
network/OS" is answered up front:

• Raw L4 to :8060 is OPEN — cat >/dev/tcp/<peer>/8060 succeeds to every peer; 11–14
  ESTABLISHED :8060 sockets per BE in ss. The stall is in the brpc application layer, not TCP.
• No host firewall — ufw inactive; nftables ruleset empty; iptables default-ACCEPT only.
• conntrack not exhausted — nf_conntrack_count 0 / max 1048576.
• No Nitro/ENA throttle — bw_in/out_allowance_exceeded, pps_allowance_exceeded,
  conntrack_allowance_exceeded, linklocal_allowance_exceeded all 0 on all 4 BEs.
• No SELinux (Ubuntu); AppArmor enabled with no Doris profile loaded.

Confounds (disclosed) and why OS is excluded

• No mitigation-toggle confound here: single_replica_insert was false from t=0, never
  toggled (the original 4.0.6 occurrence had a toggle ~2 min prior; this run removes that variable).
• OS is the only delta between this run and the Amazon Linux run (identical Terraform/config/build).
  Two reproductions whose environments differ in exactly the OS → the OS is not the cause.
• enable_brpc_connection_check=true does not fix it (it was on throughout): it detects the
  broken stub and evicts it from cache (remove brpc stub from cache), but the socket never revives.

What we expected

A BE-to-BE load-stream brpc socket that goes Broken/times out should be detected and
re-established so writes recover without operator intervention; instead it is never revived and
only a full-fleet BE restart clears it.

Questions

1. With the OS now excluded by a second-OS reproduction (Ubuntu 22.04 + Amazon Linux 2023, identical
   infra), does this confirm the failure is in the brpc load-stream layer rather than the environment?
2. Is this the brpc The download link for doris-incubating-thirdparty-20190414 is broken #1168 class (socket Broken, never revived), and is there a fixing PR or a
   Doris version that resolves it?
3. Should enable_brpc_connection_check be able to recover this socket, or is eviction-without-revive
   the expected (insufficient) behaviour here?

Willing to submit a PR

No, but happy to test a patch or run any targeted capture you want on the live wedged cluster.

What we captured / can provide

Attached 64708-ubuntu-4.0.6-evidence.tar.gz
(sha256 03896c90b393d8577f007ebffe69bb82b0831fb9ba1697d6a9dd000c7fab2950): per-BE be.WARNING
load-stream failures + brpc :8060 /vars, the parked-stack backtrace, FE cluster-state
(Alive=true + empty cluster_balance), the environment-exclusion probes, and the persistence
timeline. Full per-BE gdb 'thread apply all bt' dumps, /vars, /rpcz, and complete be.WARNING
are available on request.

64708-ubuntu-4.0.6-evidence.tar.gz