Skip to content

feat: Add application service (bridges) support#394

Open
sampaiodiego wants to merge 79 commits into
mainfrom
add-application-service-support
Open

feat: Add application service (bridges) support#394
sampaiodiego wants to merge 79 commits into
mainfrom
add-application-service-support

Conversation

@sampaiodiego

@sampaiodiego sampaiodiego commented Jun 1, 2026

Copy link
Copy Markdown
Member

Summary by CodeRabbit

  • New Features
    • Added Matrix Application Service support end-to-end (registration, ping, transaction delivery, namespace matching, and bridge querying).
    • Added client-server appservice-driven endpoints for registration, rooms, profile, events, and directory room alias updates.
    • Added support for custom event sending and automatic ephemeral EDU routing (typing, receipts, presence).
  • Bug Fixes
    • Improved batching/serialization and per-appservice delivery retries with exponential backoff and recovery behavior.
    • Improved alias reservation/release and sender/ghost user handling.
  • Documentation
    • Added a Bridge Architecture Guide plus appservice API compliance notes.
  • Tests
    • Added coverage for event routing and namespace matching.

@coderabbitai

coderabbitai Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR adds a new @rocket.chat/appservice package for Matrix Application Service support, wires it into federation SDK and homeserver flows, adds bridge-focused docs, and updates core transport and room helpers to support the new routing and delivery paths.

Changes

Application Service Bridge Integration

Layer / File(s) Summary
Bridge architecture guide
docs/bridge-architecture-guide.md
Adds a Matrix Bridge Architecture Guide covering AS registration, transaction delivery, bridge operations, query endpoints, database schema, and repo-specific implementation mapping.
Appservice package contracts
packages/appservice/package.json, packages/appservice/tsconfig.json, packages/appservice/src/index.ts, packages/appservice/src/config-provider.ts, packages/appservice/src/models/appservice.model.ts, bundle.ts, tsconfig.json, tsconfig.base.json, packages/core/src/events/homeserver-event-signatures.ts, packages/core/src/index.ts, packages/appservice/TODO.md
Introduces the appservice package entrypoint, metadata, TS project config, config provider interface/token, domain model types, and bundling/project-reference wiring. Also adds shared homeserver event signatures and the core path alias for the new package.
Appservice persistence and namespace matching
packages/appservice/src/repositories/appservice-state.repository.ts, packages/appservice/src/repositories/appservice-txn.repository.ts, packages/appservice/src/services/registration.service.ts, packages/appservice/src/services/namespace-matcher.service.ts, packages/appservice/src/services/namespace-matcher.service.spec.ts
Adds MongoDB-backed state/transaction repositories, config-driven registration caching, namespace regex compilation/matching, sender-user ownership helpers, and related tests.
Appservice routing and delivery
packages/appservice/src/utils/edu-to-appservice.ts, packages/appservice/src/utils/edu-to-appservice.spec.ts, packages/appservice/src/services/event-router.service.ts, packages/appservice/src/services/event-router.service.spec.ts, packages/appservice/src/services/bridge-query.service.ts, packages/appservice/src/services/ping.service.ts
Converts federation EDUs to ephemeral events, batches and serializes transaction delivery per bridge, and adds bridge query/ping HTTP helpers, with tests for conversion and routing behavior.
Federation SDK wiring and repositories
packages/federation-sdk/src/index.ts, packages/federation-sdk/src/repositories/room-alias.repository.ts, packages/federation-sdk/src/repositories/user.repository.ts, packages/federation-sdk/src/services/config.service.ts, packages/federation-sdk/src/services/directory.service.ts, packages/federation-sdk/src/services/event-emitter.service.ts, packages/federation-sdk/src/queues/per-destination.queue.ts, packages/federation-sdk/tsconfig.json, packages/federation-sdk/package.json, packages/federation-sdk/src/services/appservice-room.service.ts, packages/federation-sdk/src/services/event-sender.service.ts, packages/federation-sdk/src/services/event-sender.service.spec.ts, packages/federation-sdk/src/services/message.service.ts, packages/federation-sdk/src/services/profiles.service.ts, packages/federation-sdk/src/services/edu.service.ts, packages/federation-sdk/src/services/room.service.ts, packages/federation-sdk/src/utils/event-schemas.ts, packages/federation-sdk/src/sdk.ts
Registers appservice config and collections in the DI container, adds room alias and directory repositories, extends user repo sender-user provisioning, adds XMPP config schema, fixes retry timer state reset, adds custom event sending and room-creation delegation, routes messages/EDUs through the event router, and updates room/profile behavior.
Homeserver client-server endpoints and auth
packages/homeserver/src/middlewares/appserviceAuth.ts, packages/homeserver/src/controllers/client/*.ts, packages/homeserver/src/homeserver.module.ts, packages/homeserver/tsconfig.json
Adds appservice authentication middleware and client-server controllers for register, events, rooms, profile, and directory, then wires them into the homeserver app.
Core transport and room helpers
packages/core/src/utils/fetch.ts, packages/room/src/manager/room-state.ts
Reworks fetch to select HTTP/HTTPS transport with SNI support and shared body-reading/error helpers, and adds canonical alias extraction to room state.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EduService
  participant EventRouterService
  participant NamespaceMatcherService
  participant TransactionSenderService
  participant Bridge
  EduService->>EventRouterService: routeEphemeral(typing|presence|receipt EDU)
  EventRouterService->>NamespaceMatcherService: getInterestedAppServices(...)
  NamespaceMatcherService-->>EventRouterService: interested appservices
  EventRouterService->>TransactionSenderService: sendTransaction(appservice, batch)
  TransactionSenderService->>Bridge: PUT /_matrix/app/v1/transactions/{txnId}
  Bridge-->>TransactionSenderService: response
Loading
sequenceDiagram
  participant Client
  participant HomeserverController
  participant resolveAppServiceAuth
  participant FederationSDK
  Client->>HomeserverController: appservice-authenticated request
  HomeserverController->>resolveAppServiceAuth: resolveAppServiceAuth(...)
  resolveAppServiceAuth->>FederationSDK: getRegistrationByAsToken(token)
  FederationSDK-->>resolveAppServiceAuth: registration
  resolveAppServiceAuth-->>HomeserverController: auth result or error
  HomeserverController-->>Client: Matrix response
Loading

Possibly related PRs

  • RocketChat/homeserver#239: Both PRs touch the HomeserverEventSignatures type and its wiring across packages/federation-sdk/src/index.ts and core event signatures.
  • RocketChat/homeserver#292: Both PRs modify FederationSDK bootstrap and public SDK wiring in packages/federation-sdk/src/index.ts and packages/federation-sdk/src/sdk.ts.

Suggested labels: type: feature

Suggested reviewers: ggazzo, rodrigok

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding application service and bridge support.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov-commenter

codecov-commenter commented Jun 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.94946% with 510 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.06%. Comparing base (9052057) to head (db52ec5).

Files with missing lines Patch % Lines
...ce/src/repositories/appservice-state.repository.ts 5.15% 92 Missing ⚠️
...es/appservice/src/services/registration.service.ts 19.79% 77 Missing ⚠️
...es/appservice/src/services/event-router.service.ts 60.56% 56 Missing ⚠️
...ckages/federation-sdk/src/services/room.service.ts 77.60% 43 Missing ⚠️
packages/federation-sdk/src/sdk.ts 52.50% 38 Missing ⚠️
...federation-sdk/src/repositories/user.repository.ts 7.69% 36 Missing ⚠️
packages/federation-sdk/src/index.ts 43.54% 35 Missing ⚠️
...ration-sdk/src/services/appservice-room.service.ts 30.43% 32 Missing ⚠️
...tion-sdk/src/repositories/room-alias.repository.ts 17.85% 23 Missing ⚠️
...vice/src/repositories/appservice-txn.repository.ts 20.00% 20 Missing ⚠️
... and 8 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #394      +/-   ##
==========================================
+ Coverage   50.64%   51.06%   +0.41%     
==========================================
  Files         101      113      +12     
  Lines       11533    12708    +1175     
==========================================
+ Hits         5841     6489     +648     
- Misses       5692     6219     +527     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sampaiodiego sampaiodiego force-pushed the add-application-service-support branch from 1fa0373 to 4ee3063 Compare June 2, 2026 19:50

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 15 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/federation-sdk/src/services/event-sender.service.ts
Comment thread packages/federation-sdk/src/services/room.service.ts
Comment thread packages/appservice/src/services/namespace-matcher.service.ts
Comment thread packages/appservice/src/services/namespace-matcher.service.ts
Comment thread docs/bridge-architecture-guide.md Outdated
Comment thread packages/appservice/src/services/transaction-sender.service.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 14 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/appservice/src/services/event-router.service.ts">

<violation number="1" location="packages/appservice/src/services/event-router.service.ts:41">
P2: Removing the `routeEvent` wrapper breaks the mock in `packages/federation-sdk/src/services/event-sender.service.spec.ts`, which still expects `routeEvent` on `EventRouterService`. The actual `EventSenderService` code now calls `routePersistent`, so that test will throw a runtime error when it tries to invoke the missing method on the mock. Please update the test to mock and assert `routePersistent` instead.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/federation-sdk/src/index.ts
Comment thread packages/federation-sdk/src/services/room.service.ts Outdated
Comment thread packages/appservice/src/services/event-router.service.ts
Comment thread packages/appservice/src/services/namespace-matcher.service.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/appservice/src/services/transaction-sender.service.ts (1)

236-245: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Transport failures are logged/recorded as HTTP undefined.

When fetch() fails at the transport layer it resolves with ok: false and status: undefined (it does not throw), so this branch — not the catch — runs, logging status: undefined and calling markDown(..., 'HTTP undefined'). The down-marking is correct, but the reason is misleading for the common connection-failure case. Consider distinguishing response.status === undefined (connection failed) from a genuine non-2xx, mirroring ping.service.ts.

🪵 Suggested distinction
-		this.logger.warn({
-			msg: `Transaction delivery failed`,
-			asId: registration._id,
-			txnId,
-			status: response.status,
-		});
-
-		await this.txnRepo.markFailed(registration._id, txnId);
-		await this.stateRepo.markDown(registration._id, `HTTP ${response.status}`);
-		return false;
+		const reason = response.status === undefined ? 'Failed to connect to appservice' : `HTTP ${response.status}`;
+		this.logger.warn({ msg: 'Transaction delivery failed', asId: registration._id, txnId, status: response.status, reason });
+
+		await this.txnRepo.markFailed(registration._id, txnId);
+		await this.stateRepo.markDown(registration._id, reason);
+		return false;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/appservice/src/services/transaction-sender.service.ts` around lines
236 - 245, The failure handling in transaction-sender.service.ts logs and marks
transport errors as HTTP undefined, which is misleading when fetch() returns a
response with status unset. Update the non-ok response branch around the
logger.warn, txnRepo.markFailed, and stateRepo.markDown calls to distinguish
response.status === undefined from a real HTTP status, similar to
ping.service.ts. Use a clearer reason string for connection/transport failures
while keeping genuine non-2xx responses recorded with their actual HTTP status.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@packages/appservice/src/services/transaction-sender.service.ts`:
- Around line 236-245: The failure handling in transaction-sender.service.ts
logs and marks transport errors as HTTP undefined, which is misleading when
fetch() returns a response with status unset. Update the non-ok response branch
around the logger.warn, txnRepo.markFailed, and stateRepo.markDown calls to
distinguish response.status === undefined from a real HTTP status, similar to
ping.service.ts. Use a clearer reason string for connection/transport failures
while keeping genuine non-2xx responses recorded with their actual HTTP status.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0d5d88df-77c2-4d0a-b4cd-cc7a1ff75e11

📥 Commits

Reviewing files that changed from the base of the PR and between dfc5386 and 9fc2919.

📒 Files selected for processing (21)
  • docs/bridge-architecture-guide.md
  • packages/appservice/src/index.ts
  • packages/appservice/src/models/appservice.model.ts
  • packages/appservice/src/repositories/appservice-state.repository.ts
  • packages/appservice/src/repositories/appservice-txn.repository.ts
  • packages/appservice/src/services/event-router.service.spec.ts
  • packages/appservice/src/services/event-router.service.ts
  • packages/appservice/src/services/namespace-matcher.service.spec.ts
  • packages/appservice/src/services/namespace-matcher.service.ts
  • packages/appservice/src/services/ping.service.ts
  • packages/appservice/src/services/registration.service.ts
  • packages/appservice/src/services/transaction-sender.service.ts
  • packages/core/src/utils/fetch.ts
  • packages/federation-sdk/src/index.ts
  • packages/federation-sdk/src/repositories/room-alias.repository.ts
  • packages/federation-sdk/src/services/directory.service.ts
  • packages/federation-sdk/src/services/edu.service.ts
  • packages/federation-sdk/src/services/event-sender.service.spec.ts
  • packages/federation-sdk/src/services/event-sender.service.ts
  • packages/federation-sdk/src/services/message.service.ts
  • packages/federation-sdk/src/services/room.service.ts
🚧 Files skipped from review as they are similar to previous changes (13)
  • packages/appservice/src/index.ts
  • packages/federation-sdk/src/services/directory.service.ts
  • packages/appservice/src/repositories/appservice-txn.repository.ts
  • packages/appservice/src/services/event-router.service.spec.ts
  • packages/appservice/src/models/appservice.model.ts
  • packages/federation-sdk/src/services/edu.service.ts
  • packages/federation-sdk/src/services/event-sender.service.spec.ts
  • packages/appservice/src/repositories/appservice-state.repository.ts
  • packages/appservice/src/services/registration.service.ts
  • packages/federation-sdk/src/services/event-sender.service.ts
  • packages/appservice/src/services/namespace-matcher.service.spec.ts
  • packages/appservice/src/services/namespace-matcher.service.ts
  • packages/federation-sdk/src/services/room.service.ts
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
  • GitHub Check: cubic · AI code reviewer
  • GitHub Check: Code Quality Checks(lint, test, tsc)
🔇 Additional comments (8)
packages/appservice/src/services/ping.service.ts (1)

52-74: LGTM!

packages/federation-sdk/src/repositories/room-alias.repository.ts (1)

25-35: LGTM!

docs/bridge-architecture-guide.md (1)

115-131: LGTM!

packages/federation-sdk/src/index.ts (1)

156-192: LGTM!

packages/core/src/utils/fetch.ts (1)

123-184: LGTM!

Also applies to: 233-248

packages/appservice/src/services/event-router.service.ts (1)

30-68: LGTM!

packages/appservice/src/services/transaction-sender.service.ts (1)

60-87: LGTM!

Also applies to: 149-200

packages/federation-sdk/src/services/message.service.ts (1)

54-54: LGTM!

Also applies to: 128-128, 182-182, 262-262, 289-289, 334-334, 372-372

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 17 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/federation-sdk/src/index.ts">

<violation number="1" location="packages/federation-sdk/src/index.ts:192">
P2: `init()` starts a background retry scheduler via `transactionSender.startRetryScheduler()`, but there is no corresponding teardown path exposed. The underlying `setInterval` in `TransactionSenderService` is not `.unref()`'d, so it will keep the Node process alive indefinitely. In tests, workers, or short-lived scripts this prevents clean exit, and the existing `stopRetryScheduler()` method is never called. Consider either exposing a shutdown hook from `init()` that calls `stopRetryScheduler()`, or `.unref()`'ing the timer so it does not block process exit.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread packages/appservice/src/services/event-router.service.ts
Comment thread packages/appservice/src/repositories/appservice-state.repository.ts
Comment thread packages/federation-sdk/src/index.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6 issues found across 11 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/federation-sdk/src/sdk.ts">

<violation number="1" location="packages/federation-sdk/src/sdk.ts:312">
P1: Recovery is entirely process-local (`recoverers` is an in-memory `Map`) while the DOWN state and transaction backlog live in shared MongoDB. This means if the instance that marked a bridge DOWN exits before its local `setTimeout` recoverer fires, no other instance will pick up the shared backlog — `sendTransaction` simply queues new events without starting a recoverer, and `startRecoverersForDownServices` is only invoked at `setConfig` time. In a multi-instance deployment the bridge can stay permanently stuck in DOWN with undelivered transactions until a config reload or restart. Consider adding a lightweight periodic background poller or a state-change watcher so another instance can adopt recovery of bridges that are DOWN but have no live recoverer.</violation>
</file>

<file name="packages/appservice/src/services/ping.service.ts">

<violation number="1" location="packages/appservice/src/services/ping.service.ts:59">
P2: The `/timed out/i` regex only catches the inactivity timeout from the fetch wrapper (`'Request timed out after 20s'`) and misses the Node.js transport-level `ETIMEDOUT` errors (e.g., `'connect ETIMEDOUT ...'`). Those genuine connection timeouts will be returned as `M_CONNECTION_FAILED` even though the type union advertises a dedicated `M_CONNECTION_TIMEOUT` code. Broaden the check so transport-level timeouts are also classified correctly — for example by also matching `ETIMEDOUT` or `\btimeout\b` in the reason string.</violation>
</file>

<file name="packages/appservice/src/services/transaction-sender.service.ts">

<violation number="1" location="packages/appservice/src/services/transaction-sender.service.ts:101">
P2: Delivery failures no longer preserve or update the persisted appservice error cause and timestamp after the first failure.

`putTransaction` returns `false` for both HTTP and transport exceptions without calling `markDown`, so the specific HTTP status or exception message that previously reached the state record is lost. The first failure is instead stored as the generic message `Failed to deliver transaction ${txnId}` via `startRecoverer`. Subsequent retry failures in `retry`/`backoffAndReschedule` only reschedule and never call `markDown` again, so `lastError` and `lastErrorAt` remain stuck at the initial outage time even as the bridge continues to fail.

Since `AppServiceState` fields are exposed through the state repository, operators and health consumers receive stale, low-information diagnostics. Consider updating `lastError`/`lastErrorAt` on each retry failure (e.g. in `putTransaction` or `backoffAndReschedule`) so the persisted state reflects the most recent actionable cause.</violation>

<violation number="2" location="packages/appservice/src/services/transaction-sender.service.ts:128">
P1: Runtime bridge configuration changes do not update or cancel already-running recoverers. `RegistrationService.initialize()` rebuilds the registration cache (and is documented to be safe for repeated calls, e.g. after `setConfig`), but `TransactionSenderService.startRecoverersForDownServices` skips any `asId` that already has a recoverer. This means:

- Changing the bridge URL or rotating `hsToken` while the bridge is down leaves the recoverer delivering queued backlog events to the old URL with the old credentials.
- Removing the bridge config entirely still leaves the recoverer alive; when its timer fires and finds an empty queue, it calls `markUp` and recreates state that `initialize` had just cleaned up.

Consider adding a method to `TransactionSenderService` that reconciles or cancels recoverers when registrations are rebuilt, or refresh the `CachedAppService` reference inside the recoverer before each retry.</violation>

<violation number="3" location="packages/appservice/src/services/transaction-sender.service.ts:165">
P1: A transient database error can permanently stall an appservice recoverer. The `scheduleRetry` timer callback only logs errors escaping `retry()`; it does not reschedule. Inside `retry()`, MongoDB operations (`getOldestPending`, `complete`, `markUp`) are not individually protected, so any thrown error kills the recoverer with no retry scheduled. You could close the gap by rescheduling in the `.catch()` handler, or by wrapping the body of `retry()` in a catch-all that calls `backoffAndReschedule`.</violation>

<violation number="4" location="packages/appservice/src/services/transaction-sender.service.ts:193">
P1: A TOCTOU race condition between `getOldestPending` returning null and `markUp` committing can permanently strand a transaction. If a concurrent `sendTransaction` runs after `recoverers.delete(asId)` but before `markUp` completes, it still sees the bridge as `down`, persists the transaction, and returns without inline delivery. Once `markUp` finishes the bridge is UP with no recoverer running, and `sendTransaction` only ever delivers its own transaction — it never drains backlog — so the orphaned row remains in MongoDB forever. Consider making the empty-queue/mark-up transition atomic, or doing another `getOldestPending` check after `markUp` before returning, so that any transaction inserted during the window gets picked up.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

// Registrations only exist after the config is applied, so this is the
// boot hook for resuming recovery of bridges persisted as down. Idempotent
// (per-bridge recoverer guard), so repeated setConfig calls are safe.
await this.transactionSenderService.startRecoverersForDownServices(this.registrationService.getAll());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Recovery is entirely process-local (recoverers is an in-memory Map) while the DOWN state and transaction backlog live in shared MongoDB. This means if the instance that marked a bridge DOWN exits before its local setTimeout recoverer fires, no other instance will pick up the shared backlog — sendTransaction simply queues new events without starting a recoverer, and startRecoverersForDownServices is only invoked at setConfig time. In a multi-instance deployment the bridge can stay permanently stuck in DOWN with undelivered transactions until a config reload or restart. Consider adding a lightweight periodic background poller or a state-change watcher so another instance can adopt recovery of bridges that are DOWN but have no live recoverer.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/federation-sdk/src/sdk.ts, line 312:

<comment>Recovery is entirely process-local (`recoverers` is an in-memory `Map`) while the DOWN state and transaction backlog live in shared MongoDB. This means if the instance that marked a bridge DOWN exits before its local `setTimeout` recoverer fires, no other instance will pick up the shared backlog — `sendTransaction` simply queues new events without starting a recoverer, and `startRecoverersForDownServices` is only invoked at `setConfig` time. In a multi-instance deployment the bridge can stay permanently stuck in DOWN with undelivered transactions until a config reload or restart. Consider adding a lightweight periodic background poller or a state-change watcher so another instance can adopt recovery of bridges that are DOWN but have no live recoverer.</comment>

<file context>
@@ -304,6 +306,10 @@ export class FederationSDK {
+		// Registrations only exist after the config is applied, so this is the
+		// boot hook for resuming recovery of bridges persisted as down. Idempotent
+		// (per-bridge recoverer guard), so repeated setConfig calls are safe.
+		await this.transactionSenderService.startRecoverersForDownServices(this.registrationService.getAll());
 	}
 
</file context>

* Boot path: resume recovery for bridges persisted as DOWN. Bridges
* persisted as UP are left alone (matches Synapse).
*/
async startRecoverersForDownServices(appservices: CachedAppService[]): Promise<void> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Runtime bridge configuration changes do not update or cancel already-running recoverers. RegistrationService.initialize() rebuilds the registration cache (and is documented to be safe for repeated calls, e.g. after setConfig), but TransactionSenderService.startRecoverersForDownServices skips any asId that already has a recoverer. This means:

  • Changing the bridge URL or rotating hsToken while the bridge is down leaves the recoverer delivering queued backlog events to the old URL with the old credentials.
  • Removing the bridge config entirely still leaves the recoverer alive; when its timer fires and finds an empty queue, it calls markUp and recreates state that initialize had just cleaned up.

Consider adding a method to TransactionSenderService that reconciles or cancels recoverers when registrations are rebuilt, or refresh the CachedAppService reference inside the recoverer before each retry.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/appservice/src/services/transaction-sender.service.ts, line 128:

<comment>Runtime bridge configuration changes do not update or cancel already-running recoverers. `RegistrationService.initialize()` rebuilds the registration cache (and is documented to be safe for repeated calls, e.g. after `setConfig`), but `TransactionSenderService.startRecoverersForDownServices` skips any `asId` that already has a recoverer. This means:

- Changing the bridge URL or rotating `hsToken` while the bridge is down leaves the recoverer delivering queued backlog events to the old URL with the old credentials.
- Removing the bridge config entirely still leaves the recoverer alive; when its timer fires and finds an empty queue, it calls `markUp` and recreates state that `initialize` had just cleaned up.

Consider adding a method to `TransactionSenderService` that reconciles or cancels recoverers when registrations are rebuilt, or refresh the `CachedAppService` reference inside the recoverer before each retry.</comment>

<file context>
@@ -69,138 +84,168 @@ export class TransactionSenderService {
-	private async pollAppService(appservice: CachedAppService): Promise<void> {
-		const asId = appservice.registration._id;
-		const state = await this.stateRepo.getState(asId);
+	async startRecoverersForDownServices(appservices: CachedAppService[]): Promise<void> {
+		for (const appservice of appservices) {
+			const asId = appservice.registration._id;
</file context>

for (;;) {
// eslint-disable-next-line no-await-in-loop
const txn = await this.txnRepo.getOldestPending(asId);
if (!txn) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: A TOCTOU race condition between getOldestPending returning null and markUp committing can permanently strand a transaction. If a concurrent sendTransaction runs after recoverers.delete(asId) but before markUp completes, it still sees the bridge as down, persists the transaction, and returns without inline delivery. Once markUp finishes the bridge is UP with no recoverer running, and sendTransaction only ever delivers its own transaction — it never drains backlog — so the orphaned row remains in MongoDB forever. Consider making the empty-queue/mark-up transition atomic, or doing another getOldestPending check after markUp before returning, so that any transaction inserted during the window gets picked up.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/appservice/src/services/transaction-sender.service.ts, line 193:

<comment>A TOCTOU race condition between `getOldestPending` returning null and `markUp` committing can permanently strand a transaction. If a concurrent `sendTransaction` runs after `recoverers.delete(asId)` but before `markUp` completes, it still sees the bridge as `down`, persists the transaction, and returns without inline delivery. Once `markUp` finishes the bridge is UP with no recoverer running, and `sendTransaction` only ever delivers its own transaction — it never drains backlog — so the orphaned row remains in MongoDB forever. Consider making the empty-queue/mark-up transition atomic, or doing another `getOldestPending` check after `markUp` before returning, so that any transaction inserted during the window gets picked up.</comment>

<file context>
@@ -69,138 +84,168 @@ export class TransactionSenderService {
+			for (;;) {
+				// eslint-disable-next-line no-await-in-loop
+				const txn = await this.txnRepo.getOldestPending(asId);
+				if (!txn) {
+					this.recoverers.delete(asId);
 					// eslint-disable-next-line no-await-in-loop
</file context>


private scheduleRetry(recoverer: Recoverer): void {
const delayMs = 2 ** recoverer.backoffCounter * 1000;
recoverer.timer = setTimeout(() => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: A transient database error can permanently stall an appservice recoverer. The scheduleRetry timer callback only logs errors escaping retry(); it does not reschedule. Inside retry(), MongoDB operations (getOldestPending, complete, markUp) are not individually protected, so any thrown error kills the recoverer with no retry scheduled. You could close the gap by rescheduling in the .catch() handler, or by wrapping the body of retry() in a catch-all that calls backoffAndReschedule.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/appservice/src/services/transaction-sender.service.ts, line 165:

<comment>A transient database error can permanently stall an appservice recoverer. The `scheduleRetry` timer callback only logs errors escaping `retry()`; it does not reschedule. Inside `retry()`, MongoDB operations (`getOldestPending`, `complete`, `markUp`) are not individually protected, so any thrown error kills the recoverer with no retry scheduled. You could close the gap by rescheduling in the `.catch()` handler, or by wrapping the body of `retry()` in a catch-all that calls `backoffAndReschedule`.</comment>

<file context>
@@ -69,138 +84,168 @@ export class TransactionSenderService {
+
+	private scheduleRetry(recoverer: Recoverer): void {
+		const delayMs = 2 ** recoverer.backoffCounter * 1000;
+		recoverer.timer = setTimeout(() => {
+			recoverer.timer = null;
+			void this.retry(recoverer).catch((err) => {
</file context>

// with status undefined, and the failure reason is only retrievable
// through the rejecting body accessors (see errorResponse in utils/fetch.ts).
if (response.status === undefined) {
const reason = await response.text().catch((r) => String(r));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The /timed out/i regex only catches the inactivity timeout from the fetch wrapper ('Request timed out after 20s') and misses the Node.js transport-level ETIMEDOUT errors (e.g., 'connect ETIMEDOUT ...'). Those genuine connection timeouts will be returned as M_CONNECTION_FAILED even though the type union advertises a dedicated M_CONNECTION_TIMEOUT code. Broaden the check so transport-level timeouts are also classified correctly — for example by also matching ETIMEDOUT or \btimeout\b in the reason string.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/appservice/src/services/ping.service.ts, line 59:

<comment>The `/timed out/i` regex only catches the inactivity timeout from the fetch wrapper (`'Request timed out after 20s'`) and misses the Node.js transport-level `ETIMEDOUT` errors (e.g., `'connect ETIMEDOUT ...'`). Those genuine connection timeouts will be returned as `M_CONNECTION_FAILED` even though the type union advertises a dedicated `M_CONNECTION_TIMEOUT` code. Broaden the check so transport-level timeouts are also classified correctly — for example by also matching `ETIMEDOUT` or `\btimeout\b` in the reason string.</comment>

<file context>
@@ -50,18 +53,22 @@ export class PingService {
-					errcode: 'M_CONNECTION_FAILED',
-					error: 'Failed to connect to appservice',
-				};
+				const reason = await response.text().catch((r) => String(r));
+				if (/timed out/i.test(reason)) {
+					return { errcode: 'M_CONNECTION_TIMEOUT', error: reason };
</file context>

}

// The row stays as the head of the backlog; its ephemeral riders are lost.
await this.startRecoverer(appservice, `Failed to deliver transaction ${txnId}`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Delivery failures no longer preserve or update the persisted appservice error cause and timestamp after the first failure.

putTransaction returns false for both HTTP and transport exceptions without calling markDown, so the specific HTTP status or exception message that previously reached the state record is lost. The first failure is instead stored as the generic message Failed to deliver transaction ${txnId} via startRecoverer. Subsequent retry failures in retry/backoffAndReschedule only reschedule and never call markDown again, so lastError and lastErrorAt remain stuck at the initial outage time even as the bridge continues to fail.

Since AppServiceState fields are exposed through the state repository, operators and health consumers receive stale, low-information diagnostics. Consider updating lastError/lastErrorAt on each retry failure (e.g. in putTransaction or backoffAndReschedule) so the persisted state reflects the most recent actionable cause.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/appservice/src/services/transaction-sender.service.ts, line 101:

<comment>Delivery failures no longer preserve or update the persisted appservice error cause and timestamp after the first failure.

`putTransaction` returns `false` for both HTTP and transport exceptions without calling `markDown`, so the specific HTTP status or exception message that previously reached the state record is lost. The first failure is instead stored as the generic message `Failed to deliver transaction ${txnId}` via `startRecoverer`. Subsequent retry failures in `retry`/`backoffAndReschedule` only reschedule and never call `markDown` again, so `lastError` and `lastErrorAt` remain stuck at the initial outage time even as the bridge continues to fail.

Since `AppServiceState` fields are exposed through the state repository, operators and health consumers receive stale, low-information diagnostics. Consider updating `lastError`/`lastErrorAt` on each retry failure (e.g. in `putTransaction` or `backoffAndReschedule`) so the persisted state reflects the most recent actionable cause.</comment>

<file context>
@@ -69,138 +84,168 @@ export class TransactionSenderService {
+		}
+
+		// The row stays as the head of the backlog; its ephemeral riders are lost.
+		await this.startRecoverer(appservice, `Failed to deliver transaction ${txnId}`);
 	}
 
</file context>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants