[WIP][Cosmos] Gateway V2 endpoint-probe: topology-safe gate lifecycle (follow-up to #49437) by jeet1995 · Pull Request #49796 · Azure/azure-sdk-for-java

jeet1995 · 2026-07-13T23:44:51Z

Context

Follow-up to the review comment thread on #49437. Addresses the endpoint-probe gate lifecycle correctness issues raised in review.

Changes

Issue #1 — stale-green topology gate (`EndpointProbeClient`)

applyCycleResult previously recomputed the routing gate against the stale captured endpoint snapshot. A slow in-flight cycle (topology T1) completing after an overlapping trigger had already advanced the topology to T1 + E3 would republish a green verdict from T1, flipping the gate globally true while the newly-added region E3 was still unproven → misrouting.
Fix: added AtomicReference<Collection<URI>> latestTopology; the final gate is computed against latestTopology.get() (freshest topology), never the captured snapshot. Newly-added regions are proven on the next refresh delta (conservative-until-proven).

New-A — probe disposal cancels the in-flight cycle (`GlobalEndpointManager`)

fireThinClientProbeCycle used getAndSet(...).dispose() on every fire. Because runProbeCycle single-flights, a rapid second fire returned instantly as a no-op while the disposed handle was the genuine running cycle → rapid refreshes meant the gate never flipped green.
Fix: set(...) the tracked Disposable without disposing the predecessor; dispose only in close().

New-B — init-time opt-out permanently disabled probe (`RxDocumentClientImpl`)

Wiring was guarded such that an init-time hard opt-out (COSMOS.THINCLIENT_ENABLED=false) left the probe client null forever, so a later false → unset transition could never enable probe-gated routing.
Fix: always wire the thin-client probe HTTP client at init; the probe cycle self-no-ops when the flag is explicitly true/false and only gates routing when the flag is unset.

Tests

ThinClientProbeWiringTests: added overlappingProbeFire_doesNotCancelActiveCycle (New-A) and staleTopologyGrowth_doesNotFlipGateForUnprobedRegion (Issue #1). Both proven red/green. Full suite green.

Verification

azure-cosmos build: success.
ThinClientProbeWiringTests: full suite green (failsafe -Punit).

…ndpoint.

… GW Mode) cases.

…ryPlanSupport

… QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x002B) and x-ms-cosmos-query-version (0x002C) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. Previously these headers were only set as HTTP headers by QueryPlanRetriever and were lost when QueryPlan was routed through the proxy path, since ThinClientStoreModel serializes requests as RNTBD (not HTTP headers). IDs match server-side proxy definitions per ADO PR 1982503. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add testThinClientChangeFeedFullRange covering FeedRange.forFullRange() across multiple partition keys, and testThinClientChangeFeedPartitionKey covering FeedRange.forLogicalPartition with exact doc count + PK validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Documents all 59 thin client E2E tests across query (50), point operations (3), change feed (3), and stored procedures (3) with SQL, query features covered, and known account-side blockers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… QueryPlan proxy routing Add RNTBD token mappings for x-ms-cosmos-supported-query-features (0x00F0) and x-ms-cosmos-query-version (0x00F1) so the thin client proxy can read these values from the RNTBD body when processing QueryPlan requests. IDs are provisional (0x00F0, 0x00F1) — must be coordinated with server-side proxy team. See ADO PR 1982503 for the proxy-side design. Note: The design doc listed 0x002B/0x002C but those are already assigned to PartitionKey/PartitionKeyRangeId in the Java SDK. Using 0x00F0/0x00F1 to avoid ID collision until final server-side IDs are assigned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…BD instructions - Fix testGetCurrentDateTime: assert ISO 8601 format instead of exact match (gateway and proxy return slightly different timestamps) - Add DefaultAzureCredential support via COSMOS.USE_AAD_AUTH system property for accounts with disableLocalAuth=true - Add RNTBD class reference as .github/instructions/rntbd.instructions.md - Add pom.xml system properties for THINCLIENT_ENABLED, HTTP2_ENABLED, USE_AAD_AUTH - Add beforeSuiteReuse mode for degraded accounts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Switch baseline from Gateway V1 to Direct TCP to avoid JVM config interference (THINCLIENT_ENABLED/HTTP2_ENABLED affect Gateway V1) - Assert :10250 endpoint only on Gateway V2 results (not baseline) - Rename helpers: assertDirectAndThinClientMatch (was gateway) - Document seedTestData schema in Javadoc - Remove 'Expected to fail' comments (account has vector search enabled) - Clean up class/method Javadoc Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…TestNG group Tag the existing ThinClient*E2ETest classes (and ThinClientTestBase) with both the `thinclient` and `thinclientEndpointProbe` groups so the same test bodies run in either CI lane against a different ambient enablement path: - thinclient lane: -DCOSMOS.THINCLIENT_ENABLED=true (explicit opt-in) - thinclient-endpoint-probe lane: flag unset -> endpoint connectivity probe drives routing Delete the 5 duplicate ThinClientEndpointProbe* copies; the probe suite XML discovers by group across com.azure.cosmos.*, so it auto-picks up the retagged classes with zero duplication. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The consolidated thin-client E2E tests are dual-tagged with the thinclientEndpointProbe group, but the shared @BeforeSuite/@AfterSuite in TestSuiteBase (which creates and deletes the SHARED_* collections) was still gated only on the thinclient group. Under the probe lane's group filter the suite setup never ran, leaving the shared collections null and causing an NPE in cleanUpContainer during @BeforeClass. Adding the group makes both lanes green. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Update the gw connCfg assertions to expect gwV2Cto:PT5S instead of gwV2Cto:n/a, matching the intentional diagnostics change where the thin-client (gateway V2) connect timeout is emitted by default when COSMOS.THINCLIENT_ENABLED is unset. This also resolves the cascading sessionRetryOptionsInDiagnostics failure caused by full() leaking a system property when its assertion threw before cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ThinClient is default-enabled when COSMOS.THINCLIENT_ENABLED is unset on this branch, so the UserAgent suffix now carries the |F4 feature flag. Update the stale assertions to mirror RxDocumentClientImpl.addUserAgentSuffix dynamically (ThinClient default-on + Http2/Http2PingHealth) instead of hardcoding the suffix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Refactor EndpointProbeClient to an add-only proven-healthy endpoint cache, matching the .NET implementation (PR azure-cosmos-dotnet-v3#5970). Once an endpoint is proven ThinClient-routable it is never pruned; the routing gate is evaluated against a volatile snapshot of the current topology so a transiently vanished proven region does not gate current routing and is not re-probed when it re-appears. Probe cycles only probe the delta of new, not-yet-proven endpoints on each account refresh. Add unit tests covering vanish/re-appear and unsupported-new-region scenarios (region added without support blocks the gate; removing it restores routing). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ffix Fold ThinClient into generateHttp2OptedInUserAgentIfRequired so the helper dynamically mirrors production feature-flag suffixing, and remove a double-wrap of directClientUserAgent (already suffixed at construction). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

On the implicit/probe path (COSMOS.THINCLIENT_ENABLED unset), thin-client routing depends on the endpoint probe greenlighting all current regions. Under fault injection the probe is not guaranteed to converge, so requests may legitimately fall back to Gateway V1 (:443). Assert thin-client routing only when thin-client is explicitly opted in (THINCLIENT_ENABLED=true), the sole config where routing is deterministic because the probe is bypassed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…parsing, probe-client close, fire-and-forget probe cycle, single-flight tests - Configs: parse COSMOS.THINCLIENT_ENABLED via explicit true/false whitelist (parseTriStateThinClientEnabled); any other value logs a warning and is treated as unset (null -> probe-gated) instead of Boolean.parseBoolean silently collapsing every non-"true" string to a hard opt-out. - GlobalEndpointManager: close() now cancels the in-flight probe-cycle subscription and closes the EndpointProbeClient so a stale cycle drops its result. Probe cycle is fired fire-and-forget on force-refresh and topology refresh so init()/cross-region retry never block on the probe; routing stays on Gateway V1 until the probe proves the proxy endpoints. - Tests: add ConfigsTests.thinClientEnabledInvalidValueTreatedAsUnset and two EndpointProbeClientTests covering single-flight overlap skip and closed-mid-cycle result-drop (gate stays conservative). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…into jeet1995/thin-client-probe-flow

useThinClient() now reflects config-eligibility only; the test must gate on actual per-request routing (read locations + probe/opt-in) to match where the query lands. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ed() - Gate probe wiring in RxDocumentClientImpl on canThinClientBeUsed() so an explicit THINCLIENT_ENABLED=true still wires the probe (routing bypasses it), covering true->null runtime transitions. Removed dead canThinClientBeImplicitlyEnabled() from ThinClientConnectivityConfig and updated javadoc/comments/pom profile references accordingly. - Strengthen single-flight test with a post-release third cycle probing a new region to confirm the guard releases correctly. - Skip connectivity-probe requests in GatewayReadConsistencyStrategySpyWireTest request selectors: with the probe now wired under explicit enablement, its bodyless POST /connectivity-probe to :10250 was being captured by the V2 selectors, causing an NPE in collectHttpBody. The spy now filters probe traffic and selects the actual data request. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…e ReadFeedDatabases cleanup, de-dup endpoint-probe lane - IncrementalChangeFeedProcessorTest.getCurrentState: block on CFP start()/stop() instead of fire-and-forget subscribe() + fixed sleep, so lease bootstrap failures under shared-account throttling surface deterministically instead of leaving zero leases (size 0 assertion flake). - ReadFeedDatabasesTest: use CosmosDatabaseForTest.generateId() so leaked DBs are reclaimable by the janitor; rewrite afterClass to delete all DBs in parallel with bounded per-op timeouts and 3x method timeout. - live-thinclient-endpoint-probe-platform-matrix.json: reduce lane to -Pthinclient-endpoint-probe only, removing -Pquery and -Pconsistency-overrides which duplicated generic suites concurrently against the same shared account. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Mirror safeDeleteDatabase's swallow-everything contract: drop the blockLast(Duration) overload (which throws IllegalStateException on timeout) in favor of blockLast(), rely on the per-op timeout to bound runtime under the method timeout, and add an outer try/catch backstop so no error escapes cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ecycle - EndpointProbeClient (Issue #1): recompute the routing gate against the latest observed topology (new AtomicReference<Collection<URI>> latestTopology) instead of the stale captured snapshot, so a slow in-flight cycle can no longer republish a green verdict for a topology that has since grown an unproven region. Newly-added regions are proven on the next refresh delta. - GlobalEndpointManager (New-A): fireThinClientProbeCycle now set()s the tracked probe Disposable without disposing the previous one; disposal happens only in close(), so overlapping refreshes no longer cancel the genuine in-flight probe cycle. - RxDocumentClientImpl (New-B): always wire the thin-client probe HTTP client at init; the probe cycle self-no-ops when COSMOS.THINCLIENT_ENABLED is explicitly true/false, enabling later probe-gated enablement when unset. - ThinClientProbeWiringTests: add overlapping-fire and stale-topology-growth regression tests (both proven red/green). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0802fe95-dc4d-4ee2-81c8-f707fbffc161

Resolved conflicts in the four probe-flow files by keeping the follow-up correctness fixes (Issue #1 latest-topology gate, New-A probe-lifecycle, New-B always-wire) which supersede the merged Azure#49437 baseline in upstream/main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0802fe95-dc4d-4ee2-81c8-f707fbffc161

jeet1995 · 2026-07-13T23:57:56Z

/azp run java - cosmos - tests

azure-pipelines · 2026-07-13T23:58:31Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

Follow-up to the Gateway V2 (thin-client) endpoint connectivity-probe rollout work, tightening probe lifecycle correctness under overlapping refreshes and runtime flag transitions to prevent stale “green” gates and inadvertent cancellation of active probe cycles.

Changes:

Ensures probe gate computation uses the latest observed topology (prevents stale-green clobbering after topology growth).
Adjusts probe-cycle subscription tracking and adds a runtime opt-in/opt-out no-op gate for probe traffic.
Adds targeted unit tests for overlapping fires and stale-topology growth scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java	Always wires the thin-client probe HTTP client at init to avoid permanent disablement from init-time opt-out.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/GlobalEndpointManager.java	Updates probe fire/disposal behavior and skips probe traffic when `COSMOS.THINCLIENT_ENABLED` is explicitly set.
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/EndpointProbeClient.java	Recomputes routing gate against the latest topology reference to avoid stale verdict publication.
sdk/cosmos/azure-cosmos-tests/src/test/java/com/azure/cosmos/implementation/ThinClientProbeWiringTests.java	Adds regression tests covering overlapping probe fires and topology growth during an in-flight cycle, plus runtime opt-out behavior.

+        // no-op emit), so the previously tracked subscription is either already complete or the ONLY
+        // cycle doing real probe I/O; cancelling it here would abort the sole active probe and stall
+        // the gate. In-flight work is bounded by the per-probe timeout and honors close().
+        this.thinClientProbeCycleDisposable.set(newDisposable);


jeet1995 · 2026-07-14T00:08:36Z

@sdkReviewAgent

fireThinClientProbeCycle previously called .set(newDisposable) on every fire. An overlapping fire loses EndpointProbeClient's single-flight CAS and returns a fast no-op subscription, but .set() still overwrote the handle to the real in-flight cycle. close() then disposed the no-op, leaving the live probe I/O to self-terminate only on the per-probe timeout. Use getAndUpdate to keep the already-tracked live predecessor and only install the new handle when nothing live is tracked (won single-flight) or the predecessor is disposed. Adds regression test overlappingProbeFire_closeStillCancelsTheLiveCycle asserting handle identity after an overlapping no-op fire and that close() disposes the live cycle. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0802fe95-dc4d-4ee2-81c8-f707fbffc161

jeet1995 · 2026-07-14T00:30:01Z

/azp run java - cosmos - tests

azure-pipelines · 2026-07-14T00:30:47Z

Azure Pipelines successfully started running 1 pipeline(s).

jeet1995 and others added 30 commits January 20, 2026 18:20

Route Execute Stored Procedure requests to Thin Proxy endpoint.

4a83556

Route Execute Stored Procedure and QueryPlan requests to Thin Proxy e…

325ff0a

…ndpoint.

Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC +…

eb9a83e

… GW Mode) cases.

Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC +…

1492b36

… GW Mode) cases.

Ensure QueryPlan gets routed to Gateway Service Endpoint (in non-TC +…

2c8a5f3

… GW Mode) cases.

Obtain List<Range<String>> from List<PartitionKeyInternal>.

1e14a7b

Obtain List<Range<String>> from List<PartitionKeyInternal>.

91116c9

Adding query + thin-client tests.

688a5ac

Fixing tests.

8ae321f

Fixing tests.

e76c84c

Fixing tests.

7f3bad8

Addressing review comments.

7600598

Addressing review comments.

281bc88

Refactor thin-client E2E tests based on operation type.

1d171f5

Refactor thin-client E2E tests based on operation type.

9f1b4ee

Refactor thin-client E2E tests based on operation type.

cfe720b

Refactor thin-client E2E tests based on operation type.

8f1615b

Refactor thin-client E2E tests based on operation type.

b037a1d

Refactor thin-client E2E tests based on operation type.

8e28d82

Refactor thin-client E2E tests based on operation type.

2be1433

Refactor thin-client E2E tests based on operation type.

4b44cd5

Merge branch 'Azure:main' into AzCosmos_GatewayV2_QueryPlanSupport

afa9811

Refactor thin-client E2E tests based on operation type.

b07b104

Merge remote-tracking branch 'azure/main' into AzCosmos_GatewayV2_Que…

22db578

…ryPlanSupport

jeet1995 and others added 15 commits July 9, 2026 17:03

Refactoring.

7cc8055

Merge branch 'main' of https://github.com/jeet1995/azure-sdk-for-java …

709f556

…into jeet1995/thin-client-probe-flow

github-actions Bot added the Cosmos label Jul 13, 2026

jeet1995 marked this pull request as ready for review July 13, 2026 23:56

Copilot AI review requested due to automatic review settings July 13, 2026 23:56

jeet1995 requested review from a team and kirankumarkolli as code owners July 13, 2026 23:56

Copilot started reviewing on behalf of jeet1995 July 13, 2026 23:59 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP][Cosmos] Gateway V2 endpoint-probe: topology-safe gate lifecycle (follow-up to #49437)#49796

[WIP][Cosmos] Gateway V2 endpoint-probe: topology-safe gate lifecycle (follow-up to #49437)#49796
jeet1995 wants to merge 154 commits into
Azure:mainfrom
jeet1995:thin-client-probe-flow

jeet1995 commented Jul 13, 2026 •

edited

Loading

Uh oh!

jeet1995 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jeet1995 commented Jul 14, 2026

Uh oh!

jeet1995 commented Jul 14, 2026

Uh oh!

azure-pipelines Bot commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jeet1995 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Changes

Issue #1 — stale-green topology gate (EndpointProbeClient)

New-A — probe disposal cancels the in-flight cycle (GlobalEndpointManager)

New-B — init-time opt-out permanently disabled probe (RxDocumentClientImpl)

Tests

Verification

Uh oh!

jeet1995 commented Jul 13, 2026

Uh oh!

azure-pipelines Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jeet1995 commented Jul 14, 2026

Uh oh!

jeet1995 commented Jul 14, 2026

Uh oh!

azure-pipelines Bot commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jeet1995 commented Jul 13, 2026 •

edited

Loading

Issue #1 — stale-green topology gate (`EndpointProbeClient`)

New-A — probe disposal cancels the in-flight cycle (`GlobalEndpointManager`)

New-B — init-time opt-out permanently disabled probe (`RxDocumentClientImpl`)