From d372c0701123b4acfe32f24c4fb53ad550ca899d Mon Sep 17 00:00:00 2001 From: Max Isbey <224885523+maxisbey@users.noreply.github.com> Date: Tue, 30 Jun 2026 10:34:08 +0000 Subject: [PATCH 1/3] Run the extension-tagged conformance scenarios in CI as baselined known failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tasks-* scenarios are extension-tagged: they never match a --spec-version filter and sit on the harness's pending list, so none of the existing server legs can select them — they were silently absent from CI with no skip line and no waiver. server-sse-polling (pending, removed at 2026-07-28) was equally unreachable. Add a bare --suite all server leg, which selects every scenario shipped with the pinned harness, and baseline the nine runnable tasks-* scenarios in expected-failures.yml. The stale-entry rule turns the baseline into a forced burn-down list as tasks support lands. server-sse-polling turns out to already pass. tasks-status-notifications is deliberately not baselined: the harness skips it unconditionally, and a baseline entry for a scenario with no failing checks is itself flagged stale. --- .../actions/conformance/expected-failures.yml | 21 ++++++++++++++++++- .github/workflows/conformance.yml | 16 ++++++++++++++ 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/.github/actions/conformance/expected-failures.yml b/.github/actions/conformance/expected-failures.yml index 4ad4123d0..aa31cb757 100644 --- a/.github/actions/conformance/expected-failures.yml +++ b/.github/actions/conformance/expected-failures.yml @@ -12,4 +12,23 @@ client: [] -server: [] +server: + # SEP-2663 (io.modelcontextprotocol/tasks): the SDK does not implement the + # tasks extension yet. These extension-tagged scenarios are selected only by + # the bare `--suite all` leg — extension scenarios never match a + # --spec-version filter and the active/draft suites exclude them — so these + # entries are inert for the other legs that read this file. + # + # `tasks-status-notifications` is intentionally NOT listed: the harness + # skips it unconditionally (pending its rewrite against subscriptions/ + # listen), and a baseline entry for a scenario with no failing checks is + # flagged stale. + - tasks-lifecycle + - tasks-capability-negotiation + - tasks-wire-fields + - tasks-request-state-removal + - tasks-mrtr-input + - tasks-request-headers + - tasks-dispatch-and-envelope + - tasks-required-task-error + - tasks-mrtr-composition diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index c73c1e2db..acd271a01 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -75,6 +75,22 @@ jobs: --suite all --spec-version 2026-07-28 --expected-failures ./.github/actions/conformance/expected-failures.2026-07-28.yml + - name: Run server conformance (all suite, extension scenarios) + # A bare `--suite all` (no --spec-version) selects every scenario + # shipped with the pinned harness — including the extension-tagged + # tasks-* scenarios and pending-listed ones like server-sse-polling, + # which no other leg reaches (extension scenarios never match a + # --spec-version filter, and the pending list keeps them out of the + # active suite). Running the full set keeps unimplemented surfaces + # visible as baselined known failures in expected-failures.yml instead + # of silent exclusions, and stays robust to scenarios moving between + # harness suite lists across pin bumps. `--suite pending` would cover + # the same union slightly faster; the full set is preferred for the + # self-contained run and for parity with typescript-sdk's CI. + run: >- + ./.github/actions/conformance/run-server.sh + --suite all + --expected-failures ./.github/actions/conformance/expected-failures.yml client-conformance: runs-on: ubuntu-latest From 76b7016c36c6920762cce264e2ec079c4c3cbbf3 Mon Sep 17 00:00:00 2001 From: Max Isbey <224885523+maxisbey@users.noreply.github.com> Date: Tue, 30 Jun 2026 10:34:37 +0000 Subject: [PATCH 2/3] Arm the Mcp-Param server-validation conformance checks with an x-mcp-header fixture tool MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five SEP-2243 Mcp-Param-* server-validation checks in http-custom-header-server-validation skipped silently because no everything-server tool carried an x-mcp-header annotation — the scenario printed a green 0-passed/0-failed in every leg that selected it. Add a test_x_mcp_header tool whose region parameter carries the annotation via json_schema_extra (mirroring the typescript-sdk fixture). The checks now execute: the accept-path checks pass and the reject-path checks fail because server-side Mcp-Param validation is not implemented yet, so the scenario is baselined in both expected-failures files. The stale-entry rule forces the entries out when validation lands. --- .../expected-failures.2026-07-28.yml | 5 ++++- .../actions/conformance/expected-failures.yml | 7 +++++++ .../mcp_everything_server/server.py | 20 ++++++++++++++++++- 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/.github/actions/conformance/expected-failures.2026-07-28.yml b/.github/actions/conformance/expected-failures.2026-07-28.yml index 504b46385..d64d05773 100644 --- a/.github/actions/conformance/expected-failures.2026-07-28.yml +++ b/.github/actions/conformance/expected-failures.2026-07-28.yml @@ -22,4 +22,7 @@ client: [] -server: [] +server: + # SEP-2243 Mcp-Param-* server-side validation is not implemented yet; see + # the matching entry in expected-failures.yml for the full rationale. + - http-custom-header-server-validation diff --git a/.github/actions/conformance/expected-failures.yml b/.github/actions/conformance/expected-failures.yml index aa31cb757..7c7e16a56 100644 --- a/.github/actions/conformance/expected-failures.yml +++ b/.github/actions/conformance/expected-failures.yml @@ -13,6 +13,13 @@ client: [] server: + # SEP-2243 Mcp-Param-* server-side validation is not implemented yet. The + # everything-server's `test_x_mcp_header` tool arms these checks (without an + # x-mcp-header-annotated tool the harness skips all of them silently); the + # accept-path checks pass, the reject-path checks fail until the server + # validates Mcp-Param headers against body params. Read by the draft leg and + # the bare `--suite all` leg; the 2026-07-28 leg carries its own entry. + - http-custom-header-server-validation # SEP-2663 (io.modelcontextprotocol/tasks): the SDK does not implement the # tasks extension yet. These extension-tagged scenarios are selected only by # the bare `--suite all` leg — extension scenarios never match a diff --git a/examples/servers/everything-server/mcp_everything_server/server.py b/examples/servers/everything-server/mcp_everything_server/server.py index e4f5db84f..8621c877a 100644 --- a/examples/servers/everything-server/mcp_everything_server/server.py +++ b/examples/servers/everything-server/mcp_everything_server/server.py @@ -11,7 +11,7 @@ import hmac import json import logging -from typing import Any +from typing import Annotated, Any import click from mcp.server import ServerRequestContext @@ -327,6 +327,24 @@ def test_error_handling() -> str: raise RuntimeError("This tool intentionally returns an error for testing") +@mcp.tool() +def test_x_mcp_header( + region: Annotated[ + str, + Field( + description="Mirrored into the Mcp-Param-Region header", + json_schema_extra={"x-mcp-header": "Region"}, + ), + ] = "", +) -> str: + """Tests SEP-2243 Mcp-Param-* server-side validation. + + Arms the http-custom-header-server-validation conformance scenario, which + skips when no tool with an `x-mcp-header` annotation is found. + """ + return f"region={region}" + + @mcp.tool() async def test_missing_capability(ctx: Context) -> str: """Tests that a handler-raised MISSING_REQUIRED_CLIENT_CAPABILITY surfaces as a top-level JSON-RPC error. From d9f928a77ba874a199c98709a0da07bc867fd3cf Mon Sep 17 00:00:00 2001 From: Max Isbey <224885523+maxisbey@users.noreply.github.com> Date: Tue, 30 Jun 2026 11:47:23 +0000 Subject: [PATCH 3/3] Bump the conformance pin to 0.2.0-alpha.8 and baseline the listen checks it surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit conformance 4944b268 (0.2.0-alpha.8, unpublished on npm — same sha256-verified pkg.pr.new mechanics as the previous pin) includes upstream #372: checks whose prerequisite is missing now fail with a 'Not testable' message instead of skipping outside the pass/fail denominator. For this repo that surfaces one new known failure: server-stateless fails its three subscriptions/listen MUST checks, because the everything-server's legacy resources/subscribe handlers make it advertise resources.subscribe in server/discover while subscriptions/listen answers -32601 — advertised-but-rejected is now a failure, not a skip. Baseline the scenario in both expected-failures files until the listen runtime lands. The two listChanged SHOULD checks remain legitimately skipped (declared false). The tasks-* scenarios fail with higher check counts (their cascade skips now fail too) but were already baselined; client legs are unaffected. --- .../expected-failures.2026-07-28.yml | 3 +++ .../actions/conformance/expected-failures.yml | 10 ++++++++++ .github/workflows/conformance.yml | 18 +++++++++--------- 3 files changed, 22 insertions(+), 9 deletions(-) diff --git a/.github/actions/conformance/expected-failures.2026-07-28.yml b/.github/actions/conformance/expected-failures.2026-07-28.yml index d64d05773..e61033b39 100644 --- a/.github/actions/conformance/expected-failures.2026-07-28.yml +++ b/.github/actions/conformance/expected-failures.2026-07-28.yml @@ -23,6 +23,9 @@ client: [] server: + # SEP-2575 subscriptions/listen is not implemented yet; see the matching + # entry in expected-failures.yml for the full rationale. + - server-stateless # SEP-2243 Mcp-Param-* server-side validation is not implemented yet; see # the matching entry in expected-failures.yml for the full rationale. - http-custom-header-server-validation diff --git a/.github/actions/conformance/expected-failures.yml b/.github/actions/conformance/expected-failures.yml index 7c7e16a56..efadd7d4d 100644 --- a/.github/actions/conformance/expected-failures.yml +++ b/.github/actions/conformance/expected-failures.yml @@ -13,6 +13,16 @@ client: [] server: + # SEP-2575 subscriptions/listen is not implemented yet. The everything- + # server's legacy resources/subscribe handlers make it advertise + # `resources.subscribe` in server/discover, and as of conformance #372 a + # server that advertises a subscription capability but answers + # subscriptions/listen with -32601 fails the three listen MUST checks + # ("Not testable") instead of skipping them. Remove this entry when the + # listen runtime lands. NOTE: while listed, this entry also masks new + # failures in the scenario's other 25 (currently passing) checks — the + # baseline is per-scenario, not per-check. + - server-stateless # SEP-2243 Mcp-Param-* server-side validation is not implemented yet. The # everything-server's `test_x_mcp_header` tool arms these checks (without an # x-mcp-header-annotated tool the harness skips all of them silently); the diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index acd271a01..35f8b6dcc 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -19,17 +19,17 @@ env: # Bump deliberately and reconcile both # .github/actions/conformance/expected-failures*.yml files in the same change. # - # Temporarily pinned to the pkg.pr.new build of conformance main@b18aa918 - # (the merge of #371, which fixes the http-custom-headers fixture's - # spec-forbidden `number`-typed x-mcp-header annotations) — no published - # release includes it yet. Pinned by commit SHA so the tarball cannot move - # under us; CONFORMANCE_PKG_SHA256 pins the bytes and the fetch-and-verify - # step below downloads, checks the digest, and repoints CONFORMANCE_PKG at the + # Temporarily pinned to the pkg.pr.new build of conformance main@4944b268 + # (0.2.0-alpha.8, which includes #372: fail checks whose prerequisite is + # missing instead of skipping them) — alpha.8 is not published to npm yet. + # Pinned by commit SHA so the tarball cannot move under us; + # CONFORMANCE_PKG_SHA256 pins the bytes and the fetch-and-verify step below + # downloads, checks the digest, and repoints CONFORMANCE_PKG at the # verified local copy. Repin to the next published @modelcontextprotocol/ - # conformance release (>0.2.0-alpha.7) once it ships, then drop + # conformance release (>=0.2.0-alpha.8) once it ships, then drop # CONFORMANCE_PKG_SHA256 and the fetch-and-verify steps. - CONFORMANCE_PKG: "https://pkg.pr.new/@modelcontextprotocol/conformance@b18aa918" - CONFORMANCE_PKG_SHA256: "e9f6bc25085b4692e988cbdbd024a4203d54a52a6aaa065376cf8ecaa09bb680" + CONFORMANCE_PKG: "https://pkg.pr.new/@modelcontextprotocol/conformance@4944b268" + CONFORMANCE_PKG_SHA256: "0f70c035782d319d72ab427653c5275db5c50429d59fae0241a645b33aeda1a7" jobs: server-conformance: