Skip to content

[ci] Refactor CI of test cases in integration module#702

Open
weiqingy wants to merge 1 commit into
apache:mainfrom
weiqingy:161-impl
Open

[ci] Refactor CI of test cases in integration module#702
weiqingy wants to merge 1 commit into
apache:mainfrom
weiqingy:161-impl

Conversation

@weiqingy
Copy link
Copy Markdown
Collaborator

Linked issue: #161

Purpose of change

Splits the Python unit-test job from the integration-test job using a new @pytest.mark.integration marker.

Today, 10 tests under python/flink_agents/integrations/*/tests/ need live external services (Ollama, DashScope, OpenAI, Azure, Anthropic, Chroma) or optional Python deps (mem0), but they live in the unit-test job and silently skip in 5 of 6 matrix cells — only the Python 3.10 cell auto-pulls Ollama. When Ollama regresses, exactly one cell reports failure, which looks like environmental noise.

After this PR:

  • python/pyproject.toml — new [tool.pytest.ini_options] block registers the integration marker and enables strict_markers = true.
  • 10 test files under python/flink_agents/integrations/*/tests/ — module-level pytestmark = pytest.mark.integration (list form for test_mem0_vector_store.py to preserve its existing skipif).
  • tools/ut.sh — both if $run_e2e; then ... else ... fi blocks (uv branch + pip fallback) updated:
    • ut-python (run_e2e=false): appends -m "not integration" to the existing -k "not e2e_tests" filter. Drops 70 tests from the unit-test job (was 518, now 448).
    • it-python (run_e2e=true): runs two sequential pytest invocations — existing -k "e2e_tests_integration" (27 tests, unchanged) and new -m "integration" (70 tests) — with aggregated exit codes and an exit-5 trap on the integration arm to defend against -m selector typos that --strict-markers cannot catch.

No .github/workflows/ci.yml change required.

integrations/mcp/tests/test_mcp.py is intentionally not tagged: it self-hosts an MCP server via multiprocessing and runs deterministically without external deps.

Tests

Locally verified empirical contract (pytest --collect-only -q):

Selector Tests Notes
pytest -m "not integration" -k "not e2e_tests" 448 New ut-python arm (was 518)
pytest -m "integration" 70 New it-python arm 2
pytest -k "e2e_tests_integration" 27 it-python arm 1, unchanged
no selector 552 Unchanged

Overlap between -m "integration" and -k "e2e_tests_integration" is 0 — disjoint by directory structure.

Additional verification:

  • ./tools/lint.sh -c passes
  • ./tools/check-license.sh passes
  • ./tools/ut.sh -p passes locally (448 passed, 104 deselected, ~28s)
  • pytest --markers shows the registered integration marker with description
  • Typo'd @pytest.mark.intergration triggers a collection error (strict_markers = true enforcement is active)

API

No public API changes. This PR is CI plumbing plus a pytest marker added to existing test files.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue. labels May 24, 2026
Tag the 10 tests under python/flink_agents/integrations/*/tests/ that
require live external services (Ollama, DashScope, OpenAI, Azure,
Anthropic, Chroma) or optional Python deps (mem0) with a new
@pytest.mark.integration marker. Register the marker in
python/pyproject.toml with strict_markers enabled. Update tools/ut.sh
to split unit and integration arms via the new marker:

  - ut-python (run_e2e=false): adds -m "not integration" to the
    existing -k "not e2e_tests" filter. Drops 70 tests from the
    unit-test job (was 518, now 448), which previously skipped
    silently in 5 of 6 matrix cells.
  - it-python (run_e2e=true): runs two sequential pytest invocations
    -- existing -k "e2e_tests_integration" (27 tests, unchanged) and
    new -m "integration" (70 tests) -- with aggregated exit codes and
    an exit-5 trap on the integration arm to defend against -m
    selector typos that --strict-markers cannot catch.

No .github/workflows/ci.yml change required.

test_mcp.py is intentionally not tagged: it self-hosts an MCP server
via multiprocessing and runs deterministically without external deps.

Closes apache#161
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant