Add serialized Dag version diff engine#69864
Open
ephraimbuddy wants to merge 4 commits into
Open
Conversation
1 task
ephraimbuddy
force-pushed
the
dag-version-diff-core
branch
3 times, most recently
from
755ac9a to
77a23ad
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new serialization-layer “observed state” diff engine for comparing two stored serialized Dag payloads, plus a DagVersion model helper and unit tests to exercise both the pure diff and the DB-loading wrapper.
Changes:
- Introduces
build_serialized_dag_diffto compute a deterministic, bounded diff across serialized Dag schema versions (v1/v2 normalized to v3), with optional inclusion of raw values/digests. - Adds
DagVersion.get_diff(...)to load two DagVersion rows (including provenance and optional source) and return the diff result structure. - Adds extensive unit tests for determinism, classification/redaction behavior, truncation, legacy normalization, and DB-backed diff plumbing.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| airflow-core/src/airflow/serialization/dag_version_diff.py | New pure diff engine: canonicalization, redaction rules, change collection, and change categorization/impact. |
| airflow-core/src/airflow/models/dag_version.py | Adds DagVersion.get_diff wrapper, provenance/source helpers, and a new not-found exception type. |
| airflow-core/tests/unit/serialization/test_dag_version_diff.py | New unit tests covering diff determinism, classification, redaction, legacy normalization, and truncation. |
| airflow-core/tests/unit/models/test_dag_version.py | Adds DB-backed tests for DagVersion.get_diff (values/source status behaviors and missing-version handling). |
Dag authors and operators need a reliable way to understand how two stored Dag versions differ without downloading serialized payloads or manually comparing implementation details. This adds the shared comparison engine that later API and CLI surfaces build on. Historical serialized Dags can use different schema shapes, so comparisons normalize task, schedule, dependency, and metadata state before calculating changes. Malformed legacy payloads and missing historical data degrade to unavailable results instead of failing the whole request.
Users with version access need a useful structural comparison without learning identifiers or values derived from Dag code. The diff contract also needs to remain stable as serializer fields evolve.
Authorization state crosses an internal boundary before API integration, so invalid inputs must fail closed. Consistent not-found semantics also ensure future callers can preserve the required HTTP 404 behavior.
ephraimbuddy
force-pushed
the
dag-version-diff-core
branch
from
77a23ad to
28351e9
Compare
ephraimbuddy
requested review from
XD-DENG,
ashb and
bolkedebruin
as code owners
Contributor
Author
|
Extracted from #69818 and reworked |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the shared engine that computes an observed-state diff between two stored serialized Dag versions. Dag authors and operators need a reliable way to understand how two stored versions differ without downloading serialized payloads or manually comparing implementation details.
This is the foundation of a three-part stack; the REST API endpoint and the
airflow dags versions diffCLI command follow in separate PRs that build on this engine. It intentionally ships with no runtime caller yet — it is fully exercised by unit tests.What it does
build_serialized_dag_diff— pure, deterministic, bounded diff over two serialized payloads. Normalizes task/schedule/dependency/metadata state, converts legacy schema shapes (v1/v2 → v3), keys order-insensitive collections, categorizes each change (task, dependency, schedule, param, asset, callback, deadline, metadata, provenance) and its impact, and emits digests plus optional raw values.get_dag_version_diff— DB-facing wrapper that loads twoDagVersions, assembles provenance, and classifies optional source/values data as available, redacted, or unavailable.unavailableresult instead of failing the whole request.Testing
@pytest.mark.db_testclass coveringget_dag_version_diffend-to-end: observed-state diff, not-found handling, version-number validation, and the values/source status plumbing (available, redacted, unavailable-when-code-missing).No newsfragment: the engine has no user-facing surface on its own; user-facing notes ship with the API and CLI PRs.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) & GPT5.6 following the guidelines