[DCV-3797] Add Airflow metadata database cleanup how-to

docs/how-tos/airflow/cleanup-metadata-database.mdx

@@ -0,0 +1,85 @@
+---
+title: Clean up the Airflow metadata database in Datacoves
+sidebar_label: Cleanup Metadata Database
+description: "How to schedule cleanup of the Airflow metadata database in Datacoves with the datacoves_airflow_db_cleanup decorator to keep task_instance, log and other tables from growing unbounded."
+sidebar_position: 4
+---
+
+# Clean up the Airflow metadata database
+
+Over time the Airflow metadata database accumulates rows in tables such as `task_instance`, `log`, `job`, `xcom`, `dag_run` and `rendered_task_instance_fields`. Left unchecked it grows to several GB and queries on these tables slow down, which can eventually affect scheduler performance. Datacoves monitors the size of your metadata database and notifies account admins when it crosses recommended thresholds, but the cleanup itself is scheduled by you, in your own DAG, so you stay in control of the retention window.
+
+:::note
+This is available for Airflow 3 environments only.
+:::
+
+## Which tables are cleaned
+
+By default the cleanup targets the metadata tables that grow with every DAG run:
+
+- `task_instance`
+- `log`
+- `rendered_task_instance_fields`
+- `job`
+- `xcom`
+- `dag_run`
+
+These are the same tables Datacoves monitors for size. `dag` and `dag_version` are never touched, so your DAG history relationships are preserved. You only need to pass `table_names` if you want to restrict a run to a subset of these tables; leaving it unset cleans all of them, which is the recommended setup.
+
+Not sure how much each table holds or how far back your data goes? Run the decorator with `dry_run=True` first (see [Recommended first run](#recommended-first-run)): the task logs report, per table, how many rows would be removed, without deleting anything.
+
+## Cleanup decorator
+
+To clean up the metadata database, use an Airflow DAG with the Datacoves cleanup decorator:
+
+```python
+@task.datacoves_airflow_db_cleanup
+```
+
+By default it removes records older than 90 days from the tables listed above.
+
+The decorator can receive:
+
+- `retention_days`: keep records newer than this many days; older rows are deleted. Defaults to `90`.
+- `table_names`: restrict the cleanup to specific tables (any subset of the list above). Defaults to all of them.
+- `dry_run`: when `True`, report what would be deleted without deleting anything. Use this first.
+- `skip_archive`: when `True` (the default) records are deleted directly. When `False`, deleted rows are copied into `_airflow_deleted__*` archive tables first, giving you a recovery window at the cost of extra space.
+- `batch_size`: delete in batches of this many rows instead of all at once. Useful the first time you clean a very large table.
+- `connection_id`: the Airflow connection pointing at the api-server. Defaults to `datacoves_airflow_api`, which Datacoves creates and maintains for you.
+
+## Example DAG
+
+```python
+from pendulum import datetime
+
+from airflow.sdk import dag, task
+
+
+@dag(
+    default_args={
+        "start_date": datetime(2026, 1, 1),
+        "owner": "Noel",
+        "retries": 1,
+    },
+    description="Clean up the Airflow metadata database",
+    schedule="@weekly",
+    tags=["maintenance"],
+    catchup=False,
+)
+def airflow_db_cleanup():
+    @task.datacoves_airflow_db_cleanup(
+        retention_days=90,
+        # dry_run=True,  # uncomment for a first, non-destructive run
+    )
+    def cleanup():
+        pass
+
+    cleanup()
+
+
+airflow_db_cleanup()
+```
+
+## Recommended first run
+
+Cleanup is destructive. Run once with `dry_run=True` and check the task logs to confirm the volume of rows that would be removed, then switch to a real run. Start with a conservative `retention_days` (for example 90) and lower it gradually if the database is still large.