Skip to content

Google: Add retry-safe batch_id_prefix to DataprocCreateBatchOperator#69855

Open
amirmor1 wants to merge 2 commits into
apache:mainfrom
amirmor1:retry-with-unique-batch-id
Open

Google: Add retry-safe batch_id_prefix to DataprocCreateBatchOperator#69855
amirmor1 wants to merge 2 commits into
apache:mainfrom
amirmor1:retry-with-unique-batch-id

Conversation

@amirmor1

Copy link
Copy Markdown
Contributor

Add an opt-in batch_id_prefix to DataprocCreateBatchOperator so task retries can submit a new Dataproc batch with a unique ID instead of reusing a static batch_id.

Why

When a task retries with the same batch_id, Dataproc can return ALREADY_EXISTS and attach to the old batch. This change provides a built-in retry-safe path without changing existing explicit batch_id behavior.

What changes

  • add optional batch_id_prefix parameter
  • generate <batch_id_prefix>-<unique_suffix> per task attempt
  • keep batch_id and batch_id_prefix mutually exclusive
  • keep explicit batch_id behavior unchanged
  • update Dataproc operator docs
  • add unit tests for:
    • mutual exclusivity
    • prefix-based ID generation
    • execute flow with prefix
    • backward compatibility for explicit batch_id

Test plan

  • uv run ruff format providers/google/src/airflow/providers/google/cloud/operators/dataproc.py providers/google/tests/unit/google/cloud/operators/test_dataproc.py
  • uv run ruff check --fix providers/google/src/airflow/providers/google/cloud/operators/dataproc.py providers/google/tests/unit/google/cloud/operators/test_dataproc.py
  • uv run --project providers/google pytest providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_execute providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_execute_with_batch_id_prefix providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_build_unique_batch_id_from_prefix_preserves_prefix_verbatim providers/google/tests/unit/google/cloud/operators/test_dataproc.py::TestDataprocCreateBatchOperator::test_batch_id_and_batch_id_prefix_are_mutually_exclusive -xvs

@amirmor1 amirmor1 requested a review from shahar1 as a code owner July 14, 2026 07:19
@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants