Skip to content

Handle TaskNotFound in triggerer to prevent stale triggers from blocking batch#69852

Open
hanumanraj07 wants to merge 2 commits into
apache:mainfrom
hanumanraj07:fix-stale-trigger-blocking-workloads
Open

Handle TaskNotFound in triggerer to prevent stale triggers from blocking batch#69852
hanumanraj07 wants to merge 2 commits into
apache:mainfrom
hanumanraj07:fix-stale-trigger-blocking-workloads

Conversation

@hanumanraj07

Copy link
Copy Markdown

What

Catches TaskNotFound in the triggerer supervisor's _create_workload method so that one stale task-associated trigger cannot abort the entire batch of healthy trigger workloads.

Closes: #69841

Why

The triggerer supervisor loads the pinned serialized Dag for every task-associated trigger to check whether the task uses start_from_trigger. If the serialized Dag exists but no longer contains the deferred task, get_task() raises TaskNotFound. The exception escapes the per-batch loop in build_trigger_workloads, so one stale task instance prevents unrelated healthy triggers in the same batch from starting and can repeatedly restart the triggerer supervisor.

How

  • Wrapped get_task() in _create_workload with a try/except TaskNotFound block
  • When caught, logs a warning and sets task = None, falling through to the context-free RunTrigger return (matching behavior when serialized_dag_model is None)
  • Changed task.start_from_trigger check to task is not None and task.start_from_trigger
  • Added unit test verifying the fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prevent stale task triggers from blocking healthy workloads

1 participant