Feature request: Add deferrable support for invoking/waiting on Google Cloud Functions / Cloud Run functions

[BarBuccianti]

Description

Add provider-maintained deferrable support for invoking and waiting on Google Cloud Functions / HTTP Cloud Run functions from apache-airflow-providers-google.

Today, the Google provider supports deferrable execution for several Google Cloud services, including BigQuery, GCS, Dataflow, Pub/Sub, and Cloud Run Jobs via CloudRunExecuteJobOperator. However, there does not appear to be an equivalent deferrable operator/sensor/trigger pattern for HTTP Cloud Functions or HTTP Cloud Run functions.

CloudFunctionInvokeFunctionOperator is synchronous and documented as intended for testing purposes with limited traffic. For production workflows that trigger a function and then need to wait for asynchronous completion, users currently need to either:

maintain custom Airflow trigger/sensor/operator code, including Google auth, polling, timeout, retry, and failure semantics; or
implement an indirect durable-status pattern, such as having the function write completion state to BigQuery/GCS and waiting on that state with an existing deferrable sensor.
It would be useful to have a first-class deferrable pattern in the Google provider for this use case, for example a deferrable Cloud Functions / HTTP Cloud Run function operator or sensor that handles invocation, authenticated HTTP requests, polling/completion checks, timeout handling, retries, and failure propagation.

Use case/motivation

We have Airflow DAGs that trigger Google Cloud Functions / HTTP Cloud Run functions to perform asynchronous work. The function invocation itself is short, but the downstream processing can take longer, and Airflow needs to wait for the work to complete before continuing the DAG.

Because there is no provider-maintained deferrable Cloud Functions / HTTP Cloud Run function sensor/trigger today, using a synchronous task or regular sensor would occupy worker resources while waiting. To avoid that, we currently use a workaround where the function writes completion/status data to BigQuery, and Airflow waits on that status using an existing deferrable BigQuery sensor.

This works, but it adds extra infrastructure and indirection only to compensate for the missing deferrable Cloud Functions / HTTP function pattern. A provider-supported deferrable operator/sensor would reduce maintenance burden, avoid custom triggerer code, and make this pattern more consistent with other Google provider integrations such as BigQuery, GCS, Pub/Sub, Dataflow, dbt-style async workflows, and Cloud Run Jobs.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jroachgolf84]

cc: @shahar1

[jonathanleek]

I can take a look at this tonight

[RNHTTR]

This is a good feature request and should be implemented, but if you need a workaround solution in the meantime, you can probably use HTTPOperator and then an HTTPSensor with deferrable=true

[jonathanleek]

Based on my understanding of how things work, there are two ways to make an operator deferrable:

1. The operator passes a handle or execution id of some kind to the triggerer, which then polls a status endpoint until the job completes. We can't follow this pattern for Cloud functions, as there is no handle, and no status endpoint.

2. The open HTTP request is moved to the triggerer, which holds the idle connection open until it either gets a response or times out. I don't believe this works for this use case either, because the connection closes immediately after being sent. We could potentially implement this for synchronous long-running functions, but I don't believe it would solve @BarBuccianti 's use case as written. This is the pattern the HttpTrigger uses