Skip to content

Feature: add json_logs config support for Celery worker stdout #68912

Description

@hanxdatadog

Apache Airflow version:
3.x

Is your feature request related to a problem?

Setting AIRFLOW__LOGGING__JSON_LOGS=True has no effect on the Celery worker's stdout. The worker always emits plain-text logs. Other components (API server, core logging) already respect this setting, but celery_command.py calls configure_logging without passing json_output:

if AIRFLOW_V_3_0_PLUS:
    from airflow.sdk.log import configure_logging
    configure_logging(output=sys.stdout.buffer)   # json_output defaults to False

Proposed solution

Add a [celery] json_logs option for a celery-specific override, falling back to the existing [logging] json_logs — consistent with how [logging] CELERY_LOGGING_LEVEL / [logging] LOGGING_LEVEL already work in the same file:

if AIRFLOW_V_3_0_PLUS:
    from airflow.sdk.log import configure_logging

    json_output = conf.getboolean("celery", "json_logs", fallback=None)
    if json_output is None:
        json_output = conf.getboolean("logging", "json_logs", fallback=False)
    configure_logging(output=sys.stdout.buffer, json_output=json_output)

Changes required:

  • providers/celery/src/airflow/providers/celery/cli/celery_command.py — the lookup above
  • Celery provider config schema — add [celery] json_logs option

Who would benefit?

Operators running the Celery executor on Kubernetes or any platform that routes worker stdout to a structured log collector.

Are you willing to submit a PR?

Yes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions