Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/build-and-publish.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/cd.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/check-release-tag.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 23 additions & 1 deletion .github/workflows/checks.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/ci.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/dependency-update.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/fast-tests.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/gh-pages.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/matrix-all.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/matrix-exasol.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/matrix-python.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/merge-gate.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/periodic-validation.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/pr-merge.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions .github/workflows/report.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,8 @@ nosetests.xml

# Emacs
TAGS

# AI
.codex
.serena
specs/
7 changes: 6 additions & 1 deletion doc/changes/unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,18 @@

## Summary

In this minor release, the nox session `workflow:check` was added and is now used in the `checks.yml`.
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a project is not ready for this, they can remove the job with the .workflow-patcher.yml.

By adding a comment at the top of workflows maintained by the PTB, we make it clearer what is maintained vs not by us. Additionally, by adding when it was last updated, it gives the projects using the PTB a direct idea of when this was last touched before checking the git blame. [The PTB currently lacks a mechanism to say when a workflow was removed or not, but this comment could give an important hint. It also encourages user to NOT modify the .yml files but let the PTB maintain them.]

If this job is active in your CI, please double-check if additional files should be added into your project's `.gitattributes`.

## Bugfix

* #840: Added `export` plugin installation within `dependency-update.yml`

## Feature

* #722: Added check in `workflow:generate` to compare the generated and existing content before writing out
* #722: Added check in `workflow:generate` to compare the generated and existing content before writing out and nox session `workflow:check`
* #642: Added nox session `workflow:check` into the `checks.yml`
* #698: Added a comment in the top of all workflows maintained by the PTB

## Refactoring

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,12 @@ Add all Workflows to Your Project
poetry run -- nox -s workflow:generate -- all
After regenerating the workflows, you can verify that they are up-to-date with:
.. code-block:: shell
poetry run -- nox -s workflow:check -- all
.. warning::
Some workflows depend on other workflows. Please ensure you have all
the required workflows if you do not install all of them.
11 changes: 7 additions & 4 deletions doc/user_guide/features/github_workflows/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,15 @@ The PTB ships with configurable GitHub workflow templates covering the most comm
CI/CD setup variants for Python projects. The templates are defined in:
`exasol/toolbox/templates/github/workflows <https://github.com/exasol/python-toolbox/tree/main/exasol/toolbox/templates/github/workflows>`__.

The PTB provides a command line interface (CLI) for generating and updating actual
workflows from the templates.
The PTB provides a command line interface (CLI) for managing workflows from the templates.

.. code-block:: bash

poetry run -- nox -s workflow:generate --help
poetry run -- nox -s workflow:check --help

Use ``workflow:generate`` to create or update workflows and ``workflow:check`` to
compare the rendered workflow templates against the files in ``.github/workflows``.

.. attention::

Expand Down Expand Up @@ -62,8 +65,8 @@ Maintained by the PTB
* - ``checks.yml``
- Workflow call
- Executes many small & fast checks: builds documentation, validates
cross-references and links in the documentation to be valid, and runs various
linters (security, type checks, etc.).
cross-references and links in the documentation to be valid, runs various
linters (security, type checks, etc.), and validates PTB generated workflows.
* - ``ci.yml``
- Pull request
- Executes the continuous integration suite by calling ``merge-gate.yml`` and
Expand Down
10 changes: 10 additions & 0 deletions exasol/toolbox/config.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import inspect
import re
import warnings
from collections.abc import Callable
from pathlib import Path
Expand All @@ -19,12 +20,20 @@
)
from pydantic_core.core_schema import ValidationInfo

from exasol.toolbox import __version__
from exasol.toolbox.nox.plugin import (
METHODS_SPECIFIED_FOR_HOOKS,
PLUGIN_ATTR_NAME,
)
from exasol.toolbox.util.version import Version

WORKFLOW_HEADER_PREFIX = (
"# Generated and maintained by the exasol-toolbox.\n"
"# Last generated with exasol-toolbox version "
)

WORKFLOW_HEADER_PATTERN = rf"\A{re.escape(WORKFLOW_HEADER_PREFIX)}[^\n]+\.\n"


def get_methods_with_hook_implementation(
plugin_class: type[Any],
Expand Down Expand Up @@ -291,6 +300,7 @@ def github_template_dict(self) -> dict[str, Any]:
"minimum_python_version": self.minimum_python_version,
"os_version": self.os_version,
"python_versions": self.python_versions,
"workflow_header": f"{WORKFLOW_HEADER_PREFIX}{__version__}.",
"workflow_extension": {
"fast_tests": fast_tests_extension.is_file(),
"merge_gate": merge_gate_extension.is_file(),
Expand Down
31 changes: 27 additions & 4 deletions exasol/toolbox/nox/_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@
from noxconfig import PROJECT_CONFIG


def _create_parser() -> argparse.ArgumentParser:
def _create_parser(session_name: str) -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="nox -s workflow:generate",
usage="nox -s workflow:generate -- [-h] <workflow_choice>",
prog=f"nox -s {session_name}",
usage=f"nox -s {session_name} -- [-h] <workflow_choice>",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)

Expand All @@ -27,12 +27,35 @@ def _create_parser() -> argparse.ArgumentParser:
return parser


@nox.session(name="workflow:check", python=False)
def check_workflow(session: Session) -> None:
"""
Check the specified GitHub workflow or all of them to see if any differ from
the generated values. If any differ, an error is raised.
"""
parser = _create_parser("workflow:check")
args = parser.parse_args(session.posargs)

PROJECT_CONFIG.github_workflow_directory.mkdir(parents=True, exist_ok=True)

outdated_workflows = WorkflowOrchestrator(
workflow_choice=args.workflow_choice,
config=PROJECT_CONFIG,
).find_differing_workflows()

if outdated_workflows:
count = len(outdated_workflows)
count_label = "workflow is" if count == 1 else "workflows are"
workflow_list = "\n".join(f"- {workflow}" for workflow in outdated_workflows)
session.error(f"\n{count} {count_label} out of date:\n{workflow_list}")


@nox.session(name="workflow:generate", python=False)
def generate_workflow(session: Session) -> None:
"""
Generate or update the specified GitHub workflow or all of them.
"""
parser = _create_parser()
parser = _create_parser("workflow:generate")
args = parser.parse_args(session.posargs)

PROJECT_CONFIG.github_workflow_directory.mkdir(parents=True, exist_ok=True)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build & Publish

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/cd.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: CD

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Check Release Tag

on:
Expand Down
23 changes: 22 additions & 1 deletion exasol/toolbox/templates/github/workflows/checks.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Checks

on:
Expand Down Expand Up @@ -168,7 +169,6 @@ jobs:
id: check-format
run: poetry run -- nox -s format:check


build-package:
name: Build Package
runs-on: "(( os_version ))"
Expand All @@ -189,3 +189,24 @@ jobs:
- name: Build Package
id: build-package
run: poetry run -- nox -s package:check

check-workflows:
name: Check Workflows
runs-on: "(( os_version ))"
permissions:
contents: read
steps:
- name: Check out Repository
id: check-out-repository
uses: actions/checkout@v6

- name: Set up Python & Poetry Environment
id: set-up-python-and-poetry-environment
uses: exasol/python-toolbox/.github/actions/python-environment@v8
with:
python-version: "(( minimum_python_version ))"
poetry-version: "(( dependency_manager_version ))"

- name: Check Workflows
id: check-workflows
run: poetry run -- nox -s workflow:check -- all
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: CI

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Dependency Update

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/fast-tests.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Fast-Tests

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/gh-pages.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Publish Documentation

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/matrix-all.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build Matrix (All Versions)

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build Matrix (Exasol)

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Build Matrix (Python)

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/merge-gate.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Merge-Gate

on:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Periodic-Validation

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/pr-merge.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: PR-Merge

on:
Expand Down
1 change: 1 addition & 0 deletions exasol/toolbox/templates/github/workflows/report.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(( workflow_header ))
name: Status Report

on:
Expand Down
Loading