diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000000..bddf562c4dc4c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,121 @@ +# AGENTS.md — Red Hat Vector Fork Context + +This file provides context for AI agents and developers working in this repository. It covers only what is specific to this Red Hat fork. For generic Vector development commands, project structure, and Rust conventions, see the [upstream AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md). + +## What This Repo Is + +A fork of [vectordotdev/vector](https://github.com/vectordotdev/vector) maintained by Red Hat for OpenShift Logging. Vector runs as a DaemonSet on every OpenShift node, collecting application and infrastructure logs. It is deployed and configured by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator) — Vector is never run standalone in production. + +**Upstream remote:** `vectordev` (git@github.com:vectordotdev/vector.git) +**Fork origin:** `origin` (git@github.com:ViaQ/vector.git) + +## Critical: Branch Selection + +**Never commit code to `rh-main`.** It contains only documentation. + +| Branch | Vector Version | Rust | Use | +|--------|---------------|------|-----| +| `v0.47.0-rh` | v0.47.0 | 1.85 | Stable — OpenShift 6.2–6.5 | +| `v0.54.0-rh` | v0.54.0 | 1.92 | Next — future OpenShift releases | + +Always check out the appropriate version branch before making changes: + +```bash +git checkout v0.47.0-rh # stable +git checkout v0.54.0-rh # next +``` + +When a fix applies to both branches, submit separate PRs per branch. + +## Build Quick Reference + +The Makefile defaults to `FEATURES=ocp-logging` on Linux (not upstream's `default` feature set). + +```bash +make build # cargo build --release --no-default-features --features ocp-logging +make test # cargo nextest run ... --features ocp-logging --test-threads 1 +make fmt # rustfmt +make check-clippy # clippy lints +``` + +Key differences from upstream builds: +- **`--features ocp-logging`** — curated subset of Vector, not the full feature set +- **`--test-threads 1`** — single-threaded tests to avoid OOM in CI +- **`lld` linker** — configured in `.cargo/config.toml` (GNU ld OOMs during linking) +- **`CFLAGS=-g0 -O3`** — set in the build target for release optimization + +## `ocp-logging` Feature Scope + +This is the curated subset of Vector components enabled for OpenShift. Only these are compiled into the Red Hat binary. + +**Sources:** stdin, file_descriptor, file, journald, kubernetes_logs, prometheus, internal_metrics, demo_logs, http_server, syslog, opentelemetry + +**Transforms:** route, dedupe, filter, remap, log_to_metric, lua, throttle, reduce, detect_exceptions + +**Sinks (v0.47.0-rh):** aws_cloudwatch_logs, aws_s3, azure_monitor_logs, elasticsearch, file, kafka, loki, console, prometheus, gcp, splunk_hec, http, socket, opentelemetry + +**Sinks (v0.54.0-rh adds):** azure_logs_ingestion + +**Also enabled:** api, api-client, unix + +Components not in this list are **not available** in the Red Hat binary even though they exist in the source tree. + +## Fork-Specific Files + +These files exist in this fork but not in upstream: + +| Path | Purpose | +|------|---------| +| `patch/openssl/` | Forked openssl crate — enables FIPS compliance and TLS security profile enforcement (LOG-2552, LOG-3398) | +| `patch/hyper/` | Forked hyper crate — HTTP behavior modifications | +| `thirdparty/` | Bundled binaries: `protoc`, `cargo-nextest`, `cargo-deny` (no external downloads in CI) | +| `Dockerfile` | Developer/local build image (UBI 9 + rustup) | +| `Dockerfile.art` | ART production build (UBI 9 minimal + RPM rust-toolset, `make build-offline`) | +| `Dockerfile.in` | OSBS/Brew template with `REMOTE_SOURCE` markers for Cachito | +| `Dockerfile.unit` | CI unit test runner | +| `OWNERS` | Kubernetes-style code review assignments | +| `CLAUDE.md` | Stub pointing to this file | + +## Patched Crates (`[patch.crates-io]`) + +**Both branches:** +- `openssl = { path = "patch/openssl" }` — Red Hat's forked openssl crate for FIPS and TLS security profiles +- `hyper = { path = "patch/hyper" }` — HTTP behavior patches +- `ntapi` — pinned revision for Windows alignment bug (inherited from upstream) + +**v0.47.0-rh only** (removed in v0.54.0-rh): +- `tokio-util` — vectordotdev fork for framed-read-continue-on-error +- `nix` — vectordotdev fork for memfd gnu/musl support + +When modifying TLS or HTTP behavior, check `patch/openssl/` and `patch/hyper/` — the standard crate.io versions are not used. + +## Jira Conventions + +Issues are tracked in [Red Hat JIRA](https://issues.redhat.com) under the **LOG** project. + +- Prefix commit messages with the Jira ticket: `LOG-XXXX: description` +- Reference tickets in PR descriptions +- Use conventional commit format for PR titles: `fix(scope): description` + +## Code Review (OWNERS) + +``` +approvers: jcantrill, alanconway, xperimental +reviewers: jcantrill, Clee2691, syedriko, vparfonov, cahartma +``` + +Dockerfile changes get the `midstream/Dockerfile` label automatically. + +## Upstream References + +For generic Vector development, use these upstream docs (they apply to this fork's Rust code): + +| Topic | Upstream Document | +|-------|-------------------| +| Dev commands, project structure, common patterns | [AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md) | +| Contribution workflow, PR format | [CONTRIBUTING.md](https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md) | +| Internal engine architecture (topology, fanout, buffering) | [docs/ARCHITECTURE.md](https://github.com/vectordotdev/vector/blob/master/docs/ARCHITECTURE.md) | +| Rust coding conventions (agent-oriented) | [docs/RUST_STYLE.md](https://github.com/vectordotdev/vector/blob/master/docs/RUST_STYLE.md) | +| Code style (logging, metrics, error handling, concurrency) | [STYLE.md](https://github.com/vectordotdev/vector/blob/master/STYLE.md) | +| Developer setup, testing, benchmarking | [docs/DEVELOPING.md](https://github.com/vectordotdev/vector/blob/master/docs/DEVELOPING.md) | +| Component specification | [docs/specs/component.md](https://github.com/vectordotdev/vector/blob/master/docs/specs/component.md) | diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000000000..e6f0c0589d884 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,130 @@ +# Architecture — Red Hat Vector Fork + +This document covers what is architecturally specific to this fork. For Vector's internal engine architecture (topology, fanout, buffering, component construction), see the upstream [docs/ARCHITECTURE.md](https://github.com/vectordotdev/vector/blob/master/docs/ARCHITECTURE.md). + +## Deployment Model + +Vector runs as a **DaemonSet** on every OpenShift node, managed by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator). It is not deployed standalone. + +- Configuration is injected via ConfigMap by the operator +- The operator provides a `run.sh` entrypoint script +- Vector collects container logs from `/var/log/containers/` (file source) and infrastructure logs from journald +- Pod metadata is extracted from file paths and enriched by transforms + +## `ocp-logging` Feature + +The fork defines a custom Cargo feature `ocp-logging` in `Cargo.toml` that compiles only the Vector components needed for OpenShift. The Makefile sets `FEATURES=ocp-logging` by default on Linux. + +### Enabled Components + +**Sources:** stdin, file_descriptor, file, journald, kubernetes_logs, prometheus, internal_metrics, demo_logs, http_server, syslog, opentelemetry + +**Transforms:** route, dedupe, filter, remap, log_to_metric, lua, throttle, reduce, detect_exceptions + +**Sinks:** + +| Sink | v0.47.0-rh | v0.54.0-rh | +|------|:----------:|:----------:| +| aws_cloudwatch_logs | yes | yes | +| aws_s3 | yes | yes | +| azure_logs_ingestion | — | yes | +| azure_monitor_logs | yes | yes | +| elasticsearch | yes | yes | +| file | yes | yes | +| kafka | yes | yes | +| loki | yes | yes | +| console | yes | yes | +| prometheus | yes | yes | +| gcp | yes | yes | +| splunk_hec | yes | yes | +| http | yes | yes | +| socket | yes | yes | +| opentelemetry | yes | yes | + +Components not in `ocp-logging` exist in the source tree but are not compiled into the Red Hat binary. + +## Patched Dependencies + +The fork carries local patches for two crates via `[patch.crates-io]` in `Cargo.toml`: + +### `patch/openssl/` + +Forked from the `openssl` crate. Two purposes: + +1. **FIPS compliance** (LOG-2552): Vector uses OpenSSL instead of upstream's Ring for all TLS operations, enabling FIPS-validated cryptography. +2. **TLS security profiles** (LOG-3398): Enforces OpenShift's `TLSSecurityProfile` settings (cipher suites, TLS versions) at the Vector level, so cluster administrators control the security posture globally. + +### `patch/hyper/` + +Forked from the `hyper` crate. Contains HTTP behavior modifications for reliability in OpenShift environments. + +### Other patches (v0.47.0-rh only) + +- `tokio-util` — vectordotdev fork for framed-read-continue-on-error behavior +- `nix` — vectordotdev fork for memfd gnu/musl support + +Both were dropped in v0.54.0-rh (fixed upstream). + +## Build System + +### Dockerfiles + +| File | Base | Rust | Purpose | +|------|------|------|---------| +| `Dockerfile` | `ubi9/ubi` | rustup | Local/developer builds | +| `Dockerfile.art` | `ubi9/ubi-minimal` | RPM `rust-toolset` | ART production image (shipped to customers). Uses `make build-offline` for air-gapped builds | +| `Dockerfile.in` | `ubi9/ubi-minimal` | RPM | OSBS/Brew template. Has `## EXCLUDE BEGIN ##` markers for OSBS processing. Uses `REMOTE_SOURCE` for Cachito-prefetched dependencies | +| `Dockerfile.unit` | `ubi9/ubi` | rustup | CI unit test runner. Selective COPY for build cache efficiency | + +### Build constraints + +- **`lld` linker**: Configured in `.cargo/config.toml`. GNU ld runs out of memory during linking. +- **`protoc` bundled**: Pre-built binaries in `thirdparty/protoc/` (architecture-aware). No network download during build. +- **`cargo-nextest` and `cargo-deny` bundled**: In `thirdparty/` for CI reproducibility. +- **Single-threaded tests**: `--test-threads 1` in the Makefile `test` target to limit memory usage. + +### CI/CD + +No in-repo CI pipelines. Builds happen in Red Hat's internal ART/OSBS infrastructure: +- `Dockerfile.art` → production image via ART (Automated Release Tooling) +- `Dockerfile.in` → processed by OSBS (OpenShift Build Service) / Brew + +GitHub Actions on the dev branches are inherited from upstream and mostly not Red Hat-specific. + +## Key Design Decisions + +### OpenSSL over Ring (LOG-2552) + +**Decision:** Replace Ring with OpenSSL for all cryptographic operations. +**Why:** FIPS compliance requires a validated crypto library. Ring is not FIPS-validated. OpenSSL is. +**Trade-off:** Adds the `patch/openssl/` maintenance burden and prevents trivial upstream merges of TLS-related code. + +### TLS Security Profile Enforcement (LOG-3398) + +**Decision:** All TLS connections respect OpenShift's `TLSSecurityProfile` (cipher suites, min/max TLS version). +**Why:** OpenShift cluster administrators expect uniform TLS policy enforcement across all components. +**Depends on:** The OpenSSL patch above — upstream's Ring-based TLS does not support security profile configuration. + +### `detect_exceptions` Transform (LOG-6155) + +**Decision:** A Red Hat-specific transform for multiline exception detection (Java stack traces, Python tracebacks, etc.) with configurable `message_key`. +**Why:** Upstream's multiline handling is less flexible. OpenShift workloads heavily use Java, and stack traces must be grouped into single log events. + +## Differences Between Version Branches + +| Aspect | v0.47.0-rh | v0.54.0-rh | +|--------|-----------|-----------| +| Upstream base | Vector v0.47.0 | Vector v0.54.0 | +| Rust edition | 2021 | 2024 | +| MSRV (rust-toolchain.toml) | 1.85 | 1.92 | +| `azure_logs_ingestion` sink | — | enabled | +| `tokio-util` / `nix` patches | present | removed (fixed upstream) | +| GCP Workload Identity Federation | — | LOG-9171 | +| TLS curve configuration | — | LOG-8968 | + +## Upstream Sync Strategy + +1. **Version-based merges:** Each branch is based on a specific upstream tag (e.g., v0.47.0). Upstream changes are incorporated by creating a new branch from the next upstream tag and replaying Red Hat patches. +2. **Patch minimization:** Features are contributed upstream when possible. Once accepted, the fork patch is simplified or removed on the next branch. +3. **Cherry-picks:** Critical fixes from upstream are cherry-picked into active branches between version merges. +4. **Conflict zones:** `Cargo.toml` (feature definitions, patch section) and TLS-related code are the primary conflict areas during merges. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000000000..c3170642553f7 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000000..2e69b54b1a90b --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,67 @@ +# Contributing — Red Hat Vector Fork + +This guide covers what is specific to contributing to this fork. For generic Vector development workflow, coding standards, and testing, see the [upstream CONTRIBUTING.md](https://github.com/vectordotdev/vector/blob/master/CONTRIBUTING.md). + +## Branching Strategy + +| Branch | Purpose | +|--------|---------| +| `rh-main` | Documentation only — **never commit code here** | +| `v0.47.0-rh` | Stable development — OpenShift 6.2–6.5 | +| `v0.54.0-rh` | Next-gen development — future releases | + +## Submitting Changes + +1. Check out the target version branch: `git checkout v0.47.0-rh` +2. Create a feature branch: `git checkout -b LOG-XXXX-short-description` +3. Make changes, test with `make test` +4. Push to your fork and open a PR against the version branch (not `rh-main`) +5. Include the Jira ticket ID (LOG-XXXX) in the PR title and description + +If a fix applies to multiple version branches, submit a separate PR for each. + +## Commit Messages + +Use the Jira ticket ID and conventional commit format: + +``` +fix(file-source): handle symlink rotation during log collection + +Fixes LOG-7506 +``` + +PR titles follow upstream's [conventional commits](https://www.conventionalcommits.org) format: +``` +feat(scope): description +fix(scope): description +chore(scope): description +``` + +## Code Review + +Reviews follow the [OWNERS](OWNERS) file: + +- **Approvers:** jcantrill, alanconway, xperimental +- **Reviewers:** jcantrill, Clee2691, vparfonov, cahartma + +PRs require at least one approval. Significant changes require two approvers. + +## Red Hat-Specific Code + +When adding patches that diverge from upstream: + +- Reference the Jira ticket in code comments explaining why +- If the feature could be upstreamed, note it in the PR description +- Update [ARCHITECTURE.md](ARCHITECTURE.md) if the change affects design decisions or the `ocp-logging` feature scope + +## Contributing Back to Upstream + +We minimize fork divergence by upstreaming changes when possible: + +1. Implement and test the fix in this fork first +2. If the fix is generic (not OpenShift-specific), propose it to [upstream Vector](https://github.com/vectordotdev/vector) +3. Once accepted upstream, simplify or remove the fork patch on the next upstream merge + +## Filing Issues + +File issues at [Red Hat JIRA](https://issues.redhat.com) under the **LOG** project. Include [must-gather](https://github.com/openshift/cluster-logging-operator/tree/master/must-gather) output for debugging. diff --git a/README.md b/README.md index 2296e4f8f1211..99102434c5455 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,33 @@ -# Vector +# Vector — Red Hat OpenShift Logging -This repo is a fork of [vector](https://github.com/vectordotdev/vector) and contains patches carried by Red Hat OpenShift Logging. This is a log collector and forwarder that resides on each OpenShift node to gather application and node logs. Please refer to the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator) for details regarding the operator that deploys and configures this image. This image is intended to be run in conjunction with the configuration and `run.sh` files provided by the operator. Experiences with the image outside that context may vary. +This is a fork of [Vector](https://github.com/vectordotdev/vector) maintained for Red Hat OpenShift Logging. Vector is a high-performance observability data pipeline written in Rust. This fork carries patches for OpenShift integration, FIPS-compliant TLS (OpenSSL), and reliability improvements. -The `rh-main` branch is empty except for this file. The branches used by various Red Hat releases are summarized here: +Vector runs as a DaemonSet on each OpenShift node, collecting application and infrastructure logs. It is deployed and configured by the [cluster-logging-operator](https://github.com/openshift/cluster-logging-operator). This image is intended to run with the configuration and `run.sh` files provided by the operator. + +## Documentation + +| Document | Purpose | +|----------|---------| +| [AGENTS.md](AGENTS.md) | AI agent and developer quick reference — fork-specific context, build commands, patched crates | +| [CONTRIBUTING.md](CONTRIBUTING.md) | How to submit changes — branching, PRs, code review, Jira conventions | +| [ARCHITECTURE.md](ARCHITECTURE.md) | Fork architecture — deployment model, `ocp-logging` feature scope, build system, design decisions | +| [Upstream AGENTS.md](https://github.com/vectordotdev/vector/blob/master/AGENTS.md) | Generic Vector development commands, project structure, Rust conventions | + +## Quick Start (building from source) + +```bash +git checkout v0.47.0-rh # or v0.54.0-rh +make build # builds with --features ocp-logging +make test # runs tests with --test-threads 1 +``` + +## Release Branches + +The `rh-main` branch contains only documentation. All development happens on version-specific branches: | Release | Branch | Vector Version | Status | |---------|-------------|----------------|--------| +| next | v0.54.0-rh | v0.54.0 | Development | | 6.5 | v0.47.0-rh | v0.47.0 | Current | | 6.4 | v0.47.0-rh | v0.47.0 | Current | | 6.3 | v0.47.0-rh | v0.47.0 | EOL | @@ -30,6 +52,7 @@ This project varies from the upstream with the following features: | LOG-6789 | [Resolve error when using AWS credentials file authentication](https://github.com/ViaQ/vector/pull/197) | 6.2 | [Accepted](https://github.com/vectordotdev/vector/pull/22831) | | LOG-7013 | [Update cloudwatch logs max event size to match new AWS limit](https://github.com/ViaQ/vector/pull/201) | 6.2 | [Accepted](https://github.com/vectordotdev/vector/pull/22886) | | LOG-7417 | [Fix loki event timestamp out of range panic ](https://github.com/ViaQ/vector/pull/217) | 0.37.1-rh | [Cherry-pick](https://github.com/vectordotdev/vector/pull/20780) | + ## Issues Any issues can be filed at [Red Hat JIRA](https://issues.redhat.com). Please