docs: add wide events ADRs 029 and 030#459
Conversation
Formalize the internal wide-event audit model and unified /events API: - ADR 029: events table schema, categories, v2-statement emission, request-wide events for authenticated API calls, PII deny-by-default - ADR 030: GET /events with category filter, retention, shipped_at export, external threat-detection consumer pattern - Update resource map, ADR index, user-schema x-audit annotation Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
- Link oxidel ADR-023; drop sequence column (use created_at, id ordering) - Replace aggregate_* with entity_type/entity_id; remove sdk_* from events table - Replace shipped_at with per-sink event_deliveries table - Remove threat-detection section from ADR 030 Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds two new Architecture Decision Records (ADRs) that formalize a “wide events” audit model for nextgen’s relational storage, and updates surrounding design docs to reference the new audit/event concepts (including the x-audit allowlist approach for PII in audit payloads).
Changes:
- Add ADR 029 (internal wide-event model + emission/PII rules) and ADR 030 (events API surface, retention, and per-sink export tracking).
- Update ADR index + cross-links (ADR 028, API resource map, API design README).
- Update Flow Engine user schema annotations to use
x-audit(deny-by-default, allowlist for audit payload inclusion).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| docs/design/flowengine/user-schema.md | Switches audit-related schema annotation guidance to x-audit and documents deny-by-default payload inclusion. |
| docs/design/api/resource-map.md | Updates the resource map to a unified /events API aligned with the new ADRs. |
| docs/design/api/README.md | Updates API design README to reference ADR 030 for the partially-specified events surface (currently contains a broken relative link). |
| docs/adrs/README.md | Adds ADR 029 and ADR 030 to the ADR index. |
| docs/adrs/029-wide-events-internal-audit-primitive.md | Introduces ADR 029 defining the internal wide-events audit primitive model and emission boundaries. |
| docs/adrs/030-events-api-retention-export.md | Introduces ADR 030 defining /events API semantics, retention strategy, and per-sink export tracking. |
| docs/adrs/028-storage-v2-statements-and-dialects.md | Adds a cross-reference from ADR 028 to the new ADR 029. |
| Events are partially specified in [ADR 030](../adrs/030-events-api-retention-export.md); | ||
| OpenAPI sketch pending. |
| , delivered_at TIMESTAMPTZ NOT NULL DEFAULT now() | ||
| , PRIMARY KEY (project_id, event_id, sink_id) | ||
| , FOREIGN KEY (project_id, event_id) | ||
| REFERENCES zitadel_nextgen.events (project_id, id) |
There was a problem hiding this comment.
This retention design needs one more detail before it is implementable: the purge deletes from events, but every purge-eligible event has at least one event_deliveries child row, and this FK does not cascade. As written, the retention job will fail with a foreign-key violation. Please either add ON DELETE CASCADE here or make the retention algorithm delete matching event_deliveries rows before deleting events.
|
|
||
| **Permissions:** Callers require an `events.read` permission (exact RBAC shape | ||
| TBD — reference platform authz when specified). Team-scoped credentials may | ||
| restrict list results to events where `actor_id` or affected resources belong |
There was a problem hiding this comment.
This access rule is not safe enough for team-scoped credentials. actor_id and affected resources are not immutable team-scope facts: users/resources can move teams, and some events can involve cross-team actors or resources. To avoid leaking historical events across teams, the event row should carry the immutable authorization scope captured at emit time, for example team_id/scope_id/resource owner context, and filtering should use that stored scope instead of recomputing membership later.
| - Response header `X-Request-Id` echoes the assigned `request_id`. | ||
|
|
||
| Request events use a **separate INSERT** outside the handler's entity | ||
| transaction (fire-and-forget after response). Domain mutations co-locate their |
There was a problem hiding this comment.
For request-wide audit events, fire-and-forget after response can silently lose the only audit record for read/API access if the process exits, the DB insert fails, or shutdown races the background work. Since these events are part of the audit trail, please specify a durable path: either persist synchronously before completing the request, or enqueue into a durable outbox/queue with retry and shutdown draining. In-memory best-effort batching is fine for operational telemetry, but risky as the audit primitive.
|
|
||
| **Permissions:** Callers require an `events.read` permission (exact RBAC shape | ||
| TBD — reference platform authz when specified). Team-scoped credentials may | ||
| restrict list results to events where `actor_id` or affected resources belong |
There was a problem hiding this comment.
Related to the previous point: GET /events/{id} also needs a resolvable authorization scope. With only the path ID, the server cannot know which project/team scope to authorize before looking up the event. Please either include project_id in the get route/query, require the event ID to be globally scoped and lookup-safe, or define a prefetch pattern that cannot leak existence/timing across scopes.
| into a **RequestContext** middleware: | ||
|
|
||
| - Extract or generate `request_id` (W3C traceparent-compatible 128-bit hex). | ||
| - Stash optional `session_id`, `flow_id`, and `fingerprint` from request headers |
There was a problem hiding this comment.
Please avoid trusting caller-provided headers for forensic correlation fields. A client can forge session_id, flow_id, or fingerprint and stitch unrelated activity into another timeline. These should come from server-side session/flow/device state after authentication where possible; if any client hint is retained, it should be explicitly marked as untrusted metadata rather than a primary audit dimension.
| |-----------|-------|---------| | ||
| | Create user | `user.created` | `user_id`, schema ref — no attribute values | | ||
| | Patch attributes | `user.updated` | `changed_keys[]` only | | ||
| | Unique violation | `user.create.failed` | key name — not value | |
There was a problem hiding this comment.
This failure event needs a separate emission path. In Postgres, a uniqueness violation aborts the current transaction until rollback, so the same transaction cannot simply continue and insert user.create.failed. Please specify savepoints, preflight uniqueness checks, or a separate failure-event transaction/outbox so the audit behavior is actually achievable.
|
|
||
| Specs for auth_attempts, flat api_keys, events, audit_events, imports, and capabilities are not yet written. | ||
| Specs for auth_attempts, flat api_keys, imports, and capabilities are not yet written. | ||
| Events are partially specified in [ADR 030](../adrs/030-events-api-retention-export.md); |
There was a problem hiding this comment.
This relative link is one directory short. From docs/design/api/README.md, ../adrs/... resolves to docs/design/adrs/..., which does not exist. Please change it to ../../adrs/030-events-api-retention-export.md.
| Audit event payloads use a **deny-by-default** PII policy: user attribute values | ||
| are omitted unless a field is explicitly marked with `x-audit`. See | ||
| [ADR 029](../../adrs/029-wide-events-internal-audit-primitive.md) §8. The older | ||
| `x-sensitive` annotation is not used for audit redaction. |
There was a problem hiding this comment.
This update correctly separates audit allowlisting from x-sensitive, but it now makes the annotation table look like x-sensitive no longer exists. It is still part of the OpenAPI user-property schema and the CLI presets still emit it for phone/password fields. Please keep x-sensitive documented for its non-audit meaning, then add the clarification that it is deprecated/ignored only for audit redaction and x-audit is the audit allowlist.
| WHERE e.project_id = $1 | ||
| AND e.created_at < now() - $retention_interval | ||
| AND NOT EXISTS ( | ||
| SELECT 1 FROM configured_sinks s |
There was a problem hiding this comment.
This example reads like executable SQL, but configured_sinks is described below as an in-memory config set, not a relation. Please either label this as pseudo-SQL or rewrite it with an executable shape such as a generated VALUES/CTE list of enabled sink IDs.
| **Indexes** (define when query paths are concrete): | ||
|
|
||
| - `(project_id, created_at, id)` — keyset list pagination ([ADR 027](027-cursor-based-pagination.md)) | ||
| - Partial indexes on high-cardinality filter columns: `category`, `actor_id`, |
There was a problem hiding this comment.
The index guidance should be tightened before this becomes migration guidance. category is a small enum, and partial indexes only help for sparse predicates; several listed columns are NOT NULL DEFAULT '', so a partial index would not filter much unless the predicate excludes the empty value. Please describe the concrete intended predicates/indexes, or keep this at the level of add indexes when query paths are proven.
Summary
Adds two ADRs that formalize the oxidel-style wide event audit model for nextgen relational storage:
eventstable, correlation scopes, v2-statementInsertEventemission, authenticated request-wide events, deny-by-default PII (x-auditallowlist), checks/EAV reference rules, OTEL export-only boundary.GET /eventswithcategoryfilter, cursor pagination, per-sinkevent_deliveriesexport, configurable retention purge.Also updates the ADR index, ADR 028 cross-links, API resource map, API README, and user-schema annotation table (
x-auditreplacesx-sensitivefor audit).Review feedback addressed (@adlerhurst)
sequencecolumn — ordering via(created_at, id)per ADR 027aggregate_*/resource_typewithentity_type+entity_idsdk_name/sdk_versionfrom events table (operational logging / OTEL only)shipped_atwith per-sinkevent_deliveriestable for multi-subscriber exportValidation
sequence,aggregate_*,shipped_at, or threat-detection references in ADRs.Release notes / changeset
No changeset — documentation-only per the changeset decision table.
Notes
closes #350