Asynchronous payment processing service

Origin. Take-home assignment for a senior Python backend role (fintech), completed solo in 3 days. Published as a final snapshot - the design notes below explain the key decisions (outbox, idempotency levels, DLQ placement).

The microservice accepts a payment, reliably enqueues it (via an Outbox), and a separate consumer emulates processing through an "external gateway" (2–5 s, 90% success / 10% failed), updates the status, and notifies the client of the result through a signed webhook.

Stack: FastAPI + Pydantic v2 · SQLAlchemy 2.0 (async/asyncpg) · PostgreSQL · RabbitMQ (FastStream) · Alembic · Docker.

Architecture

Two application services (api, consumer) + postgres + rabbitmq. api - payment creation/retrieval only. consumer - everything else: the outbox relay (event publishing) and payment processing.

                 POST /api/v1/payments
  client ───────────────────────────────▶  ┌─────────┐
                                            │   api   │  single transaction:
                                            └────┬────┘  INSERT payments + INSERT outbox
                                                 ▼
                                       PostgreSQL (payments, outbox_events)
                                                 ▲ poll (FOR UPDATE SKIP LOCKED)
   ┌──────────────────────── consumer (single process) ─────────────────────────┐
   │  (A) outbox-dispatcher ──publish──▶ exchange "payments" ──▶ queue payments.new
   │  (B) subscriber payments.new:                                               │
   │        claim(pending→processing) → emulate 2–5s (90/10) → status →          │
   │        webhook(HMAC) → ack │ (technical error) retry ladder → DLQ           │
   └────────────────────────────────────────────────────────────────────────────┘
                                                 │ webhook (POST, X-Webhook-Signature)
                                                 ▼
                                          client's receiver

Layers (no circular dependencies): domain (models/schemas/enums) → services (business logic) → api / workers (entry points); infra (DB, broker, http) is injected.

Quick start

Requires Docker + Docker Compose.

docker compose up -d --build      # or: make up
docker compose ps                 # wait for status healthy

Brings up: postgres, rabbitmq (UI: http://localhost:15672, guest/guest), api (http://localhost:8000, Swagger: /docs), consumer, plus a dev webhook-sink receiver (http://localhost:9000). Migrations are applied automatically when api starts.

Default configuration (can be overridden via .env, see .env.example): API_KEY=dev-secret-api-key, WEBHOOK_SECRET=dev-webhook-hmac-secret.

API

All business endpoints require the X-API-Key header. /health, /ready, /metrics, /docs do not.

Create a payment - `POST /api/v1/payments`

Headers: X-API-Key, Idempotency-Key (required).

curl -i -X POST http://localhost:8000/api/v1/payments \
  -H "X-API-Key: dev-secret-api-key" \
  -H "Idempotency-Key: order-42" \
  -H "Content-Type: application/json" \
  -d '{
        "amount": "199.99",
        "currency": "RUB",
        "description": "order #42",
        "metadata": {"order_id": "42"},
        "webhook_url": "http://webhook-sink:9000/sink"
      }'

Response 202 Accepted:

{ "id": "53a3463a-...", "status": "pending", "created_at": "2026-...Z" }

Idempotency behavior:

Request	Response
new `Idempotency-Key`	`202` - created
same key + same body	`200` - same `id` (no duplicate)
same key + different body	`409` - `idempotency_conflict`
missing/invalid `X-API-Key`	`401`
missing `Idempotency-Key` / malformed body	`422`

Retrieve a payment - `GET /api/v1/payments/{id}`

curl http://localhost:8000/api/v1/payments/<id> -H "X-API-Key: dev-secret-api-key"

{
  "id": "53a3463a-...", "amount": "199.99", "currency": "RUB",
  "status": "succeeded", "webhook_delivered_at": "2026-...Z",
  "created_at": "2026-...Z", "processed_at": "2026-...Z", ...
}

Webhook

The consumer sends a POST to webhook_url with the body {event, id, status, amount, currency, ...} and signature headers:

X-Webhook-Timestamp: 1751320961
X-Webhook-Signature: t=1751320961,v1=<hex HMAC-SHA256("<timestamp>." + body)>

Verification on the receiver side (constant-time) - see payments.services.webhook_service.verify and the example receiver tools/webhook_sink.py.

Guarantees: Outbox, idempotency, retry/DLQ

Transactional Outbox. When a payment is created, payments and outbox_events are written in a single transaction. The broker is not touched at this point - publishing is handled by a background dispatcher (SELECT … FOR UPDATE SKIP LOCKED) that marks the event published_at. This way a message is not lost even if RabbitMQ is temporarily unavailable (at-least-once).

Idempotency - at three levels:

Endpoint: UNIQUE(payments.idempotency_key) + request_fingerprint (SHA-256 of the body) → a retry does not create a duplicate, while a changed body yields 409.
Consumer: an atomic compare-and-set pending → processing - the randomized emulation runs exactly once; redelivery does not repeat it.
Webhook: delivery is guarded by webhook_delivered_at; on retry only the webhook is sent.

Business failure ≠ technical error. A failed outcome (10%) is a normal business result: the webhook is sent (with status=failed), the message is ack-ed, and it does not go to the DLQ. Only technical failures (webhook delivery error, DB error, etc.) go to retry/DLQ.

Retry + DLQ (on the RabbitMQ side, no plugins): a technical error → the consumer republishes a copy to a retry queue with an increasing TTL (exponential backoff); after MAX_PROCESSING_ATTEMPTS (default 3) is exhausted - to payments.dlq. More below.

RabbitMQ topology

exchange payments (direct) ──"new"──▶ payments.new        # main queue
payments.retry.1  (x-message-ttl=5s,  dead-letter ─▶ payments.new)
payments.retry.2  (x-message-ttl=25s, dead-letter ─▶ payments.new)
payments.dlq                                              # "dead" after 3 attempts

The return path out of the retry queues is done via the default exchange + x-dead-letter-routing-key = queue name - so no manual bindings are needed. The ladder and the number of attempts are configurable (RETRY_TTLS_MS, MAX_PROCESSING_ATTEMPTS).

Technical-error path: payments.new → retry.1 (5s) → retry.2 (25s) → payments.dlq.

Configuration

All parameters are set via environment variables (see .env.example). Key ones:

Variable	Purpose	Default
`API_KEY`	value of `X-API-Key`	- (required)
`WEBHOOK_SECRET`	HMAC key for webhook signing	- (required)
`DATABASE_URL`	async PostgreSQL DSN	`postgresql+asyncpg://payments:payments@postgres:5432/payments`
`RABBITMQ_URL`	RabbitMQ DSN	`amqp://guest:guest@rabbitmq:5672/`
`PROCESSING_MIN/MAX_SECONDS`	emulation window	`2.0` / `5.0`
`PROCESSING_SUCCESS_RATE`	success rate	`0.9`
`MAX_PROCESSING_ATTEMPTS`	attempts before DLQ	`3`
`RETRY_TTLS_MS`	retry-level TTLs (comma-separated)	`5000,25000`
`OUTBOX_POLL_INTERVAL`	outbox poll interval, s	`1.0`
`JSON_LOGS`	JSON logs (prod)	`false`

Observability

Logs - structured (structlog), JSON in production. A correlation_id ties together records from api and consumer (the api propagates it into the X-Request-ID response header, the dispatcher - into the message's correlation_id).
Metrics - Prometheus. api exposes GET :8000/metrics (payments_created_total); consumer - GET :8001/metrics (payments_processed_total{status}, outbox_published_total, webhook_delivery_total{result}, message_retries_total, dlq_messages_total, histogram payment_processing_duration_seconds). Each process has its own metrics - both endpoints are scraped separately by Prometheus.
Health - GET /health (liveness), GET /ready (DB check).

Tests

pip install -e ".[dev]"
make test          # unit + integration (spin up real PostgreSQL/RabbitMQ via testcontainers)
make test-unit     # fast unit tests only

Coverage: webhook signature and fingerprint (unit); POST idempotency/conflict/401/422/404, the outbox write, the consumer's state machine (success / business-fail / no-webhook), retry→DLQ, idempotent redelivery, poison messages, and the dispatcher publishing to a real RabbitMQ (integration).

On Docker Desktop (Windows/macOS), testcontainers needs TESTCONTAINERS_RYUK_DISABLED=true

make test already sets this.

CI (GitHub Actions, .github/workflows/ci.yml): ruff + mypy + pytest.

Project structure

src/payments/
  config.py logging.py metrics.py exceptions.py
  domain/    models.py enums.py schemas.py events.py
  services/  payment_service.py   webhook_service.py
  infra/     db.py broker.py topology.py http_client.py
  api/       main.py dependencies.py errors.py routes/{payments,health}.py
  workers/   app.py dispatcher.py consumer.py runtime.py
migrations/  env.py versions/0001_initial.py        # Alembic (async)
tests/       unit/  integration/  conftest.py
docker/Dockerfile  docker-compose.yml  tools/webhook_sink.py  Makefile

Design decisions and assumptions

The outbox relay lives inside consumer - per the spec, api only creates/reads a payment, and "the consumer does all the rest of the work." Compose: postgres, rabbitmq, api, consumer.
Exactly two tables (payments, outbox_events) - as in the spec. The DLQ is a RabbitMQ queue, not a table; no separate tables for idempotency/webhook-DLQ were introduced.
A single retry mechanism: webhook redelivery goes through the same broker retry ladder (rather than a second system such as a scheduler) - this is simpler and more observable.
Money - NUMERIC(19,2)/Decimal, returned in JSON as a string (no loss of precision).
Status enum - VARCHAR + CHECK (native_enum=False), to avoid dealing with ALTER TYPE on a native PG enum in migrations.
Webhook: any non-2xx/timeout is treated as a technical failure (→ retry/DLQ). In production it makes sense to distinguish permanent 4xx, but for a clear demo the behavior is uniform.

Further improvements

Per-API-key rate limiting; a circuit breaker on the webhook; OpenTelemetry tracing; a RabbitMQ cluster + a PostgreSQL replica; periodic cleanup of published outbox rows; WEBHOOK_SECRET rotation (versioned signatures); an alert on payments.dlq growth.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Asynchronous payment processing service

Table of contents

Architecture

Quick start

API

Create a payment - `POST /api/v1/payments`

Retrieve a payment - `GET /api/v1/payments/{id}`

Webhook

Guarantees: Outbox, idempotency, retry/DLQ

RabbitMQ topology

Configuration

Observability

Tests

Project structure

Design decisions and assumptions

Further improvements

About

Uh oh!

Contributors

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.github/workflows		.github/workflows
docker		docker
migrations		migrations
src/payments		src/payments
tests		tests
tools		tools
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
alembic.ini		alembic.ini
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation

Asynchronous payment processing service

Table of contents

Architecture

Quick start

API

Create a payment - POST /api/v1/payments

Retrieve a payment - GET /api/v1/payments/{id}

Webhook

Guarantees: Outbox, idempotency, retry/DLQ

RabbitMQ topology

Configuration

Observability

Tests

Project structure

Design decisions and assumptions

Further improvements

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Contributors

Uh oh!

Languages

Create a payment - `POST /api/v1/payments`

Retrieve a payment - `GET /api/v1/payments/{id}`