Skip to content

refactor(stovepipe): collapse to a single Ping-only service #271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
behinddwalls merged 1 commit into from
Jun 25, 2026
Merged

refactor(stovepipe): collapse to a single Ping-only service #271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@ load("@gazelle//:def.bzl", "gazelle")
# gazelle:resolve go github.com/uber/submitqueue/api/runway/orchestrator/protopb //api/runway/orchestrator/protopb
# gazelle:resolve go github.com/uber/submitqueue/api/submitqueue/gateway/protopb //api/submitqueue/gateway/protopb
# gazelle:resolve go github.com/uber/submitqueue/api/submitqueue/orchestrator/protopb //api/submitqueue/orchestrator/protopb
# gazelle:resolve go github.com/uber/submitqueue/api/stovepipe/gateway/protopb //api/stovepipe/gateway/protopb
# gazelle:resolve go github.com/uber/submitqueue/api/stovepipe/orchestrator/protopb //api/stovepipe/orchestrator/protopb
# gazelle:resolve go github.com/uber/submitqueue/api/stovepipe/protopb //api/stovepipe/protopb

# Export marker files for test data dependencies (used by FindRepoRoot in tests)
exports_files(
Expand Down
12 changes: 4 additions & 8 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ request.Version = newVersion
submitqueue/ # repo root (Go module github.com/uber/submitqueue)
├── api/ # Published wire contracts (cross-domain/external)
│ ├── submitqueue/{gateway,orchestrator}/{proto,protopb}/ # RPC (proto)
│ ├── stovepipe/{gateway,orchestrator}/{proto,protopb}/
│ ├── stovepipe/{proto,protopb}/ # single-service RPC (proto) — no service segment yet
│ └── runway/messagequeue/ # external queue contracts (proto + protojson)
├── platform/ # SHARED cross-domain packages — no domain deps
│ ├── errs/, metrics/, consumer/, http/
Expand All @@ -50,12 +50,8 @@ submitqueue/ # repo root (Go module github.com/uber/submi
│ ├── entity/ # SubmitQueue-specific domain entities
│ ├── extension/ # SubmitQueue-specific extension impls (storage, counter, mergechecker, …)
│ └── core/ # SubmitQueue-internal shared infra (consumer wiring, request, topickey, …)
├── stovepipe/ # Stovepipe domain
│ ├── gateway/ # Gateway service: commit deployment verification entry point
│ ├── orchestrator/ # Orchestrator service: commit verification pipeline
│ ├── entity/ # Stovepipe-specific domain entities
│ ├── extension/ # Stovepipe-specific extension impls
│ └── core/ # Stovepipe-internal shared infra (placeholder; mirrors submitqueue/core)
├── stovepipe/ # Stovepipe domain (single Ping-only service for now)
│ └── controller/ # Business logic (currently just Ping); entity/extension/core added as it grows
├── tool/ # Development and CI tooling
├── example/
│ ├── submitqueue/ # Runnable SubmitQueue servers/clients + Docker Compose
Expand All @@ -67,7 +63,7 @@ submitqueue/ # repo root (Go module github.com/uber/submi
└── doc/ # Documentation
```

The `platform/` tree holds code reused across domains (infrastructure, shared entities, shared extension contracts). Each **domain** (`submitqueue/`, `stovepipe/`, …) keeps the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared only between that domain's services.
The `platform/` tree holds code reused across domains (infrastructure, shared entities, shared extension contracts). Each **domain** (`submitqueue/`, `stovepipe/`, …) grows into the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared only between that domain's services. A domain may start smaller — Stovepipe is currently a single Ping-only service with just `controller/` (and a service-segment-free `api/stovepipe/`), adding the other layers as it gains real behavior.

The `api/` tree holds **published** wire contracts — those depended on from outside the owning domain. RPC contracts live at `api/{domain}/{service}/` (`proto/` for `.proto` sources, `protopb/` for committed generated Go); a service package may hold multiple `.proto` files, all generating into the same `protopb/`. External message-queue contracts live at `api/{domain}/messagequeue/` (see Message Queue Contracts below). Internal queue contracts do **not** go here — they live under `{domain}/core/messagequeue/`.

Expand Down
112 changes: 28 additions & 84 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,8 @@ ORCHESTRATOR_COMPOSE_FILE = example/submitqueue/orchestrator/server/docker-compo
# Fixed project name for local manual testing (tests use unique random names)
SUBMITQUEUE_LOCAL_PROJECT = submitqueue

# Stovepipe compose files
STOVEPIPE_GATEWAY_COMPOSE_FILE = example/stovepipe/gateway/server/docker-compose.yml
STOVEPIPE_ORCHESTRATOR_COMPOSE_FILE = example/stovepipe/orchestrator/server/docker-compose.yml
STOVEPIPE_STACK_COMPOSE_FILE = example/stovepipe/docker-compose.yml
# Stovepipe compose file (single Ping-only service)
STOVEPIPE_COMPOSE_FILE = example/stovepipe/docker-compose.yml

# Fixed project name for local manual testing (tests use unique random names)
STOVEPIPE_LOCAL_PROJECT = stovepipe
Expand All @@ -37,7 +35,7 @@ GOIMPORTS_VERSION ?= v0.33.0
# (the out_dir convention in tool/proto/BUILD.bazel) and copied back here. A
# package may hold multiple .proto files (e.g. an RPC contract plus messagequeue
# contracts); all generated stubs land in the same protopb/ dir.
PROTO_PACKAGES = api/base/change api/base/mergestrategy api/base/messagequeue api/runway/messagequeue api/runway/orchestrator api/submitqueue/gateway api/submitqueue/orchestrator api/stovepipe/gateway api/stovepipe/orchestrator
PROTO_PACKAGES = api/base/change api/base/mergestrategy api/base/messagequeue api/runway/messagequeue api/runway/orchestrator api/submitqueue/gateway api/submitqueue/orchestrator api/stovepipe

# Set REPO_ROOT for docker-compose
export REPO_ROOT := $(shell pwd)
Expand All @@ -53,7 +51,7 @@ define assert_clean
fi
endef

.PHONY: build build-all-linux build-runway-orchestrator-linux build-submitqueue-gateway-linux build-submitqueue-orchestrator-linux build-stovepipe-gateway-linux build-stovepipe-orchestrator-linux check-gazelle check-mocks check-tidy clean clean-proto deps e2e-test fmt gazelle integration-test integration-test-submitqueue-consumer integration-test-extensions integration-test-submitqueue-gateway integration-test-submitqueue-orchestrator license-fix lint lint-fmt lint-license local-init-runway-queue-schema local-runway-orchestrator-start local-runway-orchestrator-stop local-submitqueue-clean local-submitqueue-gateway-start local-submitqueue-gateway-stop local-init-submitqueue-schemas local-init-stovepipe-queue-schema local-submitqueue-logs local-submitqueue-orchestrator-start local-submitqueue-orchestrator-stop local-submitqueue-ps local-submitqueue-restart local-submitqueue-start local-stop local-stovepipe-gateway-start local-stovepipe-orchestrator-start local-stovepipe-start mocks proto query-deps query-targets run-client-runway-orchestrator run-client-submitqueue-gateway run-client-submitqueue-orchestrator run-client-stovepipe-gateway run-client-stovepipe-orchestrator run-queue-admin test test-no-cache tidy tidy-bazel tidy-go help
.PHONY: build build-all-linux build-runway-orchestrator-linux build-submitqueue-gateway-linux build-submitqueue-orchestrator-linux build-stovepipe-linux check-gazelle check-mocks check-tidy clean clean-proto deps e2e-test fmt gazelle integration-test integration-test-submitqueue-consumer integration-test-extensions integration-test-submitqueue-gateway integration-test-submitqueue-orchestrator license-fix lint lint-fmt lint-license local-init-runway-queue-schema local-runway-orchestrator-start local-runway-orchestrator-stop local-submitqueue-clean local-submitqueue-gateway-start local-submitqueue-gateway-stop local-init-submitqueue-schemas local-submitqueue-logs local-submitqueue-orchestrator-start local-submitqueue-orchestrator-stop local-submitqueue-ps local-submitqueue-restart local-submitqueue-start local-stop local-stovepipe-logs local-stovepipe-start local-stovepipe-stop mocks proto query-deps query-targets run-client-runway-orchestrator run-client-submitqueue-gateway run-client-submitqueue-orchestrator run-client-stovepipe run-queue-admin test test-no-cache tidy tidy-bazel tidy-go help


build: ## Build all services and examples
Expand All @@ -62,7 +60,7 @@ build: ## Build all services and examples
@echo "Build complete!"

# Build Linux binaries required for Docker containers
build-all-linux: build-submitqueue-gateway-linux build-submitqueue-orchestrator-linux build-stovepipe-gateway-linux build-stovepipe-orchestrator-linux build-runway-orchestrator-linux ## Build all Linux binaries for Docker
build-all-linux: build-submitqueue-gateway-linux build-submitqueue-orchestrator-linux build-stovepipe-linux build-runway-orchestrator-linux ## Build all Linux binaries for Docker
@echo "All Linux binaries ready for Docker"

build-runway-orchestrator-linux: ## Build Runway orchestrator Linux binary for Docker
Expand All @@ -89,21 +87,13 @@ build-submitqueue-orchestrator-linux: ## Build Orchestrator Linux binary for Doc
cp -f bazel-bin/example/submitqueue/orchestrator/server/orchestrator .docker-bin/orchestrator
@echo "Orchestrator Linux binary ready at .docker-bin/orchestrator"

build-stovepipe-gateway-linux: ## Build Stovepipe gateway Linux binary for Docker
@echo "Building Stovepipe gateway Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/stovepipe/gateway/server:gateway
build-stovepipe-linux: ## Build Stovepipe Linux binary for Docker
@echo "Building Stovepipe Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/stovepipe/server:stovepipe
@mkdir -p .docker-bin
@cp -f bazel-bin/example/stovepipe/gateway/server/gateway_/gateway .docker-bin/stovepipe-gateway 2>/dev/null || \
cp -f bazel-bin/example/stovepipe/gateway/server/gateway .docker-bin/stovepipe-gateway
@echo "Stovepipe gateway Linux binary ready at .docker-bin/stovepipe-gateway"

build-stovepipe-orchestrator-linux: ## Build Stovepipe orchestrator Linux binary for Docker
@echo "Building Stovepipe orchestrator Linux binary for Docker..."
@$(BAZEL) build --platforms=@rules_go//go/toolchain:linux_amd64 //example/stovepipe/orchestrator/server:orchestrator
@mkdir -p .docker-bin
@cp -f bazel-bin/example/stovepipe/orchestrator/server/orchestrator_/orchestrator .docker-bin/stovepipe-orchestrator 2>/dev/null || \
cp -f bazel-bin/example/stovepipe/orchestrator/server/orchestrator .docker-bin/stovepipe-orchestrator
@echo "Stovepipe orchestrator Linux binary ready at .docker-bin/stovepipe-orchestrator"
@cp -f bazel-bin/example/stovepipe/server/stovepipe_/stovepipe .docker-bin/stovepipe 2>/dev/null || \
cp -f bazel-bin/example/stovepipe/server/stovepipe .docker-bin/stovepipe
@echo "Stovepipe Linux binary ready at .docker-bin/stovepipe"

check-gazelle: ## Check BUILD.bazel files are up to date
@echo "Running Gazelle to check BUILD files..."
Expand Down Expand Up @@ -223,14 +213,6 @@ local-init-submitqueue-schemas: ## Manually apply all database schemas
done
@echo "✅ All schemas applied successfully"

local-init-stovepipe-queue-schema: ## Apply queue schema only (mysql-queue) for Stovepipe compose stacks
@echo "Applying queue schema to mysql-queue (Stovepipe; no app storage/counter schema yet)..."
@for file in platform/extension/messagequeue/mysql/schema/*.sql; do \
echo " - Applying $$(basename $$file)..."; \
docker exec -i $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
done
@echo "✅ Stovepipe queue schema applied successfully"

local-init-runway-queue-schema: ## Apply queue schema only (mysql-queue) for Runway compose stacks
@echo "Applying queue schema to mysql-queue (Runway; consumes the merge queues)..."
@for file in platform/extension/messagequeue/mysql/schema/*.sql; do \
Expand Down Expand Up @@ -321,61 +303,27 @@ local-submitqueue-start: build-all-linux ## Start full stack (Gateway + Orchestr
local-stop: ## Stop all services (keep data)
@echo "Stopping all services..."
@$(COMPOSE) -f $(COMPOSE_FILE) -p $(SUBMITQUEUE_LOCAL_PROJECT) down
@$(COMPOSE) -f $(STOVEPIPE_STACK_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) down
@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) down
@$(COMPOSE) -f $(RUNWAY_ORCHESTRATOR_COMPOSE_FILE) -p $(RUNWAY_LOCAL_PROJECT) down
@echo "Services stopped. Data volumes preserved."

local-stovepipe-logs: ## View logs from all running Stovepipe services
@$(COMPOSE) -f $(STOVEPIPE_STACK_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) logs -f
local-stovepipe-logs: ## View logs from the running Stovepipe service
@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) logs -f

local-stovepipe-start: build-stovepipe-gateway-linux build-stovepipe-orchestrator-linux ## Start full Stovepipe stack (gateway + orchestrator + MySQL)
@echo "Starting full Stovepipe stack with compose..."
@$(COMPOSE) -f $(STOVEPIPE_STACK_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) up -d --build --wait
@echo "Applying queue schema to mysql-queue (no Stovepipe app schema yet)..."
@$(MAKE) -s local-init-stovepipe-queue-schema
@echo ""
@echo "✅ Full Stovepipe stack is running!"
@echo ""
@echo "Expected container NAMEs (project=$(STOVEPIPE_LOCAL_PROJECT), one replica each):"
@echo " $(STOVEPIPE_LOCAL_PROJECT)-mysql-app-1"
@echo " $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1"
@echo " $(STOVEPIPE_LOCAL_PROJECT)-gateway-service-1"
@echo " $(STOVEPIPE_LOCAL_PROJECT)-orchestrator-service-1"
local-stovepipe-start: build-stovepipe-linux ## Start Stovepipe service (single Ping-only gRPC service)
@echo "Starting Stovepipe service with compose..."
@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) up -d --build --wait
@echo ""
@$(COMPOSE) -f $(STOVEPIPE_STACK_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) ps
@echo "✅ Stovepipe service is running!"
@echo ""
@echo "Stovepipe gateway gRPC port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-gateway-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "Stovepipe orchestrator gRPC port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-orchestrator-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "MySQL App port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-mysql-app-1 3306 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "MySQL Queue port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 3306 2>/dev/null | cut -d: -f2 || echo 'unknown')"

local-stovepipe-orchestrator-start: build-stovepipe-orchestrator-linux ## Start Stovepipe orchestrator locally (orchestrator + 2 MySQL databases)
@echo "Starting Stovepipe orchestrator with compose..."
@$(COMPOSE) -f $(STOVEPIPE_ORCHESTRATOR_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) up -d --build --wait
@echo "Applying queue schema to mysql-queue (no Stovepipe app schema yet)..."
@$(MAKE) -s local-init-stovepipe-queue-schema
@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) ps
@echo ""
@echo "✅ Stovepipe orchestrator is running!"
@echo ""
@$(COMPOSE) -f $(STOVEPIPE_ORCHESTRATOR_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) ps
@echo ""
@echo "Stovepipe orchestrator gRPC port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-orchestrator-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "MySQL App port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-mysql-app-1 3306 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "MySQL Queue port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 3306 2>/dev/null | cut -d: -f2 || echo 'unknown')"

local-stovepipe-gateway-start: build-stovepipe-gateway-linux ## Start Stovepipe gateway locally (gateway + 2 MySQL databases)
@echo "Starting Stovepipe gateway with compose..."
@$(COMPOSE) -f $(STOVEPIPE_GATEWAY_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) up -d --build --wait
@echo "Applying queue schema to mysql-queue (no Stovepipe app schema yet)..."
@$(MAKE) -s local-init-stovepipe-queue-schema
@echo ""
@echo "✅ Stovepipe gateway is running!"
@echo ""
@$(COMPOSE) -f $(STOVEPIPE_GATEWAY_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) ps
@echo ""
@echo "Stovepipe gateway gRPC port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-gateway-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "MySQL App port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-mysql-app-1 3306 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "MySQL Queue port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 3306 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@echo "Stovepipe gRPC port: $$(docker port $(STOVEPIPE_LOCAL_PROJECT)-stovepipe-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'unknown')"

local-stovepipe-stop: ## Stop the Stovepipe service
@echo "Stopping Stovepipe service..."
@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) down
@echo "Stovepipe service stopped."

mocks: ## Generate mock files using mockgen
@echo "Generating mocks..."
Expand Down Expand Up @@ -411,13 +359,9 @@ run-client-submitqueue-gateway:
run-client-submitqueue-orchestrator:
@$(BAZEL) run //example/submitqueue/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8082) -message "$(or $(MESSAGE),ping)"

# Run stovepipe gateway client (connects to any running stovepipe gateway service)
run-client-stovepipe-gateway:
@$(BAZEL) run //example/stovepipe/gateway/client:gateway -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"

# Run stovepipe orchestrator client (connects to any running stovepipe orchestrator service)
run-client-stovepipe-orchestrator:
@$(BAZEL) run //example/stovepipe/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8084) -message "$(or $(MESSAGE),ping)"
# Run stovepipe client (connects to any running stovepipe service)
run-client-stovepipe:
@$(BAZEL) run //example/stovepipe/client:stovepipe -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"

# Run runway orchestrator client (connects to any running runway orchestrator service)
run-client-runway-orchestrator:
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Designed for large monorepos and fast-moving teams where concurrent changes can

## Repository layout

Cross-domain Go code (errors, metrics, consumer framework, HTTP helpers, shared entities, shared extension contracts) lives under [`platform/`](platform/README.md). Each product domain has its own tree (`submitqueue/`, `stovepipe/`, …) with `gateway/`, `orchestrator/`, `entity/`, `extension/`, and domain-local `core/`. See [CLAUDE.md](CLAUDE.md) for conventions and import paths.
Cross-domain Go code (errors, metrics, consumer framework, HTTP helpers, shared entities, shared extension contracts) lives under [`platform/`](platform/README.md). Each product domain has its own tree (`submitqueue/`, `stovepipe/`, …) and grows into `gateway/`, `orchestrator/`, `entity/`, `extension/`, and domain-local `core/` — though a domain may start smaller (Stovepipe is currently a single Ping-only service with just `controller/`). See [CLAUDE.md](CLAUDE.md) for conventions and import paths.

## Quick Start

Expand Down
Loading
Loading