feat: add GRPC_MAX_CONCURRENT_STREAMS to cap concurrent gRPC streams

[hoangcn95]

This continues the goroutine-buildup / OOM hardening work from #987 and #1018 by adding a configurable cap on concurrent gRPC streams per connection.

Current Behavior

The gRPC server is constructed in src/server/server_impl.go from a grpcOptions slice that sets keepalive parameters, unary/stream interceptors, and (optionally) TLS credentials. It never sets grpc.MaxConcurrentStreams. grpc-go's server therefore falls back to its internal default, which is math.MaxUint32 (server.go:190 in grpc-go v1.81.1), effectively unlimited concurrent streams per HTTP/2 connection.

Problem

1. Each inbound gRPC stream (ShouldRateLimit call) is handled by its own goroutine for the lifetime of the request.
2. When the downstream Redis backend stalls (slow responses, connection exhaustion, network blip), in-flight handler goroutines block waiting on Redis instead of completing.
3. With no cap on concurrent streams, Envoy keeps opening new streams on the same HTTP/2 connections (and additional connections), and the number of blocked handler goroutines grows without bound for as long as the stall lasts.
4. Goroutine stacks plus their retained request/response state push heap usage past GOMEMLIMIT, and the process OOMs.

This is problematic because:

• REDIS_POOL_SIZE looks like it should bound concurrency, but it only bounds the number of pooled Redis connections. It does not limit how many gRPC handler goroutines can be in flight waiting for a pool slot or a Redis response.
• Operators have no existing lever to cap intake on the gRPC side. The only bounds today are keepalive/connection-age settings, which don't limit concurrency within a live connection.
• Without a concurrency ceiling, the service cannot shed load gracefully during a downstream stall. It accepts everything until it runs out of memory.

Real-world impact: during a Redis stall on CA1 (2026-07-13), two rate limit pods OOMed as a result of this unbounded goroutine growth.

Proposed Solution

Add a new environment variable that maps directly to grpc.MaxConcurrentStreams:

Env Var	Type	Default	Description
GRPC_MAX_CONCURRENT_STREAMS	uint32	0	Caps the maximum number of concurrent gRPC streams the server allows per HTTP/2 connection. 0 disables the cap and preserves the grpc-go default (unlimited).

Implementation notes:

• src/settings/settings.go: adds GrpcMaxConcurrentStreams uint32 (envconfig tag GRPC_MAX_CONCURRENT_STREAMS, default "0"), next to the existing GrpcMaxConnectionAge/GrpcMaxConnectionAgeGrace settings.
• src/server/server_impl.go: adds a small helper, maxConcurrentStreamsOptions(maxStreams uint32) []grpc.ServerOption, that returns []grpc.ServerOption{grpc.MaxConcurrentStreams(maxStreams)} when maxStreams > 0, and nil otherwise. The result is appended to grpcOptions before grpc.NewServer(grpcOptions...) is called, so the cap is a no-op unless explicitly configured.

Per-connection semantics: this is a per-HTTP/2-connection cap, not a global one. The aggregate concurrency a pod can serve is roughly (number of active connections) x GRPC_MAX_CONCURRENT_STREAMS. Operators tuning this value should account for how many client connections they expect (e.g. Envoy's connection pooling behavior) when picking a cap.

Test Result

Added unit tests covering both new pieces of logic:

• src/settings/settings_test.go: TestGrpcMaxConcurrentStreams_Default asserts GrpcMaxConcurrentStreams defaults to 0 when the env var is unset, and TestGrpcMaxConcurrentStreams_Configured asserts it parses GRPC_MAX_CONCURRENT_STREAMS=100 into uint32(100).
• src/server/server_impl_internal_test.go (package server, internal test): TestMaxConcurrentStreamsOptions_Disabled asserts maxConcurrentStreamsOptions(0) returns no options (preserving current unlimited behavior), and TestMaxConcurrentStreamsOptions_Enabled asserts maxConcurrentStreamsOptions(100) returns exactly one grpc.ServerOption. This directly exercises the branch that is appended into grpcOptions, rather than only asserting that NewServer doesn't error.

All affected packages pass with -race:

go test -race ./src/settings/... ./src/server/... ./test/...

Load test (recommended for reviewers)

Not run as part of this PR. Recommended validation before merge/rollout:

• Use toxiproxy in front of the Redis backend to inject latency/stalls, and drive load with k6 or ghz against the gRPC listener on :8081.
• Scrape :6070/debug/pprof/goroutine before, during, and after the induced stall, with and without GRPC_MAX_CONCURRENT_STREAMS set, to confirm goroutine count plateaus instead of growing unbounded once the cap is active.
• Prod goroutine/heap graphs from the CA1 2026-07-13 incident are to be attached by the author separately. No numbers or images are included here to avoid misrepresenting data that hasn't been captured in this change.

Changes

• src/settings/settings.go: add GrpcMaxConcurrentStreams uint32 setting (GRPC_MAX_CONCURRENT_STREAMS, default 0).
• src/server/server_impl.go: add maxConcurrentStreamsOptions helper and wire it into grpcOptions before grpc.NewServer(...).
• src/settings/settings_test.go: default/configured tests for the new setting.
• src/server/server_impl_internal_test.go: unit tests for the new helper's branch behavior.
• README.md: document GRPC_MAX_CONCURRENT_STREAMS next to the existing GRPC_MAX_CONNECTION_AGE/GRPC_MAX_CONNECTION_AGE_GRACE docs, including per-connection semantics.

Related

• feat: add connection timeout configuration for Redis operations #987
• feat: Add Pool On-Empty Behavior Configuration for Redis Connections #1018