uhttp: drop pooled proxy tunnels after a wall-clock jump (Lambda freeze)#1007
uhttp: drop pooled proxy tunnels after a wall-clock jump (Lambda freeze)#1007arreyder wants to merge 1 commit into
Conversation
Hosted connectors run as Lambdas whose outbound HTTPS rides CONNECT tunnels through the egress proxy. Between invocations Lambda freezes the execution environment and the monotonic clock stops. http.Transport's IdleConnTimeout ages pooled connections on the monotonic clock, so a tunnel that sat frozen for minutes looks zero seconds old — while Squid, the NLB's 350s idle timeout, or the origin has already closed the socket. On thaw the first request reuses the dead tunnel and the sync burns a task attempt on ECONNRESET / http2: client connection lost, or stalls until the h2 ping / ResponseHeaderTimeout fires. Track last-request activity on the wall clock (monotonic stripped via Round(0), since CLOCK_REALTIME is resynced across a freeze but the monotonic clock is not) and CloseIdleConnections on the base transport when the gap exceeds 15s before dispatching. A false positive costs one CONNECT+TLS handshake against syncs that make thousands of calls. OPS-2213 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| return | ||
| } | ||
| gap := now.Sub(last) | ||
| if gap <= wallClockGapThreshold { |
There was a problem hiding this comment.
🟡 Suggestion: The gap is measured between consecutive request starts (both dropPooledConnsAfterClockJump and touchWallClock write lastActivityWall), so any connector that legitimately paces requests more than 15s apart — rate-limited or paginated syncs that sleep between calls — will trip this on every request and re-do a CONNECT+TLS handshake, defeating pooling for that workload. This is an always-on behavior change for all downstream SDK consumers, not just proxy/Lambda ones. The freeze itself is far longer than 15s (the evidence cites minutes), so a larger threshold (e.g. 60–120s) would still catch freezes while sparing slow-but-alive connectors. Worth confirming the 15s value against real inter-request spacing, or gating it behind an option, since the SDK defaults propagate everywhere.
1 participant
Problem
Hosted connectors run as Lambdas whose outbound HTTPS rides CONNECT tunnels through the egress proxy (HTTPS_PROXY → Squid behind an internal NLB, :3128). Between invocations Lambda freezes the execution environment and the monotonic clock stops.
http.Transport.IdleConnTimeoutages pooled connections on the monotonic clock, so a tunnel that sat frozen for ten real-world minutes looks zero seconds old to the pool — while Squid, the NLB's fixed 350s idle timeout, or the origin has already closed the socket.On thaw the first request reuses the dead tunnel and either burns a task attempt on
connection reset by peer/http2: client connection lost, or stalls until the h2 ping (15s) /ResponseHeaderTimeout(60s) notices. The existingIdleConnTimeout: 30sand the 5-minute transport re-cycle()can't help — both measure monotonic time, which is exactly what stops during a freeze.Evidence (prod-usw2, 2026-07-14)
net/proxy-nlb-ecdfd6e, 24h: 3.34M new flows vs 851k target-side TCP RSTs (25.5%); NLB-generated RSTs only 266. Reset rate is flat around the clock — a client-behavior signature, not overload (Squid fleet at 2.3% CPU, 0 unhealthy targets).be-temporal-sync48h: ~4khttp2: timeout awaiting response headers, 1.5krequest timeout, 1.3kcontext canceled; smoking gunrun-sync: error getting connector metadata: … http2 client connection lost— the first request of a sync on a reused dead tunnel.Root-cause analysis: ductone CE-837 + Slack thread.
Fix
Track last-request activity on the wall clock (monotonic stripped via
Round(0), sincetime.Subsilently prefers the monotonic reading, which reads ~0 across a freeze —CLOCK_REALTIMEis resynced on thaw). When the gap exceeds 15s,CloseIdleConnections()on the base transport before dispatching. This subsumes "flush the pool at invocation start": the first request of a thawed invocation always sees the gap. A false positive costs one CONNECT+TLS handshake against syncs that make thousands of calls.Test
TestTransportDropsPooledConnsAfterWallClockJumpdrives a realhttptestserver, confirms the connection is pooled/reused with no gap, then simulates a freeze by backdating the wall-clock marker and asserts a new connection is dialed.Verification after rollout
aws.networkelb.tcptarget_reset_count{loadbalancer:net/proxy-nlb-ecdfd6e*}should fall from ~35k/hr, and@error:*awaiting*counts inbe-temporal-syncfor the top affected tenants should drop sharply.🤖 Generated with Claude Code