Add async TEE resolution and public refresh-loop control#304
Merged
Conversation
Backend relays (e.g. chat-api's OHTTP relay) resolve a TEE on every request from an async server, but resolve() scans the on-chain registry with a blocking web3 call whenever the requested TEE is not the active one — stalling the whole event loop — and rescans on every call. They also had no public way to start the TEE failover loop, which otherwise only starts lazily from chat/completion. - aresolve() / LLM.aresolve_tee_connection(): event-loop-safe resolve. Registry scans run in a worker thread; each pinned id's outcome (found or not-active) is cached for 30s so steady traffic costs at most one chain RPC per TTL window per id, concurrent cold lookups collapse into a single scan, and expired entries are swept so untrusted ids can't grow the cache without bound. Also starts the refresh loop. - LLM.ensure_tee_refresh_loop(): public starter for the background failover loop, for servers that never call chat/completion. - The refresh loop's own registry scan and reconnect() now run their blocking web3 calls in a worker thread too, and a failed reconnect logs a warning (the previous message claimed a client-close failure it never checked). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wo1obT6mfb81hUDUekHDiK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backend relays (chat-api's OHTTP relay is the motivating case — see OpenGradient/chat-api#179) resolve a TEE on every request from an async server, but the SDK only offered a sync path with two problems:
resolve(tee_id)scans the on-chain registry with a blocking web3 call whenever the requested TEE isn't the active one — stalling the caller's event loop — and it rescans on every call.chat/completion, so a server that only relays OHTTP ciphertext never gets failover: the active TEE resolved at startup stays pinned until restart, even after that TEE is upgraded and shut down.chat-api currently papers over both with its own wrapper (thread dispatch + TTL memo + a private
_tee.ensure_refresh_loop()poke). This PR moves that into the SDK where it belongs.Changes
aresolve(tee_id)onTEEConnectionInterface/LLM.aresolve_tee_connection(tee_id): event-loop-safe resolve for per-request use.asyncio.to_thread) behind a lock, so concurrent cold lookups collapse into a single scan.StaticTEEConnection.aresolvereturns the static connection (no I/O), matchingresolvesemantics.LLM.ensure_tee_refresh_loop(): public starter for the failover loop, for servers that never callchat/completion.reconnect()now run their blocking web3 calls in a worker thread too (previously they stalled the host's event loop every 5 minutes), and a failed reconnect logs a warning with an accurate message (the old debug message claimed a client-close failure it never checked).resolve_tee_connectiondocstring now warns async servers off the sync path.No behavior change for existing sync callers.
Testing
TestAresolvecoverage: no-pin/matching-pin fast path does no scan; pinned id scans once then caches; unknown id raisesValueErrorwith the miss cached (single scan across repeated calls); 10 concurrent cold lookups collapse to one scan; refresh loop auto-starts; static connection resolves without I/OTestTeeConnectionApicoverage for theLLM-level delegationsruff formatandmypy srcclean🤖 Generated with Claude Code
https://claude.ai/code/session_01Wo1obT6mfb81hUDUekHDiK
Generated by Claude Code