Skip to content

Add async TEE resolution and public refresh-loop control#304

Merged
adambalogh merged 1 commit into
mainfrom
claude/async-tee-resolve
Jul 15, 2026
Merged

Add async TEE resolution and public refresh-loop control#304
adambalogh merged 1 commit into
mainfrom
claude/async-tee-resolve

Conversation

@adambalogh

Copy link
Copy Markdown
Collaborator

Summary

Backend relays (chat-api's OHTTP relay is the motivating case — see OpenGradient/chat-api#179) resolve a TEE on every request from an async server, but the SDK only offered a sync path with two problems:

  1. resolve(tee_id) scans the on-chain registry with a blocking web3 call whenever the requested TEE isn't the active one — stalling the caller's event loop — and it rescans on every call.
  2. The background TEE failover loop only starts lazily from chat/completion, so a server that only relays OHTTP ciphertext never gets failover: the active TEE resolved at startup stays pinned until restart, even after that TEE is upgraded and shut down.

chat-api currently papers over both with its own wrapper (thread dispatch + TTL memo + a private _tee.ensure_refresh_loop() poke). This PR moves that into the SDK where it belongs.

Changes

  • aresolve(tee_id) on TEEConnectionInterface / LLM.aresolve_tee_connection(tee_id): event-loop-safe resolve for per-request use.
    • Registry scans run in a worker thread (asyncio.to_thread) behind a lock, so concurrent cold lookups collapse into a single scan.
    • Each pinned id's outcome — found or not-active — is cached for 30s, so steady traffic costs at most one chain RPC per TTL window per id, and hammering with a bogus id can't turn into one RPC per request.
    • Expired entries are swept once the cache outgrows any plausible number of real TEEs, so relays forwarding untrusted browser-sent ids can't grow it without bound.
    • Also starts the background refresh loop, so long-running relays fail over automatically.
    • StaticTEEConnection.aresolve returns the static connection (no I/O), matching resolve semantics.
  • LLM.ensure_tee_refresh_loop(): public starter for the failover loop, for servers that never call chat/completion.
  • The refresh loop's own registry scan and reconnect() now run their blocking web3 calls in a worker thread too (previously they stalled the host's event loop every 5 minutes), and a failed reconnect logs a warning with an accurate message (the old debug message claimed a client-close failure it never checked).
  • resolve_tee_connection docstring now warns async servers off the sync path.

No behavior change for existing sync callers.

Testing

  • Full suite: 175 passed, 1 skipped (pre-existing skip)
  • New TestAresolve coverage: no-pin/matching-pin fast path does no scan; pinned id scans once then caches; unknown id raises ValueError with the miss cached (single scan across repeated calls); 10 concurrent cold lookups collapse to one scan; refresh loop auto-starts; static connection resolves without I/O
  • New TestTeeConnectionApi coverage for the LLM-level delegations
  • ruff format and mypy src clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wo1obT6mfb81hUDUekHDiK


Generated by Claude Code

Backend relays (e.g. chat-api's OHTTP relay) resolve a TEE on every
request from an async server, but resolve() scans the on-chain registry
with a blocking web3 call whenever the requested TEE is not the active
one — stalling the whole event loop — and rescans on every call. They
also had no public way to start the TEE failover loop, which otherwise
only starts lazily from chat/completion.

- aresolve() / LLM.aresolve_tee_connection(): event-loop-safe resolve.
  Registry scans run in a worker thread; each pinned id's outcome
  (found or not-active) is cached for 30s so steady traffic costs at
  most one chain RPC per TTL window per id, concurrent cold lookups
  collapse into a single scan, and expired entries are swept so
  untrusted ids can't grow the cache without bound. Also starts the
  refresh loop.
- LLM.ensure_tee_refresh_loop(): public starter for the background
  failover loop, for servers that never call chat/completion.
- The refresh loop's own registry scan and reconnect() now run their
  blocking web3 calls in a worker thread too, and a failed reconnect
  logs a warning (the previous message claimed a client-close failure
  it never checked).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wo1obT6mfb81hUDUekHDiK
@adambalogh adambalogh marked this pull request as ready for review July 15, 2026 12:48
@adambalogh adambalogh merged commit 3566c0c into main Jul 15, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants