Skip to content

perf: add inventory cache for stateless server patterns#1636

Closed
SamMorrowDrums wants to merge 2 commits into
mainfrom
perf/inventory-cache
Closed

perf: add inventory cache for stateless server patterns#1636
SamMorrowDrums wants to merge 2 commits into
mainfrom
perf/inventory-cache

Conversation

@SamMorrowDrums

Copy link
Copy Markdown
Collaborator

Summary

Add CachedInventory to build tool/resource/prompt definitions once at startup rather than per-request. This is particularly useful for the remote server pattern where a new server instance is created per request.

Problem

In stateless deployments (like the remote MCP server), a new server and inventory is created for each incoming request. This means AllTools(t), AllResources(t), and AllPrompts(t) are called repeatedly, rebuilding all ~130 tool definitions including JSON schema generation on every request.

Related: modelcontextprotocol/go-sdk#685

Solution

Add a CachedInventory that:

  • Builds all tools/resources/prompts once at startup
  • Uses sync.Once for thread-safe initialization
  • Returns pre-populated builders for per-request use

API

// Initialize once at startup (typically in main() or server init)
github.InitInventoryCache(translator)

// Then on each request, get a builder with pre-cached tools
inv := github.CachedInventoryBuilder().
    WithReadOnly(cfg.ReadOnly).
    WithToolsets(cfg.EnabledToolsets).
    WithFeatureChecker(checker).  // Still evaluated per-request!
    Build()

Key Features

  • InitInventoryCache(t) - Called once at startup with your translator
  • CachedInventoryBuilder() - Returns a builder pre-populated with cached definitions
  • Per-request configuration still works - read-only, toolsets, feature flags, filters
  • Thread-safe via sync.Once
  • Backward compatible - NewInventory(t) still works without caching

Why This Works

The elegance is preserved because:

  1. Tools are still self-describing - The GetMe(t), GetTeams(t) pattern stays the same
  2. No code changes needed for tool definitions - They work as before
  3. Feature flags work per-request - They're evaluated in AvailableTools(ctx), not at definition time
  4. Translations resolved once - Since remote server uses NullTranslationHelper anyway, this is fine

For Remote Server Integration

// In server startup/init (once)
func init() {
    github.InitInventoryCache(translations.NullTranslationHelper)
}

// Per-request handler
func handleRequest(ctx context.Context, req Request) {
    inv := github.CachedInventoryBuilder().
        WithReadOnly(cfg.ReadOnly).
        WithToolsets(cfg.Toolsets).
        WithFeatureChecker(createFeatureChecker(ctx, req.User)).
        Build()
    
    // Feature flags are evaluated here, per-request
    tools := inv.AvailableTools(ctx)
    // ...
}

Testing

  • Added comprehensive unit tests in inventory_cache_test.go
  • All existing tests pass
  • script/lint passes

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants