Add explicit AI crawler rules to robots.txt

[IEvangelist]

What

Adds explicit AI / LLM crawler rules to src/frontend/public/robots.txt so https://aspire.dev/robots.txt states our stance per user-agent instead of relying only on the wildcard group.

Why

The recommended "AI crawler rules" practice is to declare explicit User-agent groups for AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, …) so allow/disallow decisions are on our terms rather than left implicit. Aspire already opts into AI use via Content-Signal: ai-train=yes, search=yes, ai-input=yes, ships an llms.txt, agent skills, and an MCP server — so this change makes that intent explicit and machine-readable for each major AI agent.

What changed

• Added a dedicated group that explicitly allows the major reputable AI crawlers and carries the same Content-Signal the site already declares:
  • OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
  • Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai
  • Google: Google-Extended
  • Perplexity: PerplexityBot, Perplexity-User
  • Apple: Applebot-Extended
  • Amazon: Amazonbot; Meta: Meta-ExternalAgent; Mistral: MistralAI-User; DuckDuckGo: DuckAssistBot; Cohere: cohere-ai; Common Crawl: CCBot
• Preserved the existing wildcard (User-agent: *) default group, its Content-Signal, and the Sitemap: reference.

Allowing (rather than blocking) these agents is intentional and consistent with the site's existing ai-train=yes / search=yes / ai-input=yes signals — Aspire wants to be discoverable by AI assistants. Flip any group to Disallow: / later if the policy changes.

Verification

robots.txt lives in Astro's public/ dir, so it's copied verbatim to the site root (/robots.txt) with no transform. Validated the file with Python's urllib.robotparser:

• GPTBot, ClaudeBot, CCBot, and an arbitrary UA all resolve to can_fetch = True
• Sitemap is parsed: https://aspire.dev/sitemap-index.xml

Left as draft for maintainer review of the specific agent list / policy before merge.

[aspire-repo-bot]

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #1335. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

This comment updates automatically when a new frontend build artifact is uploaded.