Upgrade query-generator SQL step to Claude Sonnet 5#49187
Upgrade query-generator SQL step to Claude Sonnet 5#49187allenhouchins wants to merge 2 commits into
Conversation
The osquery-SQL-generation call in the /query-generator page was still on claude-sonnet-4-6. Move it to claude-sonnet-5, which follows the platform/ schema constraints in the prompt more literally and should produce more reliable SQL, since Claude Sonnet 5 now defaults to adaptive thinking when `thinking` is omitted (which this helper does), add `output_config.effort` support to the shared ai.prompt helper and set it to "low" here -- the Haiku pre-filtering step already narrows the schema, so the SQL-generation step doesn't need much additional deliberation. Also bump max_tokens from 4096 to 8192 so thinking tokens don't eat into the JSON response budget and cause truncation. The schema-filtration step stays on claude-haiku-4-5, which is already the latest Haiku release.
The sqlReport call passed the system prompt as a bare object-shorthand key named systemPromptForQueryGeneration, but the ai.prompt helper's declared input is systemPrompt -- Sails silently drops unrecognized keys passed to .with(), so the "Return ONLY a raw JSON object..." system prompt was never actually reaching the model for this call. This has been broken since the query generator was switched to Anthropic (f7c20c4); the sibling filteredTables call above was unaffected since it passes systemPrompt positionally.
WalkthroughThe SQL generation controller now uses Claude Sonnet 5 with low effort. The AI prompt helper accepts an optional effort input, forwards it through Anthropic requests as 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@website/api/helpers/ai/prompt.js`:
- Around line 87-99: Update the Anthropic response handling that derives
rawPromptResponse from anthropicResponse.content to scan for the first block
whose type is "text" rather than assuming content[0] is text. Throw an explicit
error when no text block exists, preserving the existing downstream parsing flow
for valid text responses.
- Around line 97-99: Update the effort handling in the prompt helper so
output_config.effort is attached only when baseModel identifies a supported
Anthropic model; ensure claude-haiku-* requests omit it or reject the
unsupported baseModel/effort combination before sending the request.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 65ea3077-960f-44c3-9ca4-084cbb45ef4d
📒 Files selected for processing (2)
website/api/controllers/query-generator/get-llm-generated-sql.jswebsite/api/helpers/ai/prompt.js
| // Bumped from 4096 so that models with adaptive thinking on by default (e.g. Claude Sonnet 5) | ||
| // have enough headroom for thinking tokens without truncating the actual response. | ||
| max_tokens: 8192,// eslint-disable-line camelcase | ||
| messages: [ | ||
| { role: 'user', content: prompt+(expectJson? JSON_PROMPT_SUFFIX : '') } | ||
| ] | ||
| }; | ||
| if (systemPrompt) { | ||
| requestData.system = systemPrompt; | ||
| } | ||
| if (effort) { | ||
| requestData.output_config = { effort };// eslint-disable-line camelcase | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the target file and nearby Anthropic helpers
ast-grep outline website/api/helpers/ai/prompt.js --view expanded || true
rg -n "anthropicResponse\.content|content\[0\]\.text|output_config|effort|thinking|type === 'text'|text block" website/api/helpers/ai -S || true
# Read the target file around the cited lines
wc -l website/api/helpers/ai/prompt.js
sed -n '1,220p' website/api/helpers/ai/prompt.js
# Find any other Anthropic response parsing in the repo
rg -n "content\[0\]\.text|content\.find\(\(block\) => block\.type === 'text'\)|content\.find\(|block\.type === 'text'" -S .Repository: fleetdm/fleet
Length of output: 9216
🌐 Web query:
Anthropic Messages API response content blocks thinking text block adaptive thinking output_config.effort Sonnet 5
💡 Result:
In the Anthropic Messages API, adaptive thinking is a mechanism for Claude models to dynamically determine the depth of their reasoning based on query complexity [1][2]. For models supporting this, including Claude Sonnet 5, it replaces the legacy budget_tokens configuration [3][2]. To configure adaptive thinking, you use the thinking object in the request body with type: "adaptive" [1][2]. The output_config.effort parameter is used to provide soft guidance on the desired thinking depth [4][1]. Key details: 1. Configuration Structure: The effort parameter must be placed within a dedicated output_config object in the request body, not inside the thinking object [4]. 2. Effort Levels: The effort parameter supports values such as low, medium (default), high, and xhigh (availability varies by model) [1]. This allows you to influence how much reasoning Claude applies to a task [1]. 3. Model Support: On models like Claude Sonnet 5, adaptive thinking is enabled by default [2]. You can explicitly disable it by passing thinking: {type: "disabled"} if preferred [2]. Attempting to use the legacy manual thinking: {type: "enabled", budget_tokens: N} configuration on these models will typically result in a 400 error [3][2]. 4. Displaying Thinking: On newer models, the thinking block content defaults to "omitted" [3][5]. If you wish to see the reasoning text, you must explicitly set thinking: {type: "adaptive", display: "summarized"} [5]. 5. Usage: Use max_tokens to set a hard limit on the total output (thinking plus response text) [1]. Adaptive thinking, paired with effort, is the recommended approach for managing reasoning depth on supported models [3][2].
Citations:
- 1: https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking?38d7aa68_page=2&5a5b91bc_page=1
- 2: https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking?fcdaa149_sort_date=desc&vno=z7
- 3: https://platform.claude.com/docs/en/build-with-claude/extended-thinking?_rsc=1ebj0
- 4: https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-adaptive-thinking.html
- 5: https://awesomeagents.ai/migrations/openai-to-anthropic-api/
🌐 Web query:
Anthropic Messages API response content array text thinking blocks default omitted content[0].text
💡 Result:
In the Anthropic Messages API, when extended thinking is enabled, the API response contains an array of content blocks [1]. You should not assume that the first block (response.content) is a text block, as thinking blocks are distinct objects that appear in the content array [1]. Regarding the behavior of thinking content in the response: 1. Thinking Content Types: When extended thinking is enabled, the model returns specialized content blocks of type thinking or redacted_thinking, which are distinct from text blocks [1][2]. 2. Omission and Display: Whether the actual thinking text is returned depends on the configuration provided in the request's thinking parameter [3]. - If configured to return full reasoning, the thinking content is provided in the thinking block [4][5]. - If the configuration (often via an adaptive thinking or display setting) results in the content being omitted or summarized, the thinking block may be present but contain no text, or it may be omitted entirely depending on the specific API version and model behavior [3][6][1]. 3. Robust Handling: Because the structure of the content array varies based on model output and configuration, you must iterate over the response.content array and check the type of each block (e.g., if block.type == "text" or if block.type == "thinking") rather than directly accessing response.content.text [1][5][2]. Attempting to access.text on a thinking block or an unexpected block type will cause an error [1].
Citations:
- 1: https://www.anthropiccertifications.com/glossary/content-block
- 2: https://platform.claude.com/cookbook/extended-thinking-extended-thinking
- 3: https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking
- 4: https://github.com/anthropics/anthropic-sdk-python/blob/main/src/anthropic/resources/messages/messages.py
- 5: https://github.com/anthropics/anthropic-sdk-python/blob/04b468da/examples/thinking.py
- 6: https://vercel.com/docs/ai-gateway/models-and-providers/reasoning/anthropic
Extract the text block from Anthropic responses
anthropicResponse.content[0] is not guaranteed to be a text block here; with adaptive thinking enabled, the first block can be thinking/redacted_thinking, which makes rawPromptResponse undefined and breaks JSON parsing. Scan content for the first type === 'text' block and throw if none is present.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@website/api/helpers/ai/prompt.js` around lines 87 - 99, Update the Anthropic
response handling that derives rawPromptResponse from anthropicResponse.content
to scan for the first block whose type is "text" rather than assuming content[0]
is text. Throw an explicit error when no text block exists, preserving the
existing downstream parsing flow for valid text responses.
| if (effort) { | ||
| requestData.output_config = { effort };// eslint-disable-line camelcase | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the target file around the reported lines
sed -n '1,220p' website/api/helpers/ai/prompt.js
# Search for effort/output_config/model gating logic
rg -n "output_config|effort|claude-|Haiku|supports.*effort|unsupported" website/api -g '!**/node_modules/**'
# Find any model capability allowlists or validators
rg -n "model.*(allow|support|validate|capab|feature)|effort.*(allow|support|validate)|claude" website -g '!**/node_modules/**'Repository: fleetdm/fleet
Length of output: 50373
🌐 Web query:
Anthropic output_config effort unsupported models ignored or error Claude Haiku 4.5
💡 Result:
The error "output_config: Extra inputs are not permitted" occurs when an Anthropic API request includes the output_config parameter (used for structured outputs or reasoning effort) while using an integration or provider that does not support it [1][2][3]. Key reasons for this error include: 1. Provider Restrictions: Cloud providers like Amazon Bedrock, Google Vertex AI, and Azure AI Foundry often have specific request-body requirements [1][2][4][3]. If these services do not recognize the output_config field, they reject the entire request with an "Extra inputs are not permitted" or similar validation error [2][3]. 2. Model Capability Mismatches: While Claude Haiku 4.5 supports structured outputs in many environments, it may not support all features within output_config (such as effort) depending on the specific model endpoint or deployment (e.g., via GitHub Copilot or specific proxy services) [4][5][6][7]. 3. Integration Errors: Many tools and SDKs automatically inject output_config into requests (e.g., for automated session title generation or structured output enforcement) [2][3]. If the underlying service (like Vertex AI or Bedrock) has not been updated to accept this parameter, the integration will fail [2][3]. To resolve this issue: - Strip the output_config parameter from the request body before sending it to the provider if the provider's documentation confirms it is unsupported [1][4][6]. - If you are using an integration tool (like Claude Code or a proxy), ensure you are on the latest version, as many of these issues are resolved by adding request transformers that selectively strip unsupported fields [2][3][7]. - For structured outputs on restricted providers, rely on system prompt instructions to enforce the output format instead of using the output_config API parameter [2][3]. For detailed configuration, refer to the provider-specific documentation for Amazon Bedrock or Google Vertex AI, as their support for these parameters can differ from the standard Anthropic API [8][9][10].
Citations:
- 1: fix(vertex-ai): remove output_config from Vertex AI Anthropic requests BerriAI/litellm#22691
- 2: [BUG]
output_configsent to Vertex AI causesinvalid_request_erroron session title generation anthropics/claude-code#52312 - 3: [BUG] Bedrock: session title generation fails silently — output_config rejected with 'Extra inputs are not permitted' anthropics/claude-code#57611
- 4: fix(azure_ai): strip output_config.effort for Azure AI Foundry Claude models that reject it (Haiku 4.5) BerriAI/litellm#31187
- 5: Bug: claude-opus-4.7 via GitHub Copilot fails with 'output_config: Extra inputs are not permitted' anomalyco/opencode#24527
- 6: fix(claude): omit adaptive thinking + output_config.effort for haiku diegosouzapw/OmniRoute#4661
- 7: Copilot: output_config.effort "max" is not supported by model claude-opus-4.6 anomalyco/opencode#22788
- 8: https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html
- 9: https://platform.claude.com/docs/en/build-with-claude/structured-outputs
- 10: https://docs.aws.amazon.com/bedrock/latest/userguide/claude-messages-structured-outputs.html
🌐 Web query:
site:platform.claude.com effort Claude Haiku 4.5 output_config ignored unsupported models
💡 Result:
Based on current Claude Platform documentation, Claude Haiku 4.5 does not support the output_config.effort parameter [1]. In the Claude API, unsupported fields or parameters sent to a model are generally handled in one of two ways: they are either silently ignored [2] or they trigger a validation error [3][4]. When specific parameters like output_config.effort (which is typically reserved for models that support adaptive thinking, such as Claude Opus 4.8 or Claude Fable 5 [4][5]) are passed to a model that does not support them, the API may reject the request with a 400-series error [4]. If you are encountering issues, ensure your request parameters align with the capabilities of Claude Haiku 4.5: 1. Adaptive thinking and the output_config.effort parameter are not supported on Claude Haiku 4.5 [1]. 2. For structured outputs, the API uses output_config.format [6][4]. Ensure you are not conflating output_config.effort with other output_config sub-parameters. 3. If you are using a library (such as an OpenAI-compatible SDK), note that some libraries may attempt to map parameters automatically, which can lead to unexpected errors if the underlying model does not support them [2]. You should check your API request payload to ensure that the output_config object only contains fields valid for the model you are using.
Citations:
- 1: https://platform.claude.com/docs/en/about-claude/models/overview?_rsc=1ebj0
- 2: https://platform.claude.com/docs/en/cli-sdks-libraries/libraries/openai-sdk
- 3: https://platform.claude.com/docs/en/about-claude/models/migration-guide
- 4: https://platform.claude.com/docs/en/about-claude/models/migration-guide?f80ce999_sort_date=desc&fcdaa149_sort_date=desc
- 5: https://platform.claude.com/docs/en/about-claude/models/migration-guide?939688b5_page=2&fcdaa149_sort_date=desc&lang=de
- 6: https://platform.claude.com/docs/en/release-notes/overview?_rsc=1ebj0
Gate effort by model. Claude Haiku 4.5 does not accept output_config.effort, so this helper should only attach it for supported Anthropic models (or reject unsupported baseModel/effort combinations). Otherwise a claude-haiku-* caller can get a 4xx from Anthropic.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@website/api/helpers/ai/prompt.js` around lines 97 - 99, Update the effort
handling in the prompt helper so output_config.effort is attached only when
baseModel identifies a supported Anthropic model; ensure claude-haiku-* requests
omit it or reject the unsupported baseModel/effort combination before sending
the request.
There was a problem hiding this comment.
Pull request overview
This PR updates the website query generator’s LLM SQL-generation step to use a newer Anthropic model and adds support for configuring “effort” (adaptive thinking depth) in the shared ai.prompt helper.
Changes:
- Add an
effortinput tosails.helpers.ai.promptand forward it to Anthropic requests asoutput_config.effort. - Increase Anthropic
max_tokensfrom 4096 → 8192 in the shared helper. - Update
/query-generatorSQL generation to useclaude-sonnet-5, pass the correctsystemPromptkey, and seteffort: "low".
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| website/api/helpers/ai/prompt.js | Adds optional effort support for Anthropic calls and increases max_tokens headroom. |
| website/api/controllers/query-generator/get-llm-generated-sql.js | Switches the SQL-generation call to Sonnet 5, fixes system prompt forwarding, and sets low effort. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Effort is set to "low" because the schema was already narrowed down to relevant tables by the | ||
| // schema-filtration step above -- the model doesn't need to spend much effort re-deriving that context. | ||
| let sqlReport = await sails.helpers.ai.prompt.with({prompt:sqlPrompt, baseModel:'claude-sonnet-5', expectJson: true, systemPrompt: systemPromptForQueryGeneration, effort: 'low'}) | ||
| .intercept((err)=>{ |
Related issue: N/A
What this does
The
/query-generatorpage's osquery-SQL-generation step (get-llm-generated-sql.js) was onclaude-sonnet-4-6, which is now one generation behind. This PR:claude-sonnet-5. The schema-filtration step stays onclaude-haiku-4-5, which is already the latest Haiku release, so no change needed there.effortsupport to the sharedai.prompthelper, forwarded asoutput_config.efforton Anthropic requests, and sets it to"low"for the SQL-generation call. Effort controls how much the model deliberates (and how many tokens/how much latency that costs)."low"was chosen because the Haiku pre-filtering step already narrows the osquery schema down to relevant tables, so the Sonnet step isn't starting from scratch and doesn't need to spend much effort re-deriving that context.max_tokensin the Anthropic branch of the helper from 4096 to 8192. Claude Sonnet 5 turns on adaptive thinking by default when thethinkingparam is omitted (which this helper does), andmax_tokensis a hard cap on total output including thinking tokens — at 4096 there was a real risk of thinking tokens eating into the budget and truncating the JSON response the SQL step needs to return.sqlReportcall passed the system prompt as a bare object-shorthand key namedsystemPromptForQueryGeneration, but theai.prompthelper's declared input issystemPrompt. Sails silently drops unrecognized keys passed to.with(...), so the "Return ONLY a raw JSON object..." system prompt was never actually reaching the model for this call. This has been broken since the query generator was switched to Anthropic (f7c20c4731); the siblingfilteredTablescall above it was unaffected since it passessystemPromptpositionally. Now fixed tosystemPrompt: systemPromptForQueryGeneration.Why
Claude Sonnet 5 follows structured/constrained instructions (don't alias tables, use
LIKEwith wildcards, only reference documented columns, etc.) more literally than 4.6, which should make the generated SQL more reliable. It's priced the same or cheaper than 4.6 during the current introductory period.Trade-offs called out for review
sails.helpers.http.post, single blocking call over a socket), so any added thinking time is invisible wait time for the user rather than a visible "thinking" indicator.effort: "low"should keep this modest, but worth confirming with a manual QA pass on a few representative questions before merging.output_config.effort(addedeffortis a no-op if passed to it), so it was left as-is.Checklist for submitter
If some of the following don't apply, delete the relevant line.
Changes file added for user-visible changes in
changes/,orbit/changes/oree/fleetd-chrome/changes.See Changes files for more information.
Input data is properly validated,
SELECT *is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.Timeouts are implemented and retries are limited to avoid infinite loops
If paths of existing endpoints are modified without backwards compatibility, checked the frontend/CLI for any necessary changes
Testing
Added/updated automated tests
Where appropriate, automated tests simulate multiple hosts and test for host isolation (updates to one hosts's records do not affect another)
QA'd all new/changed functionality manually
Summary by CodeRabbit