feat: add CrustApiTool for Google search via CrustAPI#6532
Open
88bricks wants to merge 1 commit into
Open
Conversation
Adds a CrustApiTool that searches Google through CrustAPI (https://crustapi.com) and returns clean, structured JSON. One tool covers web, news, maps, places, shopping, images, videos, scholar, and patents via a search_type parameter. The response schema mirrors the common SERP shape (organic, knowledgeGraph, peopleAlsoAsk, relatedSearches, news), so results map straight into an agent's context. Auth is a CRUSTAPI_API_KEY env var; CrustAPI has a free tier (no card). Includes a mocked test suite mirroring the SerperDevTool tests.
📝 WalkthroughWalkthroughChangesCrustAPI search tool
Sequence Diagram(s)sequenceDiagram
participant Agent
participant CrustApiTool
participant CrustAPI
Agent->>CrustApiTool: run(search_query, search_type)
CrustApiTool->>CrustAPI: Request search results
CrustAPI-->>CrustApiTool: Return JSON response
CrustApiTool-->>Agent: Return structured results
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
lib/crewai-tools/tests/tools/crustapi_tool_test.py (1)
1-136: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider adding error-path test coverage.
Tests cover initialization and the web/news/maps happy paths well, but there's no test for
_make_api_request's error handling (HTTP error, malformed JSON, empty response) or thesave_filebranch. Exercising the JSON-decode-error path in particular would have caught the dead exception handler flagged incrustapi_tool.py.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/tests/tools/crustapi_tool_test.py` around lines 1 - 136, Extend the CrustApiTool tests to cover _make_api_request error paths: HTTP failures, malformed JSON triggering the JSON decode handling, and empty responses, asserting the documented outcomes or exceptions. Add coverage for the save_file=True execution branch, reusing the existing requests.get mocking setup and keeping the current happy-path tests unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py`:
- Around line 38-55: Add a concise class docstring directly inside CrustApiTool,
documenting that it searches Google through CrustAPI and returns structured
results. Keep the existing class attributes and behavior unchanged.
- Around line 168-182: Update the exception handling around response.json() in
the CrustAPI request flow so requests.exceptions.JSONDecodeError is caught
before the generic requests.exceptions.RequestException handler. Preserve the
specific JSON error and response-content logging in that decode branch, while
allowing other request failures to use the generic handler.
- Line 155: Update the API key lookup in the CrustAPI tool to handle a missing
CRUSTAPI_API_KEY environment variable explicitly and raise a descriptive
configuration error instead of allowing a raw KeyError. Preserve the existing
headers construction when the variable is present.
---
Nitpick comments:
In `@lib/crewai-tools/tests/tools/crustapi_tool_test.py`:
- Around line 1-136: Extend the CrustApiTool tests to cover _make_api_request
error paths: HTTP failures, malformed JSON triggering the JSON decode handling,
and empty responses, asserting the documented outcomes or exceptions. Add
coverage for the save_file=True execution branch, reusing the existing
requests.get mocking setup and keeping the current happy-path tests unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c97c9049-7b6a-4824-bd66-1895d3b6316e
📒 Files selected for processing (6)
lib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/crustapi_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/crustapi_tool/__init__.pylib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.pylib/crewai-tools/tests/tools/crustapi_tool_test.py
Comment on lines
+38
to
+55
| class CrustApiTool(BaseTool): | ||
| name: str = "Search Google with CrustAPI" | ||
| description: str = ( | ||
| "A tool that searches Google via CrustAPI and returns clean, structured JSON. " | ||
| "Supports different search types: 'search' (default web results), 'news', 'maps', " | ||
| "'places', 'shopping', 'images', 'videos', 'scholar', 'patents'." | ||
| ) | ||
| args_schema: Type[BaseModel] = CrustApiToolSchema | ||
| base_url: str = "https://crustapi.com/v1" | ||
| n_results: int = 10 | ||
| save_file: bool = False | ||
| search_type: str = "search" | ||
| country: Optional[str] = "" | ||
| location: Optional[str] = "" | ||
| locale: Optional[str] = "" | ||
| env_vars: List[EnvVar] = [ | ||
| EnvVar(name="CRUSTAPI_API_KEY", description="API key for CrustAPI", required=True), | ||
| ] |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add a class docstring to CrustApiTool.
CrustApiToolSchema has a docstring but the public CrustApiTool class itself does not. As per coding guidelines, "Document public APIs and complex logic in Python code."
📝 Proposed fix
class CrustApiTool(BaseTool):
+ """Search Google through CrustAPI and return structured, SERP-style JSON.
+
+ Supports web, news, maps, places, shopping, images, videos, scholar, and
+ patents surfaces via the `search_type` field. Requires CRUSTAPI_API_KEY.
+ """
+
name: str = "Search Google with CrustAPI"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| class CrustApiTool(BaseTool): | |
| name: str = "Search Google with CrustAPI" | |
| description: str = ( | |
| "A tool that searches Google via CrustAPI and returns clean, structured JSON. " | |
| "Supports different search types: 'search' (default web results), 'news', 'maps', " | |
| "'places', 'shopping', 'images', 'videos', 'scholar', 'patents'." | |
| ) | |
| args_schema: Type[BaseModel] = CrustApiToolSchema | |
| base_url: str = "https://crustapi.com/v1" | |
| n_results: int = 10 | |
| save_file: bool = False | |
| search_type: str = "search" | |
| country: Optional[str] = "" | |
| location: Optional[str] = "" | |
| locale: Optional[str] = "" | |
| env_vars: List[EnvVar] = [ | |
| EnvVar(name="CRUSTAPI_API_KEY", description="API key for CrustAPI", required=True), | |
| ] | |
| class CrustApiTool(BaseTool): | |
| """Search Google through CrustAPI and return structured, SERP-style JSON. | |
| Supports web, news, maps, places, shopping, images, videos, scholar, and | |
| patents surfaces via the `search_type` field. Requires CRUSTAPI_API_KEY. | |
| """ | |
| name: str = "Search Google with CrustAPI" | |
| description: str = ( | |
| "A tool that searches Google via CrustAPI and returns clean, structured JSON. " | |
| "Supports different search types: 'search' (default web results), 'news', 'maps', " | |
| "'places', 'shopping', 'images', 'videos', 'scholar', 'patents'." | |
| ) | |
| args_schema: Type[BaseModel] = CrustApiToolSchema | |
| base_url: str = "https://crustapi.com/v1" | |
| n_results: int = 10 | |
| save_file: bool = False | |
| search_type: str = "search" | |
| country: Optional[str] = "" | |
| location: Optional[str] = "" | |
| locale: Optional[str] = "" | |
| env_vars: List[EnvVar] = [ | |
| EnvVar(name="CRUSTAPI_API_KEY", description="API key for CrustAPI", required=True), | |
| ] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py`
around lines 38 - 55, Add a concise class docstring directly inside
CrustApiTool, documenting that it searches Google through CrustAPI and returns
structured results. Keep the existing class attributes and behavior unchanged.
Source: Coding guidelines
| if self.locale != "": | ||
| params["hl"] = self.locale | ||
|
|
||
| headers = {"x-api-key": os.environ["CRUSTAPI_API_KEY"]} |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the file and inspect the relevant section.
ast-grep outline lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py --view expanded || true
echo '---'
sed -n '120,190p' lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py | cat -n
echo '--- BUILDING_TOOLS references ---'
rg -n "handle missing credentials|env_vars|required" lib/crewai-tools -g 'BUILDING_TOOLS.md' -g '*.md' || trueRepository: crewAIInc/crewAI
Length of output: 8818
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect class definition and any env handling in the tool base.
rg -n "env_vars|os\.environ\[|os\.environ\.get\(|missing credentials|clear errors" lib/crewai-tools/src lib/crewai-tools -g '*.py' -g '*.md' || trueRepository: crewAIInc/crewAI
Length of output: 27501
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find the tool file and nearby declaration details.
python3 - <<'PY'
from pathlib import Path
p = Path('lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py')
print(p.exists(), p)
if p.exists():
lines = p.read_text().splitlines()
for i, line in enumerate(lines, 1):
if 1 <= i <= len(lines):
if 1 <= i <= 220:
pass
PYRepository: crewAIInc/crewAI
Length of output: 230
Guard the API key lookup. os.environ["CRUSTAPI_API_KEY"] will raise a raw KeyError if the variable is missing; raise a descriptive error instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py` at
line 155, Update the API key lookup in the CrustAPI tool to handle a missing
CRUSTAPI_API_KEY environment variable explicitly and raise a descriptive
configuration error instead of allowing a raw KeyError. Preserve the existing
headers construction when the variable is present.
Comment on lines
+168
to
+182
| except requests.exceptions.RequestException as e: | ||
| error_msg = f"Error making request to CrustAPI: {e}" | ||
| if response is not None and hasattr(response, "content"): | ||
| error_msg += f"\nResponse content: {response.content}" | ||
| logger.error(error_msg) | ||
| raise | ||
| except json.JSONDecodeError as e: | ||
| if response is not None and hasattr(response, "content"): | ||
| logger.error(f"Error decoding JSON response: {e}") | ||
| logger.error(f"Response content: {response.content}") | ||
| else: | ||
| logger.error( | ||
| f"Error decoding JSON response: {e} (No response content available)" | ||
| ) | ||
| raise |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- target file excerpt ---'
sed -n '140,210p' lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py
echo
echo '--- requests exception hierarchy / json behavior ---'
python3 - <<'PY'
import json
import requests
print("requests version:", requests.__version__)
print("requests.exceptions.JSONDecodeError MRO:")
for cls in requests.exceptions.JSONDecodeError.__mro__:
print(" ", cls.__module__ + "." + cls.__name__)
print("Is requests.exceptions.JSONDecodeError a subclass of RequestException?",
issubclass(requests.exceptions.JSONDecodeError, requests.exceptions.RequestException))
print("Is requests.exceptions.JSONDecodeError a subclass of json.JSONDecodeError?",
issubclass(requests.exceptions.JSONDecodeError, json.JSONDecodeError))
# Construct a Response and provoke response.json() to see the raised exception type.
r = requests.Response()
r._content = b'{"not valid":'
r.encoding = "utf-8"
try:
r.json()
except Exception as e:
print("response.json() raised:", type(e).__module__ + "." + type(e).__name__)
print("caught by RequestException?", isinstance(e, requests.exceptions.RequestException))
print("caught by JSONDecodeError?", isinstance(e, requests.exceptions.JSONDecodeError))
print("caught by json.JSONDecodeError?", isinstance(e, json.JSONDecodeError))
PYRepository: crewAIInc/crewAI
Length of output: 3946
Move the JSON decode handling before the generic request catch
response.json() raises requests.exceptions.JSONDecodeError, which is already a RequestException, so the later except json.JSONDecodeError branch is unreachable here. Catch the decode error around response.json() itself if you want the specific log message.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/crustapi_tool.py`
around lines 168 - 182, Update the exception handling around response.json() in
the CrustAPI request flow so requests.exceptions.JSONDecodeError is caught
before the generic requests.exceptions.RequestException handler. Preserve the
specific JSON error and response-content logging in that decode branch, while
allowing other request failures to use the generic handler.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
CrustApiToolthat searches Google through CrustAPI and returns clean, structured JSON for agents.One tool covers multiple Google surfaces via a
search_typeparameter:search(default web),news,maps,places,shopping,images,videos,scholar, andpatents. The response schema mirrors the common SERP shape (organic,knowledgeGraph,peopleAlsoAsk,relatedSearches,news), so it slots in alongside the existing search tools and results map straight into an agent's context.Usage
Auth is a
CRUSTAPI_API_KEYenvironment variable. CrustAPI has a free tier (no card), so contributors and users can try it without a paid plan.Changes
lib/crewai-tools/src/crewai_tools/tools/crustapi_tool/— the tool, an empty package__init__.py, and a README (mirrors theserper_dev_toollayout)lib/crewai-tools/tests/tools/crustapi_tool_test.py— mocked test suite mirroring theSerperDevTooltests (init, invalid type, web/news/maps formatting)lib/crewai-tools/src/crewai_tools/__init__.pyand.../tools/__init__.pyNotes
BaseTool+EnvVar+ pydanticargs_schemapattern asSerperDevTool.