tools: add Serper Google Search API as web_search provider option
This commit is contained in:
Roger M
rogerserper
parent
14f8acdecb
commit
9f10a3b488
@ -62,11 +62,12 @@ You can keep it local with `memorySearch.provider = "local"` (no API usage).
|
||||
|
||||
See [Memory](/concepts/memory).
|
||||
|
||||
### 4) Web search tool (Brave / Perplexity via OpenRouter)
|
||||
### 4) Web search tool (Brave / Perplexity via OpenRouter / Serper Google Search API)
|
||||
`web_search` uses API keys and may incur usage charges:
|
||||
|
||||
- **Brave Search API**: `BRAVE_API_KEY` or `tools.web.search.apiKey`
|
||||
- **Perplexity** (via OpenRouter): `PERPLEXITY_API_KEY` or `OPENROUTER_API_KEY`
|
||||
- **Serper Google Search API**: `SERPER_API_KEY` or `tools.web.search.apiKey`
|
||||
|
||||
**Brave free tier (generous):**
|
||||
- **2,000 requests/month**
|
||||
|
||||
43
docs/serper.md
Normal file
43
docs/serper.md
Normal file
@ -0,0 +1,43 @@
|
||||
---
|
||||
summary: "Serper Google Search API setup for web_search"
|
||||
read_when:
|
||||
- You want to use Serper Google Search API for web search
|
||||
- You need a SERPER_API_KEY or plan details
|
||||
---
|
||||
|
||||
# Serper Google Search API
|
||||
|
||||
Clawdbot can use Serper Google Search API as a search provider for `web_search`. Serper provides fast and cost-effective access to Google Search results in structured JSON format.
|
||||
|
||||
## Get an API key
|
||||
|
||||
1) Create a Serper API account at https://serper.dev/
|
||||
2) Generate an API key in your dashboard
|
||||
3) Store the key in config (recommended) or set `SERPER_API_KEY` in the Gateway environment.
|
||||
|
||||
## Config example
|
||||
|
||||
```json5
|
||||
{
|
||||
tools: {
|
||||
web: {
|
||||
search: {
|
||||
provider: "serper",
|
||||
apiKey: "SERPER_API_KEY_HERE",
|
||||
maxResults: 5,
|
||||
timeoutSeconds: 30
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Serper provides a free tier (2,500 queries) plus paid plans; check the Serper dashboard for current limits and pricing.
|
||||
- Serper is fast and cost-effective, returning traditional Google Search results (title, URL, snippet) similar to Brave Search.
|
||||
- Supports country (`country`) and language (`search_lang`) parameters for region-specific results.
|
||||
- Does not support `freshness` filtering (Brave-only feature).
|
||||
|
||||
See [Web tools](/tools/web) for the full web_search configuration.
|
||||
|
||||
@ -1,16 +1,17 @@
|
||||
---
|
||||
summary: "Web search + fetch tools (Brave Search API, Perplexity direct/OpenRouter)"
|
||||
summary: "Web search + fetch tools (Brave Search API, Perplexity direct/OpenRouter, Serper Google Search API)"
|
||||
read_when:
|
||||
- You want to enable web_search or web_fetch
|
||||
- You need Brave Search API key setup
|
||||
- You want to use Perplexity Sonar for web search
|
||||
- You want to use Serper Google Search API for web search
|
||||
---
|
||||
|
||||
# Web tools
|
||||
|
||||
Clawdbot ships two lightweight web tools:
|
||||
|
||||
- `web_search` — Search the web via Brave Search API (default) or Perplexity Sonar (direct or via OpenRouter).
|
||||
- `web_search` — Search the web via Brave Search API (default), Perplexity Sonar (direct or via OpenRouter), or Serper Google Search API.
|
||||
- `web_fetch` — HTTP fetch + readable extraction (HTML → markdown/text).
|
||||
|
||||
These are **not** browser automation. For JS-heavy sites or logins, use the
|
||||
@ -21,6 +22,7 @@ These are **not** browser automation. For JS-heavy sites or logins, use the
|
||||
- `web_search` calls your configured provider and returns results.
|
||||
- **Brave** (default): returns structured results (title, URL, snippet).
|
||||
- **Perplexity**: returns AI-synthesized answers with citations from real-time web search.
|
||||
- **Serper**: returns Google Search results (title, URL, snippet) via Serper Google Search API.
|
||||
- Results are cached by query for 15 minutes (configurable).
|
||||
- `web_fetch` does a plain HTTP GET and extracts readable content
|
||||
(HTML → markdown/text). It does **not** execute JavaScript.
|
||||
@ -32,8 +34,9 @@ These are **not** browser automation. For JS-heavy sites or logins, use the
|
||||
|----------|------|------|---------|
|
||||
| **Brave** (default) | Fast, structured results, free tier | Traditional search results | `BRAVE_API_KEY` |
|
||||
| **Perplexity** | AI-synthesized answers, citations, real-time | Requires Perplexity or OpenRouter access | `OPENROUTER_API_KEY` or `PERPLEXITY_API_KEY` |
|
||||
| **Serper** | Fast, cost-effective Google Search API, good free tier | Traditional search results | `SERPER_API_KEY` |
|
||||
|
||||
See [Brave Search setup](/brave-search) and [Perplexity Sonar](/perplexity) for provider-specific details.
|
||||
See [Brave Search setup](/brave-search), [Perplexity Sonar](/perplexity), and [Serper Google Search API](/serper) for provider-specific details.
|
||||
|
||||
Set the provider in config:
|
||||
|
||||
@ -42,7 +45,7 @@ Set the provider in config:
|
||||
tools: {
|
||||
web: {
|
||||
search: {
|
||||
provider: "brave" // or "perplexity"
|
||||
provider: "brave" // or "perplexity" or "serper"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -138,6 +141,45 @@ If no base URL is set, Clawdbot chooses a default based on the API key source:
|
||||
| `perplexity/sonar-pro` (default) | Multi-step reasoning with web search | Complex questions |
|
||||
| `perplexity/sonar-reasoning-pro` | Chain-of-thought analysis | Deep research |
|
||||
|
||||
## Using Serper Google Search API
|
||||
|
||||
Serper Google Search API provides access to Google Search results in structured JSON format. It's a fast and cost-effective option for getting traditional search results with good free tier limits.
|
||||
|
||||
### Getting a Serper API key
|
||||
|
||||
1) Create an account at https://serper.dev/
|
||||
2) Generate an API key in your dashboard
|
||||
3) Run `clawdbot configure --section web` to store the key in config (recommended), or set `SERPER_API_KEY` in your environment.
|
||||
|
||||
Serper provides a generous free tier (2,500 queries) and affordable paid plans.
|
||||
|
||||
### Setting up Serper search
|
||||
|
||||
```json5
|
||||
{
|
||||
tools: {
|
||||
web: {
|
||||
search: {
|
||||
enabled: true,
|
||||
provider: "serper",
|
||||
apiKey: "SERPER_API_KEY_HERE" // optional if SERPER_API_KEY is set
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Environment alternative:** set `SERPER_API_KEY` in the Gateway environment. For a gateway install, put it in `~/.clawdbot/.env`.
|
||||
|
||||
### Where to set the key (recommended)
|
||||
|
||||
**Recommended:** run `clawdbot configure --section web`. It stores the key in
|
||||
`~/.clawdbot/clawdbot.json` under `tools.web.search.apiKey`.
|
||||
|
||||
**Environment alternative:** set `SERPER_API_KEY` in the Gateway process
|
||||
environment. For a gateway install, put it in `~/.clawdbot/.env` (or your
|
||||
service environment). See [Env vars](/help/faq#how-does-clawdbot-load-environment-variables).
|
||||
|
||||
## web_search
|
||||
|
||||
Search the web using your configured provider.
|
||||
@ -148,6 +190,7 @@ Search the web using your configured provider.
|
||||
- API key for your chosen provider:
|
||||
- **Brave**: `BRAVE_API_KEY` or `tools.web.search.apiKey`
|
||||
- **Perplexity**: `OPENROUTER_API_KEY`, `PERPLEXITY_API_KEY`, or `tools.web.search.perplexity.apiKey`
|
||||
- **Serper**: `SERPER_API_KEY` or `tools.web.search.apiKey`
|
||||
|
||||
### Config
|
||||
|
||||
@ -174,7 +217,7 @@ Search the web using your configured provider.
|
||||
- `country` (optional): 2-letter country code for region-specific results (e.g., "DE", "US", "ALL"). If omitted, Brave chooses its default region.
|
||||
- `search_lang` (optional): ISO language code for search results (e.g., "de", "en", "fr")
|
||||
- `ui_lang` (optional): ISO language code for UI elements
|
||||
- `freshness` (optional, Brave only): filter by discovery time (`pd`, `pw`, `pm`, `py`, or `YYYY-MM-DDtoYYYY-MM-DD`)
|
||||
- `freshness` (optional, Brave only): filter by discovery time (`pd`, `pw`, `pm`, `py`, or `YYYY-MM-DDtoYYYY-MM-DD`). Not supported by Perplexity or Serper.
|
||||
|
||||
**Examples:**
|
||||
|
||||
|
||||
@ -17,11 +17,12 @@ import {
|
||||
writeCache,
|
||||
} from "./web-shared.js";
|
||||
|
||||
const SEARCH_PROVIDERS = ["brave", "perplexity"] as const;
|
||||
const SEARCH_PROVIDERS = ["brave", "perplexity", "serper"] as const;
|
||||
const DEFAULT_SEARCH_COUNT = 5;
|
||||
const MAX_SEARCH_COUNT = 10;
|
||||
|
||||
const BRAVE_SEARCH_ENDPOINT = "https://api.search.brave.com/res/v1/web/search";
|
||||
const SERPER_SEARCH_ENDPOINT = "https://google.serper.dev/search";
|
||||
const DEFAULT_PERPLEXITY_BASE_URL = "https://openrouter.ai/api/v1";
|
||||
const PERPLEXITY_DIRECT_BASE_URL = "https://api.perplexity.ai";
|
||||
const DEFAULT_PERPLEXITY_MODEL = "perplexity/sonar-pro";
|
||||
@ -103,6 +104,17 @@ type PerplexitySearchResponse = {
|
||||
|
||||
type PerplexityBaseUrlHint = "direct" | "openrouter";
|
||||
|
||||
type SerperSearchResult = {
|
||||
title?: string;
|
||||
link?: string;
|
||||
snippet?: string;
|
||||
date?: string;
|
||||
};
|
||||
|
||||
type SerperSearchResponse = {
|
||||
organic?: SerperSearchResult[];
|
||||
};
|
||||
|
||||
function resolveSearchConfig(cfg?: ClawdbotConfig): WebSearchConfig {
|
||||
const search = cfg?.tools?.web?.search;
|
||||
if (!search || typeof search !== "object") return undefined;
|
||||
@ -122,6 +134,13 @@ function resolveSearchApiKey(search?: WebSearchConfig): string | undefined {
|
||||
return fromConfig || fromEnv || undefined;
|
||||
}
|
||||
|
||||
function resolveSerperApiKey(search?: WebSearchConfig): string | undefined {
|
||||
const fromConfig =
|
||||
search && "apiKey" in search && typeof search.apiKey === "string" ? search.apiKey.trim() : "";
|
||||
const fromEnv = (process.env.SERPER_API_KEY ?? "").trim();
|
||||
return fromConfig || fromEnv || undefined;
|
||||
}
|
||||
|
||||
function missingSearchKeyPayload(provider: (typeof SEARCH_PROVIDERS)[number]) {
|
||||
if (provider === "perplexity") {
|
||||
return {
|
||||
@ -131,6 +150,13 @@ function missingSearchKeyPayload(provider: (typeof SEARCH_PROVIDERS)[number]) {
|
||||
docs: "https://docs.clawd.bot/tools/web",
|
||||
};
|
||||
}
|
||||
if (provider === "serper") {
|
||||
return {
|
||||
error: "missing_serper_api_key",
|
||||
message: `web_search (serper) needs a Serper API key. Run \`${formatCliCommand("clawdbot configure --section web")}\` to store it, or set SERPER_API_KEY in the Gateway environment.`,
|
||||
docs: "https://docs.clawd.bot/tools/web",
|
||||
};
|
||||
}
|
||||
return {
|
||||
error: "missing_brave_api_key",
|
||||
message: `web_search needs a Brave Search API key. Run \`${formatCliCommand("clawdbot configure --section web")}\` to store it, or set BRAVE_API_KEY in the Gateway environment.`,
|
||||
@ -144,6 +170,7 @@ function resolveSearchProvider(search?: WebSearchConfig): (typeof SEARCH_PROVIDE
|
||||
? search.provider.trim().toLowerCase()
|
||||
: "";
|
||||
if (raw === "perplexity") return "perplexity";
|
||||
if (raw === "serper") return "serper";
|
||||
if (raw === "brave") return "brave";
|
||||
return "brave";
|
||||
}
|
||||
@ -306,6 +333,54 @@ async function runPerplexitySearch(params: {
|
||||
return { content, citations };
|
||||
}
|
||||
|
||||
async function runSerperSearch(params: {
|
||||
query: string;
|
||||
apiKey: string;
|
||||
count: number;
|
||||
country?: string;
|
||||
search_lang?: string;
|
||||
timeoutSeconds: number;
|
||||
}): Promise<{ results: Array<{ title: string; url: string; description: string; siteName?: string }> }> {
|
||||
const body: Record<string, unknown> = {
|
||||
q: params.query,
|
||||
num: params.count,
|
||||
};
|
||||
|
||||
if (params.country) {
|
||||
body.gl = params.country.toLowerCase();
|
||||
}
|
||||
|
||||
if (params.search_lang) {
|
||||
body.hl = params.search_lang.toLowerCase();
|
||||
}
|
||||
|
||||
const res = await fetch(SERPER_SEARCH_ENDPOINT, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
"X-API-KEY": params.apiKey,
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
const detail = await readResponseText(res);
|
||||
throw new Error(`Serper API error (${res.status}): ${detail || res.statusText}`);
|
||||
}
|
||||
|
||||
const data = (await res.json()) as SerperSearchResponse;
|
||||
const organic = Array.isArray(data.organic) ? data.organic : [];
|
||||
const mapped = organic.map((entry) => ({
|
||||
title: entry.title ?? "",
|
||||
url: entry.link ?? "",
|
||||
description: entry.snippet ?? "",
|
||||
siteName: resolveSiteName(entry.link ?? ""),
|
||||
}));
|
||||
|
||||
return { results: mapped };
|
||||
}
|
||||
|
||||
async function runWebSearch(params: {
|
||||
query: string;
|
||||
count: number;
|
||||
@ -351,6 +426,27 @@ async function runWebSearch(params: {
|
||||
return payload;
|
||||
}
|
||||
|
||||
if (params.provider === "serper") {
|
||||
const { results } = await runSerperSearch({
|
||||
query: params.query,
|
||||
apiKey: params.apiKey,
|
||||
count: params.count,
|
||||
country: params.country,
|
||||
search_lang: params.search_lang,
|
||||
timeoutSeconds: params.timeoutSeconds,
|
||||
});
|
||||
|
||||
const payload = {
|
||||
query: params.query,
|
||||
provider: params.provider,
|
||||
count: results.length,
|
||||
tookMs: Date.now() - start,
|
||||
results,
|
||||
};
|
||||
writeCache(SEARCH_CACHE, cacheKey, payload, params.cacheTtlMs);
|
||||
return payload;
|
||||
}
|
||||
|
||||
if (params.provider !== "brave") {
|
||||
throw new Error("Unsupported web search provider.");
|
||||
}
|
||||
@ -419,7 +515,9 @@ export function createWebSearchTool(options?: {
|
||||
const description =
|
||||
provider === "perplexity"
|
||||
? "Search the web using Perplexity Sonar (direct or via OpenRouter). Returns AI-synthesized answers with citations from real-time web search."
|
||||
: "Search the web using Brave Search API. Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research.";
|
||||
: provider === "serper"
|
||||
? "Search the web using Serper API (Google Search wrapper). Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research."
|
||||
: "Search the web using Brave Search API. Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research.";
|
||||
|
||||
return {
|
||||
label: "Web Search",
|
||||
@ -430,7 +528,11 @@ export function createWebSearchTool(options?: {
|
||||
const perplexityAuth =
|
||||
provider === "perplexity" ? resolvePerplexityApiKey(perplexityConfig) : undefined;
|
||||
const apiKey =
|
||||
provider === "perplexity" ? perplexityAuth?.apiKey : resolveSearchApiKey(search);
|
||||
provider === "perplexity"
|
||||
? perplexityAuth?.apiKey
|
||||
: provider === "serper"
|
||||
? resolveSerperApiKey(search)
|
||||
: resolveSearchApiKey(search);
|
||||
|
||||
if (!apiKey) {
|
||||
return jsonResult(missingSearchKeyPayload(provider));
|
||||
|
||||
@ -434,7 +434,7 @@ const FIELD_HELP: Record<string, string> = {
|
||||
'Text suffix for cross-context markers (supports "{channel}").',
|
||||
"tools.message.broadcast.enabled": "Enable broadcast action (default: true).",
|
||||
"tools.web.search.enabled": "Enable the web_search tool (requires a provider API key).",
|
||||
"tools.web.search.provider": 'Search provider ("brave" or "perplexity").',
|
||||
"tools.web.search.provider": 'Search provider ("brave", "perplexity", or "serper").',
|
||||
"tools.web.search.apiKey": "Brave Search API key (fallback: BRAVE_API_KEY env var).",
|
||||
"tools.web.search.maxResults": "Default number of results to return (1-10).",
|
||||
"tools.web.search.timeoutSeconds": "Timeout in seconds for web_search requests.",
|
||||
|
||||
@ -332,8 +332,8 @@ export type ToolsConfig = {
|
||||
search?: {
|
||||
/** Enable web search tool (default: true when API key is present). */
|
||||
enabled?: boolean;
|
||||
/** Search provider ("brave" or "perplexity"). */
|
||||
provider?: "brave" | "perplexity";
|
||||
/** Search provider ("brave", "perplexity", or "serper"). */
|
||||
provider?: "brave" | "perplexity" | "serper";
|
||||
/** Brave Search API key (optional; defaults to BRAVE_API_KEY env var). */
|
||||
apiKey?: string;
|
||||
/** Default search results count (1-10). */
|
||||
|
||||
Loading…
Reference in New Issue
Block a user