web_search: add searxng provider
This commit is contained in:
parent
613724c26e
commit
8d4270bcd6
@ -1942,8 +1942,16 @@ of `every`, keep `HEARTBEAT.md` tiny, and/or choose a cheaper `model`.
|
||||
Note: `applyPatch` is only under `tools.exec`.
|
||||
|
||||
`tools.web` configures web search + fetch tools:
|
||||
- `tools.web.search.enabled` (default: true when key is present)
|
||||
- `tools.web.search.apiKey` (recommended: set via `openclaw configure --section web`, or use `BRAVE_API_KEY` env var)
|
||||
- `tools.web.search.enabled` (default true)
|
||||
- `tools.web.search.provider` (`brave` | `perplexity` | `searxng`)
|
||||
- `tools.web.search.apiKey` (Brave; recommended: set via `openclaw configure --section web`, or use `BRAVE_API_KEY` env var)
|
||||
- `tools.web.search.perplexity.apiKey` (Perplexity/OpenRouter; optional if `PERPLEXITY_API_KEY` or `OPENROUTER_API_KEY` is set)
|
||||
- `tools.web.search.perplexity.baseUrl` (optional override)
|
||||
- `tools.web.search.perplexity.model` (optional override)
|
||||
- `tools.web.search.searxng.baseUrl` (SearXNG; optional if `SEARXNG_BASE_URL` is set)
|
||||
- `tools.web.search.searxng.apiKey` (optional auth token; sent as Authorization header)
|
||||
- `tools.web.search.searxng.headers` (optional extra headers)
|
||||
- `tools.web.search.searxng.params` (optional default query params)
|
||||
- `tools.web.search.maxResults` (1–10, default 5)
|
||||
- `tools.web.search.timeoutSeconds` (default 30)
|
||||
- `tools.web.search.cacheTtlMinutes` (default 15)
|
||||
|
||||
@ -1319,10 +1319,13 @@ The Gateway watches the config and supports hot‑reload:
|
||||
|
||||
### How do I enable web search and web fetch
|
||||
|
||||
`web_fetch` works without an API key. `web_search` requires a Brave Search API
|
||||
key. **Recommended:** run `openclaw configure --section web` to store it in
|
||||
`tools.web.search.apiKey`. Environment alternative: set `BRAVE_API_KEY` for the
|
||||
Gateway process.
|
||||
`web_fetch` works without an API key. `web_search` requires provider setup:
|
||||
|
||||
- Brave: `tools.web.search.apiKey` or `BRAVE_API_KEY`
|
||||
- Perplexity: `tools.web.search.perplexity.apiKey` or `PERPLEXITY_API_KEY` / `OPENROUTER_API_KEY`
|
||||
- SearXNG: `tools.web.search.searxng.baseUrl` or `SEARXNG_BASE_URL`
|
||||
|
||||
**Recommended:** run `openclaw configure --section web` to set these in config.
|
||||
|
||||
```json5
|
||||
{
|
||||
@ -1330,6 +1333,7 @@ Gateway process.
|
||||
web: {
|
||||
search: {
|
||||
enabled: true,
|
||||
provider: "brave",
|
||||
apiKey: "BRAVE_API_KEY_HERE",
|
||||
maxResults: 5
|
||||
},
|
||||
|
||||
@ -62,11 +62,12 @@ You can keep it local with `memorySearch.provider = "local"` (no API usage).
|
||||
|
||||
See [Memory](/concepts/memory).
|
||||
|
||||
### 4) Web search tool (Brave / Perplexity via OpenRouter)
|
||||
### 4) Web search tool (Brave / Perplexity / SearXNG)
|
||||
`web_search` uses API keys and may incur usage charges:
|
||||
|
||||
- **Brave Search API**: `BRAVE_API_KEY` or `tools.web.search.apiKey`
|
||||
- **Perplexity** (via OpenRouter): `PERPLEXITY_API_KEY` or `OPENROUTER_API_KEY`
|
||||
- **SearXNG** (self-hosted): `tools.web.search.searxng.baseUrl` or `SEARXNG_BASE_URL` (no external API billing; depends on your instance)
|
||||
|
||||
**Brave free tier (generous):**
|
||||
- **2,000 requests/month**
|
||||
|
||||
@ -44,9 +44,8 @@ run on host, set an explicit per-agent override:
|
||||
|
||||
- Node `>=22`
|
||||
- `pnpm` (optional; recommended if you build from source)
|
||||
- **Recommended:** Brave Search API key for web search. Easiest path:
|
||||
`openclaw configure --section web` (stores `tools.web.search.apiKey`).
|
||||
See [Web tools](/tools/web).
|
||||
- **Recommended:** configure a `web_search` provider (Brave, Perplexity, or SearXNG) for web search.
|
||||
Easiest path: `openclaw configure --section web`. See [Web tools](/tools/web).
|
||||
|
||||
macOS: if you plan to build the apps, install Xcode / CLT. For the CLI + gateway only, Node is enough.
|
||||
Windows: use **WSL2** (Ubuntu recommended). WSL2 is strongly recommended; native Windows is untested, more problematic, and has poorer tool compatibility. Install WSL2 first, then run the Linux steps inside WSL. See [Windows (WSL2)](/platforms/windows).
|
||||
|
||||
@ -27,9 +27,9 @@ Follow‑up reconfiguration:
|
||||
openclaw configure
|
||||
```
|
||||
|
||||
Recommended: set up a Brave Search API key so the agent can use `web_search`
|
||||
(`web_fetch` works without a key). Easiest path: `openclaw configure --section web`
|
||||
which stores `tools.web.search.apiKey`. Docs: [Web tools](/tools/web).
|
||||
Recommended: set up a `web_search` provider (Brave, Perplexity, or SearXNG) so the
|
||||
agent can search the web when needed (`web_fetch` works without a key). Easiest
|
||||
path: `openclaw configure --section web`. Docs: [Web tools](/tools/web).
|
||||
|
||||
## QuickStart vs Advanced
|
||||
|
||||
|
||||
@ -205,14 +205,14 @@ Notes:
|
||||
- `process` is scoped per agent; sessions from other agents are not visible.
|
||||
|
||||
### `web_search`
|
||||
Search the web using Brave Search API.
|
||||
Search the web using your configured provider (Brave, Perplexity, or SearXNG).
|
||||
|
||||
Core parameters:
|
||||
- `query` (required)
|
||||
- `count` (1–10; default from `tools.web.search.maxResults`)
|
||||
|
||||
Notes:
|
||||
- Requires a Brave API key (recommended: `openclaw configure --section web`, or set `BRAVE_API_KEY`).
|
||||
- Requires provider setup (recommended: `openclaw configure --section web`).
|
||||
- Enable via `tools.web.search.enabled`.
|
||||
- Responses are cached (default 15 min).
|
||||
- See [Web tools](/tools/web) for setup.
|
||||
|
||||
@ -1,16 +1,17 @@
|
||||
---
|
||||
summary: "Web search + fetch tools (Brave Search API, Perplexity direct/OpenRouter)"
|
||||
summary: "Web search + fetch tools (Brave Search API, Perplexity direct/OpenRouter, SearXNG)"
|
||||
read_when:
|
||||
- You want to enable web_search or web_fetch
|
||||
- You need Brave Search API key setup
|
||||
- You want to use Perplexity Sonar for web search
|
||||
- You want to use SearXNG for web search
|
||||
---
|
||||
|
||||
# Web tools
|
||||
|
||||
OpenClaw ships two lightweight web tools:
|
||||
|
||||
- `web_search` — Search the web via Brave Search API (default) or Perplexity Sonar (direct or via OpenRouter).
|
||||
- `web_search` — Search the web via Brave Search API (default), Perplexity Sonar (direct or via OpenRouter), or a self-hosted SearXNG instance.
|
||||
- `web_fetch` — HTTP fetch + readable extraction (HTML → markdown/text).
|
||||
|
||||
These are **not** browser automation. For JS-heavy sites or logins, use the
|
||||
@ -21,6 +22,7 @@ These are **not** browser automation. For JS-heavy sites or logins, use the
|
||||
- `web_search` calls your configured provider and returns results.
|
||||
- **Brave** (default): returns structured results (title, URL, snippet).
|
||||
- **Perplexity**: returns AI-synthesized answers with citations from real-time web search.
|
||||
- **SearXNG**: returns structured results (title, URL, snippet) from your own meta-search instance.
|
||||
- Results are cached by query for 15 minutes (configurable).
|
||||
- `web_fetch` does a plain HTTP GET and extracts readable content
|
||||
(HTML → markdown/text). It does **not** execute JavaScript.
|
||||
@ -32,6 +34,7 @@ These are **not** browser automation. For JS-heavy sites or logins, use the
|
||||
|----------|------|------|---------|
|
||||
| **Brave** (default) | Fast, structured results, free tier | Traditional search results | `BRAVE_API_KEY` |
|
||||
| **Perplexity** | AI-synthesized answers, citations, real-time | Requires Perplexity or OpenRouter access | `OPENROUTER_API_KEY` or `PERPLEXITY_API_KEY` |
|
||||
| **SearXNG** | Self-hosted, works in private environments, flexible | Requires running SearXNG and enabling JSON format | None (needs base URL) |
|
||||
|
||||
See [Brave Search setup](/brave-search) and [Perplexity Sonar](/perplexity) for provider-specific details.
|
||||
|
||||
@ -42,7 +45,7 @@ Set the provider in config:
|
||||
tools: {
|
||||
web: {
|
||||
search: {
|
||||
provider: "brave" // or "perplexity"
|
||||
provider: "brave" // or "perplexity" or "searxng"
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -68,6 +71,34 @@ Example: switch to Perplexity Sonar (direct API):
|
||||
}
|
||||
```
|
||||
|
||||
## Using SearXNG
|
||||
|
||||
SearXNG exposes a simple HTTP API at `/search`. To get JSON results you need
|
||||
`format=json` enabled on your instance; some public instances disable it and will
|
||||
return `403 Forbidden`.
|
||||
|
||||
Example config:
|
||||
|
||||
```json5
|
||||
{
|
||||
tools: {
|
||||
web: {
|
||||
search: {
|
||||
enabled: true,
|
||||
provider: "searxng",
|
||||
searxng: {
|
||||
baseUrl: "http://localhost:8080",
|
||||
// Optional extra query params merged into each request:
|
||||
// params: { categories: "general", safesearch: 1 }
|
||||
// Optional auth via headers (for reverse-proxy auth):
|
||||
// headers: { "X-Api-Key": "..." }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Getting a Brave API key
|
||||
|
||||
1) Create a Brave Search API account at https://brave.com/search/api/
|
||||
@ -145,9 +176,10 @@ Search the web using your configured provider.
|
||||
### Requirements
|
||||
|
||||
- `tools.web.search.enabled` must not be `false` (default: enabled)
|
||||
- API key for your chosen provider:
|
||||
- Provider setup:
|
||||
- **Brave**: `BRAVE_API_KEY` or `tools.web.search.apiKey`
|
||||
- **Perplexity**: `OPENROUTER_API_KEY`, `PERPLEXITY_API_KEY`, or `tools.web.search.perplexity.apiKey`
|
||||
- **SearXNG**: `tools.web.search.searxng.baseUrl` or `SEARXNG_BASE_URL`
|
||||
|
||||
### Config
|
||||
|
||||
@ -175,6 +207,12 @@ Search the web using your configured provider.
|
||||
- `search_lang` (optional): ISO language code for search results (e.g., "de", "en", "fr")
|
||||
- `ui_lang` (optional): ISO language code for UI elements
|
||||
- `freshness` (optional, Brave only): filter by discovery time (`pd`, `pw`, `pm`, `py`, or `YYYY-MM-DDtoYYYY-MM-DD`)
|
||||
- `categories` (optional, SearXNG only): comma-separated categories
|
||||
- `engines` (optional, SearXNG only): comma-separated engines
|
||||
- `language` (optional, SearXNG only): language code (if omitted, falls back to `search_lang`)
|
||||
- `time_range` (optional, SearXNG only): `day`, `month`, `year`
|
||||
- `safesearch` (optional, SearXNG only): `0`, `1`, `2`
|
||||
- `pageno` (optional, SearXNG only): page number (default `1`)
|
||||
|
||||
**Examples:**
|
||||
|
||||
|
||||
@ -2,8 +2,13 @@ import { describe, expect, it } from "vitest";
|
||||
|
||||
import { __testing } from "./web-search.js";
|
||||
|
||||
const { inferPerplexityBaseUrlFromApiKey, resolvePerplexityBaseUrl, normalizeFreshness } =
|
||||
__testing;
|
||||
const {
|
||||
inferPerplexityBaseUrlFromApiKey,
|
||||
resolvePerplexityBaseUrl,
|
||||
normalizeFreshness,
|
||||
resolveSearxngBaseUrl,
|
||||
resolveSearxngHeaders,
|
||||
} = __testing;
|
||||
|
||||
describe("web_search perplexity baseUrl defaults", () => {
|
||||
it("detects a Perplexity key prefix", () => {
|
||||
@ -69,3 +74,36 @@ describe("web_search freshness normalization", () => {
|
||||
expect(normalizeFreshness("2024-03-10to2024-03-01")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("web_search searxng config resolution", () => {
|
||||
it("prefers config baseUrl and trims trailing slash", () => {
|
||||
expect(resolveSearxngBaseUrl({ baseUrl: "http://localhost:8080/" })).toBe(
|
||||
"http://localhost:8080",
|
||||
);
|
||||
});
|
||||
|
||||
it("falls back to SEARXNG_BASE_URL env var", () => {
|
||||
const prev = process.env.SEARXNG_BASE_URL;
|
||||
process.env.SEARXNG_BASE_URL = "http://searxng.local:8888/";
|
||||
try {
|
||||
expect(resolveSearxngBaseUrl(undefined)).toBe("http://searxng.local:8888");
|
||||
} finally {
|
||||
if (prev === undefined) {
|
||||
delete process.env.SEARXNG_BASE_URL;
|
||||
} else {
|
||||
process.env.SEARXNG_BASE_URL = prev;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("builds headers with optional auth and extras", () => {
|
||||
expect(resolveSearxngHeaders(undefined)).toEqual({ Accept: "application/json" });
|
||||
|
||||
expect(resolveSearxngHeaders({ apiKey: "token-123" }).Authorization).toBe("Bearer token-123");
|
||||
expect(resolveSearxngHeaders({ apiKey: "Basic abc" }).Authorization).toBe("Basic abc");
|
||||
|
||||
const headers = resolveSearxngHeaders({ headers: { "X-Test": "ok" }, apiKey: "token-123" });
|
||||
expect(headers["X-Test"]).toBe("ok");
|
||||
expect(headers.Accept).toBe("application/json");
|
||||
});
|
||||
});
|
||||
|
||||
@ -17,7 +17,7 @@ import {
|
||||
writeCache,
|
||||
} from "./web-shared.js";
|
||||
|
||||
const SEARCH_PROVIDERS = ["brave", "perplexity"] as const;
|
||||
const SEARCH_PROVIDERS = ["brave", "perplexity", "searxng"] as const;
|
||||
const DEFAULT_SEARCH_COUNT = 5;
|
||||
const MAX_SEARCH_COUNT = 10;
|
||||
|
||||
@ -41,6 +41,39 @@ const WebSearchSchema = Type.Object({
|
||||
maximum: MAX_SEARCH_COUNT,
|
||||
}),
|
||||
),
|
||||
categories: Type.Optional(
|
||||
Type.String({
|
||||
description: "Comma-separated categories to search (SearXNG only), e.g. 'general,news'.",
|
||||
}),
|
||||
),
|
||||
engines: Type.Optional(
|
||||
Type.String({
|
||||
description: "Comma-separated engines to search (SearXNG only), e.g. 'duckduckgo,bing'.",
|
||||
}),
|
||||
),
|
||||
language: Type.Optional(
|
||||
Type.String({
|
||||
description: "Language code (SearXNG), e.g. 'en', 'de'.",
|
||||
}),
|
||||
),
|
||||
time_range: Type.Optional(
|
||||
Type.String({
|
||||
description: "Time range (SearXNG only). Values: day, month, year.",
|
||||
}),
|
||||
),
|
||||
safesearch: Type.Optional(
|
||||
Type.Number({
|
||||
description: "Safe search level (SearXNG only). Values: 0, 1, 2.",
|
||||
minimum: 0,
|
||||
maximum: 2,
|
||||
}),
|
||||
),
|
||||
pageno: Type.Optional(
|
||||
Type.Number({
|
||||
description: "Search page number (SearXNG only). Default: 1.",
|
||||
minimum: 1,
|
||||
}),
|
||||
),
|
||||
country: Type.Optional(
|
||||
Type.String({
|
||||
description:
|
||||
@ -90,6 +123,13 @@ type PerplexityConfig = {
|
||||
model?: string;
|
||||
};
|
||||
|
||||
type SearxngConfig = {
|
||||
baseUrl?: string;
|
||||
apiKey?: string;
|
||||
headers?: Record<string, string>;
|
||||
params?: Record<string, string | number | boolean>;
|
||||
};
|
||||
|
||||
type PerplexityApiKeySource = "config" | "perplexity_env" | "openrouter_env" | "none";
|
||||
|
||||
type PerplexitySearchResponse = {
|
||||
@ -103,6 +143,17 @@ type PerplexitySearchResponse = {
|
||||
|
||||
type PerplexityBaseUrlHint = "direct" | "openrouter";
|
||||
|
||||
type SearxngSearchResult = {
|
||||
title?: string;
|
||||
url?: string;
|
||||
content?: string;
|
||||
publishedDate?: string;
|
||||
};
|
||||
|
||||
type SearxngSearchResponse = {
|
||||
results?: SearxngSearchResult[];
|
||||
};
|
||||
|
||||
function resolveSearchConfig(cfg?: OpenClawConfig): WebSearchConfig {
|
||||
const search = cfg?.tools?.web?.search;
|
||||
if (!search || typeof search !== "object") return undefined;
|
||||
@ -131,6 +182,14 @@ function missingSearchKeyPayload(provider: (typeof SEARCH_PROVIDERS)[number]) {
|
||||
docs: "https://docs.openclaw.ai/tools/web",
|
||||
};
|
||||
}
|
||||
if (provider === "searxng") {
|
||||
return {
|
||||
error: "missing_searxng_base_url",
|
||||
message:
|
||||
"web_search (searxng) needs a base URL. Configure tools.web.search.searxng.baseUrl or set SEARXNG_BASE_URL in the Gateway environment.",
|
||||
docs: "https://docs.openclaw.ai/tools/web",
|
||||
};
|
||||
}
|
||||
return {
|
||||
error: "missing_brave_api_key",
|
||||
message: `web_search needs a Brave Search API key. Run \`${formatCliCommand("openclaw configure --section web")}\` to store it, or set BRAVE_API_KEY in the Gateway environment.`,
|
||||
@ -145,6 +204,7 @@ function resolveSearchProvider(search?: WebSearchConfig): (typeof SEARCH_PROVIDE
|
||||
: "";
|
||||
if (raw === "perplexity") return "perplexity";
|
||||
if (raw === "brave") return "brave";
|
||||
if (raw === "searxng") return "searxng";
|
||||
return "brave";
|
||||
}
|
||||
|
||||
@ -155,6 +215,13 @@ function resolvePerplexityConfig(search?: WebSearchConfig): PerplexityConfig {
|
||||
return perplexity as PerplexityConfig;
|
||||
}
|
||||
|
||||
function resolveSearxngConfig(search?: WebSearchConfig): SearxngConfig {
|
||||
if (!search || typeof search !== "object") return {};
|
||||
const searxng = "searxng" in search ? (search as Record<string, unknown>).searxng : undefined;
|
||||
if (!searxng || typeof searxng !== "object") return {};
|
||||
return searxng as SearxngConfig;
|
||||
}
|
||||
|
||||
function resolvePerplexityApiKey(perplexity?: PerplexityConfig): {
|
||||
apiKey?: string;
|
||||
source: PerplexityApiKeySource;
|
||||
@ -221,6 +288,38 @@ function resolvePerplexityModel(perplexity?: PerplexityConfig): string {
|
||||
return fromConfig || DEFAULT_PERPLEXITY_MODEL;
|
||||
}
|
||||
|
||||
function resolveSearxngBaseUrl(searxng?: SearxngConfig): string | undefined {
|
||||
const fromConfig =
|
||||
searxng && "baseUrl" in searxng && typeof searxng.baseUrl === "string" ? searxng.baseUrl : "";
|
||||
const fromEnv = (process.env.SEARXNG_BASE_URL ?? "").trim();
|
||||
const baseUrl = (fromConfig ?? "").trim() || fromEnv;
|
||||
if (!baseUrl) return undefined;
|
||||
return baseUrl.replace(/\/$/, "");
|
||||
}
|
||||
|
||||
function resolveSearxngHeaders(searxng?: SearxngConfig): Record<string, string> {
|
||||
const headers: Record<string, string> = {
|
||||
Accept: "application/json",
|
||||
};
|
||||
|
||||
if (!searxng || typeof searxng !== "object") return headers;
|
||||
|
||||
if (searxng.headers && typeof searxng.headers === "object") {
|
||||
for (const [key, value] of Object.entries(searxng.headers)) {
|
||||
if (typeof value === "string" && value.trim()) {
|
||||
headers[key] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const apiKey = typeof searxng.apiKey === "string" ? searxng.apiKey.trim() : "";
|
||||
if (apiKey) {
|
||||
headers.Authorization = apiKey.includes(" ") ? apiKey : `Bearer ${apiKey}`;
|
||||
}
|
||||
|
||||
return headers;
|
||||
}
|
||||
|
||||
function resolveSearchCount(value: unknown, fallback: number): number {
|
||||
const parsed = typeof value === "number" && Number.isFinite(value) ? value : fallback;
|
||||
const clamped = Math.max(1, Math.min(MAX_SEARCH_COUNT, Math.floor(parsed)));
|
||||
@ -306,6 +405,72 @@ async function runPerplexitySearch(params: {
|
||||
return { content, citations };
|
||||
}
|
||||
|
||||
function addSearxngParams(
|
||||
searchParams: URLSearchParams,
|
||||
params?: Record<string, string | number | boolean>,
|
||||
): void {
|
||||
if (!params) return;
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
if (!key.trim()) continue;
|
||||
if (value === undefined) continue;
|
||||
searchParams.set(key, String(value));
|
||||
}
|
||||
}
|
||||
|
||||
async function runSearxngSearch(params: {
|
||||
query: string;
|
||||
baseUrl: string;
|
||||
count: number;
|
||||
timeoutSeconds: number;
|
||||
searchParams?: Record<string, string | number | boolean>;
|
||||
headers?: Record<string, string>;
|
||||
categories?: string;
|
||||
engines?: string;
|
||||
language?: string;
|
||||
time_range?: string;
|
||||
safesearch?: number;
|
||||
pageno?: number;
|
||||
}): Promise<{ results: Array<Record<string, unknown>> }> {
|
||||
const url = new URL(`${params.baseUrl.replace(/\/$/, "")}/search`);
|
||||
addSearxngParams(url.searchParams, params.searchParams);
|
||||
url.searchParams.set("q", params.query);
|
||||
url.searchParams.set("format", "json");
|
||||
|
||||
if (params.categories) url.searchParams.set("categories", params.categories);
|
||||
if (params.engines) url.searchParams.set("engines", params.engines);
|
||||
if (params.language) url.searchParams.set("language", params.language);
|
||||
if (params.time_range) url.searchParams.set("time_range", params.time_range);
|
||||
if (typeof params.safesearch === "number" && Number.isFinite(params.safesearch)) {
|
||||
url.searchParams.set("safesearch", String(params.safesearch));
|
||||
}
|
||||
if (typeof params.pageno === "number" && Number.isFinite(params.pageno) && params.pageno > 0) {
|
||||
url.searchParams.set("pageno", String(Math.floor(params.pageno)));
|
||||
}
|
||||
|
||||
const res = await fetch(url.toString(), {
|
||||
method: "GET",
|
||||
headers: params.headers,
|
||||
signal: withTimeout(undefined, params.timeoutSeconds * 1000),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
const detail = await readResponseText(res);
|
||||
throw new Error(`SearXNG API error (${res.status}): ${detail || res.statusText}`);
|
||||
}
|
||||
|
||||
const data = (await res.json()) as SearxngSearchResponse;
|
||||
const results = Array.isArray(data.results) ? data.results : [];
|
||||
const mapped = results.slice(0, params.count).map((entry) => ({
|
||||
title: entry.title ?? "",
|
||||
url: entry.url ?? "",
|
||||
description: entry.content ?? "",
|
||||
published: entry.publishedDate ?? undefined,
|
||||
siteName: resolveSiteName(entry.url ?? ""),
|
||||
}));
|
||||
|
||||
return { results: mapped };
|
||||
}
|
||||
|
||||
async function runWebSearch(params: {
|
||||
query: string;
|
||||
count: number;
|
||||
@ -319,11 +484,22 @@ async function runWebSearch(params: {
|
||||
freshness?: string;
|
||||
perplexityBaseUrl?: string;
|
||||
perplexityModel?: string;
|
||||
searxngBaseUrl?: string;
|
||||
searxngHeaders?: Record<string, string>;
|
||||
searxngParams?: Record<string, string | number | boolean>;
|
||||
categories?: string;
|
||||
engines?: string;
|
||||
language?: string;
|
||||
time_range?: string;
|
||||
safesearch?: number;
|
||||
pageno?: number;
|
||||
}): Promise<Record<string, unknown>> {
|
||||
const cacheKey = normalizeCacheKey(
|
||||
params.provider === "brave"
|
||||
? `${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}:${params.freshness || "default"}`
|
||||
: `${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`,
|
||||
: params.provider === "searxng"
|
||||
? `${params.provider}:${params.query}:${params.count}:${params.categories || "default"}:${params.engines || "default"}:${params.language || "default"}:${params.time_range || "default"}:${String(params.safesearch ?? "default")}:${String(params.pageno ?? "default")}`
|
||||
: `${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`,
|
||||
);
|
||||
const cached = readCache(SEARCH_CACHE, cacheKey);
|
||||
if (cached) return { ...cached.value, cached: true };
|
||||
@ -351,6 +527,33 @@ async function runWebSearch(params: {
|
||||
return payload;
|
||||
}
|
||||
|
||||
if (params.provider === "searxng") {
|
||||
const { results } = await runSearxngSearch({
|
||||
query: params.query,
|
||||
baseUrl: params.searxngBaseUrl ?? "",
|
||||
count: params.count,
|
||||
timeoutSeconds: params.timeoutSeconds,
|
||||
searchParams: params.searxngParams,
|
||||
headers: params.searxngHeaders,
|
||||
categories: params.categories,
|
||||
engines: params.engines,
|
||||
language: params.language,
|
||||
time_range: params.time_range,
|
||||
safesearch: params.safesearch,
|
||||
pageno: params.pageno,
|
||||
});
|
||||
|
||||
const payload = {
|
||||
query: params.query,
|
||||
provider: params.provider,
|
||||
count: results.length,
|
||||
tookMs: Date.now() - start,
|
||||
results,
|
||||
};
|
||||
writeCache(SEARCH_CACHE, cacheKey, payload, params.cacheTtlMs);
|
||||
return payload;
|
||||
}
|
||||
|
||||
if (params.provider !== "brave") {
|
||||
throw new Error("Unsupported web search provider.");
|
||||
}
|
||||
@ -415,11 +618,14 @@ export function createWebSearchTool(options?: {
|
||||
|
||||
const provider = resolveSearchProvider(search);
|
||||
const perplexityConfig = resolvePerplexityConfig(search);
|
||||
const searxngConfig = resolveSearxngConfig(search);
|
||||
|
||||
const description =
|
||||
provider === "perplexity"
|
||||
? "Search the web using Perplexity Sonar (direct or via OpenRouter). Returns AI-synthesized answers with citations from real-time web search."
|
||||
: "Search the web using Brave Search API. Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research.";
|
||||
: provider === "searxng"
|
||||
? "Search the web using a self-hosted SearXNG instance. Returns titles, URLs, and snippets for fast research."
|
||||
: "Search the web using Brave Search API. Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research.";
|
||||
|
||||
return {
|
||||
label: "Web Search",
|
||||
@ -431,14 +637,26 @@ export function createWebSearchTool(options?: {
|
||||
provider === "perplexity" ? resolvePerplexityApiKey(perplexityConfig) : undefined;
|
||||
const apiKey =
|
||||
provider === "perplexity" ? perplexityAuth?.apiKey : resolveSearchApiKey(search);
|
||||
const searxngBaseUrl =
|
||||
provider === "searxng" ? resolveSearxngBaseUrl(searxngConfig) : undefined;
|
||||
|
||||
if (!apiKey) {
|
||||
if (provider === "searxng") {
|
||||
if (!searxngBaseUrl) return jsonResult(missingSearchKeyPayload(provider));
|
||||
} else if (!apiKey) {
|
||||
return jsonResult(missingSearchKeyPayload(provider));
|
||||
}
|
||||
const params = args as Record<string, unknown>;
|
||||
const query = readStringParam(params, "query", { required: true });
|
||||
const count =
|
||||
readNumberParam(params, "count", { integer: true }) ?? search?.maxResults ?? undefined;
|
||||
const categories = readStringParam(params, "categories");
|
||||
const engines = readStringParam(params, "engines");
|
||||
const language =
|
||||
readStringParam(params, "language") ??
|
||||
(provider === "searxng" ? readStringParam(params, "search_lang") : undefined);
|
||||
const time_range = readStringParam(params, "time_range");
|
||||
const safesearch = readNumberParam(params, "safesearch", { integer: true });
|
||||
const pageno = readNumberParam(params, "pageno", { integer: true });
|
||||
const country = readStringParam(params, "country");
|
||||
const search_lang = readStringParam(params, "search_lang");
|
||||
const ui_lang = readStringParam(params, "ui_lang");
|
||||
@ -462,7 +680,7 @@ export function createWebSearchTool(options?: {
|
||||
const result = await runWebSearch({
|
||||
query,
|
||||
count: resolveSearchCount(count, DEFAULT_SEARCH_COUNT),
|
||||
apiKey,
|
||||
apiKey: apiKey ?? "",
|
||||
timeoutSeconds: resolveTimeoutSeconds(search?.timeoutSeconds, DEFAULT_TIMEOUT_SECONDS),
|
||||
cacheTtlMs: resolveCacheTtlMs(search?.cacheTtlMinutes, DEFAULT_CACHE_TTL_MINUTES),
|
||||
provider,
|
||||
@ -476,6 +694,15 @@ export function createWebSearchTool(options?: {
|
||||
perplexityAuth?.apiKey,
|
||||
),
|
||||
perplexityModel: resolvePerplexityModel(perplexityConfig),
|
||||
searxngBaseUrl,
|
||||
searxngHeaders: provider === "searxng" ? resolveSearxngHeaders(searxngConfig) : undefined,
|
||||
searxngParams: provider === "searxng" ? searxngConfig.params : undefined,
|
||||
categories,
|
||||
engines,
|
||||
language,
|
||||
time_range,
|
||||
safesearch: typeof safesearch === "number" ? safesearch : undefined,
|
||||
pageno: typeof pageno === "number" ? pageno : undefined,
|
||||
});
|
||||
return jsonResult(result);
|
||||
},
|
||||
@ -486,4 +713,6 @@ export const __testing = {
|
||||
inferPerplexityBaseUrlFromApiKey,
|
||||
resolvePerplexityBaseUrl,
|
||||
normalizeFreshness,
|
||||
resolveSearxngBaseUrl,
|
||||
resolveSearxngHeaders,
|
||||
} as const;
|
||||
|
||||
@ -193,6 +193,10 @@ const FIELD_LABELS: Record<string, string> = {
|
||||
"tools.web.search.maxResults": "Web Search Max Results",
|
||||
"tools.web.search.timeoutSeconds": "Web Search Timeout (sec)",
|
||||
"tools.web.search.cacheTtlMinutes": "Web Search Cache TTL (min)",
|
||||
"tools.web.search.searxng.baseUrl": "SearXNG Base URL",
|
||||
"tools.web.search.searxng.apiKey": "SearXNG API Key",
|
||||
"tools.web.search.searxng.headers": "SearXNG Extra Headers",
|
||||
"tools.web.search.searxng.params": "SearXNG Default Params",
|
||||
"tools.web.fetch.enabled": "Enable Web Fetch Tool",
|
||||
"tools.web.fetch.maxChars": "Web Fetch Max Chars",
|
||||
"tools.web.fetch.timeoutSeconds": "Web Fetch Timeout (sec)",
|
||||
@ -434,8 +438,9 @@ const FIELD_HELP: Record<string, string> = {
|
||||
"tools.message.crossContext.marker.suffix":
|
||||
'Text suffix for cross-context markers (supports "{channel}").',
|
||||
"tools.message.broadcast.enabled": "Enable broadcast action (default: true).",
|
||||
"tools.web.search.enabled": "Enable the web_search tool (requires a provider API key).",
|
||||
"tools.web.search.provider": 'Search provider ("brave" or "perplexity").',
|
||||
"tools.web.search.enabled":
|
||||
"Enable the web_search tool (requires provider configuration: API key or base URL).",
|
||||
"tools.web.search.provider": 'Search provider ("brave", "perplexity", or "searxng").',
|
||||
"tools.web.search.apiKey": "Brave Search API key (fallback: BRAVE_API_KEY env var).",
|
||||
"tools.web.search.maxResults": "Default number of results to return (1-10).",
|
||||
"tools.web.search.timeoutSeconds": "Timeout in seconds for web_search requests.",
|
||||
@ -446,6 +451,14 @@ const FIELD_HELP: Record<string, string> = {
|
||||
"Perplexity base URL override (default: https://openrouter.ai/api/v1 or https://api.perplexity.ai).",
|
||||
"tools.web.search.perplexity.model":
|
||||
'Perplexity model override (default: "perplexity/sonar-pro").',
|
||||
"tools.web.search.searxng.baseUrl":
|
||||
"SearXNG instance base URL (e.g. http://localhost:8080). Fallback: SEARXNG_BASE_URL env var.",
|
||||
"tools.web.search.searxng.apiKey":
|
||||
"Optional SearXNG auth token; sent as Authorization header (Bearer by default).",
|
||||
"tools.web.search.searxng.headers":
|
||||
"Optional extra headers for SearXNG requests (e.g. for reverse-proxy auth).",
|
||||
"tools.web.search.searxng.params":
|
||||
"Optional default query params for SearXNG /search (merged into each request).",
|
||||
"tools.web.fetch.enabled": "Enable the web_fetch tool (lightweight HTTP fetch).",
|
||||
"tools.web.fetch.maxChars": "Max characters returned by web_fetch (truncated).",
|
||||
"tools.web.fetch.timeoutSeconds": "Timeout in seconds for web_fetch requests.",
|
||||
|
||||
@ -165,7 +165,9 @@ export const ToolPolicySchema = ToolPolicyBaseSchema.superRefine((value, ctx) =>
|
||||
export const ToolsWebSearchSchema = z
|
||||
.object({
|
||||
enabled: z.boolean().optional(),
|
||||
provider: z.union([z.literal("brave"), z.literal("perplexity")]).optional(),
|
||||
provider: z
|
||||
.union([z.literal("brave"), z.literal("perplexity"), z.literal("searxng")])
|
||||
.optional(),
|
||||
apiKey: z.string().optional(),
|
||||
maxResults: z.number().int().positive().optional(),
|
||||
timeoutSeconds: z.number().int().positive().optional(),
|
||||
@ -178,6 +180,15 @@ export const ToolsWebSearchSchema = z
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
searxng: z
|
||||
.object({
|
||||
baseUrl: z.string().optional(),
|
||||
apiKey: z.string().optional(),
|
||||
headers: z.record(z.string(), z.string()).optional(),
|
||||
params: z.record(z.string(), z.union([z.string(), z.number(), z.boolean()])).optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional(),
|
||||
})
|
||||
.strict()
|
||||
.optional();
|
||||
|
||||
@ -433,29 +433,101 @@ export async function finalizeOnboardingWizard(options: FinalizeOnboardingOption
|
||||
);
|
||||
}
|
||||
|
||||
const webSearchKey = (nextConfig.tools?.web?.search?.apiKey ?? "").trim();
|
||||
const webSearchEnv = (process.env.BRAVE_API_KEY ?? "").trim();
|
||||
const hasWebSearchKey = Boolean(webSearchKey || webSearchEnv);
|
||||
const webSearch = nextConfig.tools?.web?.search;
|
||||
const webSearchProviderRaw =
|
||||
webSearch &&
|
||||
typeof webSearch === "object" &&
|
||||
"provider" in webSearch &&
|
||||
typeof webSearch.provider === "string"
|
||||
? webSearch.provider.trim().toLowerCase()
|
||||
: "";
|
||||
const webSearchProvider =
|
||||
webSearchProviderRaw === "perplexity"
|
||||
? "perplexity"
|
||||
: webSearchProviderRaw === "searxng"
|
||||
? "searxng"
|
||||
: "brave";
|
||||
|
||||
const readTrimmedString = (value: unknown): string =>
|
||||
typeof value === "string" ? value.trim() : "";
|
||||
|
||||
const braveKey =
|
||||
webSearch && typeof webSearch === "object" && "apiKey" in webSearch
|
||||
? readTrimmedString(webSearch.apiKey)
|
||||
: "";
|
||||
const braveEnv = (process.env.BRAVE_API_KEY ?? "").trim();
|
||||
const hasBraveKey = Boolean(braveKey || braveEnv);
|
||||
|
||||
const perplexityKey =
|
||||
webSearch &&
|
||||
typeof webSearch === "object" &&
|
||||
"perplexity" in webSearch &&
|
||||
webSearch.perplexity &&
|
||||
typeof webSearch.perplexity === "object" &&
|
||||
"apiKey" in webSearch.perplexity
|
||||
? readTrimmedString((webSearch.perplexity as Record<string, unknown>).apiKey)
|
||||
: "";
|
||||
const perplexityEnv = (process.env.PERPLEXITY_API_KEY ?? "").trim();
|
||||
const openrouterEnv = (process.env.OPENROUTER_API_KEY ?? "").trim();
|
||||
const hasPerplexityKey = Boolean(perplexityKey || perplexityEnv || openrouterEnv);
|
||||
|
||||
const searxngBaseUrl =
|
||||
webSearch &&
|
||||
typeof webSearch === "object" &&
|
||||
"searxng" in webSearch &&
|
||||
webSearch.searxng &&
|
||||
typeof webSearch.searxng === "object" &&
|
||||
"baseUrl" in webSearch.searxng
|
||||
? readTrimmedString((webSearch.searxng as Record<string, unknown>).baseUrl)
|
||||
: "";
|
||||
const searxngEnv = (process.env.SEARXNG_BASE_URL ?? "").trim();
|
||||
const hasSearxngBaseUrl = Boolean(searxngBaseUrl || searxngEnv);
|
||||
|
||||
const hasWebSearchSetup =
|
||||
webSearchProvider === "perplexity"
|
||||
? hasPerplexityKey
|
||||
: webSearchProvider === "searxng"
|
||||
? hasSearxngBaseUrl
|
||||
: hasBraveKey;
|
||||
await prompter.note(
|
||||
hasWebSearchKey
|
||||
hasWebSearchSetup
|
||||
? [
|
||||
"Web search is enabled, so your agent can look things up online when needed.",
|
||||
"",
|
||||
webSearchKey
|
||||
? "API key: stored in config (tools.web.search.apiKey)."
|
||||
: "API key: provided via BRAVE_API_KEY env var (Gateway environment).",
|
||||
`Provider: ${webSearchProvider}`,
|
||||
webSearchProvider === "searxng"
|
||||
? searxngBaseUrl
|
||||
? "Base URL: stored in config (tools.web.search.searxng.baseUrl)."
|
||||
: "Base URL: provided via SEARXNG_BASE_URL env var (Gateway environment)."
|
||||
: webSearchProvider === "perplexity"
|
||||
? perplexityKey
|
||||
? "API key: stored in config (tools.web.search.perplexity.apiKey)."
|
||||
: perplexityEnv
|
||||
? "API key: provided via PERPLEXITY_API_KEY env var (Gateway environment)."
|
||||
: "API key: provided via OPENROUTER_API_KEY env var (Gateway environment)."
|
||||
: braveKey
|
||||
? "API key: stored in config (tools.web.search.apiKey)."
|
||||
: "API key: provided via BRAVE_API_KEY env var (Gateway environment).",
|
||||
"Docs: https://docs.openclaw.ai/tools/web",
|
||||
].join("\n")
|
||||
: [
|
||||
"If you want your agent to be able to search the web, you’ll need an API key.",
|
||||
"If you want your agent to be able to search the web, you’ll need to configure a provider.",
|
||||
"",
|
||||
"OpenClaw uses Brave Search for the `web_search` tool. Without a Brave Search API key, web search won’t work.",
|
||||
`OpenClaw uses ${webSearchProvider} for the \`web_search\` tool.`,
|
||||
"",
|
||||
"Set it up interactively:",
|
||||
`- Run: ${formatCliCommand("openclaw configure --section web")}`,
|
||||
"- Enable web_search and paste your Brave Search API key",
|
||||
webSearchProvider === "searxng"
|
||||
? "- Enable web_search and set your SearXNG base URL"
|
||||
: webSearchProvider === "perplexity"
|
||||
? "- Enable web_search and set your Perplexity/OpenRouter API key"
|
||||
: "- Enable web_search and paste your Brave Search API key",
|
||||
"",
|
||||
"Alternative: set BRAVE_API_KEY in the Gateway environment (no config changes).",
|
||||
webSearchProvider === "searxng"
|
||||
? "Alternative: set SEARXNG_BASE_URL in the Gateway environment (no config changes)."
|
||||
: webSearchProvider === "perplexity"
|
||||
? "Alternative: set PERPLEXITY_API_KEY or OPENROUTER_API_KEY in the Gateway environment (no config changes)."
|
||||
: "Alternative: set BRAVE_API_KEY in the Gateway environment (no config changes).",
|
||||
"Docs: https://docs.openclaw.ai/tools/web",
|
||||
].join("\n"),
|
||||
"Web search (optional)",
|
||||
|
||||
Loading…
Reference in New Issue
Block a user