diff --git a/CHANGELOG.md b/CHANGELOG.md index 191c2172d..0cbf00944 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,7 @@ Status: stable. - Matrix: switch plugin SDK to @vector-im/matrix-bot-sdk. - Tlon: format thread reply IDs as @ud. (#1837) Thanks @wca4a. - Tools: add per-sender group tool policies and fix precedence. (#1757) Thanks @adam91holt. +- Tools: add SearXNG as a `web_search` provider. (#2317) - Agents: summarize dropped messages during compaction safeguard pruning. (#2509) Thanks @jogi47. - Agents: expand cron tool description with full schema docs. (#1988) Thanks @tomascupr. - Agents: honor tools.exec.safeBins in exec allowlist checks. (#2281) diff --git a/docs/gateway/configuration.md b/docs/gateway/configuration.md index 5a00ea9cd..de60fdc1f 100644 --- a/docs/gateway/configuration.md +++ b/docs/gateway/configuration.md @@ -1942,8 +1942,16 @@ of `every`, keep `HEARTBEAT.md` tiny, and/or choose a cheaper `model`. Note: `applyPatch` is only under `tools.exec`. `tools.web` configures web search + fetch tools: -- `tools.web.search.enabled` (default: true when key is present) -- `tools.web.search.apiKey` (recommended: set via `openclaw configure --section web`, or use `BRAVE_API_KEY` env var) +- `tools.web.search.enabled` (default true) +- `tools.web.search.provider` (`brave` | `perplexity` | `searxng`) +- `tools.web.search.apiKey` (Brave; recommended: set via `openclaw configure --section web`, or use `BRAVE_API_KEY` env var) +- `tools.web.search.perplexity.apiKey` (Perplexity/OpenRouter; optional if `PERPLEXITY_API_KEY` or `OPENROUTER_API_KEY` is set) +- `tools.web.search.perplexity.baseUrl` (optional override) +- `tools.web.search.perplexity.model` (optional override) +- `tools.web.search.searxng.baseUrl` (SearXNG; optional if `SEARXNG_BASE_URL` is set) +- `tools.web.search.searxng.apiKey` (optional auth token; sent as Authorization header) +- `tools.web.search.searxng.headers` (optional extra headers) +- `tools.web.search.searxng.params` (optional default query params) - `tools.web.search.maxResults` (1–10, default 5) - `tools.web.search.timeoutSeconds` (default 30) - `tools.web.search.cacheTtlMinutes` (default 15) diff --git a/docs/help/faq.md b/docs/help/faq.md index e960d6a19..15d51ef35 100644 --- a/docs/help/faq.md +++ b/docs/help/faq.md @@ -1319,10 +1319,13 @@ The Gateway watches the config and supports hot‑reload: ### How do I enable web search and web fetch -`web_fetch` works without an API key. `web_search` requires a Brave Search API -key. **Recommended:** run `openclaw configure --section web` to store it in -`tools.web.search.apiKey`. Environment alternative: set `BRAVE_API_KEY` for the -Gateway process. +`web_fetch` works without an API key. `web_search` requires provider setup: + +- Brave: `tools.web.search.apiKey` or `BRAVE_API_KEY` +- Perplexity: `tools.web.search.perplexity.apiKey` or `PERPLEXITY_API_KEY` / `OPENROUTER_API_KEY` +- SearXNG: `tools.web.search.searxng.baseUrl` or `SEARXNG_BASE_URL` + +**Recommended:** run `openclaw configure --section web` to set these in config. ```json5 { @@ -1330,6 +1333,7 @@ Gateway process. web: { search: { enabled: true, + provider: "brave", apiKey: "BRAVE_API_KEY_HERE", maxResults: 5 }, diff --git a/docs/reference/api-usage-costs.md b/docs/reference/api-usage-costs.md index d4f47b17f..1ab079ceb 100644 --- a/docs/reference/api-usage-costs.md +++ b/docs/reference/api-usage-costs.md @@ -62,11 +62,12 @@ You can keep it local with `memorySearch.provider = "local"` (no API usage). See [Memory](/concepts/memory). -### 4) Web search tool (Brave / Perplexity via OpenRouter) +### 4) Web search tool (Brave / Perplexity / SearXNG) `web_search` uses API keys and may incur usage charges: - **Brave Search API**: `BRAVE_API_KEY` or `tools.web.search.apiKey` - **Perplexity** (via OpenRouter): `PERPLEXITY_API_KEY` or `OPENROUTER_API_KEY` +- **SearXNG** (self-hosted): `tools.web.search.searxng.baseUrl` or `SEARXNG_BASE_URL` (no external API billing; depends on your instance) **Brave free tier (generous):** - **2,000 requests/month** diff --git a/docs/start/getting-started.md b/docs/start/getting-started.md index 5102632c4..836b5ebca 100644 --- a/docs/start/getting-started.md +++ b/docs/start/getting-started.md @@ -44,9 +44,8 @@ run on host, set an explicit per-agent override: - Node `>=22` - `pnpm` (optional; recommended if you build from source) -- **Recommended:** Brave Search API key for web search. Easiest path: - `openclaw configure --section web` (stores `tools.web.search.apiKey`). - See [Web tools](/tools/web). +- **Recommended:** configure a `web_search` provider (Brave, Perplexity, or SearXNG) for web search. + Easiest path: `openclaw configure --section web`. See [Web tools](/tools/web). macOS: if you plan to build the apps, install Xcode / CLT. For the CLI + gateway only, Node is enough. Windows: use **WSL2** (Ubuntu recommended). WSL2 is strongly recommended; native Windows is untested, more problematic, and has poorer tool compatibility. Install WSL2 first, then run the Linux steps inside WSL. See [Windows (WSL2)](/platforms/windows). diff --git a/docs/start/wizard.md b/docs/start/wizard.md index 69fe8d0a7..473e94488 100644 --- a/docs/start/wizard.md +++ b/docs/start/wizard.md @@ -27,9 +27,9 @@ Follow‑up reconfiguration: openclaw configure ``` -Recommended: set up a Brave Search API key so the agent can use `web_search` -(`web_fetch` works without a key). Easiest path: `openclaw configure --section web` -which stores `tools.web.search.apiKey`. Docs: [Web tools](/tools/web). +Recommended: set up a `web_search` provider (Brave, Perplexity, or SearXNG) so the +agent can search the web when needed (`web_fetch` works without a key). Easiest +path: `openclaw configure --section web`. Docs: [Web tools](/tools/web). ## QuickStart vs Advanced diff --git a/docs/tools/index.md b/docs/tools/index.md index 9e110c93d..d3ad3de1c 100644 --- a/docs/tools/index.md +++ b/docs/tools/index.md @@ -205,14 +205,14 @@ Notes: - `process` is scoped per agent; sessions from other agents are not visible. ### `web_search` -Search the web using Brave Search API. +Search the web using your configured provider (Brave, Perplexity, or SearXNG). Core parameters: - `query` (required) - `count` (1–10; default from `tools.web.search.maxResults`) Notes: -- Requires a Brave API key (recommended: `openclaw configure --section web`, or set `BRAVE_API_KEY`). +- Requires provider setup (recommended: `openclaw configure --section web`). - Enable via `tools.web.search.enabled`. - Responses are cached (default 15 min). - See [Web tools](/tools/web) for setup. diff --git a/docs/tools/web.md b/docs/tools/web.md index 3e626400a..e5f811f10 100644 --- a/docs/tools/web.md +++ b/docs/tools/web.md @@ -1,16 +1,17 @@ --- -summary: "Web search + fetch tools (Brave Search API, Perplexity direct/OpenRouter)" +summary: "Web search + fetch tools (Brave Search API, Perplexity direct/OpenRouter, SearXNG)" read_when: - You want to enable web_search or web_fetch - You need Brave Search API key setup - You want to use Perplexity Sonar for web search + - You want to use SearXNG for web search --- # Web tools OpenClaw ships two lightweight web tools: -- `web_search` — Search the web via Brave Search API (default) or Perplexity Sonar (direct or via OpenRouter). +- `web_search` — Search the web via Brave Search API (default), Perplexity Sonar (direct or via OpenRouter), or a self-hosted SearXNG instance. - `web_fetch` — HTTP fetch + readable extraction (HTML → markdown/text). These are **not** browser automation. For JS-heavy sites or logins, use the @@ -21,6 +22,7 @@ These are **not** browser automation. For JS-heavy sites or logins, use the - `web_search` calls your configured provider and returns results. - **Brave** (default): returns structured results (title, URL, snippet). - **Perplexity**: returns AI-synthesized answers with citations from real-time web search. + - **SearXNG**: returns structured results (title, URL, snippet) from your own meta-search instance. - Results are cached by query for 15 minutes (configurable). - `web_fetch` does a plain HTTP GET and extracts readable content (HTML → markdown/text). It does **not** execute JavaScript. @@ -32,6 +34,7 @@ These are **not** browser automation. For JS-heavy sites or logins, use the |----------|------|------|---------| | **Brave** (default) | Fast, structured results, free tier | Traditional search results | `BRAVE_API_KEY` | | **Perplexity** | AI-synthesized answers, citations, real-time | Requires Perplexity or OpenRouter access | `OPENROUTER_API_KEY` or `PERPLEXITY_API_KEY` | +| **SearXNG** | Self-hosted, works in private environments, flexible | Requires running SearXNG and enabling JSON format | None (needs base URL) | See [Brave Search setup](/brave-search) and [Perplexity Sonar](/perplexity) for provider-specific details. @@ -42,7 +45,7 @@ Set the provider in config: tools: { web: { search: { - provider: "brave" // or "perplexity" + provider: "brave" // or "perplexity" or "searxng" } } } @@ -68,6 +71,34 @@ Example: switch to Perplexity Sonar (direct API): } ``` +## Using SearXNG + +SearXNG exposes a simple HTTP API at `/search`. To get JSON results you need +`format=json` enabled on your instance; some public instances disable it and will +return `403 Forbidden`. + +Example config: + +```json5 +{ + tools: { + web: { + search: { + enabled: true, + provider: "searxng", + searxng: { + baseUrl: "http://localhost:8080", + // Optional extra query params merged into each request: + // params: { categories: "general", safesearch: 1 } + // Optional auth via headers (for reverse-proxy auth): + // headers: { "X-Api-Key": "..." } + } + } + } + } +} +``` + ## Getting a Brave API key 1) Create a Brave Search API account at https://brave.com/search/api/ @@ -145,9 +176,10 @@ Search the web using your configured provider. ### Requirements - `tools.web.search.enabled` must not be `false` (default: enabled) -- API key for your chosen provider: +- Provider setup: - **Brave**: `BRAVE_API_KEY` or `tools.web.search.apiKey` - **Perplexity**: `OPENROUTER_API_KEY`, `PERPLEXITY_API_KEY`, or `tools.web.search.perplexity.apiKey` + - **SearXNG**: `tools.web.search.searxng.baseUrl` or `SEARXNG_BASE_URL` ### Config @@ -175,6 +207,12 @@ Search the web using your configured provider. - `search_lang` (optional): ISO language code for search results (e.g., "de", "en", "fr") - `ui_lang` (optional): ISO language code for UI elements - `freshness` (optional, Brave only): filter by discovery time (`pd`, `pw`, `pm`, `py`, or `YYYY-MM-DDtoYYYY-MM-DD`) +- `categories` (optional, SearXNG only): comma-separated categories +- `engines` (optional, SearXNG only): comma-separated engines +- `language` (optional, SearXNG only): language code (if omitted, falls back to `search_lang`) +- `time_range` (optional, SearXNG only): `day`, `month`, `year` +- `safesearch` (optional, SearXNG only): `0`, `1`, `2` +- `pageno` (optional, SearXNG only): page number (default `1`) **Examples:** diff --git a/src/agents/tools/web-search.test.ts b/src/agents/tools/web-search.test.ts index c6d2b6405..3087cb77e 100644 --- a/src/agents/tools/web-search.test.ts +++ b/src/agents/tools/web-search.test.ts @@ -2,8 +2,13 @@ import { describe, expect, it } from "vitest"; import { __testing } from "./web-search.js"; -const { inferPerplexityBaseUrlFromApiKey, resolvePerplexityBaseUrl, normalizeFreshness } = - __testing; +const { + inferPerplexityBaseUrlFromApiKey, + resolvePerplexityBaseUrl, + normalizeFreshness, + resolveSearxngBaseUrl, + resolveSearxngHeaders, +} = __testing; describe("web_search perplexity baseUrl defaults", () => { it("detects a Perplexity key prefix", () => { @@ -69,3 +74,36 @@ describe("web_search freshness normalization", () => { expect(normalizeFreshness("2024-03-10to2024-03-01")).toBeUndefined(); }); }); + +describe("web_search searxng config resolution", () => { + it("prefers config baseUrl and trims trailing slash", () => { + expect(resolveSearxngBaseUrl({ baseUrl: "http://localhost:8080/" })).toBe( + "http://localhost:8080", + ); + }); + + it("falls back to SEARXNG_BASE_URL env var", () => { + const prev = process.env.SEARXNG_BASE_URL; + process.env.SEARXNG_BASE_URL = "http://searxng.local:8888/"; + try { + expect(resolveSearxngBaseUrl(undefined)).toBe("http://searxng.local:8888"); + } finally { + if (prev === undefined) { + delete process.env.SEARXNG_BASE_URL; + } else { + process.env.SEARXNG_BASE_URL = prev; + } + } + }); + + it("builds headers with optional auth and extras", () => { + expect(resolveSearxngHeaders(undefined)).toEqual({ Accept: "application/json" }); + + expect(resolveSearxngHeaders({ apiKey: "token-123" }).Authorization).toBe("Bearer token-123"); + expect(resolveSearxngHeaders({ apiKey: "Basic abc" }).Authorization).toBe("Basic abc"); + + const headers = resolveSearxngHeaders({ headers: { "X-Test": "ok" }, apiKey: "token-123" }); + expect(headers["X-Test"]).toBe("ok"); + expect(headers.Accept).toBe("application/json"); + }); +}); diff --git a/src/agents/tools/web-search.ts b/src/agents/tools/web-search.ts index bf5741490..8239d9f48 100644 --- a/src/agents/tools/web-search.ts +++ b/src/agents/tools/web-search.ts @@ -17,7 +17,7 @@ import { writeCache, } from "./web-shared.js"; -const SEARCH_PROVIDERS = ["brave", "perplexity"] as const; +const SEARCH_PROVIDERS = ["brave", "perplexity", "searxng"] as const; const DEFAULT_SEARCH_COUNT = 5; const MAX_SEARCH_COUNT = 10; @@ -41,6 +41,39 @@ const WebSearchSchema = Type.Object({ maximum: MAX_SEARCH_COUNT, }), ), + categories: Type.Optional( + Type.String({ + description: "Comma-separated categories to search (SearXNG only), e.g. 'general,news'.", + }), + ), + engines: Type.Optional( + Type.String({ + description: "Comma-separated engines to search (SearXNG only), e.g. 'duckduckgo,bing'.", + }), + ), + language: Type.Optional( + Type.String({ + description: "Language code (SearXNG), e.g. 'en', 'de'.", + }), + ), + time_range: Type.Optional( + Type.String({ + description: "Time range (SearXNG only). Values: day, month, year.", + }), + ), + safesearch: Type.Optional( + Type.Number({ + description: "Safe search level (SearXNG only). Values: 0, 1, 2.", + minimum: 0, + maximum: 2, + }), + ), + pageno: Type.Optional( + Type.Number({ + description: "Search page number (SearXNG only). Default: 1.", + minimum: 1, + }), + ), country: Type.Optional( Type.String({ description: @@ -90,6 +123,13 @@ type PerplexityConfig = { model?: string; }; +type SearxngConfig = { + baseUrl?: string; + apiKey?: string; + headers?: Record; + params?: Record; +}; + type PerplexityApiKeySource = "config" | "perplexity_env" | "openrouter_env" | "none"; type PerplexitySearchResponse = { @@ -103,6 +143,17 @@ type PerplexitySearchResponse = { type PerplexityBaseUrlHint = "direct" | "openrouter"; +type SearxngSearchResult = { + title?: string; + url?: string; + content?: string; + publishedDate?: string; +}; + +type SearxngSearchResponse = { + results?: SearxngSearchResult[]; +}; + function resolveSearchConfig(cfg?: OpenClawConfig): WebSearchConfig { const search = cfg?.tools?.web?.search; if (!search || typeof search !== "object") return undefined; @@ -131,6 +182,14 @@ function missingSearchKeyPayload(provider: (typeof SEARCH_PROVIDERS)[number]) { docs: "https://docs.openclaw.ai/tools/web", }; } + if (provider === "searxng") { + return { + error: "missing_searxng_base_url", + message: + "web_search (searxng) needs a base URL. Configure tools.web.search.searxng.baseUrl or set SEARXNG_BASE_URL in the Gateway environment.", + docs: "https://docs.openclaw.ai/tools/web", + }; + } return { error: "missing_brave_api_key", message: `web_search needs a Brave Search API key. Run \`${formatCliCommand("openclaw configure --section web")}\` to store it, or set BRAVE_API_KEY in the Gateway environment.`, @@ -145,6 +204,7 @@ function resolveSearchProvider(search?: WebSearchConfig): (typeof SEARCH_PROVIDE : ""; if (raw === "perplexity") return "perplexity"; if (raw === "brave") return "brave"; + if (raw === "searxng") return "searxng"; return "brave"; } @@ -155,6 +215,13 @@ function resolvePerplexityConfig(search?: WebSearchConfig): PerplexityConfig { return perplexity as PerplexityConfig; } +function resolveSearxngConfig(search?: WebSearchConfig): SearxngConfig { + if (!search || typeof search !== "object") return {}; + const searxng = "searxng" in search ? (search as Record).searxng : undefined; + if (!searxng || typeof searxng !== "object") return {}; + return searxng as SearxngConfig; +} + function resolvePerplexityApiKey(perplexity?: PerplexityConfig): { apiKey?: string; source: PerplexityApiKeySource; @@ -221,6 +288,38 @@ function resolvePerplexityModel(perplexity?: PerplexityConfig): string { return fromConfig || DEFAULT_PERPLEXITY_MODEL; } +function resolveSearxngBaseUrl(searxng?: SearxngConfig): string | undefined { + const fromConfig = + searxng && "baseUrl" in searxng && typeof searxng.baseUrl === "string" ? searxng.baseUrl : ""; + const fromEnv = (process.env.SEARXNG_BASE_URL ?? "").trim(); + const baseUrl = (fromConfig ?? "").trim() || fromEnv; + if (!baseUrl) return undefined; + return baseUrl.replace(/\/$/, ""); +} + +function resolveSearxngHeaders(searxng?: SearxngConfig): Record { + const headers: Record = { + Accept: "application/json", + }; + + if (!searxng || typeof searxng !== "object") return headers; + + if (searxng.headers && typeof searxng.headers === "object") { + for (const [key, value] of Object.entries(searxng.headers)) { + if (typeof value === "string" && value.trim()) { + headers[key] = value; + } + } + } + + const apiKey = typeof searxng.apiKey === "string" ? searxng.apiKey.trim() : ""; + if (apiKey) { + headers.Authorization = apiKey.includes(" ") ? apiKey : `Bearer ${apiKey}`; + } + + return headers; +} + function resolveSearchCount(value: unknown, fallback: number): number { const parsed = typeof value === "number" && Number.isFinite(value) ? value : fallback; const clamped = Math.max(1, Math.min(MAX_SEARCH_COUNT, Math.floor(parsed))); @@ -306,6 +405,72 @@ async function runPerplexitySearch(params: { return { content, citations }; } +function addSearxngParams( + searchParams: URLSearchParams, + params?: Record, +): void { + if (!params) return; + for (const [key, value] of Object.entries(params)) { + if (!key.trim()) continue; + if (value === undefined) continue; + searchParams.set(key, String(value)); + } +} + +async function runSearxngSearch(params: { + query: string; + baseUrl: string; + count: number; + timeoutSeconds: number; + searchParams?: Record; + headers?: Record; + categories?: string; + engines?: string; + language?: string; + time_range?: string; + safesearch?: number; + pageno?: number; +}): Promise<{ results: Array> }> { + const url = new URL(`${params.baseUrl.replace(/\/$/, "")}/search`); + addSearxngParams(url.searchParams, params.searchParams); + url.searchParams.set("q", params.query); + url.searchParams.set("format", "json"); + + if (params.categories) url.searchParams.set("categories", params.categories); + if (params.engines) url.searchParams.set("engines", params.engines); + if (params.language) url.searchParams.set("language", params.language); + if (params.time_range) url.searchParams.set("time_range", params.time_range); + if (typeof params.safesearch === "number" && Number.isFinite(params.safesearch)) { + url.searchParams.set("safesearch", String(params.safesearch)); + } + if (typeof params.pageno === "number" && Number.isFinite(params.pageno) && params.pageno > 0) { + url.searchParams.set("pageno", String(Math.floor(params.pageno))); + } + + const res = await fetch(url.toString(), { + method: "GET", + headers: params.headers, + signal: withTimeout(undefined, params.timeoutSeconds * 1000), + }); + + if (!res.ok) { + const detail = await readResponseText(res); + throw new Error(`SearXNG API error (${res.status}): ${detail || res.statusText}`); + } + + const data = (await res.json()) as SearxngSearchResponse; + const results = Array.isArray(data.results) ? data.results : []; + const mapped = results.slice(0, params.count).map((entry) => ({ + title: entry.title ?? "", + url: entry.url ?? "", + description: entry.content ?? "", + published: entry.publishedDate ?? undefined, + siteName: resolveSiteName(entry.url ?? ""), + })); + + return { results: mapped }; +} + async function runWebSearch(params: { query: string; count: number; @@ -319,11 +484,22 @@ async function runWebSearch(params: { freshness?: string; perplexityBaseUrl?: string; perplexityModel?: string; + searxngBaseUrl?: string; + searxngHeaders?: Record; + searxngParams?: Record; + categories?: string; + engines?: string; + language?: string; + time_range?: string; + safesearch?: number; + pageno?: number; }): Promise> { const cacheKey = normalizeCacheKey( params.provider === "brave" ? `${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}:${params.freshness || "default"}` - : `${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`, + : params.provider === "searxng" + ? `${params.provider}:${params.query}:${params.count}:${params.categories || "default"}:${params.engines || "default"}:${params.language || "default"}:${params.time_range || "default"}:${String(params.safesearch ?? "default")}:${String(params.pageno ?? "default")}` + : `${params.provider}:${params.query}:${params.count}:${params.country || "default"}:${params.search_lang || "default"}:${params.ui_lang || "default"}`, ); const cached = readCache(SEARCH_CACHE, cacheKey); if (cached) return { ...cached.value, cached: true }; @@ -351,6 +527,33 @@ async function runWebSearch(params: { return payload; } + if (params.provider === "searxng") { + const { results } = await runSearxngSearch({ + query: params.query, + baseUrl: params.searxngBaseUrl ?? "", + count: params.count, + timeoutSeconds: params.timeoutSeconds, + searchParams: params.searxngParams, + headers: params.searxngHeaders, + categories: params.categories, + engines: params.engines, + language: params.language, + time_range: params.time_range, + safesearch: params.safesearch, + pageno: params.pageno, + }); + + const payload = { + query: params.query, + provider: params.provider, + count: results.length, + tookMs: Date.now() - start, + results, + }; + writeCache(SEARCH_CACHE, cacheKey, payload, params.cacheTtlMs); + return payload; + } + if (params.provider !== "brave") { throw new Error("Unsupported web search provider."); } @@ -415,11 +618,14 @@ export function createWebSearchTool(options?: { const provider = resolveSearchProvider(search); const perplexityConfig = resolvePerplexityConfig(search); + const searxngConfig = resolveSearxngConfig(search); const description = provider === "perplexity" ? "Search the web using Perplexity Sonar (direct or via OpenRouter). Returns AI-synthesized answers with citations from real-time web search." - : "Search the web using Brave Search API. Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research."; + : provider === "searxng" + ? "Search the web using a self-hosted SearXNG instance. Returns titles, URLs, and snippets for fast research." + : "Search the web using Brave Search API. Supports region-specific and localized search via country and language parameters. Returns titles, URLs, and snippets for fast research."; return { label: "Web Search", @@ -431,14 +637,26 @@ export function createWebSearchTool(options?: { provider === "perplexity" ? resolvePerplexityApiKey(perplexityConfig) : undefined; const apiKey = provider === "perplexity" ? perplexityAuth?.apiKey : resolveSearchApiKey(search); + const searxngBaseUrl = + provider === "searxng" ? resolveSearxngBaseUrl(searxngConfig) : undefined; - if (!apiKey) { + if (provider === "searxng") { + if (!searxngBaseUrl) return jsonResult(missingSearchKeyPayload(provider)); + } else if (!apiKey) { return jsonResult(missingSearchKeyPayload(provider)); } const params = args as Record; const query = readStringParam(params, "query", { required: true }); const count = readNumberParam(params, "count", { integer: true }) ?? search?.maxResults ?? undefined; + const categories = readStringParam(params, "categories"); + const engines = readStringParam(params, "engines"); + const language = + readStringParam(params, "language") ?? + (provider === "searxng" ? readStringParam(params, "search_lang") : undefined); + const time_range = readStringParam(params, "time_range"); + const safesearch = readNumberParam(params, "safesearch", { integer: true }); + const pageno = readNumberParam(params, "pageno", { integer: true }); const country = readStringParam(params, "country"); const search_lang = readStringParam(params, "search_lang"); const ui_lang = readStringParam(params, "ui_lang"); @@ -462,7 +680,7 @@ export function createWebSearchTool(options?: { const result = await runWebSearch({ query, count: resolveSearchCount(count, DEFAULT_SEARCH_COUNT), - apiKey, + apiKey: apiKey ?? "", timeoutSeconds: resolveTimeoutSeconds(search?.timeoutSeconds, DEFAULT_TIMEOUT_SECONDS), cacheTtlMs: resolveCacheTtlMs(search?.cacheTtlMinutes, DEFAULT_CACHE_TTL_MINUTES), provider, @@ -476,6 +694,15 @@ export function createWebSearchTool(options?: { perplexityAuth?.apiKey, ), perplexityModel: resolvePerplexityModel(perplexityConfig), + searxngBaseUrl, + searxngHeaders: provider === "searxng" ? resolveSearxngHeaders(searxngConfig) : undefined, + searxngParams: provider === "searxng" ? searxngConfig.params : undefined, + categories, + engines, + language, + time_range, + safesearch: typeof safesearch === "number" ? safesearch : undefined, + pageno: typeof pageno === "number" ? pageno : undefined, }); return jsonResult(result); }, @@ -486,4 +713,6 @@ export const __testing = { inferPerplexityBaseUrlFromApiKey, resolvePerplexityBaseUrl, normalizeFreshness, + resolveSearxngBaseUrl, + resolveSearxngHeaders, } as const; diff --git a/src/cli/program/register.subclis.test.ts b/src/cli/program/register.subclis.test.ts index c3c0c25df..0c2e8c5e9 100644 --- a/src/cli/program/register.subclis.test.ts +++ b/src/cli/program/register.subclis.test.ts @@ -18,7 +18,17 @@ const { nodesAction, registerNodesCli } = vi.hoisted(() => { return { nodesAction: action, registerNodesCli: register }; }); +const { gatewayRunAction, registerGatewayCli } = vi.hoisted(() => { + const action = vi.fn(); + const register = vi.fn((program: Command) => { + const gateway = program.command("gateway"); + gateway.command("run").option("--port ").action(action); + }); + return { gatewayRunAction: action, registerGatewayCli: register }; +}); + vi.mock("../acp-cli.js", () => ({ registerAcpCli })); +vi.mock("../gateway-cli.js", () => ({ registerGatewayCli })); vi.mock("../nodes-cli.js", () => ({ registerNodesCli })); const { registerSubCliByName, registerSubCliCommands } = await import("./register.subclis.js"); @@ -32,6 +42,8 @@ describe("registerSubCliCommands", () => { delete process.env.OPENCLAW_DISABLE_LAZY_SUBCOMMANDS; registerAcpCli.mockClear(); acpAction.mockClear(); + registerGatewayCli.mockClear(); + gatewayRunAction.mockClear(); registerNodesCli.mockClear(); nodesAction.mockClear(); }); @@ -79,6 +91,24 @@ describe("registerSubCliCommands", () => { expect(nodesAction).toHaveBeenCalledTimes(1); }); + it("preserves options for lazy subcommands", async () => { + process.argv = ["node", "openclaw", "gateway", "run", "--port", "18889"]; + const program = new Command(); + program.name("openclaw"); + registerSubCliCommands(program, process.argv); + + expect(program.commands.map((cmd) => cmd.name())).toEqual(["gateway"]); + + await program.parseAsync(process.argv); + + expect(registerGatewayCli).toHaveBeenCalledTimes(1); + expect(gatewayRunAction).toHaveBeenCalledTimes(1); + expect(gatewayRunAction).toHaveBeenCalledWith( + expect.objectContaining({ port: "18889" }), + expect.anything(), + ); + }); + it("replaces placeholder when registering a subcommand by name", async () => { process.argv = ["node", "openclaw", "acp", "--help"]; const program = new Command(); diff --git a/src/cli/program/register.subclis.ts b/src/cli/program/register.subclis.ts index 87eec67e1..4c488fd4a 100644 --- a/src/cli/program/register.subclis.ts +++ b/src/cli/program/register.subclis.ts @@ -246,7 +246,7 @@ export async function registerSubCliByName(program: Command, name: string): Prom return true; } -function registerLazyCommand(program: Command, entry: SubCliEntry) { +function registerLazyCommand(program: Command, entry: SubCliEntry, argvOverride?: string[]) { const placeholder = program.command(entry.name).description(entry.description); placeholder.allowUnknownOption(true); placeholder.allowExcessArguments(true); @@ -255,7 +255,7 @@ function registerLazyCommand(program: Command, entry: SubCliEntry) { await entry.register(program); const actionCommand = actionArgs.at(-1) as Command | undefined; const root = actionCommand?.parent ?? program; - const rawArgs = (root as Command & { rawArgs?: string[] }).rawArgs; + const rawArgs = (root as Command & { rawArgs?: string[] }).rawArgs ?? argvOverride; const actionArgsList = resolveActionArgs(actionCommand); const fallbackArgv = actionCommand?.name() ? [actionCommand.name(), ...actionArgsList] @@ -280,11 +280,11 @@ export function registerSubCliCommands(program: Command, argv: string[] = proces if (primary && shouldRegisterPrimaryOnly(argv)) { const entry = entries.find((candidate) => candidate.name === primary); if (entry) { - registerLazyCommand(program, entry); + registerLazyCommand(program, entry, argv); return; } } for (const candidate of entries) { - registerLazyCommand(program, candidate); + registerLazyCommand(program, candidate, argv); } } diff --git a/src/config/schema.ts b/src/config/schema.ts index 1401b0574..2e6bb7a9d 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -193,6 +193,10 @@ const FIELD_LABELS: Record = { "tools.web.search.maxResults": "Web Search Max Results", "tools.web.search.timeoutSeconds": "Web Search Timeout (sec)", "tools.web.search.cacheTtlMinutes": "Web Search Cache TTL (min)", + "tools.web.search.searxng.baseUrl": "SearXNG Base URL", + "tools.web.search.searxng.apiKey": "SearXNG API Key", + "tools.web.search.searxng.headers": "SearXNG Extra Headers", + "tools.web.search.searxng.params": "SearXNG Default Params", "tools.web.fetch.enabled": "Enable Web Fetch Tool", "tools.web.fetch.maxChars": "Web Fetch Max Chars", "tools.web.fetch.timeoutSeconds": "Web Fetch Timeout (sec)", @@ -434,8 +438,9 @@ const FIELD_HELP: Record = { "tools.message.crossContext.marker.suffix": 'Text suffix for cross-context markers (supports "{channel}").', "tools.message.broadcast.enabled": "Enable broadcast action (default: true).", - "tools.web.search.enabled": "Enable the web_search tool (requires a provider API key).", - "tools.web.search.provider": 'Search provider ("brave" or "perplexity").', + "tools.web.search.enabled": + "Enable the web_search tool (requires provider configuration: API key or base URL).", + "tools.web.search.provider": 'Search provider ("brave", "perplexity", or "searxng").', "tools.web.search.apiKey": "Brave Search API key (fallback: BRAVE_API_KEY env var).", "tools.web.search.maxResults": "Default number of results to return (1-10).", "tools.web.search.timeoutSeconds": "Timeout in seconds for web_search requests.", @@ -446,6 +451,14 @@ const FIELD_HELP: Record = { "Perplexity base URL override (default: https://openrouter.ai/api/v1 or https://api.perplexity.ai).", "tools.web.search.perplexity.model": 'Perplexity model override (default: "perplexity/sonar-pro").', + "tools.web.search.searxng.baseUrl": + "SearXNG instance base URL (e.g. http://localhost:8080). Fallback: SEARXNG_BASE_URL env var.", + "tools.web.search.searxng.apiKey": + "Optional SearXNG auth token; sent as Authorization header (Bearer by default).", + "tools.web.search.searxng.headers": + "Optional extra headers for SearXNG requests (e.g. for reverse-proxy auth).", + "tools.web.search.searxng.params": + "Optional default query params for SearXNG /search (merged into each request).", "tools.web.fetch.enabled": "Enable the web_fetch tool (lightweight HTTP fetch).", "tools.web.fetch.maxChars": "Max characters returned by web_fetch (truncated).", "tools.web.fetch.timeoutSeconds": "Timeout in seconds for web_fetch requests.", diff --git a/src/config/zod-schema.agent-runtime.ts b/src/config/zod-schema.agent-runtime.ts index 7e95c3538..5ff6644a8 100644 --- a/src/config/zod-schema.agent-runtime.ts +++ b/src/config/zod-schema.agent-runtime.ts @@ -165,7 +165,9 @@ export const ToolPolicySchema = ToolPolicyBaseSchema.superRefine((value, ctx) => export const ToolsWebSearchSchema = z .object({ enabled: z.boolean().optional(), - provider: z.union([z.literal("brave"), z.literal("perplexity")]).optional(), + provider: z + .union([z.literal("brave"), z.literal("perplexity"), z.literal("searxng")]) + .optional(), apiKey: z.string().optional(), maxResults: z.number().int().positive().optional(), timeoutSeconds: z.number().int().positive().optional(), @@ -178,6 +180,15 @@ export const ToolsWebSearchSchema = z }) .strict() .optional(), + searxng: z + .object({ + baseUrl: z.string().optional(), + apiKey: z.string().optional(), + headers: z.record(z.string(), z.string()).optional(), + params: z.record(z.string(), z.union([z.string(), z.number(), z.boolean()])).optional(), + }) + .strict() + .optional(), }) .strict() .optional(); diff --git a/src/wizard/onboarding.finalize.ts b/src/wizard/onboarding.finalize.ts index c5b01d6bf..4cf43c1cc 100644 --- a/src/wizard/onboarding.finalize.ts +++ b/src/wizard/onboarding.finalize.ts @@ -433,29 +433,101 @@ export async function finalizeOnboardingWizard(options: FinalizeOnboardingOption ); } - const webSearchKey = (nextConfig.tools?.web?.search?.apiKey ?? "").trim(); - const webSearchEnv = (process.env.BRAVE_API_KEY ?? "").trim(); - const hasWebSearchKey = Boolean(webSearchKey || webSearchEnv); + const webSearch = nextConfig.tools?.web?.search; + const webSearchProviderRaw = + webSearch && + typeof webSearch === "object" && + "provider" in webSearch && + typeof webSearch.provider === "string" + ? webSearch.provider.trim().toLowerCase() + : ""; + const webSearchProvider = + webSearchProviderRaw === "perplexity" + ? "perplexity" + : webSearchProviderRaw === "searxng" + ? "searxng" + : "brave"; + + const readTrimmedString = (value: unknown): string => + typeof value === "string" ? value.trim() : ""; + + const braveKey = + webSearch && typeof webSearch === "object" && "apiKey" in webSearch + ? readTrimmedString(webSearch.apiKey) + : ""; + const braveEnv = (process.env.BRAVE_API_KEY ?? "").trim(); + const hasBraveKey = Boolean(braveKey || braveEnv); + + const perplexityKey = + webSearch && + typeof webSearch === "object" && + "perplexity" in webSearch && + webSearch.perplexity && + typeof webSearch.perplexity === "object" && + "apiKey" in webSearch.perplexity + ? readTrimmedString((webSearch.perplexity as Record).apiKey) + : ""; + const perplexityEnv = (process.env.PERPLEXITY_API_KEY ?? "").trim(); + const openrouterEnv = (process.env.OPENROUTER_API_KEY ?? "").trim(); + const hasPerplexityKey = Boolean(perplexityKey || perplexityEnv || openrouterEnv); + + const searxngBaseUrl = + webSearch && + typeof webSearch === "object" && + "searxng" in webSearch && + webSearch.searxng && + typeof webSearch.searxng === "object" && + "baseUrl" in webSearch.searxng + ? readTrimmedString((webSearch.searxng as Record).baseUrl) + : ""; + const searxngEnv = (process.env.SEARXNG_BASE_URL ?? "").trim(); + const hasSearxngBaseUrl = Boolean(searxngBaseUrl || searxngEnv); + + const hasWebSearchSetup = + webSearchProvider === "perplexity" + ? hasPerplexityKey + : webSearchProvider === "searxng" + ? hasSearxngBaseUrl + : hasBraveKey; await prompter.note( - hasWebSearchKey + hasWebSearchSetup ? [ "Web search is enabled, so your agent can look things up online when needed.", "", - webSearchKey - ? "API key: stored in config (tools.web.search.apiKey)." - : "API key: provided via BRAVE_API_KEY env var (Gateway environment).", + `Provider: ${webSearchProvider}`, + webSearchProvider === "searxng" + ? searxngBaseUrl + ? "Base URL: stored in config (tools.web.search.searxng.baseUrl)." + : "Base URL: provided via SEARXNG_BASE_URL env var (Gateway environment)." + : webSearchProvider === "perplexity" + ? perplexityKey + ? "API key: stored in config (tools.web.search.perplexity.apiKey)." + : perplexityEnv + ? "API key: provided via PERPLEXITY_API_KEY env var (Gateway environment)." + : "API key: provided via OPENROUTER_API_KEY env var (Gateway environment)." + : braveKey + ? "API key: stored in config (tools.web.search.apiKey)." + : "API key: provided via BRAVE_API_KEY env var (Gateway environment).", "Docs: https://docs.openclaw.ai/tools/web", ].join("\n") : [ - "If you want your agent to be able to search the web, you’ll need an API key.", + "If you want your agent to be able to search the web, you’ll need to configure a provider.", "", - "OpenClaw uses Brave Search for the `web_search` tool. Without a Brave Search API key, web search won’t work.", + `OpenClaw uses ${webSearchProvider} for the \`web_search\` tool.`, "", "Set it up interactively:", `- Run: ${formatCliCommand("openclaw configure --section web")}`, - "- Enable web_search and paste your Brave Search API key", + webSearchProvider === "searxng" + ? "- Enable web_search and set your SearXNG base URL" + : webSearchProvider === "perplexity" + ? "- Enable web_search and set your Perplexity/OpenRouter API key" + : "- Enable web_search and paste your Brave Search API key", "", - "Alternative: set BRAVE_API_KEY in the Gateway environment (no config changes).", + webSearchProvider === "searxng" + ? "Alternative: set SEARXNG_BASE_URL in the Gateway environment (no config changes)." + : webSearchProvider === "perplexity" + ? "Alternative: set PERPLEXITY_API_KEY or OPENROUTER_API_KEY in the Gateway environment (no config changes)." + : "Alternative: set BRAVE_API_KEY in the Gateway environment (no config changes).", "Docs: https://docs.openclaw.ai/tools/web", ].join("\n"), "Web search (optional)",