Compare commits

...

9 Commits

Author SHA1 Message Date
vignesh07
d16d721898 fix(memory/qmd): throttle embed + citations auto + restore --force 2026-01-28 02:05:58 -08:00
Vignesh Natarajan
ebf6553c9e Add how to trigger model downloads for qmd in documentation 2026-01-28 01:25:10 -08:00
Vignesh Natarajan
3211b6cb10 fix(memory-qmd): create collections via qmd CLI (no YAML) 2026-01-28 01:25:10 -08:00
Vignesh Natarajan
06d675d4ae fix(memory-qmd): write XDG index.yml + legacy compat 2026-01-28 01:25:10 -08:00
Vignesh Natarajan
eff75c395b fix(qmd): use XDG dirs for qmd home; drop ollama docs 2026-01-28 01:25:10 -08:00
Vignesh Natarajan
62e80d030c Fix build errors 2026-01-28 01:25:07 -08:00
Vignesh Natarajan
f3f1640c7b Add more tests; make fall back more resilient and visible 2026-01-28 01:13:18 -08:00
Vignesh Natarajan
4e8000e745 Make memory more resilient to failure 2026-01-28 01:13:18 -08:00
Vignesh Natarajan
1cb4af074f feat (memory): Implement new (opt-in) QMD memory backend 2026-01-28 01:13:18 -08:00
29 changed files with 1946 additions and 122 deletions

View File

@ -96,6 +96,119 @@ embeddings for memory search. For Gemini, use `GEMINI_API_KEY` or
`models.providers.google.apiKey`. When using a custom OpenAI-compatible endpoint,
set `memorySearch.remote.apiKey` (and optional `memorySearch.remote.headers`).
### QMD backend (experimental)
Set `memory.backend = "qmd"` to swap the built-in SQLite indexer for
[QMD](https://github.com/tobi/qmd): a local-first search sidecar that combines
BM25 + vectors + reranking. Markdown stays the source of truth; Moltbot shells
out to QMD for retrieval. Key points:
**Prereqs**
- Disabled by default. Opt in per-config (`memory.backend = "qmd"`).
- Install the QMD CLI separately (`bun install -g github.com/tobi/qmd` or grab
a release) and make sure the `qmd` binary is on the gateways `PATH`.
- QMD needs an SQLite build that allows extensions (`brew install sqlite` on
macOS).
- QMD runs fully locally via Bun + `node-llama-cpp` and auto-downloads GGUF
models from HuggingFace on first use (no separate Ollama daemon required).
- The gateway runs QMD in a self-contained XDG home under
`~/.clawdbot/agents/<agentId>/qmd/` by setting `XDG_CONFIG_HOME` and
`XDG_CACHE_HOME`.
- OS support: macOS and Linux work out of the box once Bun + SQLite are
installed. Windows is best supported via WSL2.
**How the sidecar runs**
- The gateway writes a self-contained QMD home under
`~/.clawdbot/agents/<agentId>/qmd/` (config + cache + sqlite DB).
- Collections are rewritten from `memory.qmd.paths` (plus default workspace
memory files) into `index.yml`, then `qmd update` + `qmd embed` run on boot and
on a configurable interval (`memory.qmd.update.interval`, default 5m).
- Searches run via `qmd query --json`. If QMD fails or the binary is missing,
Moltbot automatically falls back to the builtin SQLite manager so memory tools
keep working.
- **First search may be slow**: QMD may download local GGUF models (reranker/query
expansion) on the first `qmd query` run.
- Moltbot sets `XDG_CONFIG_HOME`/`XDG_CACHE_HOME` automatically when it runs QMD.
- If you want to pre-download models manually (and warm the same index Moltbot
uses), run a one-off query with the agents XDG dirs.
Moltbots QMD state lives under your **state dir** (usually `~/.clawdbot`, or
`~/.moltbot` on newer installs). You can point `qmd` at the exact same index
by exporting the same XDG vars Moltbot uses:
```bash
# Pick the same state dir Moltbot uses
STATE_DIR="${MOLTBOT_STATE_DIR:-${CLAWDBOT_STATE_DIR:-$HOME/.clawdbot}}"
if [ -d "$HOME/.moltbot" ] && [ ! -d "$HOME/.clawdbot" ] \
&& [ -z "${MOLTBOT_STATE_DIR:-}" ] && [ -z "${CLAWDBOT_STATE_DIR:-}" ]; then
STATE_DIR="$HOME/.moltbot"
fi
export XDG_CONFIG_HOME="$STATE_DIR/agents/main/qmd/xdg-config"
export XDG_CACHE_HOME="$STATE_DIR/agents/main/qmd/xdg-cache"
# (Optional) force an index refresh + embeddings
qmd update
qmd embed
# Warm up / trigger first-time model downloads
qmd query "test" -c memory-root --json >/dev/null 2>&1
```
**Config surface (`memory.qmd.*`)**
- `command` (default `qmd`): override the executable path.
- `includeDefaultMemory` (default `true`): auto-index `MEMORY.md` + `memory/**/*.md`.
- `paths[]`: add extra directories/files (`path`, optional `pattern`, optional
stable `name`).
- `sessions`: opt into session JSONL indexing (`enabled`, `retentionDays`,
`exportDir`).
- `update`: controls refresh cadence (`interval`, `debounceMs`, `onBoot`, `embedInterval`).
- `limits`: clamp recall payload (`maxResults`, `maxSnippetChars`,
`maxInjectedChars`, `timeoutMs`).
- `scope`: same schema as [`session.sendPolicy`](/reference/configuration#session-sendpolicy).
Default is DM-only (`deny` all, `allow` direct chats); loosen it to surface QMD
hits in groups/channels.
- Snippets sourced outside the workspace show up as
`qmd/<collection>/<relative-path>` in `memory_search` results; `memory_get`
understands that prefix and reads from the configured QMD collection root.
- When `memory.qmd.sessions.enabled = true`, Moltbot exports sanitized session
transcripts (User/Assistant turns) into a dedicated QMD collection under
`~/.clawdbot/agents/<id>/qmd/sessions/`, so `memory_search` can recall recent
conversations without touching the builtin SQLite index.
- `memory_search` snippets now include a `Source: <path#line>` footer when
`memory.citations` is `auto`/`on`; set `memory.citations = "off"` to keep
the path metadata internal (the agent still receives the path for
`memory_get`, but the snippet text omits the footer and the system prompt
warns the agent not to cite it).
**Example**
```json5
memory: {
backend: "qmd",
citations: "auto",
qmd: {
includeDefaultMemory: true,
update: { interval: "5m", debounceMs: 15000 },
limits: { maxResults: 6, timeoutMs: 4000 },
scope: {
default: "deny",
rules: [{ action: "allow", match: { chatType: "direct" } }]
},
paths: [
{ name: "docs", path: "~/notes", pattern: "**/*.md" }
]
}
}
```
**Citations & fallback**
- `memory.citations` applies regardless of backend (`auto`/`on`/`off`).
- When `qmd` runs, we tag `status().backend = "qmd"` so diagnostics show which
engine served the results. If the QMD subprocess exits or JSON output cant be
parsed, the search manager logs a warning and returns the builtin provider
(existing Markdown embeddings) until QMD recovers.
### Gemini embeddings (native)
Set the provider to `gemini` to use the Gemini embeddings API directly:

View File

@ -212,6 +212,7 @@ export function buildSystemPrompt(params: {
userTimeFormat,
contextFiles: params.contextFiles,
ttsHint,
memoryCitationsMode: params.config?.memory?.citations,
});
}

View File

@ -346,6 +346,7 @@ export async function compactEmbeddedPiSessionDirect(
userTime,
userTimeFormat,
contextFiles,
memoryCitationsMode: params.config?.memory?.citations,
});
const systemPrompt = createSystemPromptOverride(appendPrompt);

View File

@ -359,6 +359,7 @@ export async function runEmbeddedAttempt(
userTime,
userTimeFormat,
contextFiles,
memoryCitationsMode: params.config?.memory?.citations,
});
const systemPromptReport = buildSystemPromptReport({
source: "run",

View File

@ -1,4 +1,5 @@
import type { AgentTool } from "@mariozechner/pi-agent-core";
import type { MemoryCitationsMode } from "../../config/types.memory.js";
import type { ResolvedTimeFormat } from "../date-time.js";
import type { EmbeddedContextFile } from "../pi-embedded-helpers.js";
import { buildAgentSystemPrompt, type PromptMode } from "../system-prompt.js";
@ -45,6 +46,7 @@ export function buildEmbeddedSystemPrompt(params: {
userTime?: string;
userTimeFormat?: ResolvedTimeFormat;
contextFiles?: EmbeddedContextFile[];
memoryCitationsMode?: MemoryCitationsMode;
}): string {
return buildAgentSystemPrompt({
workspaceDir: params.workspaceDir,
@ -70,6 +72,7 @@ export function buildEmbeddedSystemPrompt(params: {
userTime: params.userTime,
userTimeFormat: params.userTimeFormat,
contextFiles: params.contextFiles,
memoryCitationsMode: params.memoryCitationsMode,
});
}

View File

@ -1,6 +1,7 @@
import type { ReasoningLevel, ThinkLevel } from "../auto-reply/thinking.js";
import { SILENT_REPLY_TOKEN } from "../auto-reply/tokens.js";
import { listDeliverableMessageChannels } from "../utils/message-channel.js";
import type { MemoryCitationsMode } from "../config/types.memory.js";
import type { ResolvedTimeFormat } from "./date-time.js";
import type { EmbeddedContextFile } from "./pi-embedded-helpers.js";
@ -32,16 +33,30 @@ function buildSkillsSection(params: {
];
}
function buildMemorySection(params: { isMinimal: boolean; availableTools: Set<string> }) {
function buildMemorySection(params: {
isMinimal: boolean;
availableTools: Set<string>;
citationsMode?: MemoryCitationsMode;
}) {
if (params.isMinimal) return [];
if (!params.availableTools.has("memory_search") && !params.availableTools.has("memory_get")) {
return [];
}
return [
const lines = [
"## Memory Recall",
"Before answering anything about prior work, decisions, dates, people, preferences, or todos: run memory_search on MEMORY.md + memory/*.md; then use memory_get to pull only the needed lines. If low confidence after search, say you checked.",
"",
];
if (params.citationsMode === "off") {
lines.push(
"Citations are disabled: do not mention file paths or line numbers in replies unless the user explicitly asks.",
);
} else {
lines.push(
"Citations: include Source: <path#line> when it helps the user verify memory snippets.",
);
}
lines.push("");
return lines;
}
function buildUserIdentitySection(ownerLine: string | undefined, isMinimal: boolean) {
@ -178,6 +193,7 @@ export function buildAgentSystemPrompt(params: {
level: "minimal" | "extensive";
channel: string;
};
memoryCitationsMode?: MemoryCitationsMode;
}) {
const coreToolSummaries: Record<string, string> = {
read: "Read file contents",
@ -314,7 +330,11 @@ export function buildAgentSystemPrompt(params: {
isMinimal,
readToolName,
});
const memorySection = buildMemorySection({ isMinimal, availableTools });
const memorySection = buildMemorySection({
isMinimal,
availableTools,
citationsMode: params.memoryCitationsMode,
});
const docsSection = buildDocsSection({
docsPath: params.docsPath,
isMinimal,

View File

@ -0,0 +1,65 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
const stubManager = {
search: vi.fn(async () => [
{
path: "MEMORY.md",
startLine: 5,
endLine: 7,
score: 0.9,
snippet: "@@ -5,3 @@\nAssistant: noted",
source: "memory" as const,
},
]),
readFile: vi.fn(),
status: () => ({
backend: "builtin" as const,
files: 1,
chunks: 1,
dirty: false,
workspaceDir: "/workspace",
dbPath: "/workspace/.memory/index.sqlite",
provider: "builtin",
model: "builtin",
requestedProvider: "builtin",
sources: ["memory" as const],
sourceCounts: [{ source: "memory" as const, files: 1, chunks: 1 }],
}),
sync: vi.fn(),
probeVectorAvailability: vi.fn(async () => true),
close: vi.fn(),
};
vi.mock("../../memory/index.js", () => {
return {
getMemorySearchManager: async () => ({ manager: stubManager }),
};
});
import { createMemorySearchTool } from "./memory-tool.js";
beforeEach(() => {
vi.clearAllMocks();
});
describe("memory search citations", () => {
it("appends source information when citations are enabled", async () => {
const cfg = { memory: { citations: "on" }, agents: { list: [{ id: "main", default: true }] } };
const tool = createMemorySearchTool({ config: cfg });
if (!tool) throw new Error("tool missing");
const result = await tool.execute("call_citations_on", { query: "notes" });
const details = result.details as { results: Array<{ snippet: string; citation?: string }> };
expect(details.results[0]?.snippet).toMatch(/Source: MEMORY.md#L5-L7/);
expect(details.results[0]?.citation).toBe("MEMORY.md#L5-L7");
});
it("leaves snippet untouched when citations are off", async () => {
const cfg = { memory: { citations: "off" }, agents: { list: [{ id: "main", default: true }] } };
const tool = createMemorySearchTool({ config: cfg });
if (!tool) throw new Error("tool missing");
const result = await tool.execute("call_citations_off", { query: "notes" });
const details = result.details as { results: Array<{ snippet: string; citation?: string }> };
expect(details.results[0]?.snippet).not.toMatch(/Source:/);
expect(details.results[0]?.citation).toBeUndefined();
});
});

View File

@ -1,7 +1,9 @@
import { Type } from "@sinclair/typebox";
import type { MoltbotConfig } from "../../config/config.js";
import type { MemoryCitationsMode } from "../../config/types.memory.js";
import { getMemorySearchManager } from "../../memory/index.js";
import type { MemorySearchResult } from "../../memory/types.js";
import { resolveSessionAgentId } from "../agent-scope.js";
import { resolveMemorySearchConfig } from "../memory-search.js";
import type { AnyAgentTool } from "./common.js";
@ -48,17 +50,24 @@ export function createMemorySearchTool(options: {
return jsonResult({ results: [], disabled: true, error });
}
try {
const results = await manager.search(query, {
const citationsMode = resolveMemoryCitationsMode(cfg);
const includeCitations = shouldIncludeCitations({
mode: citationsMode,
sessionKey: options.agentSessionKey,
});
const rawResults = await manager.search(query, {
maxResults,
minScore,
sessionKey: options.agentSessionKey,
});
const status = manager.status();
const results = decorateCitations(rawResults, includeCitations);
return jsonResult({
results,
provider: status.provider,
model: status.model,
fallback: status.fallback,
citations: citationsMode,
});
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
@ -110,3 +119,46 @@ export function createMemoryGetTool(options: {
},
};
}
function resolveMemoryCitationsMode(cfg: MoltbotConfig): MemoryCitationsMode {
const mode = cfg.memory?.citations;
if (mode === "on" || mode === "off" || mode === "auto") return mode;
return "auto";
}
function decorateCitations(results: MemorySearchResult[], include: boolean): MemorySearchResult[] {
if (!include) {
return results.map((entry) => ({ ...entry, citation: undefined }));
}
return results.map((entry) => {
const citation = formatCitation(entry);
const snippet = `${entry.snippet.trim()}\n\nSource: ${citation}`;
return { ...entry, citation, snippet };
});
}
function formatCitation(entry: MemorySearchResult): string {
const lineRange =
entry.startLine === entry.endLine
? `#L${entry.startLine}`
: `#L${entry.startLine}-L${entry.endLine}`;
return `${entry.path}${lineRange}`;
}
function shouldIncludeCitations(params: {
mode: MemoryCitationsMode;
sessionKey?: string;
}): boolean {
if (params.mode === "on") return true;
if (params.mode === "off") return false;
// auto: show citations in direct chats; suppress in groups/channels by default.
const chatType = deriveChatTypeFromSessionKey(params.sessionKey);
return chatType === "direct";
}
function deriveChatTypeFromSessionKey(sessionKey?: string): "direct" | "group" | "channel" {
if (!sessionKey) return "direct";
if (sessionKey.includes(":group:")) return "group";
if (sessionKey.includes(":channel:")) return "channel";
return "direct";
}

View File

@ -150,6 +150,7 @@ async function resolveContextReport(
ttsHint,
runtimeInfo,
sandboxInfo,
memoryCitationsMode: params.cfg?.memory?.citations,
});
return buildSystemPromptReport({

View File

@ -242,7 +242,7 @@ describe("memory cli", () => {
await program.parseAsync(["memory", "status", "--index"], { from: "user" });
expect(sync).toHaveBeenCalledWith(
expect.objectContaining({ reason: "cli", progress: expect.any(Function) }),
expect.objectContaining({ reason: "cli", force: false, progress: expect.any(Function) }),
);
expect(probeEmbeddingAvailability).toHaveBeenCalled();
expect(close).toHaveBeenCalled();

View File

@ -24,6 +24,7 @@ type MemoryCommandOptions = {
json?: boolean;
deep?: boolean;
index?: boolean;
force?: boolean;
verbose?: boolean;
};
@ -213,13 +214,16 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
onMissing: (error) => defaultRuntime.log(error ?? "Memory search disabled."),
onCloseError: (err) =>
defaultRuntime.error(`Memory manager close failed: ${formatErrorMessage(err)}`),
close: (manager) => manager.close(),
close: async (manager) => {
await manager.close?.();
},
run: async (manager) => {
const deep = Boolean(opts.deep || opts.index);
let embeddingProbe:
| Awaited<ReturnType<typeof manager.probeEmbeddingAvailability>>
| undefined;
let indexError: string | undefined;
const syncFn = manager.sync ? manager.sync.bind(manager) : undefined;
if (deep) {
await withProgress({ label: "Checking memory…", total: 2 }, async (progress) => {
progress.setLabel("Probing vector…");
@ -229,7 +233,7 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
embeddingProbe = await manager.probeEmbeddingAvailability();
progress.tick();
});
if (opts.index) {
if (opts.index && syncFn) {
await withProgressTotals(
{
label: "Indexing memory…",
@ -238,8 +242,9 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
},
async (update, progress) => {
try {
await manager.sync({
await syncFn({
reason: "cli",
force: Boolean(opts.force),
progress: (syncUpdate) => {
update({
completed: syncUpdate.completed,
@ -256,6 +261,8 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
}
},
);
} else if (opts.index && !syncFn) {
defaultRuntime.log("Memory backend does not support manual reindex.");
}
} else {
await manager.probeVectorAvailability();
@ -264,11 +271,10 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
const sources = (
status.sources?.length ? status.sources : ["memory"]
) as MemorySourceName[];
const scan = await scanMemorySources({
workspaceDir: status.workspaceDir,
agentId,
sources,
});
const workspaceDir = status.workspaceDir;
const scan = workspaceDir
? await scanMemorySources({ workspaceDir, agentId, sources })
: undefined;
allResults.push({ agentId, status, embeddingProbe, indexError, scan });
},
});
@ -290,26 +296,31 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
for (const result of allResults) {
const { agentId, status, embeddingProbe, indexError, scan } = result;
const filesIndexed = status.files ?? 0;
const chunksIndexed = status.chunks ?? 0;
const totalFiles = scan?.totalFiles ?? null;
const indexedLabel =
totalFiles === null
? `${status.files}/? files · ${status.chunks} chunks`
: `${status.files}/${totalFiles} files · ${status.chunks} chunks`;
? `${filesIndexed}/? files · ${chunksIndexed} chunks`
: `${filesIndexed}/${totalFiles} files · ${chunksIndexed} chunks`;
if (opts.index) {
const line = indexError ? `Memory index failed: ${indexError}` : "Memory index complete.";
defaultRuntime.log(line);
}
const requestedProvider = status.requestedProvider ?? status.provider;
const modelLabel = status.model ?? status.provider;
const storePath = status.dbPath ? shortenHomePath(status.dbPath) : "<unknown>";
const workspacePath = status.workspaceDir ? shortenHomePath(status.workspaceDir) : "<unknown>";
const sourceList = status.sources?.length ? status.sources.join(", ") : null;
const lines = [
`${heading("Memory Search")} ${muted(`(${agentId})`)}`,
`${label("Provider")} ${info(status.provider)} ${muted(
`(requested: ${status.requestedProvider})`,
)}`,
`${label("Model")} ${info(status.model)}`,
status.sources?.length ? `${label("Sources")} ${info(status.sources.join(", "))}` : null,
`${label("Provider")} ${info(status.provider)} ${muted(`(requested: ${requestedProvider})`)}`,
`${label("Model")} ${info(modelLabel)}`,
sourceList ? `${label("Sources")} ${info(sourceList)}` : null,
`${label("Indexed")} ${success(indexedLabel)}`,
`${label("Dirty")} ${status.dirty ? warn("yes") : muted("no")}`,
`${label("Store")} ${info(shortenHomePath(status.dbPath))}`,
`${label("Workspace")} ${info(shortenHomePath(status.workspaceDir))}`,
`${label("Store")} ${info(storePath)}`,
`${label("Workspace")} ${info(workspacePath)}`,
].filter(Boolean) as string[];
if (embeddingProbe) {
const state = embeddingProbe.ok ? "ready" : "unavailable";
@ -322,7 +333,7 @@ export async function runMemoryStatus(opts: MemoryCommandOptions) {
if (status.sourceCounts?.length) {
lines.push(label("By source"));
for (const entry of status.sourceCounts) {
const total = scan?.sources.find(
const total = scan?.sources?.find(
(scanEntry) => scanEntry.source === entry.source,
)?.totalFiles;
const counts =
@ -435,7 +446,7 @@ export function registerMemoryCli(program: Command) {
.option("--deep", "Probe embedding provider availability")
.option("--index", "Reindex if dirty (implies --deep)")
.option("--verbose", "Verbose logging", false)
.action(async (opts: MemoryCommandOptions) => {
.action(async (opts: MemoryCommandOptions & { force?: boolean }) => {
await runMemoryStatus(opts);
});
@ -445,7 +456,7 @@ export function registerMemoryCli(program: Command) {
.option("--agent <id>", "Agent id (default: default agent)")
.option("--force", "Force full reindex", false)
.option("--verbose", "Verbose logging", false)
.action(async (opts: MemoryCommandOptions & { force?: boolean }) => {
.action(async (opts: MemoryCommandOptions) => {
setVerbose(Boolean(opts.verbose));
const cfg = loadConfig();
const agentIds = resolveAgentIds(cfg, opts.agent);
@ -455,9 +466,12 @@ export function registerMemoryCli(program: Command) {
onMissing: (error) => defaultRuntime.log(error ?? "Memory search disabled."),
onCloseError: (err) =>
defaultRuntime.error(`Memory manager close failed: ${formatErrorMessage(err)}`),
close: (manager) => manager.close(),
close: async (manager) => {
await manager.close?.();
},
run: async (manager) => {
try {
const syncFn = manager.sync ? manager.sync.bind(manager) : undefined;
if (opts.verbose) {
const status = manager.status();
const rich = isRich();
@ -466,15 +480,17 @@ export function registerMemoryCli(program: Command) {
const info = (text: string) => colorize(rich, theme.info, text);
const warn = (text: string) => colorize(rich, theme.warn, text);
const label = (text: string) => muted(`${text}:`);
const sourceLabels = status.sources.map((source) =>
formatSourceLabel(source, status.workspaceDir, agentId),
const sourceLabels = (status.sources ?? []).map((source) =>
formatSourceLabel(source, status.workspaceDir ?? "", agentId),
);
const requestedProvider = status.requestedProvider ?? status.provider;
const modelLabel = status.model ?? status.provider;
const lines = [
`${heading("Memory Index")} ${muted(`(${agentId})`)}`,
`${label("Provider")} ${info(status.provider)} ${muted(
`(requested: ${status.requestedProvider})`,
`(requested: ${requestedProvider})`,
)}`,
`${label("Model")} ${info(status.model)}`,
`${label("Model")} ${info(modelLabel)}`,
sourceLabels.length
? `${label("Sources")} ${info(sourceLabels.join(", "))}`
: null,
@ -514,6 +530,10 @@ export function registerMemoryCli(program: Command) {
? `${lastLabel} · elapsed ${elapsed} · eta ${eta}`
: `${lastLabel} · elapsed ${elapsed}`;
};
if (!syncFn) {
defaultRuntime.log("Memory backend does not support manual reindex.");
return;
}
await withProgressTotals(
{
label: "Indexing memory…",
@ -525,9 +545,9 @@ export function registerMemoryCli(program: Command) {
progress.setLabel(buildLabel());
}, 1000);
try {
await manager.sync({
await syncFn({
reason: "cli",
force: opts.force,
force: Boolean(opts.force),
progress: (syncUpdate) => {
if (syncUpdate.label) lastLabel = syncUpdate.label;
lastCompleted = syncUpdate.completed;
@ -579,7 +599,9 @@ export function registerMemoryCli(program: Command) {
onMissing: (error) => defaultRuntime.log(error ?? "Memory search disabled."),
onCloseError: (err) =>
defaultRuntime.error(`Memory manager close failed: ${formatErrorMessage(err)}`),
close: (manager) => manager.close(),
close: async (manager) => {
await manager.close?.();
},
run: async (manager) => {
let results: Awaited<ReturnType<typeof manager.search>>;
try {

View File

@ -6,7 +6,8 @@ import { probeGateway } from "../gateway/probe.js";
import { collectChannelStatusIssues } from "../infra/channels-status-issues.js";
import { resolveOsSummary } from "../infra/os-summary.js";
import { getTailnetHostname } from "../infra/tailscale.js";
import type { MemoryIndexManager } from "../memory/manager.js";
import { getMemorySearchManager } from "../memory/index.js";
import type { MemoryProviderStatus } from "../memory/types.js";
import { runExec } from "../process/exec.js";
import type { RuntimeEnv } from "../runtime.js";
import { getAgentLocalStatuses } from "./status.agent-local.js";
@ -15,7 +16,7 @@ import { getStatusSummary } from "./status.summary.js";
import { getUpdateCheckResult } from "./status.update.js";
import { buildChannelsTable } from "./status-all/channels.js";
type MemoryStatusSnapshot = ReturnType<MemoryIndexManager["status"]> & {
type MemoryStatusSnapshot = MemoryProviderStatus & {
agentId: string;
};
@ -151,14 +152,13 @@ export async function scanStatus(
if (!memoryPlugin.enabled) return null;
if (memoryPlugin.slot !== "memory-core") return null;
const agentId = agentStatus.defaultId ?? "main";
const { MemoryIndexManager } = await import("../memory/manager.js");
const manager = await MemoryIndexManager.get({ cfg, agentId }).catch(() => null);
const { manager } = await getMemorySearchManager({ cfg, agentId });
if (!manager) return null;
try {
await manager.probeVectorAvailability();
} catch {}
const status = manager.status();
await manager.close().catch(() => {});
await manager.close?.().catch(() => {});
return { agentId, ...status };
})();
progress.tick();

View File

@ -252,6 +252,27 @@ const FIELD_LABELS: Record<string, string> = {
"Memory Search Hybrid Candidate Multiplier",
"agents.defaults.memorySearch.cache.enabled": "Memory Search Embedding Cache",
"agents.defaults.memorySearch.cache.maxEntries": "Memory Search Embedding Cache Max Entries",
memory: "Memory",
"memory.backend": "Memory Backend",
"memory.citations": "Memory Citations Mode",
"memory.qmd.command": "QMD Binary",
"memory.qmd.includeDefaultMemory": "QMD Include Default Memory",
"memory.qmd.paths": "QMD Extra Paths",
"memory.qmd.paths.path": "QMD Path",
"memory.qmd.paths.pattern": "QMD Path Pattern",
"memory.qmd.paths.name": "QMD Path Name",
"memory.qmd.sessions.enabled": "QMD Session Indexing",
"memory.qmd.sessions.exportDir": "QMD Session Export Directory",
"memory.qmd.sessions.retentionDays": "QMD Session Retention (days)",
"memory.qmd.update.interval": "QMD Update Interval",
"memory.qmd.update.debounceMs": "QMD Update Debounce (ms)",
"memory.qmd.update.onBoot": "QMD Update on Startup",
"memory.qmd.update.embedInterval": "QMD Embed Interval",
"memory.qmd.limits.maxResults": "QMD Max Results",
"memory.qmd.limits.maxSnippetChars": "QMD Max Snippet Chars",
"memory.qmd.limits.maxInjectedChars": "QMD Max Injected Chars",
"memory.qmd.limits.timeoutMs": "QMD Search Timeout (ms)",
"memory.qmd.scope": "QMD Surface Scope",
"auth.profiles": "Auth Profiles",
"auth.order": "Auth Profile Order",
"auth.cooldowns.billingBackoffHours": "Billing Backoff (hours)",
@ -537,6 +558,37 @@ const FIELD_HELP: Record<string, string> = {
"Multiplier for candidate pool size (default: 4).",
"agents.defaults.memorySearch.cache.enabled":
"Cache chunk embeddings in SQLite to speed up reindexing and frequent updates (default: true).",
memory: "Memory backend configuration (global).",
"memory.backend": 'Memory backend ("builtin" for Moltbot embeddings, "qmd" for QMD sidecar).',
"memory.citations": 'Default citation behavior ("auto", "on", or "off").',
"memory.qmd.command": "Path to the qmd binary (default: resolves from PATH).",
"memory.qmd.includeDefaultMemory":
"Whether to automatically index MEMORY.md + memory/**/*.md (default: true).",
"memory.qmd.paths":
"Additional directories/files to index with QMD (path + optional glob pattern).",
"memory.qmd.paths.path": "Absolute or ~-relative path to index via QMD.",
"memory.qmd.paths.pattern": "Glob pattern relative to the path root (default: **/*.md).",
"memory.qmd.paths.name":
"Optional stable name for the QMD collection (default derived from path).",
"memory.qmd.sessions.enabled":
"Enable QMD session transcript indexing (experimental, default: false).",
"memory.qmd.sessions.exportDir":
"Override directory for sanitized session exports before indexing.",
"memory.qmd.sessions.retentionDays":
"Retention window for exported sessions before pruning (default: unlimited).",
"memory.qmd.update.interval":
"How often the QMD sidecar refreshes indexes (duration string, default: 5m).",
"memory.qmd.update.debounceMs":
"Minimum delay between successive QMD refresh runs (default: 15000).",
"memory.qmd.update.onBoot": "Run QMD update once on gateway startup (default: true).",
"memory.qmd.update.embedInterval":
"How often QMD embeddings are refreshed (duration string, default: 60m). Set to 0 to disable periodic embed.",
"memory.qmd.limits.maxResults": "Max QMD results returned to the agent loop (default: 6).",
"memory.qmd.limits.maxSnippetChars": "Max characters per snippet pulled from QMD (default: 700).",
"memory.qmd.limits.maxInjectedChars": "Max total characters injected from QMD hits per turn.",
"memory.qmd.limits.timeoutMs": "Per-query timeout for QMD searches (default: 4000).",
"memory.qmd.scope":
"Session/channel scope for QMD recall (same syntax as session.sendPolicy; default: direct-only).",
"agents.defaults.memorySearch.cache.maxEntries":
"Optional cap on cached embeddings (best-effort).",
"agents.defaults.memorySearch.sync.onSearch":

View File

@ -23,6 +23,7 @@ import type { NodeHostConfig } from "./types.node-host.js";
import type { PluginsConfig } from "./types.plugins.js";
import type { SkillsConfig } from "./types.skills.js";
import type { ToolsConfig } from "./types.tools.js";
import type { MemoryConfig } from "./types.memory.js";
export type MoltbotConfig = {
meta?: {
@ -95,6 +96,7 @@ export type MoltbotConfig = {
canvasHost?: CanvasHostConfig;
talk?: TalkConfig;
gateway?: GatewayConfig;
memory?: MemoryConfig;
};
export type ConfigValidationIssue = {

View File

@ -0,0 +1,46 @@
import type { SessionSendPolicyConfig } from "./types.base.js";
export type MemoryBackend = "builtin" | "qmd";
export type MemoryCitationsMode = "auto" | "on" | "off";
export type MemoryConfig = {
backend?: MemoryBackend;
citations?: MemoryCitationsMode;
qmd?: MemoryQmdConfig;
};
export type MemoryQmdConfig = {
command?: string;
includeDefaultMemory?: boolean;
paths?: MemoryQmdIndexPath[];
sessions?: MemoryQmdSessionConfig;
update?: MemoryQmdUpdateConfig;
limits?: MemoryQmdLimitsConfig;
scope?: SessionSendPolicyConfig;
};
export type MemoryQmdIndexPath = {
path: string;
name?: string;
pattern?: string;
};
export type MemoryQmdSessionConfig = {
enabled?: boolean;
exportDir?: string;
retentionDays?: number;
};
export type MemoryQmdUpdateConfig = {
interval?: string;
debounceMs?: number;
onBoot?: boolean;
embedInterval?: string;
};
export type MemoryQmdLimitsConfig = {
maxResults?: number;
maxSnippetChars?: number;
maxInjectedChars?: number;
timeoutMs?: number;
};

View File

@ -28,3 +28,4 @@ export * from "./types.telegram.js";
export * from "./types.tts.js";
export * from "./types.tools.js";
export * from "./types.whatsapp.js";
export * from "./types.memory.js";

View File

@ -16,6 +16,31 @@ const SessionResetConfigSchema = z
})
.strict();
export const SessionSendPolicySchema = z
.object({
default: z.union([z.literal("allow"), z.literal("deny")]).optional(),
rules: z
.array(
z
.object({
action: z.union([z.literal("allow"), z.literal("deny")]),
match: z
.object({
channel: z.string().optional(),
chatType: z
.union([z.literal("direct"), z.literal("group"), z.literal("channel")])
.optional(),
keyPrefix: z.string().optional(),
})
.strict()
.optional(),
})
.strict(),
)
.optional(),
})
.strict();
export const SessionSchema = z
.object({
scope: z.union([z.literal("per-sender"), z.literal("global")]).optional(),
@ -51,31 +76,7 @@ export const SessionSchema = z
])
.optional(),
mainKey: z.string().optional(),
sendPolicy: z
.object({
default: z.union([z.literal("allow"), z.literal("deny")]).optional(),
rules: z
.array(
z
.object({
action: z.union([z.literal("allow"), z.literal("deny")]),
match: z
.object({
channel: z.string().optional(),
chatType: z
.union([z.literal("direct"), z.literal("group"), z.literal("channel")])
.optional(),
keyPrefix: z.string().optional(),
})
.strict()
.optional(),
})
.strict(),
)
.optional(),
})
.strict()
.optional(),
sendPolicy: SessionSendPolicySchema.optional(),
agentToAgent: z
.object({
maxPingPongTurns: z.number().int().min(0).max(5).optional(),

View File

@ -5,7 +5,12 @@ import { AgentsSchema, AudioSchema, BindingsSchema, BroadcastSchema } from "./zo
import { HexColorSchema, ModelsConfigSchema } from "./zod-schema.core.js";
import { HookMappingSchema, HooksGmailSchema, InternalHooksSchema } from "./zod-schema.hooks.js";
import { ChannelsSchema } from "./zod-schema.providers.js";
import { CommandsSchema, MessagesSchema, SessionSchema } from "./zod-schema.session.js";
import {
CommandsSchema,
MessagesSchema,
SessionSchema,
SessionSendPolicySchema,
} from "./zod-schema.session.js";
const BrowserSnapshotDefaultsSchema = z
.object({
@ -27,6 +32,61 @@ const NodeHostSchema = z
.strict()
.optional();
const MemoryQmdPathSchema = z
.object({
path: z.string(),
name: z.string().optional(),
pattern: z.string().optional(),
})
.strict();
const MemoryQmdSessionSchema = z
.object({
enabled: z.boolean().optional(),
exportDir: z.string().optional(),
retentionDays: z.number().int().nonnegative().optional(),
})
.strict();
const MemoryQmdUpdateSchema = z
.object({
interval: z.string().optional(),
debounceMs: z.number().int().nonnegative().optional(),
onBoot: z.boolean().optional(),
embedInterval: z.string().optional(),
})
.strict();
const MemoryQmdLimitsSchema = z
.object({
maxResults: z.number().int().positive().optional(),
maxSnippetChars: z.number().int().positive().optional(),
maxInjectedChars: z.number().int().positive().optional(),
timeoutMs: z.number().int().nonnegative().optional(),
})
.strict();
const MemoryQmdSchema = z
.object({
command: z.string().optional(),
includeDefaultMemory: z.boolean().optional(),
paths: z.array(MemoryQmdPathSchema).optional(),
sessions: MemoryQmdSessionSchema.optional(),
update: MemoryQmdUpdateSchema.optional(),
limits: MemoryQmdLimitsSchema.optional(),
scope: SessionSendPolicySchema.optional(),
})
.strict();
const MemorySchema = z
.object({
backend: z.union([z.literal("builtin"), z.literal("qmd")]).optional(),
citations: z.union([z.literal("auto"), z.literal("on"), z.literal("off")]).optional(),
qmd: MemoryQmdSchema.optional(),
})
.strict()
.optional();
export const MoltbotSchema = z
.object({
meta: z
@ -445,6 +505,7 @@ export const MoltbotSchema = z
})
.strict()
.optional(),
memory: MemorySchema,
skills: z
.object({
allowBundled: z.array(z.string()).optional(),

View File

@ -84,10 +84,14 @@ export async function resolveDiscordTarget(
const likelyUsername = isLikelyUsername(trimmed);
const shouldLookup = isExplicitUserLookup(trimmed, parseOptions) || likelyUsername;
// Parse directly if it's already a known format. Use a safe parse so ambiguous
// numeric targets don't throw when we still want to attempt username lookup.
const directParse = safeParseDiscordTarget(trimmed, parseOptions);
if (directParse && directParse.kind !== "channel" && !likelyUsername) {
return directParse;
}
if (!shouldLookup) {
return directParse ?? parseDiscordTarget(trimmed, parseOptions);
}

View File

@ -0,0 +1,58 @@
import path from "node:path";
import { describe, expect, it } from "vitest";
import type { MoltbotConfig } from "../config/config.js";
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
import { resolveMemoryBackendConfig } from "./backend-config.js";
describe("resolveMemoryBackendConfig", () => {
it("defaults to builtin backend when config missing", () => {
const cfg = { agents: { defaults: { workspace: "/tmp/memory-test" } } } as MoltbotConfig;
const resolved = resolveMemoryBackendConfig({ cfg, agentId: "main" });
expect(resolved.backend).toBe("builtin");
expect(resolved.citations).toBe("auto");
expect(resolved.qmd).toBeUndefined();
});
it("resolves qmd backend with default collections", () => {
const cfg = {
agents: { defaults: { workspace: "/tmp/memory-test" } },
memory: {
backend: "qmd",
qmd: {},
},
} as MoltbotConfig;
const resolved = resolveMemoryBackendConfig({ cfg, agentId: "main" });
expect(resolved.backend).toBe("qmd");
expect(resolved.qmd?.collections.length).toBeGreaterThanOrEqual(3);
expect(resolved.qmd?.command).toBe("qmd");
expect(resolved.qmd?.update.intervalMs).toBeGreaterThan(0);
});
it("resolves custom paths relative to workspace", () => {
const cfg = {
agents: {
defaults: { workspace: "/workspace/root" },
list: [{ id: "main", workspace: "/workspace/root" }],
},
memory: {
backend: "qmd",
qmd: {
paths: [
{
path: "notes",
name: "custom-notes",
pattern: "**/*.md",
},
],
},
},
} as MoltbotConfig;
const resolved = resolveMemoryBackendConfig({ cfg, agentId: "main" });
const custom = resolved.qmd?.collections.find((c) => c.name.startsWith("custom-notes"));
expect(custom).toBeDefined();
const workspaceRoot = resolveAgentWorkspaceDir(cfg, "main");
expect(custom?.path).toBe(path.resolve(workspaceRoot, "notes"));
});
});

View File

@ -0,0 +1,255 @@
import path from "node:path";
import { parseDurationMs } from "../cli/parse-duration.js";
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
import type { MoltbotConfig } from "../config/config.js";
import type {
MemoryBackend,
MemoryCitationsMode,
MemoryQmdConfig,
MemoryQmdIndexPath,
} from "../config/types.memory.js";
import type { SessionSendPolicyConfig } from "../config/types.base.js";
import { resolveUserPath } from "../utils.js";
export type ResolvedMemoryBackendConfig = {
backend: MemoryBackend;
citations: MemoryCitationsMode;
qmd?: ResolvedQmdConfig;
};
export type ResolvedQmdCollection = {
name: string;
path: string;
pattern: string;
kind: "memory" | "custom" | "sessions";
};
export type ResolvedQmdUpdateConfig = {
intervalMs: number;
debounceMs: number;
onBoot: boolean;
embedIntervalMs: number;
};
export type ResolvedQmdLimitsConfig = {
maxResults: number;
maxSnippetChars: number;
maxInjectedChars: number;
timeoutMs: number;
};
export type ResolvedQmdSessionConfig = {
enabled: boolean;
exportDir?: string;
retentionDays?: number;
};
export type ResolvedQmdConfig = {
command: string;
collections: ResolvedQmdCollection[];
sessions: ResolvedQmdSessionConfig;
update: ResolvedQmdUpdateConfig;
limits: ResolvedQmdLimitsConfig;
includeDefaultMemory: boolean;
scope?: SessionSendPolicyConfig;
};
const DEFAULT_BACKEND: MemoryBackend = "builtin";
const DEFAULT_CITATIONS: MemoryCitationsMode = "auto";
const DEFAULT_QMD_INTERVAL = "5m";
const DEFAULT_QMD_DEBOUNCE_MS = 15_000;
const DEFAULT_QMD_TIMEOUT_MS = 4_000;
const DEFAULT_QMD_EMBED_INTERVAL = "60m";
const DEFAULT_QMD_LIMITS: ResolvedQmdLimitsConfig = {
maxResults: 6,
maxSnippetChars: 700,
maxInjectedChars: 4_000,
timeoutMs: DEFAULT_QMD_TIMEOUT_MS,
};
const DEFAULT_QMD_SCOPE: SessionSendPolicyConfig = {
default: "deny",
rules: [
{
action: "allow",
match: { chatType: "direct" },
},
],
};
function sanitizeName(input: string): string {
const lower = input.toLowerCase().replace(/[^a-z0-9-]+/g, "-");
const trimmed = lower.replace(/^-+|-+$/g, "");
return trimmed || "collection";
}
function ensureUniqueName(base: string, existing: Set<string>): string {
let name = sanitizeName(base);
if (!existing.has(name)) {
existing.add(name);
return name;
}
let suffix = 2;
while (existing.has(`${name}-${suffix}`)) {
suffix += 1;
}
const unique = `${name}-${suffix}`;
existing.add(unique);
return unique;
}
function resolvePath(raw: string, workspaceDir: string): string {
const trimmed = raw.trim();
if (!trimmed) throw new Error("path required");
if (trimmed.startsWith("~") || path.isAbsolute(trimmed)) {
return path.normalize(resolveUserPath(trimmed));
}
return path.normalize(path.resolve(workspaceDir, trimmed));
}
function resolveIntervalMs(raw: string | undefined): number {
const value = raw?.trim();
if (!value) return parseDurationMs(DEFAULT_QMD_INTERVAL, { defaultUnit: "m" });
try {
return parseDurationMs(value, { defaultUnit: "m" });
} catch {
return parseDurationMs(DEFAULT_QMD_INTERVAL, { defaultUnit: "m" });
}
}
function resolveEmbedIntervalMs(raw: string | undefined): number {
const value = raw?.trim();
if (!value) return parseDurationMs(DEFAULT_QMD_EMBED_INTERVAL, { defaultUnit: "m" });
try {
return parseDurationMs(value, { defaultUnit: "m" });
} catch {
return parseDurationMs(DEFAULT_QMD_EMBED_INTERVAL, { defaultUnit: "m" });
}
}
function resolveDebounceMs(raw: number | undefined): number {
if (typeof raw === "number" && Number.isFinite(raw) && raw >= 0) {
return Math.floor(raw);
}
return DEFAULT_QMD_DEBOUNCE_MS;
}
function resolveLimits(raw?: MemoryQmdConfig["limits"]): ResolvedQmdLimitsConfig {
const parsed: ResolvedQmdLimitsConfig = { ...DEFAULT_QMD_LIMITS };
if (raw?.maxResults && raw.maxResults > 0) parsed.maxResults = Math.floor(raw.maxResults);
if (raw?.maxSnippetChars && raw.maxSnippetChars > 0) {
parsed.maxSnippetChars = Math.floor(raw.maxSnippetChars);
}
if (raw?.maxInjectedChars && raw.maxInjectedChars > 0) {
parsed.maxInjectedChars = Math.floor(raw.maxInjectedChars);
}
if (raw?.timeoutMs && raw.timeoutMs > 0) {
parsed.timeoutMs = Math.floor(raw.timeoutMs);
}
return parsed;
}
function resolveSessionConfig(
cfg: MemoryQmdConfig["sessions"],
workspaceDir: string,
): ResolvedQmdSessionConfig {
const enabled = Boolean(cfg?.enabled);
const exportDirRaw = cfg?.exportDir?.trim();
const exportDir = exportDirRaw ? resolvePath(exportDirRaw, workspaceDir) : undefined;
const retentionDays =
cfg?.retentionDays && cfg.retentionDays > 0 ? Math.floor(cfg.retentionDays) : undefined;
return {
enabled,
exportDir,
retentionDays,
};
}
function resolveCustomPaths(
rawPaths: MemoryQmdIndexPath[] | undefined,
workspaceDir: string,
existing: Set<string>,
): ResolvedQmdCollection[] {
if (!rawPaths?.length) return [];
const collections: ResolvedQmdCollection[] = [];
rawPaths.forEach((entry, index) => {
const trimmedPath = entry?.path?.trim();
if (!trimmedPath) return;
let resolved: string;
try {
resolved = resolvePath(trimmedPath, workspaceDir);
} catch {
return;
}
const pattern = entry.pattern?.trim() || "**/*.md";
const baseName = entry.name?.trim() || `custom-${index + 1}`;
const name = ensureUniqueName(baseName, existing);
collections.push({
name,
path: resolved,
pattern,
kind: "custom",
});
});
return collections;
}
function resolveDefaultCollections(
include: boolean,
workspaceDir: string,
existing: Set<string>,
): ResolvedQmdCollection[] {
if (!include) return [];
const entries: Array<{ path: string; pattern: string; base: string }> = [
{ path: workspaceDir, pattern: "MEMORY.md", base: "memory-root" },
{ path: workspaceDir, pattern: "memory.md", base: "memory-alt" },
{ path: path.join(workspaceDir, "memory"), pattern: "**/*.md", base: "memory-dir" },
];
return entries.map((entry) => ({
name: ensureUniqueName(entry.base, existing),
path: entry.path,
pattern: entry.pattern,
kind: "memory",
}));
}
export function resolveMemoryBackendConfig(params: {
cfg: MoltbotConfig;
agentId: string;
}): ResolvedMemoryBackendConfig {
const backend = params.cfg.memory?.backend ?? DEFAULT_BACKEND;
const citations = params.cfg.memory?.citations ?? DEFAULT_CITATIONS;
if (backend !== "qmd") {
return { backend: "builtin", citations };
}
const workspaceDir = resolveAgentWorkspaceDir(params.cfg, params.agentId);
const qmdCfg = params.cfg.memory?.qmd;
const includeDefaultMemory = qmdCfg?.includeDefaultMemory !== false;
const nameSet = new Set<string>();
const collections = [
...resolveDefaultCollections(includeDefaultMemory, workspaceDir, nameSet),
...resolveCustomPaths(qmdCfg?.paths, workspaceDir, nameSet),
];
const resolved: ResolvedQmdConfig = {
command: (qmdCfg?.command?.trim() || "qmd").split(/\s+/)[0] || "qmd",
collections,
includeDefaultMemory,
sessions: resolveSessionConfig(qmdCfg?.sessions, workspaceDir),
update: {
intervalMs: resolveIntervalMs(qmdCfg?.update?.interval),
debounceMs: resolveDebounceMs(qmdCfg?.update?.debounceMs),
onBoot: qmdCfg?.update?.onBoot !== false,
embedIntervalMs: resolveEmbedIntervalMs(qmdCfg?.update?.embedInterval),
},
limits: resolveLimits(qmdCfg?.limits),
scope: qmdCfg?.scope ?? DEFAULT_QMD_SCOPE,
};
return {
backend: "qmd",
citations,
qmd: resolved,
};
}

View File

@ -1,2 +1,7 @@
export type { MemoryIndexManager, MemorySearchResult } from "./manager.js";
export { MemoryIndexManager } from "./manager.js";
export type {
MemoryEmbeddingProbeResult,
MemorySearchManager,
MemorySearchResult,
} from "./types.js";
export { getMemorySearchManager, type MemorySearchManagerResult } from "./search-manager.js";

View File

@ -45,17 +45,14 @@ import { searchKeyword, searchVector } from "./manager-search.js";
import { ensureMemoryIndexSchema } from "./memory-schema.js";
import { requireNodeSqlite } from "./sqlite.js";
import { loadSqliteVecExtension } from "./sqlite-vec.js";
type MemorySource = "memory" | "sessions";
export type MemorySearchResult = {
path: string;
startLine: number;
endLine: number;
score: number;
snippet: string;
source: MemorySource;
};
import type {
MemoryEmbeddingProbeResult,
MemoryProviderStatus,
MemorySearchManager,
MemorySearchResult,
MemorySource,
MemorySyncProgressUpdate,
} from "./types.js";
type MemoryIndexMeta = {
model: string;
@ -75,12 +72,6 @@ type SessionFileEntry = {
content: string;
};
type MemorySyncProgressUpdate = {
completed: number;
total: number;
label?: string;
};
type MemorySyncProgressState = {
completed: number;
total: number;
@ -115,7 +106,7 @@ const INDEX_CACHE = new Map<string, MemoryIndexManager>();
const vectorToBlob = (embedding: number[]): Buffer =>
Buffer.from(new Float32Array(embedding).buffer);
export class MemoryIndexManager {
export class MemoryIndexManager implements MemorySearchManager {
private readonly cacheKey: string;
private readonly cfg: MoltbotConfig;
private readonly agentId: string;
@ -415,39 +406,7 @@ export class MemoryIndexManager {
return { text: slice.join("\n"), path: relPath };
}
status(): {
files: number;
chunks: number;
dirty: boolean;
workspaceDir: string;
dbPath: string;
provider: string;
model: string;
requestedProvider: string;
sources: MemorySource[];
sourceCounts: Array<{ source: MemorySource; files: number; chunks: number }>;
cache?: { enabled: boolean; entries?: number; maxEntries?: number };
fts?: { enabled: boolean; available: boolean; error?: string };
fallback?: { from: string; reason?: string };
vector?: {
enabled: boolean;
available?: boolean;
extensionPath?: string;
loadError?: string;
dims?: number;
};
batch?: {
enabled: boolean;
failures: number;
limit: number;
wait: boolean;
concurrency: number;
pollIntervalMs: number;
timeoutMs: number;
lastError?: string;
lastProvider?: string;
};
} {
status(): MemoryProviderStatus {
const sourceFilter = this.buildSourceFilter();
const files = this.db
.prepare(`SELECT COUNT(*) as c FROM files WHERE 1=1${sourceFilter.sql}`)
@ -489,9 +448,10 @@ export class MemoryIndexManager {
return sources.map((source) => ({ source, ...bySource.get(source)! }));
})();
return {
backend: "builtin",
files: files?.c ?? 0,
chunks: chunks?.c ?? 0,
dirty: this.dirty,
dirty: this.dirty || this.sessionsDirty,
workspaceDir: this.workspaceDir,
dbPath: this.settings.store.path,
provider: this.provider.id,
@ -545,7 +505,7 @@ export class MemoryIndexManager {
return this.ensureVectorReady();
}
async probeEmbeddingAvailability(): Promise<{ ok: boolean; error?: string }> {
async probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
try {
await this.embedBatchWithRetry(["ping"]);
return { ok: true };

View File

@ -0,0 +1,98 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { EventEmitter } from "node:events";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("node:child_process", () => {
const spawn = vi.fn((_cmd: string, _args: string[]) => {
const stdout = new EventEmitter();
const stderr = new EventEmitter();
const child = new EventEmitter() as {
stdout: EventEmitter;
stderr: EventEmitter;
kill: () => void;
emit: (event: string, code: number) => boolean;
};
child.stdout = stdout;
child.stderr = stderr;
child.kill = () => {
child.emit("close", 0);
};
setImmediate(() => {
stdout.emit("data", "");
stderr.emit("data", "");
child.emit("close", 0);
});
return child;
});
return { spawn };
});
import { spawn as mockedSpawn } from "node:child_process";
import type { MoltbotConfig } from "../config/config.js";
import { resolveMemoryBackendConfig } from "./backend-config.js";
import { QmdMemoryManager } from "./qmd-manager.js";
const spawnMock = mockedSpawn as unknown as vi.Mock;
describe("QmdMemoryManager", () => {
let tmpRoot: string;
let workspaceDir: string;
let stateDir: string;
let cfg: MoltbotConfig;
const agentId = "main";
beforeEach(async () => {
spawnMock.mockClear();
tmpRoot = await fs.mkdtemp(path.join(os.tmpdir(), "qmd-manager-test-"));
workspaceDir = path.join(tmpRoot, "workspace");
await fs.mkdir(workspaceDir, { recursive: true });
stateDir = path.join(tmpRoot, "state");
await fs.mkdir(stateDir, { recursive: true });
process.env.MOLTBOT_STATE_DIR = stateDir;
cfg = {
agents: {
list: [{ id: agentId, default: true, workspace: workspaceDir }],
},
memory: {
backend: "qmd",
qmd: {
includeDefaultMemory: false,
update: { interval: "0s", debounceMs: 60_000, onBoot: false },
paths: [{ path: workspaceDir, pattern: "**/*.md", name: "workspace" }],
},
},
} as MoltbotConfig;
});
afterEach(async () => {
delete process.env.MOLTBOT_STATE_DIR;
await fs.rm(tmpRoot, { recursive: true, force: true });
});
it("debounces back-to-back sync calls", async () => {
const resolved = resolveMemoryBackendConfig({ cfg, agentId });
const manager = await QmdMemoryManager.create({ cfg, agentId, resolved });
expect(manager).toBeTruthy();
if (!manager) throw new Error("manager missing");
const baselineCalls = spawnMock.mock.calls.length;
await manager.sync({ reason: "manual" });
expect(spawnMock.mock.calls.length).toBe(baselineCalls + 2);
await manager.sync({ reason: "manual-again" });
expect(spawnMock.mock.calls.length).toBe(baselineCalls + 2);
(manager as unknown as { lastUpdateAt: number | null }).lastUpdateAt =
Date.now() - (resolved.qmd?.update.debounceMs ?? 0) - 10;
await manager.sync({ reason: "after-wait" });
// By default we refresh embeddings less frequently than index updates.
expect(spawnMock.mock.calls.length).toBe(baselineCalls + 3);
await manager.close();
});
});

684
src/memory/qmd-manager.ts Normal file
View File

@ -0,0 +1,684 @@
import { spawn } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { MoltbotConfig } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
import {
listSessionFilesForAgent,
buildSessionEntry,
type SessionFileEntry,
} from "./session-files.js";
import { requireNodeSqlite } from "./sqlite.js";
import type {
MemoryEmbeddingProbeResult,
MemoryProviderStatus,
MemorySearchManager,
MemorySearchResult,
MemorySource,
MemorySyncProgressUpdate,
} from "./types.js";
type SqliteDatabase = import("node:sqlite").DatabaseSync;
import type { ResolvedMemoryBackendConfig, ResolvedQmdConfig } from "./backend-config.js";
const log = createSubsystemLogger("memory");
const SNIPPET_HEADER_RE = /@@\s*-([0-9]+),([0-9]+)/;
type QmdQueryResult = {
docid?: string;
score?: number;
file?: string;
snippet?: string;
body?: string;
};
type CollectionRoot = {
path: string;
kind: MemorySource;
};
type SessionExporterConfig = {
dir: string;
retentionMs?: number;
collectionName: string;
};
export class QmdMemoryManager implements MemorySearchManager {
static async create(params: {
cfg: MoltbotConfig;
agentId: string;
resolved: ResolvedMemoryBackendConfig;
}): Promise<QmdMemoryManager | null> {
const resolved = params.resolved.qmd;
if (!resolved) return null;
const manager = new QmdMemoryManager({ cfg: params.cfg, agentId: params.agentId, resolved });
await manager.initialize();
return manager;
}
private readonly cfg: MoltbotConfig;
private readonly agentId: string;
private readonly qmd: ResolvedQmdConfig;
private readonly workspaceDir: string;
private readonly stateDir: string;
private readonly agentStateDir: string;
private readonly qmdDir: string;
private readonly xdgConfigHome: string;
private readonly xdgCacheHome: string;
private readonly indexPath: string;
private readonly env: NodeJS.ProcessEnv;
private readonly collectionRoots = new Map<string, CollectionRoot>();
private readonly sources = new Set<MemorySource>();
private readonly docPathCache = new Map<
string,
{ rel: string; abs: string; source: MemorySource }
>();
private readonly sessionExporter: SessionExporterConfig | null;
private updateTimer: NodeJS.Timeout | null = null;
private pendingUpdate: Promise<void> | null = null;
private closed = false;
private db: SqliteDatabase | null = null;
private lastUpdateAt: number | null = null;
private lastEmbedAt: number | null = null;
private constructor(params: {
cfg: MoltbotConfig;
agentId: string;
resolved: ResolvedQmdConfig;
}) {
this.cfg = params.cfg;
this.agentId = params.agentId;
this.qmd = params.resolved;
this.workspaceDir = resolveAgentWorkspaceDir(params.cfg, params.agentId);
this.stateDir = resolveStateDir(process.env, os.homedir);
this.agentStateDir = path.join(this.stateDir, "agents", this.agentId);
this.qmdDir = path.join(this.agentStateDir, "qmd");
// QMD uses XDG base dirs for its internal state.
// Collections are managed via `qmd collection add` and stored inside the index DB.
// - config: $XDG_CONFIG_HOME (contexts, etc.)
// - cache: $XDG_CACHE_HOME/qmd/index.sqlite
this.xdgConfigHome = path.join(this.qmdDir, "xdg-config");
this.xdgCacheHome = path.join(this.qmdDir, "xdg-cache");
this.indexPath = path.join(this.xdgCacheHome, "qmd", "index.sqlite");
this.env = {
...process.env,
XDG_CONFIG_HOME: this.xdgConfigHome,
XDG_CACHE_HOME: this.xdgCacheHome,
NO_COLOR: "1",
};
this.sessionExporter = this.qmd.sessions.enabled
? {
dir: this.qmd.sessions.exportDir ?? path.join(this.qmdDir, "sessions"),
retentionMs: this.qmd.sessions.retentionDays
? this.qmd.sessions.retentionDays * 24 * 60 * 60 * 1000
: undefined,
collectionName: this.pickSessionCollectionName(),
}
: null;
if (this.sessionExporter) {
this.qmd.collections = [
...this.qmd.collections,
{
name: this.sessionExporter.collectionName,
path: this.sessionExporter.dir,
pattern: "**/*.md",
kind: "sessions",
},
];
}
}
private async initialize(): Promise<void> {
await fs.mkdir(this.xdgConfigHome, { recursive: true });
await fs.mkdir(this.xdgCacheHome, { recursive: true });
await fs.mkdir(path.dirname(this.indexPath), { recursive: true });
this.bootstrapCollections();
await this.ensureCollections();
if (this.qmd.update.onBoot) {
await this.runUpdate("boot", true);
}
if (this.qmd.update.intervalMs > 0) {
this.updateTimer = setInterval(() => {
void this.runUpdate("interval").catch((err) => {
log.warn(`qmd update failed (${String(err)})`);
});
}, this.qmd.update.intervalMs);
}
}
private bootstrapCollections(): void {
this.collectionRoots.clear();
this.sources.clear();
for (const collection of this.qmd.collections) {
const kind: MemorySource = collection.kind === "sessions" ? "sessions" : "memory";
this.collectionRoots.set(collection.name, { path: collection.path, kind });
this.sources.add(kind);
}
}
private async ensureCollections(): Promise<void> {
// QMD collections are persisted inside the index database and must be created
// via the CLI. Prefer listing existing collections when supported, otherwise
// fall back to best-effort idempotent `qmd collection add`.
const existing = new Set<string>();
try {
const result = await this.runQmd(["collection", "list", "--json"]);
const parsed = JSON.parse(result.stdout) as unknown;
if (Array.isArray(parsed)) {
for (const entry of parsed) {
if (typeof entry === "string") existing.add(entry);
else if (entry && typeof entry === "object") {
const name = (entry as { name?: unknown }).name;
if (typeof name === "string") existing.add(name);
}
}
}
} catch {
// ignore; older qmd versions might not support list --json.
}
for (const collection of this.qmd.collections) {
if (existing.has(collection.name)) continue;
try {
await this.runQmd([
"collection",
"add",
collection.path,
"--name",
collection.name,
"--mask",
collection.pattern,
]);
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
// Idempotency: qmd exits non-zero if the collection name already exists.
if (message.toLowerCase().includes("already exists")) continue;
if (message.toLowerCase().includes("exists")) continue;
log.warn(`qmd collection add failed for ${collection.name}: ${message}`);
}
}
}
async search(
query: string,
opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
): Promise<MemorySearchResult[]> {
if (!this.isScopeAllowed(opts?.sessionKey)) return [];
const trimmed = query.trim();
if (!trimmed) return [];
await this.pendingUpdate?.catch(() => undefined);
const limit = Math.min(
this.qmd.limits.maxResults,
opts?.maxResults ?? this.qmd.limits.maxResults,
);
const args = ["query", trimmed, "--json", "-n", String(limit)];
let stdout: string;
try {
const result = await this.runQmd(args, { timeoutMs: this.qmd.limits.timeoutMs });
stdout = result.stdout;
} catch (err) {
log.warn(`qmd query failed: ${String(err)}`);
throw err instanceof Error ? err : new Error(String(err));
}
let parsed: QmdQueryResult[] = [];
try {
parsed = JSON.parse(stdout);
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
log.warn(`qmd query returned invalid JSON: ${message}`);
throw new Error(`qmd query returned invalid JSON: ${message}`);
}
const results: MemorySearchResult[] = [];
for (const entry of parsed) {
const doc = await this.resolveDocLocation(entry.docid);
if (!doc) continue;
const snippet = entry.snippet?.slice(0, this.qmd.limits.maxSnippetChars) ?? "";
const lines = this.extractSnippetLines(snippet);
const score = typeof entry.score === "number" ? entry.score : 0;
const minScore = opts?.minScore ?? 0;
if (score < minScore) continue;
results.push({
path: doc.rel,
startLine: lines.startLine,
endLine: lines.endLine,
score,
snippet,
source: doc.source,
});
}
return this.clampResultsByInjectedChars(results.slice(0, limit));
}
async sync(params?: {
reason?: string;
force?: boolean;
progress?: (update: MemorySyncProgressUpdate) => void;
}): Promise<void> {
if (params?.progress) {
params.progress({ completed: 0, total: 1, label: "Updating QMD index…" });
}
await this.runUpdate(params?.reason ?? "manual", params?.force);
if (params?.progress) {
params.progress({ completed: 1, total: 1, label: "QMD index updated" });
}
}
async readFile(params: {
relPath: string;
from?: number;
lines?: number;
}): Promise<{ text: string; path: string }> {
const relPath = params.relPath?.trim();
if (!relPath) throw new Error("path required");
const absPath = this.resolveReadPath(relPath);
const content = await fs.readFile(absPath, "utf-8");
if (!params.from && !params.lines) {
return { text: content, path: relPath };
}
const lines = content.split("\n");
const start = Math.max(1, params.from ?? 1);
const count = Math.max(1, params.lines ?? lines.length);
const slice = lines.slice(start - 1, start - 1 + count);
return { text: slice.join("\n"), path: relPath };
}
status(): MemoryProviderStatus {
const counts = this.readCounts();
return {
backend: "qmd",
provider: "qmd",
model: "qmd",
requestedProvider: "qmd",
files: counts.totalDocuments,
chunks: counts.totalDocuments,
dirty: false,
workspaceDir: this.workspaceDir,
dbPath: this.indexPath,
sources: Array.from(this.sources),
sourceCounts: counts.sourceCounts,
vector: { enabled: true, available: true },
batch: {
enabled: false,
failures: 0,
limit: 0,
wait: false,
concurrency: 0,
pollIntervalMs: 0,
timeoutMs: 0,
},
custom: {
qmd: {
collections: this.qmd.collections.length,
lastUpdateAt: this.lastUpdateAt,
},
},
};
}
async probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
return { ok: true };
}
async probeVectorAvailability(): Promise<boolean> {
return true;
}
async close(): Promise<void> {
if (this.closed) return;
this.closed = true;
if (this.updateTimer) {
clearInterval(this.updateTimer);
this.updateTimer = null;
}
await this.pendingUpdate?.catch(() => undefined);
if (this.db) {
this.db.close();
this.db = null;
}
}
private async runUpdate(reason: string, force?: boolean): Promise<void> {
if (this.pendingUpdate && !force) return this.pendingUpdate;
if (this.shouldSkipUpdate(force)) {
return;
}
const run = async () => {
if (this.sessionExporter) {
await this.exportSessions();
}
await this.runQmd(["update"], { timeoutMs: 120_000 });
const embedIntervalMs = this.qmd.update.embedIntervalMs;
const shouldEmbed =
Boolean(force) ||
this.lastEmbedAt === null ||
(embedIntervalMs > 0 && Date.now() - this.lastEmbedAt > embedIntervalMs);
if (shouldEmbed) {
try {
await this.runQmd(["embed"], { timeoutMs: 120_000 });
this.lastEmbedAt = Date.now();
} catch (err) {
log.warn(`qmd embed failed (${reason}): ${String(err)}`);
}
}
this.lastUpdateAt = Date.now();
this.docPathCache.clear();
};
this.pendingUpdate = run().finally(() => {
this.pendingUpdate = null;
});
await this.pendingUpdate;
}
private async runQmd(
args: string[],
opts?: { timeoutMs?: number },
): Promise<{ stdout: string; stderr: string }> {
return await new Promise((resolve, reject) => {
const child = spawn(this.qmd.command, args, {
env: this.env,
cwd: this.workspaceDir,
});
let stdout = "";
let stderr = "";
const timer = opts?.timeoutMs
? setTimeout(() => {
child.kill("SIGKILL");
reject(new Error(`qmd ${args.join(" ")} timed out after ${opts.timeoutMs}ms`));
}, opts.timeoutMs)
: null;
child.stdout.on("data", (data) => {
stdout += data.toString();
});
child.stderr.on("data", (data) => {
stderr += data.toString();
});
child.on("error", (err) => {
if (timer) clearTimeout(timer);
reject(err);
});
child.on("close", (code) => {
if (timer) clearTimeout(timer);
if (code === 0) {
resolve({ stdout, stderr });
} else {
reject(new Error(`qmd ${args.join(" ")} failed (code ${code}): ${stderr || stdout}`));
}
});
});
}
private ensureDb(): SqliteDatabase {
if (this.db) return this.db;
const { DatabaseSync } = requireNodeSqlite();
this.db = new DatabaseSync(this.indexPath, { readOnly: true });
return this.db;
}
private async exportSessions(): Promise<void> {
if (!this.sessionExporter) return;
const exportDir = this.sessionExporter.dir;
await fs.mkdir(exportDir, { recursive: true });
const files = await listSessionFilesForAgent(this.agentId);
const keep = new Set<string>();
const cutoff = this.sessionExporter.retentionMs
? Date.now() - this.sessionExporter.retentionMs
: null;
for (const sessionFile of files) {
const entry = await buildSessionEntry(sessionFile);
if (!entry) continue;
if (cutoff && entry.mtimeMs < cutoff) continue;
const target = path.join(exportDir, `${path.basename(sessionFile, ".jsonl")}.md`);
await fs.writeFile(target, this.renderSessionMarkdown(entry), "utf-8");
keep.add(target);
}
const exported = await fs.readdir(exportDir).catch(() => []);
for (const name of exported) {
if (!name.endsWith(".md")) continue;
const full = path.join(exportDir, name);
if (!keep.has(full)) {
await fs.rm(full, { force: true });
}
}
}
private renderSessionMarkdown(entry: SessionFileEntry): string {
const header = `# Session ${path.basename(entry.absPath, path.extname(entry.absPath))}`;
const body = entry.content?.trim().length ? entry.content.trim() : "(empty)";
return `${header}\n\n${body}\n`;
}
private pickSessionCollectionName(): string {
const existing = new Set(this.qmd.collections.map((collection) => collection.name));
if (!existing.has("sessions")) return "sessions";
let counter = 2;
let candidate = `sessions-${counter}`;
while (existing.has(candidate)) {
counter += 1;
candidate = `sessions-${counter}`;
}
return candidate;
}
private async resolveDocLocation(
docid?: string,
): Promise<{ rel: string; abs: string; source: MemorySource } | null> {
if (!docid) return null;
const normalized = docid.startsWith("#") ? docid.slice(1) : docid;
if (!normalized) return null;
const cached = this.docPathCache.get(normalized);
if (cached) return cached;
const db = this.ensureDb();
const row = db
.prepare("SELECT collection, path FROM documents WHERE hash LIKE ? AND active = 1 LIMIT 1")
.get(`${normalized}%`) as { collection: string; path: string } | undefined;
if (!row) return null;
const location = this.toDocLocation(row.collection, row.path);
if (!location) return null;
this.docPathCache.set(normalized, location);
return location;
}
private extractSnippetLines(snippet: string): { startLine: number; endLine: number } {
const match = SNIPPET_HEADER_RE.exec(snippet);
if (match) {
const start = Number(match[1]);
const count = Number(match[2]);
if (Number.isFinite(start) && Number.isFinite(count)) {
return { startLine: start, endLine: start + count - 1 };
}
}
const lines = snippet.split("\n").length;
return { startLine: 1, endLine: lines };
}
private readCounts(): {
totalDocuments: number;
sourceCounts: Array<{ source: MemorySource; files: number; chunks: number }>;
} {
try {
const db = this.ensureDb();
const rows = db
.prepare(
"SELECT collection, COUNT(*) as c FROM documents WHERE active = 1 GROUP BY collection",
)
.all() as Array<{ collection: string; c: number }>;
const bySource = new Map<MemorySource, { files: number; chunks: number }>();
for (const source of this.sources) {
bySource.set(source, { files: 0, chunks: 0 });
}
let total = 0;
for (const row of rows) {
const root = this.collectionRoots.get(row.collection);
const source = root?.kind ?? "memory";
const entry = bySource.get(source) ?? { files: 0, chunks: 0 };
entry.files += row.c ?? 0;
entry.chunks += row.c ?? 0;
bySource.set(source, entry);
total += row.c ?? 0;
}
return {
totalDocuments: total,
sourceCounts: Array.from(bySource.entries()).map(([source, value]) => ({
source,
files: value.files,
chunks: value.chunks,
})),
};
} catch (err) {
log.warn(`failed to read qmd index stats: ${String(err)}`);
return {
totalDocuments: 0,
sourceCounts: Array.from(this.sources).map((source) => ({ source, files: 0, chunks: 0 })),
};
}
}
private isScopeAllowed(sessionKey?: string): boolean {
const scope = this.qmd.scope;
if (!scope) return true;
const channel = this.deriveChannelFromKey(sessionKey);
const chatType = this.deriveChatTypeFromKey(sessionKey);
const normalizedKey = sessionKey ?? "";
for (const rule of scope.rules ?? []) {
if (!rule) continue;
const match = rule.match ?? {};
if (match.channel && match.channel !== channel) continue;
if (match.chatType && match.chatType !== chatType) continue;
if (match.keyPrefix && !normalizedKey.startsWith(match.keyPrefix)) continue;
return rule.action === "allow";
}
const fallback = scope.default ?? "allow";
return fallback === "allow";
}
private deriveChannelFromKey(key?: string) {
if (!key) return undefined;
const parts = key.split(":").filter(Boolean);
if (parts.length >= 3 && (parts[1] === "group" || parts[1] === "channel")) {
return parts[0]?.toLowerCase();
}
return undefined;
}
private deriveChatTypeFromKey(key?: string) {
if (!key) return undefined;
if (key.includes(":group:")) return "group";
if (key.includes(":channel:")) return "channel";
return "direct";
}
private toDocLocation(
collection: string,
collectionRelativePath: string,
): { rel: string; abs: string; source: MemorySource } | null {
const root = this.collectionRoots.get(collection);
if (!root) return null;
const normalizedRelative = collectionRelativePath.replace(/\\/g, "/");
const absPath = path.normalize(path.resolve(root.path, collectionRelativePath));
const relativeToWorkspace = path.relative(this.workspaceDir, absPath);
const relPath = this.buildSearchPath(
collection,
normalizedRelative,
relativeToWorkspace,
absPath,
);
return { rel: relPath, abs: absPath, source: root.kind };
}
private buildSearchPath(
collection: string,
collectionRelativePath: string,
relativeToWorkspace: string,
absPath: string,
): string {
const insideWorkspace = this.isInsideWorkspace(relativeToWorkspace);
if (insideWorkspace) {
const normalized = relativeToWorkspace.replace(/\\/g, "/");
if (!normalized) return path.basename(absPath);
return normalized;
}
const sanitized = collectionRelativePath.replace(/^\/+/, "");
return `qmd/${collection}/${sanitized}`;
}
private isInsideWorkspace(relativePath: string): boolean {
if (!relativePath) return true;
if (relativePath.startsWith("..")) return false;
if (relativePath.startsWith(`..${path.sep}`)) return false;
return !path.isAbsolute(relativePath);
}
private resolveReadPath(relPath: string): string {
if (relPath.startsWith("qmd/")) {
const [, collection, ...rest] = relPath.split("/");
if (!collection || rest.length === 0) {
throw new Error("invalid qmd path");
}
const root = this.collectionRoots.get(collection);
if (!root) throw new Error(`unknown qmd collection: ${collection}`);
const joined = rest.join("/");
const resolved = path.resolve(root.path, joined);
if (!this.isWithinRoot(root.path, resolved)) {
throw new Error("qmd path escapes collection");
}
return resolved;
}
const absPath = path.resolve(this.workspaceDir, relPath);
if (!this.isWithinWorkspace(absPath)) {
throw new Error("path escapes workspace");
}
return absPath;
}
private isWithinWorkspace(absPath: string): boolean {
const normalizedWorkspace = this.workspaceDir.endsWith(path.sep)
? this.workspaceDir
: `${this.workspaceDir}${path.sep}`;
if (absPath === this.workspaceDir) return true;
const candidate = absPath.endsWith(path.sep) ? absPath : `${absPath}${path.sep}`;
return candidate.startsWith(normalizedWorkspace);
}
private isWithinRoot(root: string, candidate: string): boolean {
const normalizedRoot = root.endsWith(path.sep) ? root : `${root}${path.sep}`;
if (candidate === root) return true;
const next = candidate.endsWith(path.sep) ? candidate : `${candidate}${path.sep}`;
return next.startsWith(normalizedRoot);
}
private clampResultsByInjectedChars(results: MemorySearchResult[]): MemorySearchResult[] {
const budget = this.qmd.limits.maxInjectedChars;
if (!budget || budget <= 0) return results;
let remaining = budget;
const clamped: MemorySearchResult[] = [];
for (const entry of results) {
if (remaining <= 0) break;
const snippet = entry.snippet ?? "";
if (snippet.length <= remaining) {
clamped.push(entry);
remaining -= snippet.length;
} else {
const trimmed = snippet.slice(0, Math.max(0, remaining));
clamped.push({ ...entry, snippet: trimmed });
break;
}
}
return clamped;
}
private shouldSkipUpdate(force?: boolean): boolean {
if (force) return false;
const debounceMs = this.qmd.update.debounceMs;
if (debounceMs <= 0) return false;
if (!this.lastUpdateAt) return false;
return Date.now() - this.lastUpdateAt < debounceMs;
}
}

View File

@ -0,0 +1,65 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
const mockPrimary = {
search: vi.fn(async () => []),
readFile: vi.fn(async () => ({ text: "", path: "MEMORY.md" })),
status: vi.fn(() => ({
backend: "qmd" as const,
provider: "qmd",
model: "qmd",
requestedProvider: "qmd",
files: 0,
chunks: 0,
dirty: false,
workspaceDir: "/tmp",
dbPath: "/tmp/index.sqlite",
sources: ["memory" as const],
sourceCounts: [{ source: "memory" as const, files: 0, chunks: 0 }],
})),
sync: vi.fn(async () => {}),
probeEmbeddingAvailability: vi.fn(async () => ({ ok: true })),
probeVectorAvailability: vi.fn(async () => true),
close: vi.fn(async () => {}),
};
vi.mock("./qmd-manager.js", () => ({
QmdMemoryManager: {
create: vi.fn(async () => mockPrimary),
},
}));
vi.mock("./manager.js", () => ({
MemoryIndexManager: {
get: vi.fn(async () => null),
},
}));
import { QmdMemoryManager } from "./qmd-manager.js";
import { getMemorySearchManager } from "./search-manager.js";
beforeEach(() => {
mockPrimary.search.mockClear();
mockPrimary.readFile.mockClear();
mockPrimary.status.mockClear();
mockPrimary.sync.mockClear();
mockPrimary.probeEmbeddingAvailability.mockClear();
mockPrimary.probeVectorAvailability.mockClear();
mockPrimary.close.mockClear();
QmdMemoryManager.create.mockClear();
});
describe("getMemorySearchManager caching", () => {
it("reuses the same QMD manager instance for repeated calls", async () => {
const cfg = {
memory: { backend: "qmd", qmd: {} },
agents: { list: [{ id: "main", default: true, workspace: "/tmp/workspace" }] },
} as const;
const first = await getMemorySearchManager({ cfg, agentId: "main" });
const second = await getMemorySearchManager({ cfg, agentId: "main" });
expect(first.manager).toBe(second.manager);
// eslint-disable-next-line @typescript-eslint/unbound-method
expect(QmdMemoryManager.create).toHaveBeenCalledTimes(1);
});
});

View File

@ -1,8 +1,18 @@
import { createSubsystemLogger } from "../logging/subsystem.js";
import type { MoltbotConfig } from "../config/config.js";
import type { MemoryIndexManager } from "./manager.js";
import { resolveMemoryBackendConfig } from "./backend-config.js";
import type { ResolvedQmdConfig } from "./backend-config.js";
import type {
MemoryEmbeddingProbeResult,
MemorySearchManager,
MemorySyncProgressUpdate,
} from "./types.js";
const log = createSubsystemLogger("memory");
const QMD_MANAGER_CACHE = new Map<string, MemorySearchManager>();
export type MemorySearchManagerResult = {
manager: MemoryIndexManager | null;
manager: MemorySearchManager | null;
error?: string;
};
@ -10,6 +20,40 @@ export async function getMemorySearchManager(params: {
cfg: MoltbotConfig;
agentId: string;
}): Promise<MemorySearchManagerResult> {
const resolved = resolveMemoryBackendConfig(params);
if (resolved.backend === "qmd" && resolved.qmd) {
const cacheKey = buildQmdCacheKey(params.agentId, resolved.qmd);
const cached = QMD_MANAGER_CACHE.get(cacheKey);
if (cached) {
return { manager: cached };
}
try {
const { QmdMemoryManager } = await import("./qmd-manager.js");
const primary = await QmdMemoryManager.create({
cfg: params.cfg,
agentId: params.agentId,
resolved,
});
if (primary) {
const wrapper = new FallbackMemoryManager(
{
primary,
fallbackFactory: async () => {
const { MemoryIndexManager } = await import("./manager.js");
return await MemoryIndexManager.get(params);
},
},
() => QMD_MANAGER_CACHE.delete(cacheKey),
);
QMD_MANAGER_CACHE.set(cacheKey, wrapper);
return { manager: wrapper };
}
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
log.warn(`qmd memory unavailable; falling back to builtin: ${message}`);
}
}
try {
const { MemoryIndexManager } = await import("./manager.js");
const manager = await MemoryIndexManager.get(params);
@ -19,3 +63,131 @@ export async function getMemorySearchManager(params: {
return { manager: null, error: message };
}
}
class FallbackMemoryManager implements MemorySearchManager {
private fallback: MemorySearchManager | null = null;
private primaryFailed = false;
private lastError?: string;
constructor(
private readonly deps: {
primary: MemorySearchManager;
fallbackFactory: () => Promise<MemorySearchManager | null>;
},
private readonly onClose?: () => void,
) {}
async search(
query: string,
opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
) {
if (!this.primaryFailed) {
try {
return await this.deps.primary.search(query, opts);
} catch (err) {
this.primaryFailed = true;
this.lastError = err instanceof Error ? err.message : String(err);
log.warn(`qmd memory failed; switching to builtin index: ${this.lastError}`);
await this.deps.primary.close?.().catch(() => {});
}
}
const fallback = await this.ensureFallback();
if (fallback) {
return await fallback.search(query, opts);
}
throw new Error(this.lastError ?? "memory search unavailable");
}
async readFile(params: { relPath: string; from?: number; lines?: number }) {
if (!this.primaryFailed) {
return await this.deps.primary.readFile(params);
}
const fallback = await this.ensureFallback();
if (fallback) {
return await fallback.readFile(params);
}
throw new Error(this.lastError ?? "memory read unavailable");
}
status() {
if (!this.primaryFailed) {
return this.deps.primary.status();
}
const fallbackStatus = this.fallback?.status();
const fallbackInfo = { from: "qmd", reason: this.lastError ?? "unknown" };
if (fallbackStatus) {
const custom = fallbackStatus.custom ?? {};
return {
...fallbackStatus,
fallback: fallbackInfo,
custom: {
...custom,
fallback: { disabled: true, reason: this.lastError ?? "unknown" },
},
};
}
const primaryStatus = this.deps.primary.status();
const custom = primaryStatus.custom ?? {};
return {
...primaryStatus,
fallback: fallbackInfo,
custom: {
...custom,
fallback: { disabled: true, reason: this.lastError ?? "unknown" },
},
};
}
async sync(params?: {
reason?: string;
force?: boolean;
progress?: (update: MemorySyncProgressUpdate) => void;
}) {
if (!this.primaryFailed) {
await this.deps.primary.sync?.(params);
return;
}
const fallback = await this.ensureFallback();
await fallback?.sync?.(params);
}
async probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult> {
if (!this.primaryFailed) {
return await this.deps.primary.probeEmbeddingAvailability();
}
const fallback = await this.ensureFallback();
if (fallback) {
return await fallback.probeEmbeddingAvailability();
}
return { ok: false, error: this.lastError ?? "memory embeddings unavailable" };
}
async probeVectorAvailability() {
if (!this.primaryFailed) {
return await this.deps.primary.probeVectorAvailability();
}
const fallback = await this.ensureFallback();
return (await fallback?.probeVectorAvailability()) ?? false;
}
async close() {
await this.deps.primary.close?.();
await this.fallback?.close?.();
this.onClose?.();
}
private async ensureFallback(): Promise<MemorySearchManager | null> {
if (this.fallback) return this.fallback;
const fallback = await this.deps.fallbackFactory();
if (!fallback) {
log.warn("memory fallback requested but builtin index is unavailable");
return null;
}
this.fallback = fallback;
return this.fallback;
}
}
function buildQmdCacheKey(agentId: string, config: ResolvedQmdConfig): string {
return `${agentId}:${JSON.stringify(config)}`;
}

View File

@ -3,6 +3,7 @@ import path from "node:path";
import { resolveSessionTranscriptsDirForAgent } from "../config/sessions/paths.js";
import { createSubsystemLogger } from "../logging/subsystem.js";
import { redactSensitiveText } from "../logging/redact.js";
import { hashText } from "./internal.js";
const log = createSubsystemLogger("memory");
@ -87,8 +88,9 @@ export async function buildSessionEntry(absPath: string): Promise<SessionFileEnt
if (message.role !== "user" && message.role !== "assistant") continue;
const text = extractSessionText(message.content);
if (!text) continue;
const safe = redactSensitiveText(text, { mode: "tools" });
const label = message.role === "user" ? "User" : "Assistant";
collected.push(`${label}: ${text}`);
collected.push(`${label}: ${safe}`);
}
const content = collected.join("\n");
return {

79
src/memory/types.ts Normal file
View File

@ -0,0 +1,79 @@
export type MemorySource = "memory" | "sessions";
export type MemorySearchResult = {
path: string;
startLine: number;
endLine: number;
score: number;
snippet: string;
source: MemorySource;
citation?: string;
};
export type MemoryEmbeddingProbeResult = {
ok: boolean;
error?: string;
};
export type MemorySyncProgressUpdate = {
completed: number;
total: number;
label?: string;
};
export type MemoryProviderStatus = {
backend: "builtin" | "qmd";
provider: string;
model?: string;
requestedProvider?: string;
files?: number;
chunks?: number;
dirty?: boolean;
workspaceDir?: string;
dbPath?: string;
sources?: MemorySource[];
sourceCounts?: Array<{ source: MemorySource; files: number; chunks: number }>;
cache?: { enabled: boolean; entries?: number; maxEntries?: number };
fts?: { enabled: boolean; available: boolean; error?: string };
fallback?: { from: string; reason?: string };
vector?: {
enabled: boolean;
available?: boolean;
extensionPath?: string;
loadError?: string;
dims?: number;
};
batch?: {
enabled: boolean;
failures: number;
limit: number;
wait: boolean;
concurrency: number;
pollIntervalMs: number;
timeoutMs: number;
lastError?: string;
lastProvider?: string;
};
custom?: Record<string, unknown>;
};
export interface MemorySearchManager {
search(
query: string,
opts?: { maxResults?: number; minScore?: number; sessionKey?: string },
): Promise<MemorySearchResult[]>;
readFile(params: {
relPath: string;
from?: number;
lines?: number;
}): Promise<{ text: string; path: string }>;
status(): MemoryProviderStatus;
sync?(params?: {
reason?: string;
force?: boolean;
progress?: (update: MemorySyncProgressUpdate) => void;
}): Promise<void>;
probeEmbeddingAvailability(): Promise<MemoryEmbeddingProbeResult>;
probeVectorAvailability(): Promise<boolean>;
close?(): Promise<void>;
}