diff --git a/docs/memory-cognee.md b/docs/memory-cognee.md index 623e6c1c7..cc5a1e943 100644 --- a/docs/memory-cognee.md +++ b/docs/memory-cognee.md @@ -1,66 +1,37 @@ --- -summary: "Cognee knowledge graph memory: setup, Docker config, and usage" +summary: "Cognee memory quick setup and usage" read_when: - Setting up Cognee memory provider - Configuring knowledge graph memory - - Running Cognee with Docker --- # Cognee Memory Provider -MoltBot supports [**Cognee**](https://www.cognee.ai/) as an optional memory provider. Unlike the default SQLite-based vector memory, Cognee builds a knowledge graph with entity extraction and semantic relationships, providing richer contextual memory for your AI agent. +Moltbot supports [Cognee](https://www.cognee.ai/) - [open source AI memory](https://github.com/topoteretes/cognee) - as an optional memory provider. Cognee builds knowledge graph memory backed by embeddings from any data and can be run locally with Docker. Learn more from [Cognee Documentation](https://docs.cognee.ai/). -## What is Cognee? +## Quickstart with Docker -Cognee is an [open source AI memory](https://github.com/topoteretes/cognee) framework that: -- Ingests unstructured/structured data from 30+ sources -- Extracts entities (people, places, concepts) from documents -- Builds a knowledge graph of relationships backed by embeddings -- Enables semantic search with LLM-powered reasoning -- Supports multiple search modes (e.g., GRAPH_COMPLETION, chunks, summaries) - -Learn more at [docs.cognee.ai](https://docs.cognee.ai/). - -## Setup Options - -### Option 1: Local Docker (Recommended) - -Use the repo example compose file: +Run the example compose file: ```bash docker compose -f examples/cognee-docker-compose.yaml up -d ``` - -This binds to `127.0.0.1:8000`, so Cognee is only reachable from your machine. - -**Verify:** +Verify: ```bash curl http://localhost:8000/health ``` -You can use `http://localhost:8000/docs` to register user and login to get your token. - -### Option 2: Cognee Cloud - -Use the hosted Cognee service: - -1. Sign up at [platform.cognee.ai](https://platform.cognee.ai/) -2. Get your API key from the dashboard -3. Use base URL: `https://cognee--cognee-saas-backend-serve.modal.run` - ## Configuration -Moltbot reads `~/.clawdbot/moltbot.json` (JSON5). For local + secure setup, use an env var for the Cognee token and reference it in config. - -### Step 1: Put tokens in `~/.clawdbot/.env` +Put the token in `~/.clawdbot/.env`: ```bash COGNEE_API_KEY="your-cognee-access-token" CLAWDBOT_GATEWAY_TOKEN="your-random-gateway-token" ``` -### Step 2: Configure Cognee in `~/.clawdbot/moltbot.json` +Configure `~/.clawdbot/moltbot.json` (JSON5): ```json5 { @@ -86,195 +57,30 @@ CLAWDBOT_GATEWAY_TOKEN="your-random-gateway-token" } ``` -### Step 3: Start the gateway with env loaded +Start the gateway with env loaded: ```zsh set -a; source "$HOME/.clawdbot/.env"; set +a pnpm moltbot gateway --port 18789 --token "$CLAWDBOT_GATEWAY_TOKEN" --verbose ``` - - -## Configuration Options - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `baseUrl` | string | `http://localhost:8000` | Cognee API endpoint | -| `apiKey` | string | - | Access token (required if Cognee auth is enabled) | -| `datasetName` | string | `"clawdbot"` | Dataset name for organizing memories | -| `searchType` | string | `"GRAPH_COMPLETION"` | Search mode: `GRAPH_COMPLETION`, `chunks`, or `summaries` | -| `maxResults` | number | `6` | Maximum search results returned | -| `autoCognify` | boolean | `true` | Auto-process documents after adding | -| `cognifyBatchSize` | number | `100` | Batch size for processing | -| `timeoutSeconds` | number | `180` | Request timeout in seconds | - -## Search Types - -Cognee offers these modes. GRAPH_COMPLETION is the default mode. - -Learn more from [cognee documentation](https://docs.cognee.ai/core-concepts/main-operations/search) - ## Usage -### Memory Files +Cognee indexes `MEMORY.md` in workspace root, `memory/*.md`, and session transcripts when `sources: ["sessions"]` is enabled. -Cognee automatically syncs your memory files: -- `MEMORY.md` or `memory.md` in workspace root -- All `*.md` files in `memory/` directory - -### Session Transcripts (Optional) - -Enable session memory to index conversation history: - -```yaml -agents: - defaults: - memorySearch: - provider: cognee - sources: [memory, sessions] # Include sessions - experimental: - sessionMemory: true -``` - -### Manual Sync and Status +1. Initial index and status: ```bash -# Manually sync cognee memory for the current agent pnpm moltbot memory status --index --json ``` -## How It Works +2. Memory updates: -1. **Add**: Memory files are sent to Cognee with metadata -2. **Cognify**: Cognee processes documents: - - Extracts entities (people, places, concepts) - - Identifies relationships - - Builds knowledge graph -3. **Search**: Agent queries use semantic search: - - Searches knowledge graph - - Returns relevant insights/chunks/summaries - - Includes metadata and scores - -## Comparison: Cognee vs SQLite Memory - -| Feature | Cognee | SQLite (Default) | -|---------|--------|------------------| -| **Setup** | Requires Docker/cloud | Built-in, no setup | -| **Offline** | Yes (Optional cloud) | Yes (fully local) | -| **Search** | Knowledge graph + LLM | Vector + BM25 hybrid | -| **Entities** | Extracted automatically | Not available | -| **Relationships** | Yes (graph-based) | No | -| **Speed** | Slower (API calls) | Faster (local DB) | -| **Memory** | Stored locally (Optional cognee configs) | SQLite file | -| **Best for** | Rich context, reasoning | Fast lookup, privacy | +```bash +pnpm moltbot memory status --index --update-cognee --json +``` ## Troubleshooting -### Connection Failed - -**Error**: `Failed to connect to Cognee at http://localhost:8000` - -**Solutions**: -1. Verify Docker is running: `docker ps | grep cognee` -2. Check Cognee logs: `docker logs cognee` -3. Test manually: `curl http://localhost:8000/health` -4. Ensure port 8000 is not blocked - -### Slow Performance - -**Solutions**: -1. Reduce `maxResults` (try 3-5 instead of 10+) -2. Use `searchType: "chunks"` for faster results -3. Set `autoCognify: false` and cognify manually -4. Check Docker resource limits - -### Out of Memory - -**Solutions**: -1. Increase Docker memory limit (Docker Desktop settings) -2. Reduce `cognifyBatchSize` (try 50 instead of 100) -3. Process fewer files at once -4. Clear old datasets via Cognee API - -### 401 Unauthorized (Cognee auth) - -**Cause**: Cognee auth is enabled, but Moltbot is missing/using an invalid `COGNEE_API_KEY`. - -**Fix**: -1. Log in to Cognee and get a fresh access token. -2. Update `COGNEE_API_KEY` in `.clawd/.env`. -3. Restart the gateway with env loaded (see Step 3 above). - -## Advanced Configuration - -### Per-Agent Override - -```yaml -agents: - defaults: - memorySearch: - provider: openai # Default for all agents - - agents: - research-bot: - memorySearch: - provider: cognee # Override for this agent - cognee: - searchType: GRAPH_COMPLETION - maxResults: 10 -``` - -## Docker Production Tips - -### Health Checks - -Add health checks to docker-compose.yml: - -```yaml -services: - cognee: - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 40s -``` - -### Resource Limits - -```yaml -services: - cognee: - deploy: - resources: - limits: - cpus: '2' - memory: 4G - reservations: - cpus: '1' - memory: 2G -``` - -### Persistent Storage - -Mount volumes for persistence: - -```yaml -volumes: - - ./cognee_data:/app/cognee/.cognee_system - - ./cognee_logs:/app/logs -``` - -## Resources - -- [Cognee Documentation](https://docs.cognee.ai/) -- [Cognee GitHub](https://github.com/topoteretes/cognee) -- [Clawdbot Memory Guide](/memory) -- [Docker Setup Guide](/install/docker) - -## Feedback - -Cognee integration is new. Report issues at: -- Clawdbot: [github.com/clawdbot/clawdbot/issues](https://github.com/clawdbot/clawdbot/issues) -- Cognee: [github.com/topoteretes/cognee/issues](https://github.com/topoteretes/cognee/issues) +- Connection test: `curl http://localhost:8000/health` +- Reset cached values that Moltbot reuses: `mv "$HOME/.clawdbot/memory/cognee/main.json" "$HOME/.clawdbot/memory/cognee/main.json.bak"` \ No newline at end of file diff --git a/examples/cognee-config.yaml b/examples/cognee-config.yaml index 5412b992b..411e83ca9 100644 --- a/examples/cognee-config.yaml +++ b/examples/cognee-config.yaml @@ -35,7 +35,7 @@ agents: cognifyBatchSize: 100 # Request timeout in seconds - timeoutSeconds: 30 + timeoutSeconds: 180 # Enable experimental session memory indexing experimental: diff --git a/src/memory/cognee-client.test.ts b/src/memory/cognee-client.test.ts deleted file mode 100644 index 91b5c1a78..000000000 --- a/src/memory/cognee-client.test.ts +++ /dev/null @@ -1,227 +0,0 @@ -import { describe, expect, it, vi, beforeEach, afterEach } from "vitest"; -import { CogneeClient } from "./cognee-client.js"; -import { request } from "undici"; - -vi.mock("undici", async () => { - const actual = await vi.importActual("undici"); - return { - ...actual, - request: vi.fn(), - }; -}); - -describe("CogneeClient", () => { - beforeEach(() => { - vi.clearAllMocks(); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - describe("add", () => { - it("should add data successfully", async () => { - const mockResponse = { - statusCode: 200, - body: { - json: vi.fn().mockResolvedValue({ - dataset_id: "test-dataset-id", - dataset_name: "test-dataset", - message: "Data added successfully", - }), - text: vi.fn(), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient({ - baseUrl: "http://localhost:8000", - apiKey: "test-key", - }); - - const result = await client.add({ - data: "Test data", - datasetName: "test-dataset", - }); - - expect(result).toEqual({ - datasetId: "test-dataset-id", - datasetName: "test-dataset", - message: "Data added successfully", - }); - expect(request).toHaveBeenCalledWith( - "http://localhost:8000/api/v1/add", - expect.objectContaining({ - method: "POST", - headers: expect.objectContaining({ - Authorization: "Bearer test-key", - "X-Api-Key": "test-key", - }), - }), - ); - }); - - it("should handle errors", async () => { - const mockResponse = { - statusCode: 500, - body: { - text: vi.fn().mockResolvedValue("Internal server error"), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient(); - - await expect( - client.add({ - data: "Test data", - datasetName: "test-dataset", - }), - ).rejects.toThrow( - "Cognee add request failed: Cognee add failed with status 500: Internal server error", - ); - }); - }); - - describe("cognify", () => { - it("should run cognify successfully", async () => { - const mockResponse = { - statusCode: 200, - body: { - json: vi.fn().mockResolvedValue({ - status: "success", - message: "Cognify completed", - }), - text: vi.fn(), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient({ - baseUrl: "http://localhost:8000", - }); - - const result = await client.cognify({ - datasetIds: ["dataset-1"], - }); - - expect(result).toEqual({ - status: "success", - message: "Cognify completed", - }); - }); - }); - - describe("search", () => { - it("should search successfully", async () => { - const mockResponse = { - statusCode: 200, - body: { - json: vi.fn().mockResolvedValue({ - results: [ - { - id: "result-1", - text: "Test result", - score: 0.9, - metadata: { path: "test.md" }, - }, - ], - query: "test query", - search_type: "GRAPH_COMPLETION", - }), - text: vi.fn(), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient(); - - const result = await client.search({ - queryText: "test query", - searchType: "GRAPH_COMPLETION", - }); - - expect(result.results).toHaveLength(1); - expect(result.results[0]).toEqual({ - id: "result-1", - text: "Test result", - score: 0.9, - metadata: { path: "test.md" }, - }); - expect(result.query).toBe("test query"); - }); - - it("should use default search type", async () => { - const mockResponse = { - statusCode: 200, - body: { - json: vi.fn().mockResolvedValue({ - results: [], - query: "test", - search_type: "GRAPH_COMPLETION", - }), - text: vi.fn(), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient(); - await client.search({ queryText: "test" }); - - expect(request).toHaveBeenCalledWith( - expect.any(String), - expect.objectContaining({ - body: expect.stringContaining('"searchType":"GRAPH_COMPLETION"'), - }), - ); - }); - }); - - describe("status", () => { - it("should get status successfully", async () => { - const mockResponse = { - statusCode: 200, - body: { - json: vi.fn().mockResolvedValue({ status: "healthy" }), - text: vi.fn(), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient(); - - const result = await client.status(); - - expect(result).toEqual({ - status: "healthy", - }); - }); - }); - - describe("healthCheck", () => { - it("should return true when status is successful", async () => { - const mockResponse = { - statusCode: 200, - body: { - json: vi.fn().mockResolvedValue({ status: "healthy" }), - text: vi.fn(), - }, - }; - vi.mocked(request).mockResolvedValue(mockResponse as any); - - const client = new CogneeClient(); - const result = await client.healthCheck(); - - expect(result).toBe(true); - }); - - it("should return false when status fails", async () => { - vi.mocked(request).mockRejectedValue(new Error("Connection failed")); - - const client = new CogneeClient(); - const result = await client.healthCheck(); - - expect(result).toBe(false); - }); - }); -}); diff --git a/src/memory/cognee-provider.test.ts b/src/memory/cognee-provider.test.ts index 3bc90c1b0..500bdc356 100644 --- a/src/memory/cognee-provider.test.ts +++ b/src/memory/cognee-provider.test.ts @@ -1,27 +1,34 @@ -import { describe, expect, it, vi, beforeEach } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { CogneeMemoryProvider } from "./cognee-provider.js"; import type { ClawdbotConfig } from "../config/config.js"; +const searchMock = vi.fn(); + vi.mock("./cognee-client.js", () => { class CogneeClient { - healthCheck = vi.fn().mockResolvedValue(true); - add = vi.fn().mockResolvedValue({ - datasetId: "test-dataset-id", - datasetName: "test-dataset", - message: "Success", - }); - cognify = vi.fn().mockResolvedValue({ - status: "success", - message: "Cognify completed", - }); - search = vi.fn().mockResolvedValue({ + search = searchMock; + } + + return { CogneeClient }; +}); + +describe("CogneeMemoryProvider", () => { + it("maps search results into memory snippets", async () => { + const mockConfig: ClawdbotConfig = { + agents: { + defaults: { + workspace: "/tmp/test-workspace", + }, + }, + }; + searchMock.mockResolvedValue({ results: [ { id: "result-1", - text: "Test result text", + text: "A".repeat(800), score: 0.85, metadata: { - path: "test.md", + path: "memory/test.md", source: "memory", }, }, @@ -29,124 +36,51 @@ vi.mock("./cognee-client.js", () => { query: "test query", searchType: "GRAPH_COMPLETION", }); - status = vi.fn().mockResolvedValue({ - status: "healthy", - version: "1.0.0", - datasets: [ + + const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); + + const results = await provider.search("test query"); + + expect(results).toHaveLength(1); + expect(results[0]).toMatchObject({ + path: "memory/test.md", + source: "memory", + score: 0.85, + }); + expect(results[0].snippet.length).toBeGreaterThan(700); + expect(results[0].snippet.endsWith("...")).toBe(true); + }); + + it("defaults missing metadata to safe values", async () => { + searchMock.mockResolvedValue({ + results: [ { - id: "test-dataset-id", - name: "clawdbot", - documentCount: 5, + id: "result-2", + text: "Short result", + score: 0.2, }, ], + query: "missing metadata", + searchType: "GRAPH_COMPLETION", }); - } - return { CogneeClient }; -}); - -vi.mock("./internal.js", () => ({ - listMemoryFiles: vi.fn().mockResolvedValue([]), - buildFileEntry: vi.fn(), - hashText: vi.fn().mockReturnValue("test-hash"), -})); - -vi.mock("node:fs/promises", () => ({ - default: { - readdir: vi.fn().mockResolvedValue([]), - readFile: vi.fn().mockResolvedValue("Test file content"), - stat: vi.fn().mockResolvedValue({ mtimeMs: 1234567890, size: 100 }), - }, -})); - -describe("CogneeMemoryProvider", () => { - const mockConfig: ClawdbotConfig = { - agents: { - defaults: { - workspace: "/tmp/test-workspace", + const mockConfig: ClawdbotConfig = { + agents: { + defaults: { + workspace: "/tmp/test-workspace", + }, }, - }, - }; + }; + const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); - beforeEach(() => { - vi.clearAllMocks(); - }); + const results = await provider.search("missing metadata"); - describe("constructor", () => { - it("should initialize with default configuration", () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); - - expect(provider).toBeDefined(); - }); - - it("should initialize with custom configuration", () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"], { - baseUrl: "http://custom:8000", - apiKey: "custom-key", - datasetName: "custom-dataset", - searchType: "chunks", - maxResults: 10, - }); - - expect(provider).toBeDefined(); - }); - }); - - describe("healthCheck", () => { - it("should perform health check", async () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); - - const healthy = await provider.healthCheck(); - - expect(healthy).toBe(true); - }); - }); - - describe("search", () => { - it("should search and transform results", async () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); - - const results = await provider.search("test query"); - - expect(results).toHaveLength(1); - expect(results[0]).toMatchObject({ - path: "test.md", - source: "memory", - score: 0.85, - snippet: "Test result text", - }); - }); - - it("should respect maxResults setting", async () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"], { - maxResults: 5, - }); - - const results = await provider.search("test query"); - - expect(results.length).toBeLessThanOrEqual(5); - }); - }); - - describe("cognify", () => { - it("should run cognify", async () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); - - await expect(provider.cognify()).resolves.not.toThrow(); - }); - }); - - describe("getStatus", () => { - it("should return status information", async () => { - const provider = new CogneeMemoryProvider(mockConfig, "test-agent", ["memory"]); - - const status = await provider.getStatus(); - - expect(status).toMatchObject({ - connected: true, - datasetName: "clawdbot", - syncedFileCount: 0, - }); + expect(results).toHaveLength(1); + expect(results[0]).toMatchObject({ + path: "unknown", + source: "memory", + score: 0.2, + snippet: "Short result", }); }); });