25afb60ece
Adds a new plugin that integrates with Weights & Biases Weave for comprehensive LLM tracing and observability. Plugin features: - Full system prompt capture via new llm_request hook - Multi-turn conversation tracing (all LLM requests) - Tool call spans with inputs, outputs, duration - Session lifecycle tracking - Support for all LLM providers (Claude, GPT, Gemini, Bedrock, etc.) - Self-hosted W&B instance support via baseUrl config - Automatic project creation with RESTRICTED visibility - Graceful degradation when core llm_request hook unavailable Core changes: - Add llm_request hook to plugin system for LLM payload observability - Wrap streamFn in agent runner to capture requests to all providers Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
128 lines
3.9 KiB
Markdown
128 lines
3.9 KiB
Markdown
# W&B Weave Plugin for Clawdbot
|
|
|
|
Integrates [Weights & Biases Weave](https://weave-docs.wandb.ai/) for comprehensive LLM observability and tracing.
|
|
|
|
## Features
|
|
|
|
- **Full System Prompt Capture** - Captures the complete system prompt, tracks changes across requests
|
|
- **Multi-Turn Conversation Tracing** - Tracks all LLM requests in a conversation, not just the first
|
|
- **Tool Call Spans** - Logs each tool call with inputs, outputs, and duration as child spans
|
|
- **Session Lifecycle Tracking** - Monitors session start/end with aggregated metrics
|
|
- **All LLM Providers** - Works with Claude, GPT, Gemini, Bedrock, and all pi-ai supported models
|
|
|
|
## Installation
|
|
|
|
The plugin is included in Clawdbot. Enable it in your configuration:
|
|
|
|
```json
|
|
{
|
|
"plugins": {
|
|
"entries": {
|
|
"weave": {
|
|
"enabled": true,
|
|
"config": {
|
|
"apiKey": "wandb_...",
|
|
"entity": "your-username"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Configuration
|
|
|
|
| Option | Type | Default | Description |
|
|
|--------|------|---------|-------------|
|
|
| `apiKey` | string | **required** | W&B API key ([get one here](https://wandb.ai/settings)) |
|
|
| `entity` | string | **required** | W&B entity (username or team name) |
|
|
| `project` | string | `"clawdbot"` | Project name for traces |
|
|
| `autoTrace` | boolean | `true` | Automatically trace all agent runs |
|
|
| `traceToolCalls` | boolean | `true` | Log tool calls as child spans |
|
|
| `traceSessions` | boolean | `true` | Track session lifecycle |
|
|
| `baseUrl` | string | - | Custom W&B server URL (for self-hosted) |
|
|
| `sampleRate` | number | `1.0` | Trace sampling rate (0.0 to 1.0) |
|
|
| `debug` | boolean | `false` | Enable verbose debug logging |
|
|
|
|
## Trace Data
|
|
|
|
Each agent run trace includes:
|
|
|
|
### Inputs
|
|
- User messages
|
|
- Session context (compaction summaries)
|
|
|
|
### Outputs
|
|
- Assistant messages
|
|
- Final response
|
|
- Tool call results
|
|
|
|
### Metadata
|
|
- **systemPrompt** - Full system prompt (latest version if changed)
|
|
- **systemPromptChanges** - Number of times the system prompt changed
|
|
- **llmRequests** - Array of all LLM API requests (each with full system prompt)
|
|
- **llmRequestCount** - Number of LLM requests made
|
|
- **llmTools** - Available tools/functions
|
|
- **toolCalls** - Detailed tool execution data
|
|
|
|
### Tool Call Spans
|
|
Each tool call is logged as a child span with:
|
|
- Tool name
|
|
- Input parameters
|
|
- Output/result
|
|
- Duration
|
|
- Success/error status
|
|
|
|
## Supported LLM Providers
|
|
|
|
The plugin works with all providers supported by pi-ai:
|
|
|
|
| Provider | API | Models |
|
|
|----------|-----|--------|
|
|
| Anthropic | `anthropic-messages` | Claude Opus 4.5, Claude Sonnet 4.5, etc. |
|
|
| Google | `google-generative-ai` | Gemini 3 Pro, Gemini 3 Flash, etc. |
|
|
| OpenAI | `openai-completions` | GPT-5.2, GPT-4o, o3, etc. |
|
|
| OpenAI | `openai-responses` | Newer response API |
|
|
| AWS | `bedrock-converse-stream` | Bedrock models |
|
|
| Google Cloud | `google-vertex` | Vertex AI models |
|
|
| Others | `openai-completions` | OpenRouter, DeepSeek, Grok, Mistral, etc. |
|
|
|
|
## Viewing Traces
|
|
|
|
1. Go to [wandb.ai](https://wandb.ai)
|
|
2. Navigate to your project (e.g., `your-username/clawdbot`)
|
|
3. Click on "Weave" in the left sidebar
|
|
4. Browse traces, view spans, and analyze metrics
|
|
|
|
## Troubleshooting
|
|
|
|
### No traces appearing
|
|
- Verify your API key is correct
|
|
- Check that `autoTrace` is `true`
|
|
- Look for errors in Clawdbot logs with `[weave]` prefix
|
|
|
|
### Missing system prompt
|
|
- The `llm_request` hook must be enabled in Clawdbot core
|
|
- Check `systemPromptChanges` field to see if prompt changed during conversation
|
|
|
|
### Partial traces
|
|
- Check `sampleRate` - if less than 1.0, some traces are skipped
|
|
- Ensure agent completes normally (crashes may lose trace data)
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Build
|
|
cd extensions/weave
|
|
pnpm build
|
|
|
|
# Watch mode
|
|
pnpm dev
|
|
```
|
|
|
|
## Links
|
|
|
|
- [W&B Weave Documentation](https://weave-docs.wandb.ai/)
|
|
- [Weave TypeScript SDK](https://www.npmjs.com/package/weave)
|
|
- [Clawdbot Plugin SDK](../../docs/plugins.md)
|