openclaw/docs/enterprise/observability.md

Observability Guide

This guide covers monitoring, metrics, tracing, and alerting for enterprise deployments.

Health Endpoints

The gateway exposes HTTP health endpoints for container orchestration:

Liveness Probe

GET /health

Returns 200 OK if the gateway process is alive:

{
  "status": "healthy",
  "version": "2024.1.15",
  "uptimeMs": 3600000
}

Readiness Probe

GET /ready

Returns 200 OK if channels are connected and ready:

{
  "status": "healthy",
  "checks": {
    "channels": {
      "telegram": "connected",
      "discord": "connected",
      "slack": "degraded"
    }
  }
}

Returns 503 Service Unavailable if critical channels are down.

Deep Health Check

GET /health/deep
Authorization: Bearer <gateway-token>

Returns detailed system status (requires authentication):

{
  "status": "healthy",
  "version": "2024.1.15",
  "uptimeMs": 3600000,
  "checks": {
    "channels": {
      "telegram": { "status": "connected", "latencyMs": 45 },
      "discord": { "status": "connected", "latencyMs": 62 }
    },
    "memory": {
      "heapUsed": 128000000,
      "heapTotal": 256000000,
      "rss": 312000000
    }
  }
}

Kubernetes Configuration

livenessProbe:
  httpGet:
    path: /health
    port: 18789
  initialDelaySeconds: 10
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 18789
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 2

Prometheus Metrics

The gateway exposes Prometheus metrics at /metrics:

GET /metrics
Authorization: Bearer <gateway-token>

Available Metrics

Token Usage

# Total tokens used
clawdbot_tokens_total{type="input"} 125000
clawdbot_tokens_total{type="output"} 45000
clawdbot_tokens_total{type="cache_read"} 80000
clawdbot_tokens_total{type="cache_write"} 15000

Cost Tracking

# Total cost in USD
clawdbot_cost_usd_total{channel="telegram",provider="anthropic",model="claude-sonnet-4-20250514"} 1.25

Message Processing

# Messages processed
clawdbot_message_processed_total{channel="telegram",outcome="completed"} 500
clawdbot_message_processed_total{channel="telegram",outcome="error"} 5

# Message processing duration
clawdbot_message_duration_seconds_bucket{channel="telegram",le="1"} 450
clawdbot_message_duration_seconds_bucket{channel="telegram",le="5"} 495
clawdbot_message_duration_seconds_bucket{channel="telegram",le="30"} 500

Webhooks

# Webhook events
clawdbot_webhook_received_total{channel="telegram"} 1000
clawdbot_webhook_processed_total{channel="telegram"} 995
clawdbot_webhook_error_total{channel="telegram"} 5

Sessions

# Active sessions by state
clawdbot_session_state{state="idle"} 10
clawdbot_session_state{state="processing"} 2
clawdbot_session_state{state="waiting"} 1

Queue

# Queue depth
clawdbot_queue_depth{lane="default"} 3

System

# Gateway uptime
clawdbot_gateway_uptime_seconds 3600

Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: 'clawdbot'
    static_configs:
      - targets: ['gateway-host:18789']
    bearer_token: '<gateway-token>'
    metrics_path: '/metrics'

Grafana Dashboard

Import this dashboard for key metrics:

{
  "title": "Clawdbot Gateway",
  "panels": [
    {
      "title": "Token Usage Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(clawdbot_tokens_total[5m])",
          "legendFormat": "{{type}}"
        }
      ]
    },
    {
      "title": "Message Throughput",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(clawdbot_message_processed_total[5m])",
          "legendFormat": "{{channel}} - {{outcome}}"
        }
      ]
    },
    {
      "title": "Cost per Hour",
      "type": "stat",
      "targets": [
        {
          "expr": "increase(clawdbot_cost_usd_total[1h])"
        }
      ]
    },
    {
      "title": "Active Sessions",
      "type": "gauge",
      "targets": [
        {
          "expr": "sum(clawdbot_session_state)"
        }
      ]
    }
  ]
}

Distributed Tracing

The gateway supports W3C Trace Context for distributed tracing.

Trace ID Propagation

All requests include a trace ID that flows through:

• Audit logs (traceId field)
• Diagnostic events
• Console/file logs
• Outgoing HTTP requests

Log Correlation

Trace IDs appear in logs for correlation:

[gateway] [a1b2c3d4] Processing message from telegram
[agent] [a1b2c3d4] Invoking claude-sonnet-4-20250514
[gateway] [a1b2c3d4] Message completed in 2.3s

Audit Log Correlation

Query audit logs by trace ID:

cat ~/.clawdbot/audit.jsonl | jq 'select(.traceId == "4bf92f3577b34da6a3ce929d0e0e4736")'

OpenTelemetry Integration

For full APM integration, use the diagnostics-otel extension:

plugins:
  diagnostics-otel:
    enabled: true
    endpoint: "http://otel-collector:4317"

This exports:

• Traces to your APM (Jaeger, Zipkin, Datadog, etc.)
• Metrics in OTLP format
• Logs with trace correlation

Logging

Log Levels

Configure log verbosity:

logging:
  level: info  # trace, debug, info, warn, error, silent
  console:
    level: warn  # Separate console level
    style: pretty  # pretty, compact, json

Structured Logging

JSON log format for log aggregation:

logging:
  console:
    style: json

Output:

{"time":"2024-01-15T10:30:00.000Z","level":"info","subsystem":"gateway","message":"Gateway started","traceId":"a1b2c3d4"}

Log Files

Logs are written to ~/.clawdbot/logs/:

~/.clawdbot/logs/
├── clawdbot.log        # Current log
├── clawdbot.1.log      # Rotated logs
└── clawdbot.2.log

Log Aggregation

Ship logs to your aggregation system:

Filebeat:

filebeat.inputs:
  - type: log
    paths:
      - /root/.clawdbot/logs/*.log
    json.keys_under_root: true
    json.add_error_key: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Fluentd:

<source>
  @type tail
  path /root/.clawdbot/logs/*.log
  pos_file /var/log/fluentd/clawdbot.pos
  tag clawdbot
  <parse>
    @type json
  </parse>
</source>

Alerting

Prometheus Alerting Rules

# alerts.yml
groups:
  - name: clawdbot
    rules:
      - alert: ClawdbotDown
        expr: up{job="clawdbot"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Clawdbot gateway is down"

      - alert: HighErrorRate
        expr: rate(clawdbot_message_processed_total{outcome="error"}[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High message error rate"

      - alert: ChannelDisconnected
        expr: clawdbot_channel_connected == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Channel {{ $labels.channel }} disconnected"

      - alert: HighCostRate
        expr: increase(clawdbot_cost_usd_total[1h]) > 10
        for: 1m
        labels:
          severity: info
        annotations:
          summary: "High API cost in last hour: ${{ $value }}"

Key Metrics to Monitor

Metric	Alert Threshold	Severity
Gateway uptime	Down > 1m	Critical
Message error rate	> 10%	Warning
Channel disconnected	> 5m	Warning
Hourly cost	> $10	Info
Queue depth	> 100	Warning
Processing latency p99	> 30s	Warning

Diagnostic Events

The gateway emits real-time diagnostic events for monitoring:

// Subscribe to diagnostic events
import { onDiagnosticEvent } from 'clawdbot/diagnostic-events';

onDiagnosticEvent((event) => {
  console.log(event.type, event);
});

Event Types

Event	Description
model.usage	Token usage and cost
webhook.received	Incoming webhook
webhook.processed	Webhook processing complete
webhook.error	Webhook processing error
message.queued	Message added to queue
message.processed	Message processing complete
session.state	Session state change
session.stuck	Session stuck detection
diagnostic.heartbeat	Periodic status summary

Troubleshooting

Common Issues

No metrics at /metrics:

• Check authentication header
• Verify gateway.metrics.enabled: true

Missing trace IDs:

• Ensure requests flow through the gateway
• Check log format includes trace context

High memory usage:

• Review session retention settings
• Check for stuck sessions
• Monitor queue depth

Debug Commands

# Check gateway health
curl -s http://localhost:18789/health | jq

# View recent audit events
tail -100 ~/.clawdbot/audit.jsonl | jq

# Check channel status
clawdbot channels status --deep

# View diagnostic events
clawdbot gateway events --follow

Next Steps

• Enterprise Deployment - Deployment patterns
• Security Hardening - Security configuration
• Gateway Troubleshooting - Detailed troubleshooting