romtuck/openclaw

Fork 0

ronitchidara 5aae2aa197 security: comprehensive pre-PR security audit fixes (rebased)

2026-01-28 17:03:07 +05:30

9.1 KiB

Raw Blame History

Observability Guide

This guide covers monitoring, metrics, tracing, and alerting for enterprise deployments.

Health Endpoints

The gateway exposes HTTP health endpoints for container orchestration:

Liveness Probe

GET /health

Returns 200 OK if the gateway process is alive:

{
  "status": "healthy",
  "version": "2024.1.15",
  "uptimeMs": 3600000
}

Readiness Probe

GET /ready

Returns 200 OK if channels are connected and ready:

{
  "status": "healthy",
  "checks": {
    "channels": {
      "telegram": "connected",
      "discord": "connected",
      "slack": "degraded"
    }
  }
}

Returns 503 Service Unavailable if critical channels are down.

Deep Health Check

GET /health/deep
Authorization: Bearer <gateway-token>

Returns detailed system status (requires authentication):

{
  "status": "healthy",
  "version": "2024.1.15",
  "uptimeMs": 3600000,
  "checks": {
    "channels": {
      "telegram": { "status": "connected", "latencyMs": 45 },
      "discord": { "status": "connected", "latencyMs": 62 }
    },
    "memory": {
      "heapUsed": 128000000,
      "heapTotal": 256000000,
      "rss": 312000000
    }
  }
}

Kubernetes Configuration

livenessProbe:
  httpGet:
    path: /health
    port: 18789
  initialDelaySeconds: 10
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /ready
    port: 18789
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 2

Prometheus Metrics

The gateway exposes Prometheus metrics at /metrics:

GET /metrics
Authorization: Bearer <gateway-token>

Available Metrics

Token Usage

# Total tokens used
clawdbot_tokens_total{type="input"} 125000
clawdbot_tokens_total{type="output"} 45000
clawdbot_tokens_total{type="cache_read"} 80000
clawdbot_tokens_total{type="cache_write"} 15000

Cost Tracking

# Total cost in USD
clawdbot_cost_usd_total{channel="telegram",provider="anthropic",model="claude-sonnet-4-20250514"} 1.25

Message Processing

# Messages processed
clawdbot_message_processed_total{channel="telegram",outcome="completed"} 500
clawdbot_message_processed_total{channel="telegram",outcome="error"} 5

# Message processing duration
clawdbot_message_duration_seconds_bucket{channel="telegram",le="1"} 450
clawdbot_message_duration_seconds_bucket{channel="telegram",le="5"} 495
clawdbot_message_duration_seconds_bucket{channel="telegram",le="30"} 500

Webhooks

# Webhook events
clawdbot_webhook_received_total{channel="telegram"} 1000
clawdbot_webhook_processed_total{channel="telegram"} 995
clawdbot_webhook_error_total{channel="telegram"} 5

Sessions

# Active sessions by state
clawdbot_session_state{state="idle"} 10
clawdbot_session_state{state="processing"} 2
clawdbot_session_state{state="waiting"} 1

Queue

# Queue depth
clawdbot_queue_depth{lane="default"} 3

System

# Gateway uptime
clawdbot_gateway_uptime_seconds 3600

Prometheus Configuration

# prometheus.yml
scrape_configs:
  - job_name: 'clawdbot'
    static_configs:
      - targets: ['gateway-host:18789']
    bearer_token: '<gateway-token>'
    metrics_path: '/metrics'

Grafana Dashboard

Import this dashboard for key metrics:

{
  "title": "Moltbot Gateway",
  "panels": [
    {
      "title": "Token Usage Rate",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(clawdbot_tokens_total[5m])",
          "legendFormat": "{{type}}"
        }
      ]
    },
    {
      "title": "Message Throughput",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(clawdbot_message_processed_total[5m])",
          "legendFormat": "{{channel}} - {{outcome}}"
        }
      ]
    },
    {
      "title": "Cost per Hour",
      "type": "stat",
      "targets": [
        {
          "expr": "increase(clawdbot_cost_usd_total[1h])"
        }
      ]
    },
    {
      "title": "Active Sessions",
      "type": "gauge",
      "targets": [
        {
          "expr": "sum(clawdbot_session_state)"
        }
      ]
    }
  ]
}

Distributed Tracing

The gateway supports W3C Trace Context for distributed tracing.

Trace ID Propagation

All requests include a trace ID that flows through:

Audit logs (traceId field)
Diagnostic events
Console/file logs
Outgoing HTTP requests

Log Correlation

Trace IDs appear in logs for correlation:

[gateway] [a1b2c3d4] Processing message from telegram
[agent] [a1b2c3d4] Invoking claude-sonnet-4-20250514
[gateway] [a1b2c3d4] Message completed in 2.3s

Audit Log Correlation

Query audit logs by trace ID:

cat ~/.clawdbot/audit.jsonl | jq 'select(.traceId == "4bf92f3577b34da6a3ce929d0e0e4736")'

OpenTelemetry Integration

For full APM integration, use the diagnostics-otel extension:

plugins:
  diagnostics-otel:
    enabled: true
    endpoint: "http://otel-collector:4317"

This exports:

Traces to your APM (Jaeger, Zipkin, Datadog, etc.)
Metrics in OTLP format
Logs with trace correlation

Logging

Log Levels

Configure log verbosity:

logging:
  level: info  # trace, debug, info, warn, error, silent
  console:
    level: warn  # Separate console level
    style: pretty  # pretty, compact, json

Structured Logging

JSON log format for log aggregation:

logging:
  console:
    style: json

Output:

{"time":"2024-01-15T10:30:00.000Z","level":"info","subsystem":"gateway","message":"Gateway started","traceId":"a1b2c3d4"}

Log Files

Logs are written to ~/.clawdbot/logs/:

~/.clawdbot/logs/
├── clawdbot.log        # Current log
├── clawdbot.1.log      # Rotated logs
└── clawdbot.2.log

Log Aggregation

Ship logs to your aggregation system:

Filebeat:

filebeat.inputs:
  - type: log
    paths:
      - /root/.clawdbot/logs/*.log
    json.keys_under_root: true
    json.add_error_key: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Fluentd:

<source>
  @type tail
  path /root/.clawdbot/logs/*.log
  pos_file /var/log/fluentd/clawdbot.pos
  tag clawdbot
  <parse>
    @type json
  </parse>
</source>

Alerting

Prometheus Alerting Rules

# alerts.yml
groups:
  - name: clawdbot
    rules:
      - alert: ClawdbotDown
        expr: up{job="clawdbot"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Moltbot gateway is down"

      - alert: HighErrorRate
        expr: rate(clawdbot_message_processed_total{outcome="error"}[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High message error rate"

      - alert: ChannelDisconnected
        expr: clawdbot_channel_connected == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Channel {{ $labels.channel }} disconnected"

      - alert: HighCostRate
        expr: increase(clawdbot_cost_usd_total[1h]) > 10
        for: 1m
        labels:
          severity: info
        annotations:
          summary: "High API cost in last hour: ${{ $value }}"

Key Metrics to Monitor

Metric	Alert Threshold	Severity
Gateway uptime	Down > 1m	Critical
Message error rate	> 10%	Warning
Channel disconnected	> 5m	Warning
Hourly cost	> $10	Info
Queue depth	> 100	Warning
Processing latency p99	> 30s	Warning

Diagnostic Events

The gateway emits real-time diagnostic events for monitoring:

// Subscribe to diagnostic events
import { onDiagnosticEvent } from 'clawdbot/diagnostic-events';

onDiagnosticEvent((event) => {
  console.log(event.type, event);
});

Event Types

Event	Description
`model.usage`	Token usage and cost
`webhook.received`	Incoming webhook
`webhook.processed`	Webhook processing complete
`webhook.error`	Webhook processing error
`message.queued`	Message added to queue
`message.processed`	Message processing complete
`session.state`	Session state change
`session.stuck`	Session stuck detection
`diagnostic.heartbeat`	Periodic status summary

Troubleshooting

Common Issues

No metrics at /metrics:

Check authentication header
Verify gateway.metrics.enabled: true

Missing trace IDs:

Ensure requests flow through the gateway
Check log format includes trace context

High memory usage:

Review session retention settings
Check for stuck sessions
Monitor queue depth

Debug Commands

# Check gateway health
curl -s http://localhost:18789/health | jq

# View recent audit events
tail -100 ~/.clawdbot/audit.jsonl | jq

# Check channel status
clawdbot channels status --deep

# View diagnostic events
clawdbot gateway events --follow

Next Steps

Enterprise Deployment - Deployment patterns
Security Hardening - Security configuration
Gateway Troubleshooting - Detailed troubleshooting

9.1 KiB Raw Blame History