openclaw/docs/architecture/whatsapp-message-flow.md

WhatsApp Message Flow Architecture

This document describes the complete lifecycle of a WhatsApp message in Moltbot, from reception through AI processing to response delivery.

Overview

Moltbot uses the Baileys library to connect to WhatsApp Web. When a message arrives, it flows through several layers before reaching the Pi agent framework for AI processing, then returns through a dispatcher for delivery.

┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐    ┌─────────┐
│WhatsApp │───▶│ Baileys │───▶│ Monitor │───▶│ Router  │───▶│Pi Agent │
│ Server  │    │ Socket  │    │ Layer   │    │         │    │ Runtime │
└─────────┘    └─────────┘    └─────────┘    └─────────┘    └─────────┘
     ▲                                                            │
     │                                                            │
     └────────────────────────────────────────────────────────────┘
                         Response Delivery

Layer Architecture

Layer	Purpose	Key Files
Baileys	WhatsApp Web protocol	src/web/session.ts
Monitor	Message extraction & deduplication	src/web/inbound/monitor.ts
Channel	Connection orchestration	src/web/auto-reply/monitor.ts
Router	Agent & session resolution	src/routing/resolve-route.ts
Processor	Context building & dispatch	src/web/auto-reply/monitor/process-message.ts
Agent	AI execution via Pi framework	src/agents/pi-embedded-runner/run.ts
Dispatcher	Response queuing & delivery	src/auto-reply/reply/reply-dispatcher.ts

Detailed Flow

Step 1: Baileys Connection

File: src/web/session.ts Function: createWaSocket()

The Baileys socket manages the WhatsApp Web connection:

// Creates socket with multi-file auth state
const { state, saveCreds } = await useMultiFileAuthState(authDir)
const { version } = await fetchLatestBaileysVersion()

const sock = makeWASocket({
  version,
  auth: state,
  printQRInTerminal: printQr,
  // ... other options
})

Responsibilities:

• QR code authentication for first-time login
• Credential persistence in ~/.clawdbot/sessions/
• Connection state management (connecting, open, close)
• Event emission for messages.upsert, connection.update, creds.update

---

Step 2: Message Reception

File: src/web/inbound/monitor.ts Function: monitorWebInbox()

When Baileys emits a messages.upsert event, handleMessagesUpsert() processes it:

sock.ev.on('messages.upsert', async ({ messages, type }) => {
  for (const msg of messages) {
    // Filter status/broadcast messages
    if (isStatusMessage(msg)) continue

    // Deduplicate
    if (isRecentInboundMessage(msg.key.id)) continue

    // Extract content
    const body = extractText(msg)
    const mediaPath = await downloadInboundMedia(msg)
    const location = extractLocationData(msg)

    // Build WebInboundMsg and call handler
    onMessage(webInboundMsg)
  }
})

Extracts:

• Message text (body)
• Sender info (senderJid, senderE164, senderName)
• Group metadata (groupSubject, groupParticipants)
• Media content (mediaPath, mediaType, mediaUrl)
• Reply context (replyToId, replyToBody, replyToSender)
• Location data (latitude, longitude)
• Mentions (mentionedJids, wasMentioned)

---

Step 3: Channel Monitoring

File: src/web/auto-reply/monitor.ts Function: monitorWebChannel()

Orchestrates the WhatsApp channel lifecycle:

export async function monitorWebChannel(
  verbose: boolean,
  listenerFactory: ListenerFactory,
  keepAlive: KeepAliveController,
  replyResolver: ReplyResolver,
  runtime: ChannelRuntime,
  abortSignal?: AbortSignal,
  tuning?: MonitorTuning
) {
  // Load config
  const config = await loadConfig()

  // Create message handler
  const onMessage = createWebOnMessageHandler({ ... })

  // Start monitoring with reconnect logic
  while (!abortSignal?.aborted) {
    const listener = await monitorWebInbox({ onMessage, ... })
    // Handle disconnects with exponential backoff
  }
}

Responsibilities:

• Connection retry with exponential backoff
• Heartbeat logging
• Watchdog timer for 30+ minute inactivity
• Status tracking (connected, reconnectAttempts, lastMessageAt)

---

Step 4: Message Handling & Routing

File: src/web/auto-reply/monitor/on-message.ts Function: createWebOnMessageHandler()

File: src/routing/resolve-route.ts Function: resolveAgentRoute()

Routes the message to the correct agent:

// Resolve conversation and peer IDs
const conversationId = resolveConversationId(msg)
const peerId = resolvePeerId(msg)

// Route to agent
const route = await resolveAgentRoute({
  channel: 'whatsapp',
  accountId: msg.accountId,
  peer: { kind: msg.chatType, id: peerId }
})

// For groups, apply gating
if (msg.chatType === 'group') {
  const gateResult = applyGroupGating(msg, route, config)
  if (gateResult.blocked) return
}

// Process message with resolved route
await processMessage({ msg, route, ... })

Routing Logic:

• Matches against configured bindings (peer, guild, team, account, channel)
• Falls back to default agent if no match
• Returns agentId and sessionKey for persistence

Group Gating:

• Checks if bot was mentioned
• Validates against allowlist
• Applies activation rules

---

Step 5: Message Processing

File: src/web/auto-reply/monitor/process-message.ts Function: processMessage()

Builds context and dispatches to the agent:

export async function processMessage(params: ProcessMessageParams) {
  const { msg, route, ... } = params

  // Build history context from recent messages
  const historyContext = await buildHistoryContextFromEntries(...)

  // Check command authorization
  const authorized = await resolveWhatsAppCommandAuthorized(msg, config)

  // Build MsgContext
  const ctx: MsgContext = {
    body: msg.body,
    from: msg.from,
    agentId: route.agentId,
    sessionKey: route.sessionKey,
    mediaPath: msg.mediaPath,
    // ... other fields
  }

  // Send ACK reaction (optional)
  await maybeSendAckReaction(msg, config)

  // Dispatch to agent
  await dispatchReplyWithBufferedBlockDispatcher({
    ctx,
    onBlockReply: (reply) => deliverWebReply(msg, reply),
    // ...
  })
}

---

Step 6: Agent Execution

File: src/auto-reply/reply/get-reply.ts Function: getReplyFromConfig()

File: src/auto-reply/reply/get-reply-run.ts Function: runPreparedReply()

Sets up and runs the AI agent:

export async function getReplyFromConfig(ctx: MsgContext, opts: ReplyOpts) {
  // Resolve model (Claude, GPT, Gemini, etc.)
  const model = await resolveDefaultModel(ctx.agentId)

  // Ensure agent workspace exists
  await ensureAgentWorkspace(ctx.agentId)

  // Create typing controller
  const typing = createTypingController(ctx.sendComposing)

  // Apply media understanding (images, audio, video, PDFs)
  const mediaContext = await applyMediaUnderstanding(ctx.mediaPath, model)

  // Apply link understanding (URL previews)
  const linkContext = await applyLinkUnderstanding(ctx.body)

  // Run the agent
  return runPreparedReply({ ctx, model, typing, ... })
}

---

Step 7: Pi Framework Execution

File: src/agents/pi-embedded-runner/run.ts Function: runEmbeddedPiAgent()

Executes the AI agent using the Pi framework:

export async function runEmbeddedPiAgent(
  params: RunEmbeddedPiAgentParams
): Promise<EmbeddedPiRunResult> {
  // Resolve auth profile and API key
  const apiKey = await getApiKeyForModel(params.model)

  // Load or create session
  const session = await loadSession(params.sessionKey)

  // Build system prompt with skills
  const systemPrompt = await createSystemPromptOverride(params)

  // Run agent attempt with streaming
  const result = await runEmbeddedAttempt({
    session,
    systemPrompt,
    message: params.message,
    tools: params.tools,
    onChunk: (chunk) => params.onBlockReply?.(chunk),
    onToolCall: (call) => params.onToolResult?.(call),
  })

  // Save session transcript
  await saveSession(params.sessionKey, result.session)

  return result
}

Pi Framework Components:

• @mariozechner/pi-agent-core - Agent loop, tool execution
• @mariozechner/pi-ai - Multi-provider LLM API
• @mariozechner/pi-coding-agent - Session management, tools
• @mariozechner/pi-tui - Terminal UI (optional)

Default Tools:

• read - Read files
• write - Write files
• edit - Edit files
• bash - Execute shell commands
• Custom skills from skills/ directory

---

Step 8: Response Dispatch

File: src/auto-reply/reply/reply-dispatcher.ts Function: createReplyDispatcher()

Queues and delivers responses:

export function createReplyDispatcher(options: DispatcherOptions) {
  const queue: ReplyPayload[] = []
  let sendChain = Promise.resolve()

  return {
    sendBlockReply(reply: ReplyPayload) {
      queue.push(reply)
      sendChain = sendChain.then(() => deliver(reply))
    },

    sendToolResult(result: ToolResult) {
      // Optional: send tool updates to user
    },

    sendFinalReply(reply: ReplyPayload) {
      queue.push(reply)
      sendChain = sendChain.then(() => deliver(reply))
    },

    async waitForIdle() {
      await sendChain
    }
  }
}

Features:

• Serializes deliveries to maintain order
• Adds human-like delays between messages
• Tracks pending deliveries for idle signal
• Supports streaming block replies

---

Step 9: WhatsApp Delivery

File: src/web/auto-reply/deliver-reply.ts Function: deliverWebReply()

File: src/web/inbound/send-api.ts Function: createWebSendApi()

Sends the response back to WhatsApp:

export async function deliverWebReply(params: DeliverParams) {
  const { msg, reply } = params

  // Convert markdown tables for WhatsApp
  const text = convertMarkdownTables(reply.text)

  // Chunk long messages
  const chunks = chunkMarkdownTextWithMode(text, 4096)

  for (const chunk of chunks) {
    // Retry with backoff
    await retry(async () => {
      await msg.reply(chunk)
    }, { retries: 3, backoff: 'exponential' })
  }
}

// In send-api.ts
export function createWebSendApi(params: SendApiParams) {
  return {
    async sendMessage(to, text, media) {
      const jid = toWhatsappJid(to)
      const payload = buildPayload(text, media)

      const result = await sock.sendMessage(jid, payload)
      return result.key.id
    }
  }
}

---

Sequence Diagrams

Complete Message Lifecycle

┌──────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐ ┌────────┐ ┌─────────┐
│ WhatsApp │ │ Baileys │ │ Monitor │ │ Router │ │ Agent  │ │   LLM   │
└────┬─────┘ └────┬────┘ └────┬────┘ └───┬────┘ └───┬────┘ └────┬────┘
     │            │           │          │          │           │
     │  message   │           │          │          │           │
     │───────────▶│           │          │          │           │
     │            │           │          │          │           │
     │            │  upsert   │          │          │           │
     │            │──────────▶│          │          │           │
     │            │           │          │          │           │
     │            │           │  route   │          │           │
     │            │           │─────────▶│          │           │
     │            │           │          │          │           │
     │            │           │  agentId │          │           │
     │            │           │◀─────────│          │           │
     │            │           │          │          │           │
     │            │           │  dispatch│          │           │
     │            │           │─────────────────────▶           │
     │            │           │          │          │           │
     │            │           │          │          │  request  │
     │            │           │          │          │──────────▶│
     │            │           │          │          │           │
     │            │           │          │          │  stream   │
     │            │           │          │          │◀──────────│
     │            │           │          │          │           │
     │            │           │  reply   │          │           │
     │            │           │◀─────────────────────           │
     │            │           │          │          │           │
     │            │  send     │          │          │           │
     │            │◀──────────│          │          │           │
     │            │           │          │          │           │
     │  deliver   │           │          │          │           │
     │◀───────────│           │          │          │           │
     │            │           │          │          │           │

Group Message with Mention Check

┌──────────┐ ┌─────────┐ ┌─────────┐ ┌────────┐
│ WhatsApp │ │ Monitor │ │ Gating  │ │ Agent  │
└────┬─────┘ └────┬────┘ └────┬────┘ └───┬────┘
     │            │           │          │
     │ @bot hello │           │          │
     │───────────▶│           │          │
     │            │           │          │
     │            │ is group? │          │
     │            │───────────▶          │
     │            │           │          │
     │            │ mentioned?│          │
     │            │───────────▶          │
     │            │           │          │
     │            │   yes ✓   │          │
     │            │◀───────────          │
     │            │           │          │
     │            │ dispatch  │          │
     │            │──────────────────────▶
     │            │           │          │
     │            │   reply   │          │
     │◀───────────────────────────────────
     │            │           │          │

Tool Execution Flow

┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│  Agent  │ │   LLM   │ │  Tool   │ │Dispatcher│
└────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
     │           │           │           │
     │  request  │           │           │
     │──────────▶│           │           │
     │           │           │           │
     │ tool_call │           │           │
     │◀──────────│           │           │
     │           │           │           │
     │  execute  │           │           │
     │──────────────────────▶│           │
     │           │           │           │
     │  result   │           │           │
     │◀──────────────────────│           │
     │           │           │           │
     │ tool_result           │           │
     │──────────▶│           │           │
     │           │           │           │
     │  continue │           │           │
     │◀──────────│           │           │
     │           │           │           │
     │ final text│           │           │
     │◀──────────│           │           │
     │           │           │           │
     │  deliver  │           │           │
     │──────────────────────────────────▶│
     │           │           │           │

---

Key Data Structures

WebInboundMsg

Represents an incoming WhatsApp message:

interface WebInboundMsg {
  id?: string
  from: string              // Sender JID
  to: string                // Recipient JID
  body: string              // Message text
  chatType: 'dm' | 'group'

  // Sender info
  senderE164?: string       // Phone number (+1234567890)
  senderJid?: string        // WhatsApp JID
  senderName?: string       // Display name
  selfE164?: string         // Bot's phone number

  // Group info
  groupSubject?: string     // Group name
  groupParticipants?: string[]
  groupMembers?: GroupMember[]

  // Media
  mediaPath?: string        // Local file path
  mediaType?: string        // MIME type
  mediaUrl?: string         // Remote URL

  // Reply context
  replyToId?: string        // Quoted message ID
  replyToBody?: string      // Quoted message text
  replyToSender?: string    // Quoted message sender

  // Location
  location?: {
    latitude: number
    longitude: number
    name?: string
    address?: string
  }

  // Mentions
  mentionedJids?: string[]
  wasMentioned?: boolean

  // Account
  accountId: string

  // Methods
  reply(text: string, media?: Buffer): Promise<void>
  sendComposing(): void     // Show typing indicator
}

ResolvedAgentRoute

Routing decision for a message:

interface ResolvedAgentRoute {
  agentId: string           // Target agent ID
  channel: string           // 'whatsapp'
  accountId: string         // WhatsApp account ID
  sessionKey: string        // Unique per conversation
  mainSessionKey: string    // For collapsing multiple DMs
  matchedBy:
    | 'binding.peer'
    | 'binding.guild'
    | 'binding.team'
    | 'binding.account'
    | 'binding.channel'
    | 'default'
}

MsgContext

Context passed to the agent:

interface MsgContext {
  body: string              // User message
  from: string              // Sender identifier
  agentId: string           // Target agent
  sessionKey: string        // Session for persistence

  // Media
  mediaPath?: string
  mediaType?: string
  mediaContext?: string     // AI-generated description

  // Conversation
  historyContext?: string   // Recent message history
  replyToBody?: string      // Quoted message

  // Authorization
  authorized: boolean
  elevated: boolean

  // Callbacks
  sendComposing?: () => void
  onBlockReply?: (reply: ReplyPayload) => void
  onToolResult?: (result: ToolResult) => void
}

ReplyPayload

Response to deliver:

interface ReplyPayload {
  text?: string             // Response text
  mediaUrl?: string         // Single media URL
  mediaUrls?: string[]      // Multiple media URLs
  mediaBuffer?: Buffer      // Raw media data
  mediaType?: string        // MIME type

  // Formatting
  markdown?: boolean        // Enable markdown
  monospace?: boolean       // Code block style
}

---

Configuration

Agent Routing

Configure in ~/.clawdbot/config.json5:

{
  routing: {
    bindings: [
      {
        // Route specific contact to agent
        peer: { kind: 'dm', id: '+1234567890' },
        agentId: 'work-agent'
      },
      {
        // Route group to agent
        peer: { kind: 'group', id: 'group-jid@g.us' },
        agentId: 'group-agent'
      }
    ],
    default: {
      agentId: 'main'  // Fallback agent
    }
  }
}

Group Gating

{
  channels: {
    whatsapp: {
      groups: {
        // Require mention to respond
        requireMention: true,

        // Allowlist specific groups
        allowlist: ['group-jid@g.us'],

        // Or blocklist
        blocklist: ['spam-group@g.us']
      }
    }
  }
}

Media Understanding

{
  agents: {
    defaults: {
      mediaUnderstanding: {
        // Enable vision for images
        images: true,

        // Transcribe audio
        audio: true,

        // Extract video frames
        video: true,

        // Parse PDFs
        documents: true
      }
    }
  }
}

---

Error Handling

Connection Failures

// Exponential backoff for reconnection
const backoff = [1000, 2000, 5000, 10000, 30000, 60000]
let attempt = 0

while (!aborted) {
  try {
    await connect()
    attempt = 0  // Reset on success
  } catch (err) {
    const delay = backoff[Math.min(attempt, backoff.length - 1)]
    await sleep(delay)
    attempt++
  }
}

Message Delivery Failures

// Retry with backoff
await retry(
  () => msg.reply(text),
  {
    retries: 3,
    backoff: 'exponential',
    onRetry: (err, attempt) => {
      log.warn(`Delivery failed (attempt ${attempt}): ${err.message}`)
    }
  }
)

Agent Failures

// Failover to backup model
try {
  return await runWithModel('claude-3-opus')
} catch (err) {
  if (isRateLimitError(err)) {
    return await runWithModel('gpt-4')  // Failover
  }
  throw err
}

---

Observability

Logging

Key log points:

• Connection state changes
• Message reception (with IDs)
• Routing decisions
• Agent execution start/end
• Tool calls
• Delivery confirmation

Metrics

Tracked metrics:

• whatsapp.messages.received - Inbound message count
• whatsapp.messages.sent - Outbound message count
• whatsapp.connection.uptime - Connection duration
• agent.execution.duration - Agent response time
• agent.tokens.used - Token consumption

Health Checks

# Check channel status
moltbot channels status --probe

# Check gateway health
curl http://localhost:18789/health

---

Related Documentation

• Gateway Architecture
• Agent Configuration
• Skills System
• Pi Framework
• Baileys Library