openclaw/apps/android/ASSISTANT-APP-PLAN.md

Android Assistant App Support for Clawdbot

Overview

Implement Android's VoiceInteractionService to make Clawdbot appear in "Settings > Apps > Default Apps > Digital assistant app" like ChatGPT and Perplexity. When users long-press the home button, Clawdbot will launch with screen context captured via the Assist API.

Architecture Decisions (User-Confirmed)

1. UI: Launch MainActivity with chat overlay expanded (reuse existing UI)
2. Screen Context: Always capture via Assist API and include in query
3. Service: Extend NodeForegroundService to also be a VoiceInteractionService
4. Offline: Show error dialog prompting user to connect to gateway

Implementation Flow

User long-presses Home
  ↓
Android invokes VoiceInteractionSession.onHandleAssist()
  ↓
Capture screen context (AssistStructure + AssistContent) → JSON
  ↓
Launch MainActivity with Intent (action=ACTION_ASSISTANT, context JSON)
  ↓
Check gateway connection
  ├─ Connected: Open chat sheet with context message
  └─ Offline: Show error dialog with "Open Settings" button

Implementation Steps

1. AndroidManifest Changes

File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/AndroidManifest.xml

Add permission:

<uses-permission android:name="android.permission.BIND_VOICE_INTERACTION" />

Update NodeForegroundService declaration:

<service
    android:name=".NodeForegroundService"
    android:exported="true"
    android:foregroundServiceType="dataSync|microphone|mediaProjection"
    android:permission="android.permission.BIND_VOICE_INTERACTION">
    <intent-filter>
        <action android:name="android.service.voice.VoiceInteractionService" />
    </intent-filter>
    <meta-data
        android:name="android.voice_interaction"
        android:resource="@xml/voice_interaction_service" />
</service>

Add VoiceInteractionSessionService:

<service
    android:name=".assistant.ClawdbotVoiceInteractionSessionService"
    android:exported="false"
    android:permission="android.permission.BIND_VOICE_INTERACTION" />

Update MainActivity for assistant intent:

<activity
    android:name=".MainActivity"
    android:exported="true"
    android:launchMode="singleTop">
    <!-- Existing MAIN/LAUNCHER intent-filter stays -->
    <intent-filter>
        <action android:name="com.clawdbot.android.ACTION_ASSISTANT" />
        <category android:name="android.intent.category.DEFAULT" />
    </intent-filter>
</activity>

2. Create XML Resource

New File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/res/xml/voice_interaction_service.xml

<?xml version="1.0" encoding="utf-8"?>
<voice-interaction-service
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:sessionService=".assistant.ClawdbotVoiceInteractionSessionService"
    android:supportsAssist="true"
    android:supportsLaunchVoiceAssistFromKeyguard="false"
    android:supportsLocalInteraction="false" />

3. Update NodeForegroundService

File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/NodeForegroundService.kt

Change line 23 from:

class NodeForegroundService : Service() {

To:

class NodeForegroundService : VoiceInteractionService() {

Add import:

import android.service.voice.VoiceInteractionService

Add lifecycle callbacks (after line 66, in the class body):

override fun onReady() {
    super.onReady()
    android.util.Log.i("ClawdbotVoiceInteraction", "VoiceInteractionService ready")
}

override fun onShutdown() {
    android.util.Log.i("ClawdbotVoiceInteraction", "VoiceInteractionService shutdown")
    super.onShutdown()
}

4. Create VoiceInteractionSessionService

New File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/assistant/ClawdbotVoiceInteractionSessionService.kt

package com.clawdbot.android.assistant

import android.os.Bundle
import android.service.voice.VoiceInteractionSession
import android.service.voice.VoiceInteractionSessionService

class ClawdbotVoiceInteractionSessionService : VoiceInteractionSessionService() {
    override fun onNewSession(args: Bundle?): VoiceInteractionSession {
        android.util.Log.i("ClawdbotAssistant", "Creating new VoiceInteractionSession")
        return ClawdbotVoiceInteractionSession(this)
    }
}

5. Create VoiceInteractionSession

New File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/assistant/ClawdbotVoiceInteractionSession.kt

Core logic:

• Capture screen context from AssistStructure (view hierarchy text) and AssistContent (URLs, structured data)
• Serialize to JSON (timestamp, packageName, webUri, visibleText array)
• Launch MainActivity with ACTION_ASSISTANT intent + context JSON extra
• Extract up to 100 text nodes to avoid overwhelming the chat
• Finish session immediately after launching activity

Key features:

• onHandleAssist(): Main entry point, captures context and launches MainActivity
• captureScreenContext(): Builds JSON from AssistStructure/AssistContent
• extractTextFromViewNode(): Recursively walks view hierarchy for text
• Error handling with try-catch and fallback

6. MainActivity Integration

File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/MainActivity.kt

Add in onCreate() (after line 37):

handleAssistantIntent(intent)

Add onNewIntent() method (for singleTop launch mode):

override fun onNewIntent(intent: Intent) {
    super.onNewIntent(intent)
    setIntent(intent)
    handleAssistantIntent(intent)
}

Add helper methods:

private fun handleAssistantIntent(intent: Intent?) {
    if (intent?.action == "com.clawdbot.android.ACTION_ASSISTANT") {
        val screenContext = intent.getStringExtra("screen_context")
        lifecycleScope.launch {
            delay(100) // Wait for runtime initialization
            if (!viewModel.isConnected.value) {
                showAssistantOfflineDialog()
            } else {
                viewModel.openChatWithContext(screenContext)
            }
        }
    }
}

private fun showAssistantOfflineDialog() {
    androidx.appcompat.app.AlertDialog.Builder(this)
        .setTitle("Gateway Not Connected")
        .setMessage("Clawdbot Assistant requires a connection to your gateway. Please connect first.")
        .setPositiveButton("Open Settings") { _, _ ->
            viewModel.requestShowSettings()
        }
        .setNegativeButton("Cancel", null)
        .show()
}

Add imports:

import android.content.Intent
import androidx.appcompat.app.AlertDialog
import kotlinx.coroutines.delay

7. MainViewModel Updates

File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/MainViewModel.kt

Add state flows:

private val _assistantContext = MutableStateFlow<String?>(null)
val assistantContext: StateFlow<String?> = _assistantContext.asStateFlow()

private val _showSettingsRequest = MutableStateFlow(false)
val showSettingsRequest: StateFlow<Boolean> = _showSettingsRequest.asStateFlow()

Add methods:

fun openChatWithContext(screenContext: String?) {
    _assistantContext.value = screenContext
}

fun requestShowSettings() {
    _showSettingsRequest.value = true
}

fun clearShowSettingsRequest() {
    _showSettingsRequest.value = false
}

fun clearAssistantContext() {
    _assistantContext.value = null
}

8. RootScreen UI Integration

File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/ui/RootScreen.kt

Add state collection:

val assistantContext by viewModel.assistantContext.collectAsState()
val showSettingsRequest by viewModel.showSettingsRequest.collectAsState()

Add LaunchedEffect to auto-open chat:

LaunchedEffect(assistantContext) {
    if (assistantContext != null) {
        sheet = Sheet.Chat
    }
}

LaunchedEffect(showSettingsRequest) {
    if (showSettingsRequest) {
        sheet = Sheet.Settings
        viewModel.clearShowSettingsRequest()
    }
}

9. ChatSheet Context Formatting

File: /home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/ui/chat/ChatSheetContent.kt

Add state collection and auto-send:

val assistantContext by viewModel.assistantContext.collectAsState()

LaunchedEffect(assistantContext) {
    assistantContext?.let { contextJson ->
        val formattedMessage = formatAssistantMessage(contextJson)
        viewModel.sendChat(
            message = formattedMessage,
            thinking = thinkingLevel,
            attachments = emptyList()
        )
        viewModel.clearAssistantContext()
    }
}

Add formatter helper:

private fun formatAssistantMessage(contextJson: String): String {
    return try {
        val json = Json.parseToJsonElement(contextJson).jsonObject
        buildString {
            appendLine("I'm looking at:")
            json["packageName"]?.jsonPrimitive?.contentOrNull?.let {
                appendLine("App: $it")
            }
            json["webUri"]?.jsonPrimitive?.contentOrNull?.let {
                appendLine("URL: $it")
            }
            json["visibleText"]?.jsonArray?.take(20)?.forEach {
                it.jsonPrimitive?.contentOrNull?.let { text ->
                    appendLine("- $text")
                }
            }
            appendLine("\nScreen context JSON:")
            appendLine("```json")
            appendLine(contextJson)
            appendLine("```")
        }
    } catch (e: Exception) {
        "Screen context:\n```json\n$contextJson\n```"
    }
}

Testing Plan

Setup

1. Install debug build: ./gradlew installDebug
2. Open Settings > Apps > Default Apps > Digital assistant app
3. Select "Clawdbot"

Test Cases

Test 1: Basic Invocation (Connected)

• Precondition: Connected to gateway
• Open any app (Chrome, Gmail)
• Long-press Home button
• Expected: MainActivity opens, chat sheet shows, message auto-sent with screen context

Test 2: Offline Invocation

• Precondition: NOT connected to gateway
• Long-press Home button
• Expected: Alert dialog "Gateway Not Connected" with "Open Settings" button

Test 3: Screen Context - Web Browser

• Open Chrome with webpage
• Invoke assistant
• Expected: Context includes webUri, packageName, visible text from page

Test 4: Multiple Invocations

• Invoke from one app, let response complete
• Switch to different app
• Invoke again
• Expected: New context from second app

Test 5: Service Persistence

• Reboot device
• Check Settings > Apps > Default Apps
• Expected: Clawdbot still appears as option

Debugging

# View logs
adb logcat | grep -E "Clawdbot|VoiceInteraction|Assistant"

# Check service status
adb shell dumpsys activity services | grep -A 20 clawdbot

# Verify manifest
aapt dump xmltree app/build/outputs/apk/debug/app-debug.apk AndroidManifest.xml | grep -A 10 voice

Critical Files Summary

To Modify (5 files)

1. AndroidManifest.xml - Add permissions, service declarations, metadata
2. NodeForegroundService.kt - Extend VoiceInteractionService, add lifecycle methods
3. MainActivity.kt - Handle assistant intents, show offline dialog
4. MainViewModel.kt - Add assistant context state flows and methods
5. RootScreen.kt - Auto-open chat sheet when context provided

To Create (3 files)

1. res/xml/voice_interaction_service.xml - Voice interaction metadata
2. assistant/ClawdbotVoiceInteractionSessionService.kt - Session service (lightweight)
3. assistant/ClawdbotVoiceInteractionSession.kt - Core session logic (screen context capture)

Optional Enhancement

• ChatSheetContent.kt - Format context message for better UX (can use basic formatting initially)

Key Technical Notes

• VoiceInteractionService is a Service subclass, so extending it from NodeForegroundService is fully compatible
• BIND_VOICE_INTERACTION permission is automatically granted when app is set as default assistant
• AssistStructure can be large; extract data immediately and discard object
• Screen context limited to 100 text nodes to prevent overwhelming chat
• Intent flags use SINGLE_TOP to reuse existing MainActivity instance
• Error handling: Wrap all Assist API calls in try-catch for robustness

Security & Privacy

• Screen context may contain sensitive data (passwords, PII)
• Data only sent to user's own gateway (existing trust model)
• No local storage of screen context (memory only)
• User must explicitly invoke assistant (intentional action)
• Offline mode prevents transmission if not connected

Success Criteria

✅ Clawdbot appears in "Digital assistant app" settings ✅ Long-press home launches Clawdbot with chat ✅ Screen context captured and formatted in message ✅ Offline handling shows error dialog ✅ Existing app functionality unaffected ✅ Service lifecycle stable across reboots