Android: disable VoiceInteractionService for OPPO/ColorOS compatibility
OPPO's PhoneWindowManager sets IsQuickLaunchSupport=false for apps with VoiceInteractionService, blocking long-press power button from triggering the assistant. Disabling the service makes OPPO fall back to the standard ACTION_ASSIST intent path (same as App Manager). - Disabled ClawdbotVoiceInteractionService and SessionService - MainActivity now handles ASSIST intents directly - Tested working on OnePlus 13 (ColorOS)
This commit is contained in:
parent
a3b467167f
commit
b02d042239
411
apps/android/ASSISTANT-APP-PLAN.md
Normal file
411
apps/android/ASSISTANT-APP-PLAN.md
Normal file
@ -0,0 +1,411 @@
|
|||||||
|
# Android Assistant App Support for Clawdbot
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Implement Android's VoiceInteractionService to make Clawdbot appear in "Settings > Apps > Default Apps > Digital assistant app" like ChatGPT and Perplexity. When users long-press the home button, Clawdbot will launch with screen context captured via the Assist API.
|
||||||
|
|
||||||
|
## Architecture Decisions (User-Confirmed)
|
||||||
|
|
||||||
|
1. **UI**: Launch MainActivity with chat overlay expanded (reuse existing UI)
|
||||||
|
2. **Screen Context**: Always capture via Assist API and include in query
|
||||||
|
3. **Service**: Extend NodeForegroundService to also be a VoiceInteractionService
|
||||||
|
4. **Offline**: Show error dialog prompting user to connect to gateway
|
||||||
|
|
||||||
|
## Implementation Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
User long-presses Home
|
||||||
|
↓
|
||||||
|
Android invokes VoiceInteractionSession.onHandleAssist()
|
||||||
|
↓
|
||||||
|
Capture screen context (AssistStructure + AssistContent) → JSON
|
||||||
|
↓
|
||||||
|
Launch MainActivity with Intent (action=ACTION_ASSISTANT, context JSON)
|
||||||
|
↓
|
||||||
|
Check gateway connection
|
||||||
|
├─ Connected: Open chat sheet with context message
|
||||||
|
└─ Offline: Show error dialog with "Open Settings" button
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation Steps
|
||||||
|
|
||||||
|
### 1. AndroidManifest Changes
|
||||||
|
|
||||||
|
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/AndroidManifest.xml`
|
||||||
|
|
||||||
|
Add permission:
|
||||||
|
```xml
|
||||||
|
<uses-permission android:name="android.permission.BIND_VOICE_INTERACTION" />
|
||||||
|
```
|
||||||
|
|
||||||
|
Update NodeForegroundService declaration:
|
||||||
|
```xml
|
||||||
|
<service
|
||||||
|
android:name=".NodeForegroundService"
|
||||||
|
android:exported="true"
|
||||||
|
android:foregroundServiceType="dataSync|microphone|mediaProjection"
|
||||||
|
android:permission="android.permission.BIND_VOICE_INTERACTION">
|
||||||
|
<intent-filter>
|
||||||
|
<action android:name="android.service.voice.VoiceInteractionService" />
|
||||||
|
</intent-filter>
|
||||||
|
<meta-data
|
||||||
|
android:name="android.voice_interaction"
|
||||||
|
android:resource="@xml/voice_interaction_service" />
|
||||||
|
</service>
|
||||||
|
```
|
||||||
|
|
||||||
|
Add VoiceInteractionSessionService:
|
||||||
|
```xml
|
||||||
|
<service
|
||||||
|
android:name=".assistant.ClawdbotVoiceInteractionSessionService"
|
||||||
|
android:exported="false"
|
||||||
|
android:permission="android.permission.BIND_VOICE_INTERACTION" />
|
||||||
|
```
|
||||||
|
|
||||||
|
Update MainActivity for assistant intent:
|
||||||
|
```xml
|
||||||
|
<activity
|
||||||
|
android:name=".MainActivity"
|
||||||
|
android:exported="true"
|
||||||
|
android:launchMode="singleTop">
|
||||||
|
<!-- Existing MAIN/LAUNCHER intent-filter stays -->
|
||||||
|
<intent-filter>
|
||||||
|
<action android:name="com.clawdbot.android.ACTION_ASSISTANT" />
|
||||||
|
<category android:name="android.intent.category.DEFAULT" />
|
||||||
|
</intent-filter>
|
||||||
|
</activity>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Create XML Resource
|
||||||
|
|
||||||
|
**New File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/res/xml/voice_interaction_service.xml`
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<voice-interaction-service
|
||||||
|
xmlns:android="http://schemas.android.com/apk/res/android"
|
||||||
|
android:sessionService=".assistant.ClawdbotVoiceInteractionSessionService"
|
||||||
|
android:supportsAssist="true"
|
||||||
|
android:supportsLaunchVoiceAssistFromKeyguard="false"
|
||||||
|
android:supportsLocalInteraction="false" />
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Update NodeForegroundService
|
||||||
|
|
||||||
|
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/NodeForegroundService.kt`
|
||||||
|
|
||||||
|
Change line 23 from:
|
||||||
|
```kotlin
|
||||||
|
class NodeForegroundService : Service() {
|
||||||
|
```
|
||||||
|
|
||||||
|
To:
|
||||||
|
```kotlin
|
||||||
|
class NodeForegroundService : VoiceInteractionService() {
|
||||||
|
```
|
||||||
|
|
||||||
|
Add import:
|
||||||
|
```kotlin
|
||||||
|
import android.service.voice.VoiceInteractionService
|
||||||
|
```
|
||||||
|
|
||||||
|
Add lifecycle callbacks (after line 66, in the class body):
|
||||||
|
```kotlin
|
||||||
|
override fun onReady() {
|
||||||
|
super.onReady()
|
||||||
|
android.util.Log.i("ClawdbotVoiceInteraction", "VoiceInteractionService ready")
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onShutdown() {
|
||||||
|
android.util.Log.i("ClawdbotVoiceInteraction", "VoiceInteractionService shutdown")
|
||||||
|
super.onShutdown()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Create VoiceInteractionSessionService
|
||||||
|
|
||||||
|
**New File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/assistant/ClawdbotVoiceInteractionSessionService.kt`
|
||||||
|
|
||||||
|
```kotlin
|
||||||
|
package com.clawdbot.android.assistant
|
||||||
|
|
||||||
|
import android.os.Bundle
|
||||||
|
import android.service.voice.VoiceInteractionSession
|
||||||
|
import android.service.voice.VoiceInteractionSessionService
|
||||||
|
|
||||||
|
class ClawdbotVoiceInteractionSessionService : VoiceInteractionSessionService() {
|
||||||
|
override fun onNewSession(args: Bundle?): VoiceInteractionSession {
|
||||||
|
android.util.Log.i("ClawdbotAssistant", "Creating new VoiceInteractionSession")
|
||||||
|
return ClawdbotVoiceInteractionSession(this)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Create VoiceInteractionSession
|
||||||
|
|
||||||
|
**New File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/assistant/ClawdbotVoiceInteractionSession.kt`
|
||||||
|
|
||||||
|
Core logic:
|
||||||
|
- Capture screen context from AssistStructure (view hierarchy text) and AssistContent (URLs, structured data)
|
||||||
|
- Serialize to JSON (timestamp, packageName, webUri, visibleText array)
|
||||||
|
- Launch MainActivity with ACTION_ASSISTANT intent + context JSON extra
|
||||||
|
- Extract up to 100 text nodes to avoid overwhelming the chat
|
||||||
|
- Finish session immediately after launching activity
|
||||||
|
|
||||||
|
Key features:
|
||||||
|
- `onHandleAssist()`: Main entry point, captures context and launches MainActivity
|
||||||
|
- `captureScreenContext()`: Builds JSON from AssistStructure/AssistContent
|
||||||
|
- `extractTextFromViewNode()`: Recursively walks view hierarchy for text
|
||||||
|
- Error handling with try-catch and fallback
|
||||||
|
|
||||||
|
### 6. MainActivity Integration
|
||||||
|
|
||||||
|
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/MainActivity.kt`
|
||||||
|
|
||||||
|
Add in `onCreate()` (after line 37):
|
||||||
|
```kotlin
|
||||||
|
handleAssistantIntent(intent)
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `onNewIntent()` method (for singleTop launch mode):
|
||||||
|
```kotlin
|
||||||
|
override fun onNewIntent(intent: Intent) {
|
||||||
|
super.onNewIntent(intent)
|
||||||
|
setIntent(intent)
|
||||||
|
handleAssistantIntent(intent)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add helper methods:
|
||||||
|
```kotlin
|
||||||
|
private fun handleAssistantIntent(intent: Intent?) {
|
||||||
|
if (intent?.action == "com.clawdbot.android.ACTION_ASSISTANT") {
|
||||||
|
val screenContext = intent.getStringExtra("screen_context")
|
||||||
|
lifecycleScope.launch {
|
||||||
|
delay(100) // Wait for runtime initialization
|
||||||
|
if (!viewModel.isConnected.value) {
|
||||||
|
showAssistantOfflineDialog()
|
||||||
|
} else {
|
||||||
|
viewModel.openChatWithContext(screenContext)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun showAssistantOfflineDialog() {
|
||||||
|
androidx.appcompat.app.AlertDialog.Builder(this)
|
||||||
|
.setTitle("Gateway Not Connected")
|
||||||
|
.setMessage("Clawdbot Assistant requires a connection to your gateway. Please connect first.")
|
||||||
|
.setPositiveButton("Open Settings") { _, _ ->
|
||||||
|
viewModel.requestShowSettings()
|
||||||
|
}
|
||||||
|
.setNegativeButton("Cancel", null)
|
||||||
|
.show()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add imports:
|
||||||
|
```kotlin
|
||||||
|
import android.content.Intent
|
||||||
|
import androidx.appcompat.app.AlertDialog
|
||||||
|
import kotlinx.coroutines.delay
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7. MainViewModel Updates
|
||||||
|
|
||||||
|
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/MainViewModel.kt`
|
||||||
|
|
||||||
|
Add state flows:
|
||||||
|
```kotlin
|
||||||
|
private val _assistantContext = MutableStateFlow<String?>(null)
|
||||||
|
val assistantContext: StateFlow<String?> = _assistantContext.asStateFlow()
|
||||||
|
|
||||||
|
private val _showSettingsRequest = MutableStateFlow(false)
|
||||||
|
val showSettingsRequest: StateFlow<Boolean> = _showSettingsRequest.asStateFlow()
|
||||||
|
```
|
||||||
|
|
||||||
|
Add methods:
|
||||||
|
```kotlin
|
||||||
|
fun openChatWithContext(screenContext: String?) {
|
||||||
|
_assistantContext.value = screenContext
|
||||||
|
}
|
||||||
|
|
||||||
|
fun requestShowSettings() {
|
||||||
|
_showSettingsRequest.value = true
|
||||||
|
}
|
||||||
|
|
||||||
|
fun clearShowSettingsRequest() {
|
||||||
|
_showSettingsRequest.value = false
|
||||||
|
}
|
||||||
|
|
||||||
|
fun clearAssistantContext() {
|
||||||
|
_assistantContext.value = null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8. RootScreen UI Integration
|
||||||
|
|
||||||
|
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/ui/RootScreen.kt`
|
||||||
|
|
||||||
|
Add state collection:
|
||||||
|
```kotlin
|
||||||
|
val assistantContext by viewModel.assistantContext.collectAsState()
|
||||||
|
val showSettingsRequest by viewModel.showSettingsRequest.collectAsState()
|
||||||
|
```
|
||||||
|
|
||||||
|
Add LaunchedEffect to auto-open chat:
|
||||||
|
```kotlin
|
||||||
|
LaunchedEffect(assistantContext) {
|
||||||
|
if (assistantContext != null) {
|
||||||
|
sheet = Sheet.Chat
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
LaunchedEffect(showSettingsRequest) {
|
||||||
|
if (showSettingsRequest) {
|
||||||
|
sheet = Sheet.Settings
|
||||||
|
viewModel.clearShowSettingsRequest()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 9. ChatSheet Context Formatting
|
||||||
|
|
||||||
|
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/ui/chat/ChatSheetContent.kt`
|
||||||
|
|
||||||
|
Add state collection and auto-send:
|
||||||
|
```kotlin
|
||||||
|
val assistantContext by viewModel.assistantContext.collectAsState()
|
||||||
|
|
||||||
|
LaunchedEffect(assistantContext) {
|
||||||
|
assistantContext?.let { contextJson ->
|
||||||
|
val formattedMessage = formatAssistantMessage(contextJson)
|
||||||
|
viewModel.sendChat(
|
||||||
|
message = formattedMessage,
|
||||||
|
thinking = thinkingLevel,
|
||||||
|
attachments = emptyList()
|
||||||
|
)
|
||||||
|
viewModel.clearAssistantContext()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add formatter helper:
|
||||||
|
```kotlin
|
||||||
|
private fun formatAssistantMessage(contextJson: String): String {
|
||||||
|
return try {
|
||||||
|
val json = Json.parseToJsonElement(contextJson).jsonObject
|
||||||
|
buildString {
|
||||||
|
appendLine("I'm looking at:")
|
||||||
|
json["packageName"]?.jsonPrimitive?.contentOrNull?.let {
|
||||||
|
appendLine("App: $it")
|
||||||
|
}
|
||||||
|
json["webUri"]?.jsonPrimitive?.contentOrNull?.let {
|
||||||
|
appendLine("URL: $it")
|
||||||
|
}
|
||||||
|
json["visibleText"]?.jsonArray?.take(20)?.forEach {
|
||||||
|
it.jsonPrimitive?.contentOrNull?.let { text ->
|
||||||
|
appendLine("- $text")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
appendLine("\nScreen context JSON:")
|
||||||
|
appendLine("```json")
|
||||||
|
appendLine(contextJson)
|
||||||
|
appendLine("```")
|
||||||
|
}
|
||||||
|
} catch (e: Exception) {
|
||||||
|
"Screen context:\n```json\n$contextJson\n```"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Plan
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
1. Install debug build: `./gradlew installDebug`
|
||||||
|
2. Open Settings > Apps > Default Apps > Digital assistant app
|
||||||
|
3. Select "Clawdbot"
|
||||||
|
|
||||||
|
### Test Cases
|
||||||
|
|
||||||
|
**Test 1: Basic Invocation (Connected)**
|
||||||
|
- Precondition: Connected to gateway
|
||||||
|
- Open any app (Chrome, Gmail)
|
||||||
|
- Long-press Home button
|
||||||
|
- Expected: MainActivity opens, chat sheet shows, message auto-sent with screen context
|
||||||
|
|
||||||
|
**Test 2: Offline Invocation**
|
||||||
|
- Precondition: NOT connected to gateway
|
||||||
|
- Long-press Home button
|
||||||
|
- Expected: Alert dialog "Gateway Not Connected" with "Open Settings" button
|
||||||
|
|
||||||
|
**Test 3: Screen Context - Web Browser**
|
||||||
|
- Open Chrome with webpage
|
||||||
|
- Invoke assistant
|
||||||
|
- Expected: Context includes webUri, packageName, visible text from page
|
||||||
|
|
||||||
|
**Test 4: Multiple Invocations**
|
||||||
|
- Invoke from one app, let response complete
|
||||||
|
- Switch to different app
|
||||||
|
- Invoke again
|
||||||
|
- Expected: New context from second app
|
||||||
|
|
||||||
|
**Test 5: Service Persistence**
|
||||||
|
- Reboot device
|
||||||
|
- Check Settings > Apps > Default Apps
|
||||||
|
- Expected: Clawdbot still appears as option
|
||||||
|
|
||||||
|
### Debugging
|
||||||
|
```bash
|
||||||
|
# View logs
|
||||||
|
adb logcat | grep -E "Clawdbot|VoiceInteraction|Assistant"
|
||||||
|
|
||||||
|
# Check service status
|
||||||
|
adb shell dumpsys activity services | grep -A 20 clawdbot
|
||||||
|
|
||||||
|
# Verify manifest
|
||||||
|
aapt dump xmltree app/build/outputs/apk/debug/app-debug.apk AndroidManifest.xml | grep -A 10 voice
|
||||||
|
```
|
||||||
|
|
||||||
|
## Critical Files Summary
|
||||||
|
|
||||||
|
### To Modify (5 files)
|
||||||
|
1. `AndroidManifest.xml` - Add permissions, service declarations, metadata
|
||||||
|
2. `NodeForegroundService.kt` - Extend VoiceInteractionService, add lifecycle methods
|
||||||
|
3. `MainActivity.kt` - Handle assistant intents, show offline dialog
|
||||||
|
4. `MainViewModel.kt` - Add assistant context state flows and methods
|
||||||
|
5. `RootScreen.kt` - Auto-open chat sheet when context provided
|
||||||
|
|
||||||
|
### To Create (3 files)
|
||||||
|
1. `res/xml/voice_interaction_service.xml` - Voice interaction metadata
|
||||||
|
2. `assistant/ClawdbotVoiceInteractionSessionService.kt` - Session service (lightweight)
|
||||||
|
3. `assistant/ClawdbotVoiceInteractionSession.kt` - Core session logic (screen context capture)
|
||||||
|
|
||||||
|
### Optional Enhancement
|
||||||
|
- `ChatSheetContent.kt` - Format context message for better UX (can use basic formatting initially)
|
||||||
|
|
||||||
|
## Key Technical Notes
|
||||||
|
|
||||||
|
- **VoiceInteractionService** is a Service subclass, so extending it from NodeForegroundService is fully compatible
|
||||||
|
- **BIND_VOICE_INTERACTION** permission is automatically granted when app is set as default assistant
|
||||||
|
- **AssistStructure** can be large; extract data immediately and discard object
|
||||||
|
- **Screen context** limited to 100 text nodes to prevent overwhelming chat
|
||||||
|
- **Intent flags** use SINGLE_TOP to reuse existing MainActivity instance
|
||||||
|
- **Error handling**: Wrap all Assist API calls in try-catch for robustness
|
||||||
|
|
||||||
|
## Security & Privacy
|
||||||
|
|
||||||
|
- Screen context may contain sensitive data (passwords, PII)
|
||||||
|
- Data only sent to user's own gateway (existing trust model)
|
||||||
|
- No local storage of screen context (memory only)
|
||||||
|
- User must explicitly invoke assistant (intentional action)
|
||||||
|
- Offline mode prevents transmission if not connected
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
✅ Clawdbot appears in "Digital assistant app" settings
|
||||||
|
✅ Long-press home launches Clawdbot with chat
|
||||||
|
✅ Screen context captured and formatted in message
|
||||||
|
✅ Offline handling shows error dialog
|
||||||
|
✅ Existing app functionality unaffected
|
||||||
|
✅ Service lifecycle stable across reboots
|
||||||
@ -115,6 +115,9 @@ dependencies {
|
|||||||
// Unicast DNS-SD (Wide-Area Bonjour) for tailnet discovery domains.
|
// Unicast DNS-SD (Wide-Area Bonjour) for tailnet discovery domains.
|
||||||
implementation("dnsjava:dnsjava:3.6.4")
|
implementation("dnsjava:dnsjava:3.6.4")
|
||||||
|
|
||||||
|
// BouncyCastle for Ed25519 (software provider, no hardware dependency)
|
||||||
|
implementation("org.bouncycastle:bcprov-jdk18on:1.79")
|
||||||
|
|
||||||
testImplementation("junit:junit:4.13.2")
|
testImplementation("junit:junit:4.13.2")
|
||||||
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.10.2")
|
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.10.2")
|
||||||
testImplementation("io.kotest:kotest-runner-junit5-jvm:6.0.7")
|
testImplementation("io.kotest:kotest-runner-junit5-jvm:6.0.7")
|
||||||
|
|||||||
@ -16,6 +16,7 @@
|
|||||||
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||||
<uses-permission android:name="android.permission.SEND_SMS" />
|
<uses-permission android:name="android.permission.SEND_SMS" />
|
||||||
<uses-permission android:name="android.permission.BIND_VOICE_INTERACTION" />
|
<uses-permission android:name="android.permission.BIND_VOICE_INTERACTION" />
|
||||||
|
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
|
||||||
<uses-feature
|
<uses-feature
|
||||||
android:name="android.hardware.camera"
|
android:name="android.hardware.camera"
|
||||||
android:required="false" />
|
android:required="false" />
|
||||||
@ -38,8 +39,11 @@
|
|||||||
android:name=".NodeForegroundService"
|
android:name=".NodeForegroundService"
|
||||||
android:exported="false"
|
android:exported="false"
|
||||||
android:foregroundServiceType="dataSync|microphone|mediaProjection" />
|
android:foregroundServiceType="dataSync|microphone|mediaProjection" />
|
||||||
|
<!-- DISABLED: OPPO/ColorOS blocks VoiceInteractionService with IsQuickLaunchSupport=false
|
||||||
|
Using ASSIST intent filter on MainActivity instead (like App Manager does) -->
|
||||||
<service
|
<service
|
||||||
android:name=".assistant.ClawdbotVoiceInteractionService"
|
android:name=".assistant.ClawdbotVoiceInteractionService"
|
||||||
|
android:enabled="false"
|
||||||
android:exported="true"
|
android:exported="true"
|
||||||
android:permission="android.permission.BIND_VOICE_INTERACTION">
|
android:permission="android.permission.BIND_VOICE_INTERACTION">
|
||||||
<intent-filter>
|
<intent-filter>
|
||||||
@ -51,6 +55,7 @@
|
|||||||
</service>
|
</service>
|
||||||
<service
|
<service
|
||||||
android:name=".assistant.ClawdbotVoiceInteractionSessionService"
|
android:name=".assistant.ClawdbotVoiceInteractionSessionService"
|
||||||
|
android:enabled="false"
|
||||||
android:exported="true"
|
android:exported="true"
|
||||||
android:permission="android.permission.BIND_VOICE_INTERACTION" />
|
android:permission="android.permission.BIND_VOICE_INTERACTION" />
|
||||||
<service
|
<service
|
||||||
@ -68,13 +73,7 @@
|
|||||||
android:excludeFromRecents="true"
|
android:excludeFromRecents="true"
|
||||||
android:noHistory="true"
|
android:noHistory="true"
|
||||||
android:taskAffinity="">
|
android:taskAffinity="">
|
||||||
<intent-filter>
|
<!-- Keep for backward compatibility, but MainActivity now handles ASSIST directly -->
|
||||||
<action android:name="android.intent.action.ASSIST" />
|
|
||||||
<action android:name="android.intent.action.VOICE_ASSIST" />
|
|
||||||
<action android:name="android.intent.action.VOICE_COMMAND" />
|
|
||||||
<action android:name="android.intent.action.SEARCH_LONG_PRESS" />
|
|
||||||
<category android:name="android.intent.category.DEFAULT" />
|
|
||||||
</intent-filter>
|
|
||||||
</activity>
|
</activity>
|
||||||
<activity
|
<activity
|
||||||
android:name=".MainActivity"
|
android:name=".MainActivity"
|
||||||
@ -84,6 +83,14 @@
|
|||||||
<action android:name="android.intent.action.MAIN" />
|
<action android:name="android.intent.action.MAIN" />
|
||||||
<category android:name="android.intent.category.LAUNCHER" />
|
<category android:name="android.intent.category.LAUNCHER" />
|
||||||
</intent-filter>
|
</intent-filter>
|
||||||
|
<!-- Digital Assistant - long press power button -->
|
||||||
|
<intent-filter>
|
||||||
|
<action android:name="android.intent.action.ASSIST" />
|
||||||
|
<action android:name="android.intent.action.VOICE_ASSIST" />
|
||||||
|
<action android:name="android.intent.action.VOICE_COMMAND" />
|
||||||
|
<action android:name="android.intent.action.SEARCH_LONG_PRESS" />
|
||||||
|
<category android:name="android.intent.category.DEFAULT" />
|
||||||
|
</intent-filter>
|
||||||
<intent-filter>
|
<intent-filter>
|
||||||
<action android:name="com.clawdbot.android.ACTION_ASSISTANT" />
|
<action android:name="com.clawdbot.android.ACTION_ASSISTANT" />
|
||||||
<category android:name="android.intent.category.DEFAULT" />
|
<category android:name="android.intent.category.DEFAULT" />
|
||||||
|
|||||||
@ -0,0 +1,44 @@
|
|||||||
|
package com.clawdbot.android.assistant
|
||||||
|
|
||||||
|
import android.content.Intent
|
||||||
|
import android.os.Bundle
|
||||||
|
import android.speech.RecognitionService
|
||||||
|
import android.speech.RecognizerIntent
|
||||||
|
import android.speech.SpeechRecognizer
|
||||||
|
import android.util.Log
|
||||||
|
|
||||||
|
/**
|
||||||
|
* RecognitionService implementation required for VoiceInteractionService.
|
||||||
|
*
|
||||||
|
* Per Android documentation: "Given that all VIAs are also voice recognizer services,
|
||||||
|
* you must also include a RecognitionService in your manifest."
|
||||||
|
*
|
||||||
|
* This is a minimal implementation that satisfies the system requirement.
|
||||||
|
* Actual speech recognition is handled by VoiceInteractionSession.
|
||||||
|
*/
|
||||||
|
class ClawdbotRecognitionService : RecognitionService() {
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val TAG = "ClawdbotRecognition"
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onStartListening(recognizerIntent: Intent?, listener: Callback?) {
|
||||||
|
Log.i(TAG, "onStartListening called")
|
||||||
|
|
||||||
|
// This is a minimal implementation - actual recognition happens in VoiceInteractionSession
|
||||||
|
// We just need this service to exist for the VoiceInteractionService to be valid
|
||||||
|
|
||||||
|
// Notify that we're not implementing speech recognition here
|
||||||
|
listener?.error(SpeechRecognizer.ERROR_SERVER)
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onCancel(listener: Callback?) {
|
||||||
|
Log.i(TAG, "onCancel called")
|
||||||
|
listener?.endOfSpeech()
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onStopListening(listener: Callback?) {
|
||||||
|
Log.i(TAG, "onStopListening called")
|
||||||
|
listener?.endOfSpeech()
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,35 @@
|
|||||||
|
package com.clawdbot.android.assistant
|
||||||
|
|
||||||
|
import android.os.Bundle
|
||||||
|
import android.service.voice.VoiceInteractionService
|
||||||
|
import android.util.Log
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Standalone VoiceInteractionService that makes Clawdbot appear in
|
||||||
|
* "Settings > Apps > Default Apps > Digital assistant app".
|
||||||
|
*
|
||||||
|
* This service must be separate from NodeForegroundService because
|
||||||
|
* VoiceInteractionService cannot have foregroundServiceType attributes.
|
||||||
|
*/
|
||||||
|
class ClawdbotVoiceInteractionService : VoiceInteractionService() {
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val TAG = "ClawdbotVoiceInteraction"
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onCreate() {
|
||||||
|
super.onCreate()
|
||||||
|
Log.i(TAG, "VoiceInteractionService created")
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onReady() {
|
||||||
|
super.onReady()
|
||||||
|
Log.i(TAG, "VoiceInteractionService ready")
|
||||||
|
// The service should now be ready to handle showSession() calls from the system
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onShutdown() {
|
||||||
|
Log.i(TAG, "VoiceInteractionService shutdown")
|
||||||
|
super.onShutdown()
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,149 @@
|
|||||||
|
package com.clawdbot.android.assistant
|
||||||
|
|
||||||
|
import android.app.assist.AssistContent
|
||||||
|
import android.app.assist.AssistStructure
|
||||||
|
import android.content.Context
|
||||||
|
import android.content.Intent
|
||||||
|
import android.graphics.Bitmap
|
||||||
|
import android.os.Bundle
|
||||||
|
import android.service.voice.VoiceInteractionSession
|
||||||
|
import android.util.Log
|
||||||
|
import com.clawdbot.android.MainActivity
|
||||||
|
import kotlinx.serialization.json.JsonPrimitive
|
||||||
|
import kotlinx.serialization.json.buildJsonArray
|
||||||
|
import kotlinx.serialization.json.buildJsonObject
|
||||||
|
import kotlinx.serialization.json.put
|
||||||
|
|
||||||
|
class ClawdbotVoiceInteractionSession(context: Context) : VoiceInteractionSession(context) {
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
private const val TAG = "ClawdbotAssistSession"
|
||||||
|
const val ACTION_ASSISTANT = "com.clawdbot.android.ACTION_ASSISTANT"
|
||||||
|
const val EXTRA_SCREEN_CONTEXT = "screen_context"
|
||||||
|
}
|
||||||
|
|
||||||
|
@Suppress("OVERRIDE_DEPRECATION")
|
||||||
|
override fun onHandleAssist(
|
||||||
|
data: Bundle?,
|
||||||
|
structure: AssistStructure?,
|
||||||
|
content: AssistContent?
|
||||||
|
) {
|
||||||
|
try {
|
||||||
|
Log.i(TAG, "onHandleAssist called")
|
||||||
|
|
||||||
|
// Capture screen context
|
||||||
|
val screenContext = captureScreenContext(structure, content)
|
||||||
|
|
||||||
|
// Launch MainActivity with context
|
||||||
|
val intent = Intent(context, MainActivity::class.java).apply {
|
||||||
|
action = ACTION_ASSISTANT
|
||||||
|
putExtra(EXTRA_SCREEN_CONTEXT, screenContext)
|
||||||
|
flags = Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_SINGLE_TOP
|
||||||
|
}
|
||||||
|
|
||||||
|
context.startActivity(intent)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
Log.e(TAG, "Error handling assist", e)
|
||||||
|
// Show error toast
|
||||||
|
android.widget.Toast.makeText(
|
||||||
|
context,
|
||||||
|
"Clawdbot Assistant error: ${e.message}",
|
||||||
|
android.widget.Toast.LENGTH_LONG
|
||||||
|
).show()
|
||||||
|
} finally {
|
||||||
|
// Close the session (we're delegating to MainActivity)
|
||||||
|
finish()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun onHandleScreenshot(screenshot: Bitmap?) {
|
||||||
|
Log.i(TAG, "Screenshot captured: ${screenshot?.width}x${screenshot?.height}")
|
||||||
|
// Optional: Could save screenshot and include in context
|
||||||
|
// For now, we rely on AssistStructure text extraction
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun captureScreenContext(
|
||||||
|
structure: AssistStructure?,
|
||||||
|
content: AssistContent?
|
||||||
|
): String {
|
||||||
|
val contextData = buildJsonObject {
|
||||||
|
// Basic metadata
|
||||||
|
put("timestamp", System.currentTimeMillis())
|
||||||
|
|
||||||
|
// Content data (URLs, structured data)
|
||||||
|
content?.let { assistContent ->
|
||||||
|
assistContent.webUri?.let { uri ->
|
||||||
|
put("webUri", uri.toString())
|
||||||
|
}
|
||||||
|
assistContent.clipData?.let { clip ->
|
||||||
|
if (clip.itemCount > 0) {
|
||||||
|
put("clipText", clip.getItemAt(0).text?.toString() ?: "")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// JSON-LD structured data (if available)
|
||||||
|
assistContent.structuredData?.let { jsonLd ->
|
||||||
|
put("structuredData", jsonLd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Structure data (view hierarchy, text content)
|
||||||
|
structure?.let { assistStruct ->
|
||||||
|
put("packageName", assistStruct.activityComponent.packageName)
|
||||||
|
put("className", assistStruct.activityComponent.className)
|
||||||
|
|
||||||
|
// Extract visible text from view hierarchy
|
||||||
|
val textNodes = mutableListOf<String>()
|
||||||
|
extractTextFromStructure(assistStruct, textNodes)
|
||||||
|
|
||||||
|
put("visibleText", buildJsonArray {
|
||||||
|
textNodes.forEach { text -> add(JsonPrimitive(text)) }
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return contextData.toString()
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun extractTextFromStructure(
|
||||||
|
structure: AssistStructure,
|
||||||
|
textNodes: MutableList<String>,
|
||||||
|
maxNodes: Int = 100
|
||||||
|
) {
|
||||||
|
if (textNodes.size >= maxNodes) return
|
||||||
|
|
||||||
|
val windowCount = structure.windowNodeCount
|
||||||
|
for (i in 0 until windowCount) {
|
||||||
|
val windowNode = structure.getWindowNodeAt(i)
|
||||||
|
val rootViewNode = windowNode.rootViewNode
|
||||||
|
extractTextFromViewNode(rootViewNode, textNodes, maxNodes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun extractTextFromViewNode(
|
||||||
|
viewNode: AssistStructure.ViewNode,
|
||||||
|
textNodes: MutableList<String>,
|
||||||
|
maxNodes: Int
|
||||||
|
) {
|
||||||
|
if (textNodes.size >= maxNodes) return
|
||||||
|
|
||||||
|
// Extract text from this node
|
||||||
|
viewNode.text?.toString()?.trim()?.takeIf { it.isNotEmpty() }?.let {
|
||||||
|
textNodes.add(it)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract content description
|
||||||
|
viewNode.contentDescription?.toString()?.trim()?.takeIf { it.isNotEmpty() }?.let {
|
||||||
|
if (!textNodes.contains(it)) {
|
||||||
|
textNodes.add(it)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Recursively process children
|
||||||
|
val childCount = viewNode.childCount
|
||||||
|
for (i in 0 until childCount) {
|
||||||
|
viewNode.getChildAt(i)?.let { child ->
|
||||||
|
extractTextFromViewNode(child, textNodes, maxNodes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,12 @@
|
|||||||
|
package com.clawdbot.android.assistant
|
||||||
|
|
||||||
|
import android.os.Bundle
|
||||||
|
import android.service.voice.VoiceInteractionSession
|
||||||
|
import android.service.voice.VoiceInteractionSessionService
|
||||||
|
|
||||||
|
class ClawdbotVoiceInteractionSessionService : VoiceInteractionSessionService() {
|
||||||
|
override fun onNewSession(args: Bundle?): VoiceInteractionSession {
|
||||||
|
android.util.Log.i("ClawdbotAssistant", "Creating new VoiceInteractionSession")
|
||||||
|
return ClawdbotVoiceInteractionSession(this)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -2,10 +2,12 @@ package com.clawdbot.android.gateway
|
|||||||
|
|
||||||
import android.content.Context
|
import android.content.Context
|
||||||
import android.util.Base64
|
import android.util.Base64
|
||||||
|
import org.bouncycastle.jce.provider.BouncyCastleProvider
|
||||||
import java.io.File
|
import java.io.File
|
||||||
import java.security.KeyFactory
|
import java.security.KeyFactory
|
||||||
import java.security.KeyPairGenerator
|
import java.security.KeyPairGenerator
|
||||||
import java.security.MessageDigest
|
import java.security.MessageDigest
|
||||||
|
import java.security.Security
|
||||||
import java.security.Signature
|
import java.security.Signature
|
||||||
import java.security.spec.PKCS8EncodedKeySpec
|
import java.security.spec.PKCS8EncodedKeySpec
|
||||||
import kotlinx.serialization.Serializable
|
import kotlinx.serialization.Serializable
|
||||||
@ -23,6 +25,21 @@ class DeviceIdentityStore(context: Context) {
|
|||||||
private val json = Json { ignoreUnknownKeys = true }
|
private val json = Json { ignoreUnknownKeys = true }
|
||||||
private val identityFile = File(context.filesDir, "clawdbot/identity/device.json")
|
private val identityFile = File(context.filesDir, "clawdbot/identity/device.json")
|
||||||
|
|
||||||
|
companion object {
|
||||||
|
// Use our own provider instance to avoid conflicts with Android's built-in BC
|
||||||
|
private val bcProvider = BouncyCastleProvider()
|
||||||
|
private val ED25519_SPKI_PREFIX = byteArrayOf(
|
||||||
|
0x30, 0x2a, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x03, 0x21, 0x00,
|
||||||
|
)
|
||||||
|
|
||||||
|
init {
|
||||||
|
// Remove any conflicting BC provider and add our full BouncyCastle
|
||||||
|
Security.removeProvider("BC")
|
||||||
|
Security.insertProviderAt(bcProvider, 1)
|
||||||
|
android.util.Log.d("DeviceIdentityStore", "Registered BouncyCastle provider: ${bcProvider.info}")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@Synchronized
|
@Synchronized
|
||||||
fun loadOrCreate(): DeviceIdentity {
|
fun loadOrCreate(): DeviceIdentity {
|
||||||
val existing = load()
|
val existing = load()
|
||||||
@ -44,13 +61,14 @@ class DeviceIdentityStore(context: Context) {
|
|||||||
return try {
|
return try {
|
||||||
val privateKeyBytes = Base64.decode(identity.privateKeyPkcs8Base64, Base64.DEFAULT)
|
val privateKeyBytes = Base64.decode(identity.privateKeyPkcs8Base64, Base64.DEFAULT)
|
||||||
val keySpec = PKCS8EncodedKeySpec(privateKeyBytes)
|
val keySpec = PKCS8EncodedKeySpec(privateKeyBytes)
|
||||||
val keyFactory = KeyFactory.getInstance("Ed25519")
|
val keyFactory = KeyFactory.getInstance("Ed25519", bcProvider)
|
||||||
val privateKey = keyFactory.generatePrivate(keySpec)
|
val privateKey = keyFactory.generatePrivate(keySpec)
|
||||||
val signature = Signature.getInstance("Ed25519")
|
val signature = Signature.getInstance("Ed25519", bcProvider)
|
||||||
signature.initSign(privateKey)
|
signature.initSign(privateKey)
|
||||||
signature.update(payload.toByteArray(Charsets.UTF_8))
|
signature.update(payload.toByteArray(Charsets.UTF_8))
|
||||||
base64UrlEncode(signature.sign())
|
base64UrlEncode(signature.sign())
|
||||||
} catch (_: Throwable) {
|
} catch (e: Throwable) {
|
||||||
|
android.util.Log.e("DeviceIdentityStore", "signPayload failed", e)
|
||||||
null
|
null
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -93,11 +111,18 @@ class DeviceIdentityStore(context: Context) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private fun generate(): DeviceIdentity {
|
private fun generate(): DeviceIdentity {
|
||||||
val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair()
|
android.util.Log.d("DeviceIdentityStore", "Generating new Ed25519 key using BouncyCastle")
|
||||||
|
|
||||||
|
val kpg = KeyPairGenerator.getInstance("Ed25519", bcProvider)
|
||||||
|
val keyPair = kpg.generateKeyPair()
|
||||||
|
|
||||||
val spki = keyPair.public.encoded
|
val spki = keyPair.public.encoded
|
||||||
val rawPublic = stripSpkiPrefix(spki)
|
val rawPublic = stripSpkiPrefix(spki)
|
||||||
val deviceId = sha256Hex(rawPublic)
|
val deviceId = sha256Hex(rawPublic)
|
||||||
val privateKey = keyPair.private.encoded
|
val privateKey = keyPair.private.encoded
|
||||||
|
|
||||||
|
android.util.Log.d("DeviceIdentityStore", "Generated Ed25519 key, deviceId=${deviceId.take(8)}..., pubKeyLen=${rawPublic.size}")
|
||||||
|
|
||||||
return DeviceIdentity(
|
return DeviceIdentity(
|
||||||
deviceId = deviceId,
|
deviceId = deviceId,
|
||||||
publicKeyRawBase64 = Base64.encodeToString(rawPublic, Base64.NO_WRAP),
|
publicKeyRawBase64 = Base64.encodeToString(rawPublic, Base64.NO_WRAP),
|
||||||
@ -136,11 +161,4 @@ class DeviceIdentityStore(context: Context) {
|
|||||||
private fun base64UrlEncode(data: ByteArray): String {
|
private fun base64UrlEncode(data: ByteArray): String {
|
||||||
return Base64.encodeToString(data, Base64.URL_SAFE or Base64.NO_WRAP or Base64.NO_PADDING)
|
return Base64.encodeToString(data, Base64.URL_SAFE or Base64.NO_WRAP or Base64.NO_PADDING)
|
||||||
}
|
}
|
||||||
|
|
||||||
companion object {
|
|
||||||
private val ED25519_SPKI_PREFIX =
|
|
||||||
byteArrayOf(
|
|
||||||
0x30, 0x2a, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x03, 0x21, 0x00,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -253,11 +253,16 @@ class GatewaySession(
|
|||||||
|
|
||||||
private inner class Listener : WebSocketListener() {
|
private inner class Listener : WebSocketListener() {
|
||||||
override fun onOpen(webSocket: WebSocket, response: Response) {
|
override fun onOpen(webSocket: WebSocket, response: Response) {
|
||||||
|
Log.d("ClawdbotGateway", "[$loggerTag] WebSocket opened, endpoint=${endpoint.host}:${endpoint.port}")
|
||||||
scope.launch {
|
scope.launch {
|
||||||
try {
|
try {
|
||||||
|
Log.d("ClawdbotGateway", "[$loggerTag] Awaiting connect nonce...")
|
||||||
val nonce = awaitConnectNonce()
|
val nonce = awaitConnectNonce()
|
||||||
|
Log.d("ClawdbotGateway", "[$loggerTag] Got nonce=$nonce, sending connect...")
|
||||||
sendConnect(nonce)
|
sendConnect(nonce)
|
||||||
|
Log.d("ClawdbotGateway", "[$loggerTag] Connect completed successfully")
|
||||||
} catch (err: Throwable) {
|
} catch (err: Throwable) {
|
||||||
|
Log.e("ClawdbotGateway", "[$loggerTag] Connect failed: ${err.message}", err)
|
||||||
connectDeferred.completeExceptionally(err)
|
connectDeferred.completeExceptionally(err)
|
||||||
closeQuietly()
|
closeQuietly()
|
||||||
}
|
}
|
||||||
@ -373,6 +378,7 @@ class GatewaySession(
|
|||||||
)
|
)
|
||||||
val signature = identityStore.signPayload(payload, identity)
|
val signature = identityStore.signPayload(payload, identity)
|
||||||
val publicKey = identityStore.publicKeyBase64Url(identity)
|
val publicKey = identityStore.publicKeyBase64Url(identity)
|
||||||
|
Log.d("ClawdbotGateway", "[$loggerTag] Device identity: id=${identity.deviceId.take(8)}..., signature=${signature?.take(20) ?: "NULL"}, publicKey=${publicKey?.take(20) ?: "NULL"}")
|
||||||
val deviceJson =
|
val deviceJson =
|
||||||
if (!signature.isNullOrBlank() && !publicKey.isNullOrBlank()) {
|
if (!signature.isNullOrBlank() && !publicKey.isNullOrBlank()) {
|
||||||
buildJsonObject {
|
buildJsonObject {
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user