Android: disable VoiceInteractionService for OPPO/ColorOS compatibility
OPPO's PhoneWindowManager sets IsQuickLaunchSupport=false for apps with VoiceInteractionService, blocking long-press power button from triggering the assistant. Disabling the service makes OPPO fall back to the standard ACTION_ASSIST intent path (same as App Manager). - Disabled ClawdbotVoiceInteractionService and SessionService - MainActivity now handles ASSIST intents directly - Tested working on OnePlus 13 (ColorOS)
This commit is contained in:
yishai
parent
a3b467167f
commit
b02d042239
411
apps/android/ASSISTANT-APP-PLAN.md
Normal file
411
apps/android/ASSISTANT-APP-PLAN.md
Normal file
@ -0,0 +1,411 @@
|
||||
# Android Assistant App Support for Clawdbot
|
||||
|
||||
## Overview
|
||||
|
||||
Implement Android's VoiceInteractionService to make Clawdbot appear in "Settings > Apps > Default Apps > Digital assistant app" like ChatGPT and Perplexity. When users long-press the home button, Clawdbot will launch with screen context captured via the Assist API.
|
||||
|
||||
## Architecture Decisions (User-Confirmed)
|
||||
|
||||
1. **UI**: Launch MainActivity with chat overlay expanded (reuse existing UI)
|
||||
2. **Screen Context**: Always capture via Assist API and include in query
|
||||
3. **Service**: Extend NodeForegroundService to also be a VoiceInteractionService
|
||||
4. **Offline**: Show error dialog prompting user to connect to gateway
|
||||
|
||||
## Implementation Flow
|
||||
|
||||
```
|
||||
User long-presses Home
|
||||
↓
|
||||
Android invokes VoiceInteractionSession.onHandleAssist()
|
||||
↓
|
||||
Capture screen context (AssistStructure + AssistContent) → JSON
|
||||
↓
|
||||
Launch MainActivity with Intent (action=ACTION_ASSISTANT, context JSON)
|
||||
↓
|
||||
Check gateway connection
|
||||
├─ Connected: Open chat sheet with context message
|
||||
└─ Offline: Show error dialog with "Open Settings" button
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### 1. AndroidManifest Changes
|
||||
|
||||
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/AndroidManifest.xml`
|
||||
|
||||
Add permission:
|
||||
```xml
|
||||
<uses-permission android:name="android.permission.BIND_VOICE_INTERACTION" />
|
||||
```
|
||||
|
||||
Update NodeForegroundService declaration:
|
||||
```xml
|
||||
<service
|
||||
android:name=".NodeForegroundService"
|
||||
android:exported="true"
|
||||
android:foregroundServiceType="dataSync|microphone|mediaProjection"
|
||||
android:permission="android.permission.BIND_VOICE_INTERACTION">
|
||||
<intent-filter>
|
||||
<action android:name="android.service.voice.VoiceInteractionService" />
|
||||
</intent-filter>
|
||||
<meta-data
|
||||
android:name="android.voice_interaction"
|
||||
android:resource="@xml/voice_interaction_service" />
|
||||
</service>
|
||||
```
|
||||
|
||||
Add VoiceInteractionSessionService:
|
||||
```xml
|
||||
<service
|
||||
android:name=".assistant.ClawdbotVoiceInteractionSessionService"
|
||||
android:exported="false"
|
||||
android:permission="android.permission.BIND_VOICE_INTERACTION" />
|
||||
```
|
||||
|
||||
Update MainActivity for assistant intent:
|
||||
```xml
|
||||
<activity
|
||||
android:name=".MainActivity"
|
||||
android:exported="true"
|
||||
android:launchMode="singleTop">
|
||||
<!-- Existing MAIN/LAUNCHER intent-filter stays -->
|
||||
<intent-filter>
|
||||
<action android:name="com.clawdbot.android.ACTION_ASSISTANT" />
|
||||
<category android:name="android.intent.category.DEFAULT" />
|
||||
</intent-filter>
|
||||
</activity>
|
||||
```
|
||||
|
||||
### 2. Create XML Resource
|
||||
|
||||
**New File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/res/xml/voice_interaction_service.xml`
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<voice-interaction-service
|
||||
xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
android:sessionService=".assistant.ClawdbotVoiceInteractionSessionService"
|
||||
android:supportsAssist="true"
|
||||
android:supportsLaunchVoiceAssistFromKeyguard="false"
|
||||
android:supportsLocalInteraction="false" />
|
||||
```
|
||||
|
||||
### 3. Update NodeForegroundService
|
||||
|
||||
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/NodeForegroundService.kt`
|
||||
|
||||
Change line 23 from:
|
||||
```kotlin
|
||||
class NodeForegroundService : Service() {
|
||||
```
|
||||
|
||||
To:
|
||||
```kotlin
|
||||
class NodeForegroundService : VoiceInteractionService() {
|
||||
```
|
||||
|
||||
Add import:
|
||||
```kotlin
|
||||
import android.service.voice.VoiceInteractionService
|
||||
```
|
||||
|
||||
Add lifecycle callbacks (after line 66, in the class body):
|
||||
```kotlin
|
||||
override fun onReady() {
|
||||
super.onReady()
|
||||
android.util.Log.i("ClawdbotVoiceInteraction", "VoiceInteractionService ready")
|
||||
}
|
||||
|
||||
override fun onShutdown() {
|
||||
android.util.Log.i("ClawdbotVoiceInteraction", "VoiceInteractionService shutdown")
|
||||
super.onShutdown()
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Create VoiceInteractionSessionService
|
||||
|
||||
**New File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/assistant/ClawdbotVoiceInteractionSessionService.kt`
|
||||
|
||||
```kotlin
|
||||
package com.clawdbot.android.assistant
|
||||
|
||||
import android.os.Bundle
|
||||
import android.service.voice.VoiceInteractionSession
|
||||
import android.service.voice.VoiceInteractionSessionService
|
||||
|
||||
class ClawdbotVoiceInteractionSessionService : VoiceInteractionSessionService() {
|
||||
override fun onNewSession(args: Bundle?): VoiceInteractionSession {
|
||||
android.util.Log.i("ClawdbotAssistant", "Creating new VoiceInteractionSession")
|
||||
return ClawdbotVoiceInteractionSession(this)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Create VoiceInteractionSession
|
||||
|
||||
**New File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/assistant/ClawdbotVoiceInteractionSession.kt`
|
||||
|
||||
Core logic:
|
||||
- Capture screen context from AssistStructure (view hierarchy text) and AssistContent (URLs, structured data)
|
||||
- Serialize to JSON (timestamp, packageName, webUri, visibleText array)
|
||||
- Launch MainActivity with ACTION_ASSISTANT intent + context JSON extra
|
||||
- Extract up to 100 text nodes to avoid overwhelming the chat
|
||||
- Finish session immediately after launching activity
|
||||
|
||||
Key features:
|
||||
- `onHandleAssist()`: Main entry point, captures context and launches MainActivity
|
||||
- `captureScreenContext()`: Builds JSON from AssistStructure/AssistContent
|
||||
- `extractTextFromViewNode()`: Recursively walks view hierarchy for text
|
||||
- Error handling with try-catch and fallback
|
||||
|
||||
### 6. MainActivity Integration
|
||||
|
||||
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/MainActivity.kt`
|
||||
|
||||
Add in `onCreate()` (after line 37):
|
||||
```kotlin
|
||||
handleAssistantIntent(intent)
|
||||
```
|
||||
|
||||
Add `onNewIntent()` method (for singleTop launch mode):
|
||||
```kotlin
|
||||
override fun onNewIntent(intent: Intent) {
|
||||
super.onNewIntent(intent)
|
||||
setIntent(intent)
|
||||
handleAssistantIntent(intent)
|
||||
}
|
||||
```
|
||||
|
||||
Add helper methods:
|
||||
```kotlin
|
||||
private fun handleAssistantIntent(intent: Intent?) {
|
||||
if (intent?.action == "com.clawdbot.android.ACTION_ASSISTANT") {
|
||||
val screenContext = intent.getStringExtra("screen_context")
|
||||
lifecycleScope.launch {
|
||||
delay(100) // Wait for runtime initialization
|
||||
if (!viewModel.isConnected.value) {
|
||||
showAssistantOfflineDialog()
|
||||
} else {
|
||||
viewModel.openChatWithContext(screenContext)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun showAssistantOfflineDialog() {
|
||||
androidx.appcompat.app.AlertDialog.Builder(this)
|
||||
.setTitle("Gateway Not Connected")
|
||||
.setMessage("Clawdbot Assistant requires a connection to your gateway. Please connect first.")
|
||||
.setPositiveButton("Open Settings") { _, _ ->
|
||||
viewModel.requestShowSettings()
|
||||
}
|
||||
.setNegativeButton("Cancel", null)
|
||||
.show()
|
||||
}
|
||||
```
|
||||
|
||||
Add imports:
|
||||
```kotlin
|
||||
import android.content.Intent
|
||||
import androidx.appcompat.app.AlertDialog
|
||||
import kotlinx.coroutines.delay
|
||||
```
|
||||
|
||||
### 7. MainViewModel Updates
|
||||
|
||||
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/MainViewModel.kt`
|
||||
|
||||
Add state flows:
|
||||
```kotlin
|
||||
private val _assistantContext = MutableStateFlow<String?>(null)
|
||||
val assistantContext: StateFlow<String?> = _assistantContext.asStateFlow()
|
||||
|
||||
private val _showSettingsRequest = MutableStateFlow(false)
|
||||
val showSettingsRequest: StateFlow<Boolean> = _showSettingsRequest.asStateFlow()
|
||||
```
|
||||
|
||||
Add methods:
|
||||
```kotlin
|
||||
fun openChatWithContext(screenContext: String?) {
|
||||
_assistantContext.value = screenContext
|
||||
}
|
||||
|
||||
fun requestShowSettings() {
|
||||
_showSettingsRequest.value = true
|
||||
}
|
||||
|
||||
fun clearShowSettingsRequest() {
|
||||
_showSettingsRequest.value = false
|
||||
}
|
||||
|
||||
fun clearAssistantContext() {
|
||||
_assistantContext.value = null
|
||||
}
|
||||
```
|
||||
|
||||
### 8. RootScreen UI Integration
|
||||
|
||||
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/ui/RootScreen.kt`
|
||||
|
||||
Add state collection:
|
||||
```kotlin
|
||||
val assistantContext by viewModel.assistantContext.collectAsState()
|
||||
val showSettingsRequest by viewModel.showSettingsRequest.collectAsState()
|
||||
```
|
||||
|
||||
Add LaunchedEffect to auto-open chat:
|
||||
```kotlin
|
||||
LaunchedEffect(assistantContext) {
|
||||
if (assistantContext != null) {
|
||||
sheet = Sheet.Chat
|
||||
}
|
||||
}
|
||||
|
||||
LaunchedEffect(showSettingsRequest) {
|
||||
if (showSettingsRequest) {
|
||||
sheet = Sheet.Settings
|
||||
viewModel.clearShowSettingsRequest()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 9. ChatSheet Context Formatting
|
||||
|
||||
**File**: `/home/yishai/code_projects/clawdbot/apps/android/app/src/main/java/com/clawdbot/android/ui/chat/ChatSheetContent.kt`
|
||||
|
||||
Add state collection and auto-send:
|
||||
```kotlin
|
||||
val assistantContext by viewModel.assistantContext.collectAsState()
|
||||
|
||||
LaunchedEffect(assistantContext) {
|
||||
assistantContext?.let { contextJson ->
|
||||
val formattedMessage = formatAssistantMessage(contextJson)
|
||||
viewModel.sendChat(
|
||||
message = formattedMessage,
|
||||
thinking = thinkingLevel,
|
||||
attachments = emptyList()
|
||||
)
|
||||
viewModel.clearAssistantContext()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Add formatter helper:
|
||||
```kotlin
|
||||
private fun formatAssistantMessage(contextJson: String): String {
|
||||
return try {
|
||||
val json = Json.parseToJsonElement(contextJson).jsonObject
|
||||
buildString {
|
||||
appendLine("I'm looking at:")
|
||||
json["packageName"]?.jsonPrimitive?.contentOrNull?.let {
|
||||
appendLine("App: $it")
|
||||
}
|
||||
json["webUri"]?.jsonPrimitive?.contentOrNull?.let {
|
||||
appendLine("URL: $it")
|
||||
}
|
||||
json["visibleText"]?.jsonArray?.take(20)?.forEach {
|
||||
it.jsonPrimitive?.contentOrNull?.let { text ->
|
||||
appendLine("- $text")
|
||||
}
|
||||
}
|
||||
appendLine("\nScreen context JSON:")
|
||||
appendLine("```json")
|
||||
appendLine(contextJson)
|
||||
appendLine("```")
|
||||
}
|
||||
} catch (e: Exception) {
|
||||
"Screen context:\n```json\n$contextJson\n```"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Plan
|
||||
|
||||
### Setup
|
||||
1. Install debug build: `./gradlew installDebug`
|
||||
2. Open Settings > Apps > Default Apps > Digital assistant app
|
||||
3. Select "Clawdbot"
|
||||
|
||||
### Test Cases
|
||||
|
||||
**Test 1: Basic Invocation (Connected)**
|
||||
- Precondition: Connected to gateway
|
||||
- Open any app (Chrome, Gmail)
|
||||
- Long-press Home button
|
||||
- Expected: MainActivity opens, chat sheet shows, message auto-sent with screen context
|
||||
|
||||
**Test 2: Offline Invocation**
|
||||
- Precondition: NOT connected to gateway
|
||||
- Long-press Home button
|
||||
- Expected: Alert dialog "Gateway Not Connected" with "Open Settings" button
|
||||
|
||||
**Test 3: Screen Context - Web Browser**
|
||||
- Open Chrome with webpage
|
||||
- Invoke assistant
|
||||
- Expected: Context includes webUri, packageName, visible text from page
|
||||
|
||||
**Test 4: Multiple Invocations**
|
||||
- Invoke from one app, let response complete
|
||||
- Switch to different app
|
||||
- Invoke again
|
||||
- Expected: New context from second app
|
||||
|
||||
**Test 5: Service Persistence**
|
||||
- Reboot device
|
||||
- Check Settings > Apps > Default Apps
|
||||
- Expected: Clawdbot still appears as option
|
||||
|
||||
### Debugging
|
||||
```bash
|
||||
# View logs
|
||||
adb logcat | grep -E "Clawdbot|VoiceInteraction|Assistant"
|
||||
|
||||
# Check service status
|
||||
adb shell dumpsys activity services | grep -A 20 clawdbot
|
||||
|
||||
# Verify manifest
|
||||
aapt dump xmltree app/build/outputs/apk/debug/app-debug.apk AndroidManifest.xml | grep -A 10 voice
|
||||
```
|
||||
|
||||
## Critical Files Summary
|
||||
|
||||
### To Modify (5 files)
|
||||
1. `AndroidManifest.xml` - Add permissions, service declarations, metadata
|
||||
2. `NodeForegroundService.kt` - Extend VoiceInteractionService, add lifecycle methods
|
||||
3. `MainActivity.kt` - Handle assistant intents, show offline dialog
|
||||
4. `MainViewModel.kt` - Add assistant context state flows and methods
|
||||
5. `RootScreen.kt` - Auto-open chat sheet when context provided
|
||||
|
||||
### To Create (3 files)
|
||||
1. `res/xml/voice_interaction_service.xml` - Voice interaction metadata
|
||||
2. `assistant/ClawdbotVoiceInteractionSessionService.kt` - Session service (lightweight)
|
||||
3. `assistant/ClawdbotVoiceInteractionSession.kt` - Core session logic (screen context capture)
|
||||
|
||||
### Optional Enhancement
|
||||
- `ChatSheetContent.kt` - Format context message for better UX (can use basic formatting initially)
|
||||
|
||||
## Key Technical Notes
|
||||
|
||||
- **VoiceInteractionService** is a Service subclass, so extending it from NodeForegroundService is fully compatible
|
||||
- **BIND_VOICE_INTERACTION** permission is automatically granted when app is set as default assistant
|
||||
- **AssistStructure** can be large; extract data immediately and discard object
|
||||
- **Screen context** limited to 100 text nodes to prevent overwhelming chat
|
||||
- **Intent flags** use SINGLE_TOP to reuse existing MainActivity instance
|
||||
- **Error handling**: Wrap all Assist API calls in try-catch for robustness
|
||||
|
||||
## Security & Privacy
|
||||
|
||||
- Screen context may contain sensitive data (passwords, PII)
|
||||
- Data only sent to user's own gateway (existing trust model)
|
||||
- No local storage of screen context (memory only)
|
||||
- User must explicitly invoke assistant (intentional action)
|
||||
- Offline mode prevents transmission if not connected
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✅ Clawdbot appears in "Digital assistant app" settings
|
||||
✅ Long-press home launches Clawdbot with chat
|
||||
✅ Screen context captured and formatted in message
|
||||
✅ Offline handling shows error dialog
|
||||
✅ Existing app functionality unaffected
|
||||
✅ Service lifecycle stable across reboots
|
||||
@ -115,6 +115,9 @@ dependencies {
|
||||
// Unicast DNS-SD (Wide-Area Bonjour) for tailnet discovery domains.
|
||||
implementation("dnsjava:dnsjava:3.6.4")
|
||||
|
||||
// BouncyCastle for Ed25519 (software provider, no hardware dependency)
|
||||
implementation("org.bouncycastle:bcprov-jdk18on:1.79")
|
||||
|
||||
testImplementation("junit:junit:4.13.2")
|
||||
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.10.2")
|
||||
testImplementation("io.kotest:kotest-runner-junit5-jvm:6.0.7")
|
||||
|
||||
@ -16,6 +16,7 @@
|
||||
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||
<uses-permission android:name="android.permission.SEND_SMS" />
|
||||
<uses-permission android:name="android.permission.BIND_VOICE_INTERACTION" />
|
||||
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
|
||||
<uses-feature
|
||||
android:name="android.hardware.camera"
|
||||
android:required="false" />
|
||||
@ -38,8 +39,11 @@
|
||||
android:name=".NodeForegroundService"
|
||||
android:exported="false"
|
||||
android:foregroundServiceType="dataSync|microphone|mediaProjection" />
|
||||
<!-- DISABLED: OPPO/ColorOS blocks VoiceInteractionService with IsQuickLaunchSupport=false
|
||||
Using ASSIST intent filter on MainActivity instead (like App Manager does) -->
|
||||
<service
|
||||
android:name=".assistant.ClawdbotVoiceInteractionService"
|
||||
android:enabled="false"
|
||||
android:exported="true"
|
||||
android:permission="android.permission.BIND_VOICE_INTERACTION">
|
||||
<intent-filter>
|
||||
@ -51,6 +55,7 @@
|
||||
</service>
|
||||
<service
|
||||
android:name=".assistant.ClawdbotVoiceInteractionSessionService"
|
||||
android:enabled="false"
|
||||
android:exported="true"
|
||||
android:permission="android.permission.BIND_VOICE_INTERACTION" />
|
||||
<service
|
||||
@ -68,13 +73,7 @@
|
||||
android:excludeFromRecents="true"
|
||||
android:noHistory="true"
|
||||
android:taskAffinity="">
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.ASSIST" />
|
||||
<action android:name="android.intent.action.VOICE_ASSIST" />
|
||||
<action android:name="android.intent.action.VOICE_COMMAND" />
|
||||
<action android:name="android.intent.action.SEARCH_LONG_PRESS" />
|
||||
<category android:name="android.intent.category.DEFAULT" />
|
||||
</intent-filter>
|
||||
<!-- Keep for backward compatibility, but MainActivity now handles ASSIST directly -->
|
||||
</activity>
|
||||
<activity
|
||||
android:name=".MainActivity"
|
||||
@ -84,6 +83,14 @@
|
||||
<action android:name="android.intent.action.MAIN" />
|
||||
<category android:name="android.intent.category.LAUNCHER" />
|
||||
</intent-filter>
|
||||
<!-- Digital Assistant - long press power button -->
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.ASSIST" />
|
||||
<action android:name="android.intent.action.VOICE_ASSIST" />
|
||||
<action android:name="android.intent.action.VOICE_COMMAND" />
|
||||
<action android:name="android.intent.action.SEARCH_LONG_PRESS" />
|
||||
<category android:name="android.intent.category.DEFAULT" />
|
||||
</intent-filter>
|
||||
<intent-filter>
|
||||
<action android:name="com.clawdbot.android.ACTION_ASSISTANT" />
|
||||
<category android:name="android.intent.category.DEFAULT" />
|
||||
|
||||
@ -0,0 +1,44 @@
|
||||
package com.clawdbot.android.assistant
|
||||
|
||||
import android.content.Intent
|
||||
import android.os.Bundle
|
||||
import android.speech.RecognitionService
|
||||
import android.speech.RecognizerIntent
|
||||
import android.speech.SpeechRecognizer
|
||||
import android.util.Log
|
||||
|
||||
/**
|
||||
* RecognitionService implementation required for VoiceInteractionService.
|
||||
*
|
||||
* Per Android documentation: "Given that all VIAs are also voice recognizer services,
|
||||
* you must also include a RecognitionService in your manifest."
|
||||
*
|
||||
* This is a minimal implementation that satisfies the system requirement.
|
||||
* Actual speech recognition is handled by VoiceInteractionSession.
|
||||
*/
|
||||
class ClawdbotRecognitionService : RecognitionService() {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "ClawdbotRecognition"
|
||||
}
|
||||
|
||||
override fun onStartListening(recognizerIntent: Intent?, listener: Callback?) {
|
||||
Log.i(TAG, "onStartListening called")
|
||||
|
||||
// This is a minimal implementation - actual recognition happens in VoiceInteractionSession
|
||||
// We just need this service to exist for the VoiceInteractionService to be valid
|
||||
|
||||
// Notify that we're not implementing speech recognition here
|
||||
listener?.error(SpeechRecognizer.ERROR_SERVER)
|
||||
}
|
||||
|
||||
override fun onCancel(listener: Callback?) {
|
||||
Log.i(TAG, "onCancel called")
|
||||
listener?.endOfSpeech()
|
||||
}
|
||||
|
||||
override fun onStopListening(listener: Callback?) {
|
||||
Log.i(TAG, "onStopListening called")
|
||||
listener?.endOfSpeech()
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,35 @@
|
||||
package com.clawdbot.android.assistant
|
||||
|
||||
import android.os.Bundle
|
||||
import android.service.voice.VoiceInteractionService
|
||||
import android.util.Log
|
||||
|
||||
/**
|
||||
* Standalone VoiceInteractionService that makes Clawdbot appear in
|
||||
* "Settings > Apps > Default Apps > Digital assistant app".
|
||||
*
|
||||
* This service must be separate from NodeForegroundService because
|
||||
* VoiceInteractionService cannot have foregroundServiceType attributes.
|
||||
*/
|
||||
class ClawdbotVoiceInteractionService : VoiceInteractionService() {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "ClawdbotVoiceInteraction"
|
||||
}
|
||||
|
||||
override fun onCreate() {
|
||||
super.onCreate()
|
||||
Log.i(TAG, "VoiceInteractionService created")
|
||||
}
|
||||
|
||||
override fun onReady() {
|
||||
super.onReady()
|
||||
Log.i(TAG, "VoiceInteractionService ready")
|
||||
// The service should now be ready to handle showSession() calls from the system
|
||||
}
|
||||
|
||||
override fun onShutdown() {
|
||||
Log.i(TAG, "VoiceInteractionService shutdown")
|
||||
super.onShutdown()
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,149 @@
|
||||
package com.clawdbot.android.assistant
|
||||
|
||||
import android.app.assist.AssistContent
|
||||
import android.app.assist.AssistStructure
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.graphics.Bitmap
|
||||
import android.os.Bundle
|
||||
import android.service.voice.VoiceInteractionSession
|
||||
import android.util.Log
|
||||
import com.clawdbot.android.MainActivity
|
||||
import kotlinx.serialization.json.JsonPrimitive
|
||||
import kotlinx.serialization.json.buildJsonArray
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
class ClawdbotVoiceInteractionSession(context: Context) : VoiceInteractionSession(context) {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "ClawdbotAssistSession"
|
||||
const val ACTION_ASSISTANT = "com.clawdbot.android.ACTION_ASSISTANT"
|
||||
const val EXTRA_SCREEN_CONTEXT = "screen_context"
|
||||
}
|
||||
|
||||
@Suppress("OVERRIDE_DEPRECATION")
|
||||
override fun onHandleAssist(
|
||||
data: Bundle?,
|
||||
structure: AssistStructure?,
|
||||
content: AssistContent?
|
||||
) {
|
||||
try {
|
||||
Log.i(TAG, "onHandleAssist called")
|
||||
|
||||
// Capture screen context
|
||||
val screenContext = captureScreenContext(structure, content)
|
||||
|
||||
// Launch MainActivity with context
|
||||
val intent = Intent(context, MainActivity::class.java).apply {
|
||||
action = ACTION_ASSISTANT
|
||||
putExtra(EXTRA_SCREEN_CONTEXT, screenContext)
|
||||
flags = Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_SINGLE_TOP
|
||||
}
|
||||
|
||||
context.startActivity(intent)
|
||||
} catch (e: Exception) {
|
||||
Log.e(TAG, "Error handling assist", e)
|
||||
// Show error toast
|
||||
android.widget.Toast.makeText(
|
||||
context,
|
||||
"Clawdbot Assistant error: ${e.message}",
|
||||
android.widget.Toast.LENGTH_LONG
|
||||
).show()
|
||||
} finally {
|
||||
// Close the session (we're delegating to MainActivity)
|
||||
finish()
|
||||
}
|
||||
}
|
||||
|
||||
override fun onHandleScreenshot(screenshot: Bitmap?) {
|
||||
Log.i(TAG, "Screenshot captured: ${screenshot?.width}x${screenshot?.height}")
|
||||
// Optional: Could save screenshot and include in context
|
||||
// For now, we rely on AssistStructure text extraction
|
||||
}
|
||||
|
||||
private fun captureScreenContext(
|
||||
structure: AssistStructure?,
|
||||
content: AssistContent?
|
||||
): String {
|
||||
val contextData = buildJsonObject {
|
||||
// Basic metadata
|
||||
put("timestamp", System.currentTimeMillis())
|
||||
|
||||
// Content data (URLs, structured data)
|
||||
content?.let { assistContent ->
|
||||
assistContent.webUri?.let { uri ->
|
||||
put("webUri", uri.toString())
|
||||
}
|
||||
assistContent.clipData?.let { clip ->
|
||||
if (clip.itemCount > 0) {
|
||||
put("clipText", clip.getItemAt(0).text?.toString() ?: "")
|
||||
}
|
||||
}
|
||||
// JSON-LD structured data (if available)
|
||||
assistContent.structuredData?.let { jsonLd ->
|
||||
put("structuredData", jsonLd)
|
||||
}
|
||||
}
|
||||
|
||||
// Structure data (view hierarchy, text content)
|
||||
structure?.let { assistStruct ->
|
||||
put("packageName", assistStruct.activityComponent.packageName)
|
||||
put("className", assistStruct.activityComponent.className)
|
||||
|
||||
// Extract visible text from view hierarchy
|
||||
val textNodes = mutableListOf<String>()
|
||||
extractTextFromStructure(assistStruct, textNodes)
|
||||
|
||||
put("visibleText", buildJsonArray {
|
||||
textNodes.forEach { text -> add(JsonPrimitive(text)) }
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return contextData.toString()
|
||||
}
|
||||
|
||||
private fun extractTextFromStructure(
|
||||
structure: AssistStructure,
|
||||
textNodes: MutableList<String>,
|
||||
maxNodes: Int = 100
|
||||
) {
|
||||
if (textNodes.size >= maxNodes) return
|
||||
|
||||
val windowCount = structure.windowNodeCount
|
||||
for (i in 0 until windowCount) {
|
||||
val windowNode = structure.getWindowNodeAt(i)
|
||||
val rootViewNode = windowNode.rootViewNode
|
||||
extractTextFromViewNode(rootViewNode, textNodes, maxNodes)
|
||||
}
|
||||
}
|
||||
|
||||
private fun extractTextFromViewNode(
|
||||
viewNode: AssistStructure.ViewNode,
|
||||
textNodes: MutableList<String>,
|
||||
maxNodes: Int
|
||||
) {
|
||||
if (textNodes.size >= maxNodes) return
|
||||
|
||||
// Extract text from this node
|
||||
viewNode.text?.toString()?.trim()?.takeIf { it.isNotEmpty() }?.let {
|
||||
textNodes.add(it)
|
||||
}
|
||||
|
||||
// Extract content description
|
||||
viewNode.contentDescription?.toString()?.trim()?.takeIf { it.isNotEmpty() }?.let {
|
||||
if (!textNodes.contains(it)) {
|
||||
textNodes.add(it)
|
||||
}
|
||||
}
|
||||
|
||||
// Recursively process children
|
||||
val childCount = viewNode.childCount
|
||||
for (i in 0 until childCount) {
|
||||
viewNode.getChildAt(i)?.let { child ->
|
||||
extractTextFromViewNode(child, textNodes, maxNodes)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,12 @@
|
||||
package com.clawdbot.android.assistant
|
||||
|
||||
import android.os.Bundle
|
||||
import android.service.voice.VoiceInteractionSession
|
||||
import android.service.voice.VoiceInteractionSessionService
|
||||
|
||||
class ClawdbotVoiceInteractionSessionService : VoiceInteractionSessionService() {
|
||||
override fun onNewSession(args: Bundle?): VoiceInteractionSession {
|
||||
android.util.Log.i("ClawdbotAssistant", "Creating new VoiceInteractionSession")
|
||||
return ClawdbotVoiceInteractionSession(this)
|
||||
}
|
||||
}
|
||||
@ -2,10 +2,12 @@ package com.clawdbot.android.gateway
|
||||
|
||||
import android.content.Context
|
||||
import android.util.Base64
|
||||
import org.bouncycastle.jce.provider.BouncyCastleProvider
|
||||
import java.io.File
|
||||
import java.security.KeyFactory
|
||||
import java.security.KeyPairGenerator
|
||||
import java.security.MessageDigest
|
||||
import java.security.Security
|
||||
import java.security.Signature
|
||||
import java.security.spec.PKCS8EncodedKeySpec
|
||||
import kotlinx.serialization.Serializable
|
||||
@ -23,6 +25,21 @@ class DeviceIdentityStore(context: Context) {
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
private val identityFile = File(context.filesDir, "clawdbot/identity/device.json")
|
||||
|
||||
companion object {
|
||||
// Use our own provider instance to avoid conflicts with Android's built-in BC
|
||||
private val bcProvider = BouncyCastleProvider()
|
||||
private val ED25519_SPKI_PREFIX = byteArrayOf(
|
||||
0x30, 0x2a, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x03, 0x21, 0x00,
|
||||
)
|
||||
|
||||
init {
|
||||
// Remove any conflicting BC provider and add our full BouncyCastle
|
||||
Security.removeProvider("BC")
|
||||
Security.insertProviderAt(bcProvider, 1)
|
||||
android.util.Log.d("DeviceIdentityStore", "Registered BouncyCastle provider: ${bcProvider.info}")
|
||||
}
|
||||
}
|
||||
|
||||
@Synchronized
|
||||
fun loadOrCreate(): DeviceIdentity {
|
||||
val existing = load()
|
||||
@ -44,13 +61,14 @@ class DeviceIdentityStore(context: Context) {
|
||||
return try {
|
||||
val privateKeyBytes = Base64.decode(identity.privateKeyPkcs8Base64, Base64.DEFAULT)
|
||||
val keySpec = PKCS8EncodedKeySpec(privateKeyBytes)
|
||||
val keyFactory = KeyFactory.getInstance("Ed25519")
|
||||
val keyFactory = KeyFactory.getInstance("Ed25519", bcProvider)
|
||||
val privateKey = keyFactory.generatePrivate(keySpec)
|
||||
val signature = Signature.getInstance("Ed25519")
|
||||
val signature = Signature.getInstance("Ed25519", bcProvider)
|
||||
signature.initSign(privateKey)
|
||||
signature.update(payload.toByteArray(Charsets.UTF_8))
|
||||
base64UrlEncode(signature.sign())
|
||||
} catch (_: Throwable) {
|
||||
} catch (e: Throwable) {
|
||||
android.util.Log.e("DeviceIdentityStore", "signPayload failed", e)
|
||||
null
|
||||
}
|
||||
}
|
||||
@ -93,11 +111,18 @@ class DeviceIdentityStore(context: Context) {
|
||||
}
|
||||
|
||||
private fun generate(): DeviceIdentity {
|
||||
val keyPair = KeyPairGenerator.getInstance("Ed25519").generateKeyPair()
|
||||
android.util.Log.d("DeviceIdentityStore", "Generating new Ed25519 key using BouncyCastle")
|
||||
|
||||
val kpg = KeyPairGenerator.getInstance("Ed25519", bcProvider)
|
||||
val keyPair = kpg.generateKeyPair()
|
||||
|
||||
val spki = keyPair.public.encoded
|
||||
val rawPublic = stripSpkiPrefix(spki)
|
||||
val deviceId = sha256Hex(rawPublic)
|
||||
val privateKey = keyPair.private.encoded
|
||||
|
||||
android.util.Log.d("DeviceIdentityStore", "Generated Ed25519 key, deviceId=${deviceId.take(8)}..., pubKeyLen=${rawPublic.size}")
|
||||
|
||||
return DeviceIdentity(
|
||||
deviceId = deviceId,
|
||||
publicKeyRawBase64 = Base64.encodeToString(rawPublic, Base64.NO_WRAP),
|
||||
@ -136,11 +161,4 @@ class DeviceIdentityStore(context: Context) {
|
||||
private fun base64UrlEncode(data: ByteArray): String {
|
||||
return Base64.encodeToString(data, Base64.URL_SAFE or Base64.NO_WRAP or Base64.NO_PADDING)
|
||||
}
|
||||
|
||||
companion object {
|
||||
private val ED25519_SPKI_PREFIX =
|
||||
byteArrayOf(
|
||||
0x30, 0x2a, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x03, 0x21, 0x00,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@ -253,11 +253,16 @@ class GatewaySession(
|
||||
|
||||
private inner class Listener : WebSocketListener() {
|
||||
override fun onOpen(webSocket: WebSocket, response: Response) {
|
||||
Log.d("ClawdbotGateway", "[$loggerTag] WebSocket opened, endpoint=${endpoint.host}:${endpoint.port}")
|
||||
scope.launch {
|
||||
try {
|
||||
Log.d("ClawdbotGateway", "[$loggerTag] Awaiting connect nonce...")
|
||||
val nonce = awaitConnectNonce()
|
||||
Log.d("ClawdbotGateway", "[$loggerTag] Got nonce=$nonce, sending connect...")
|
||||
sendConnect(nonce)
|
||||
Log.d("ClawdbotGateway", "[$loggerTag] Connect completed successfully")
|
||||
} catch (err: Throwable) {
|
||||
Log.e("ClawdbotGateway", "[$loggerTag] Connect failed: ${err.message}", err)
|
||||
connectDeferred.completeExceptionally(err)
|
||||
closeQuietly()
|
||||
}
|
||||
@ -373,6 +378,7 @@ class GatewaySession(
|
||||
)
|
||||
val signature = identityStore.signPayload(payload, identity)
|
||||
val publicKey = identityStore.publicKeyBase64Url(identity)
|
||||
Log.d("ClawdbotGateway", "[$loggerTag] Device identity: id=${identity.deviceId.take(8)}..., signature=${signature?.take(20) ?: "NULL"}, publicKey=${publicKey?.take(20) ?: "NULL"}")
|
||||
val deviceJson =
|
||||
if (!signature.isNullOrBlank() && !publicKey.isNullOrBlank()) {
|
||||
buildJsonObject {
|
||||
|
||||
Loading…
Reference in New Issue
Block a user