fix(voice-intents): honest errors, modal lifecycle, and missing LLM tools
End-to-end fix for the voice -> bridge -> agent pipeline. Twelve changes
across phone, plugin, and bridge layers so the voice fast-path, the LLM
tool-calling path, and the safety modal all actually work.
Voice fast-path:
- Remove Scroll voice intent (nobody says it aloud; /scroll route stays
for LLM android_scroll tool calls).
- Add SMS_INDIRECT regex to catch "send Hannah a text saying hi" phrasing
and "message" as a direct verb ("message Sam saying hi").
- Classify resolver failures via ContactResolution / AppResolution sealed
types so voice speaks specific per-category messages (permission,
service, not-found, no-phone, other) instead of "couldn't find" for
every failure mode.
- Pre-check SEND_SMS before the 5s countdown so permission-denied doesn't
silent-fail at the end of the confirmation flow.
- Flip voice state to Thinking immediately on classifier fall-through so
the UI shows progress during SSE connect latency.
- Enrich IntentResult.Handled with details: Map<String, String> for
structured chat-trace rendering (app label, package, match tier,
contact, resolved number, body, error code).
- Rewrite chat-trace formatter to render markdown per category.
Bridge safety modal:
- Fix showConfirmation threading -- ComposeView setContent must run on
Main, was called from Dispatchers.Default via voice local-dispatch,
threw, and got swallowed as "likely overlay permission missing".
- Fix OverlayLifecycleOwner.start() init order -- current androidx.savedstate
asserts performAttach runs while lifecycle is still INITIALIZED; the
old code advanced to CREATED first and tripped the assertion, killing
every destructive-verb modal attempt silently.
Voice mode UI:
- Replace single responseText slot with compact rolling transcript
observing ChatViewModel.messages (last 6), rendered via new
CompactTranscriptRow + StreamingResponseRow composables.
- Voice-action traces render via MarkdownContent. User messages keep
the "YOU" caption (fix mislabeling where voice-intent user messages
were captioned "ACTION").
- Preserve local voice-intent trace messages across
ChatHandler.loadMessageHistory reloads so "Opened Chrome" bubbles
don't vanish when session_end reload fires after fall-through.
LLM bridge path -- honest errors:
- respondFromResult emits structured error_code + required_permission
alongside the existing free-text error when ActionExecutor errors
match known patterns (permission_denied, service_unavailable,
user_denied). LLM gets both human-readable text AND a machine-readable
classification.
LLM bridge path -- missing tools:
- Fix plugin _check_requirements: was hitting /ping which returns
{pong, ts}, looking for phone_connected and accessibilityService
fields that do not exist there. Result: the gate always returned
False and hid all 13 non-setup tools from every gateway platform.
Now hits /bridge/status and requires BOTH phone_connected AND
bridge.accessibility_granted -- tools vanish from the LLM's schema
when a11y is revoked (common post-Studio-reinstall) instead of
letting the LLM confidently dispatch commands that 503.
- Add 4 new plugin tools: android_search_contacts, android_send_sms,
android_call, android_return_to_hermes. The first three wrap phone
routes that were fully implemented (with safety modals and direct
SmsManager / CALL_PHONE dispatch) but never exposed to the agent,
forcing it to drive the Messages app UI step-by-step. The fourth
lets the agent foreground Hermes Relay as the final step of a
phone-control task so the user sees the reply in-context without
manually switching apps.
- New /return_to_hermes route on BridgeCommandHandler, exempt from
the master-toggle check so the agent can always wrap up cleanly.
Tool count goes from 14 to 18. Plugin + gateway verified on the server
(systemctl --user restart hermes-gateway, _check_requirements() returns
True, all 18 tools register with no missing or orphan handlers).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Bailey Dixon
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
2e51ba9115
commit
5c763eb265
@@ -17,6 +17,7 @@ import kotlinx.coroutines.flow.asStateFlow
|
||||
import kotlinx.coroutines.flow.first
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.plus
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.coroutines.withTimeout
|
||||
import java.util.concurrent.ConcurrentHashMap
|
||||
import java.util.concurrent.atomic.AtomicLong
|
||||
@@ -200,13 +201,28 @@ class BridgeSafetyManager(
|
||||
return false
|
||||
}
|
||||
|
||||
// ComposeView creation + setContent inside BridgeStatusOverlay.showConfirmation
|
||||
// must run on the Main thread. This awaitConfirmation call can originate
|
||||
// from either the WSS-incoming BridgeCommandHandler path (already on Main
|
||||
// via ChannelMultiplexer's dispatcher) OR from the in-process voice-intent
|
||||
// local-dispatch path (Dispatchers.Default, via RealVoiceBridgeIntentHandler's
|
||||
// own scope). Using Dispatchers.Main.immediate makes the first case a no-op
|
||||
// and only schedules on Main for the second — one line handles both callers.
|
||||
//
|
||||
// Before this fix the off-thread call threw from ComposeView.setContent,
|
||||
// the outer runCatching swallowed it, and the log message mislabelled the
|
||||
// cause as "likely overlay permission missing" which sent debugging up a
|
||||
// wrong tree (2026-04-15 on-device test with overlay permission granted
|
||||
// but voice SMS still never showed the modal).
|
||||
val shown = runCatching {
|
||||
host.showConfirmation(pending) { resolution ->
|
||||
resolveConfirmation(requestId, resolution)
|
||||
withContext(Dispatchers.Main.immediate) {
|
||||
host.showConfirmation(pending) { resolution ->
|
||||
resolveConfirmation(requestId, resolution)
|
||||
}
|
||||
}
|
||||
}
|
||||
if (shown.isFailure) {
|
||||
Log.w(TAG, "awaitConfirmation: overlay host refused to show modal (likely overlay permission missing)")
|
||||
Log.w(TAG, "awaitConfirmation: host.showConfirmation threw — denying", shown.exceptionOrNull())
|
||||
pendingConfirmations.remove(requestId)
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -240,10 +240,28 @@ class BridgeStatusOverlay(context: Context) : ConfirmationOverlayHost {
|
||||
* [IllegalStateException] at `AndroidComposeView.onAttachedToWindow:2234`
|
||||
* on every overlay attach.
|
||||
*
|
||||
* The required init sequence is: move the lifecycle to CREATED first,
|
||||
* call `SavedStateRegistryController.performRestore(null)` (empty
|
||||
* bundle = fresh state), THEN advance to RESUMED. Doing it in any other
|
||||
* order trips a second assertion in `SavedStateRegistryController`.
|
||||
* ## Init sequence — DO NOT REORDER
|
||||
*
|
||||
* Current androidx.savedstate requires:
|
||||
*
|
||||
* 1. `savedStateController.performRestore(null)` — while the owner is
|
||||
* still in [Lifecycle.State.INITIALIZED]. Internally this calls
|
||||
* `performAttach()` which hard-asserts `currentState == INITIALIZED`
|
||||
* and throws `IllegalStateException: Restarter must be created only
|
||||
* during owner's initialization stage` if you've already advanced
|
||||
* past it.
|
||||
* 2. `registry.currentState = CREATED`
|
||||
* 3. `registry.currentState = RESUMED`
|
||||
*
|
||||
* An older androidx.savedstate release required the OPPOSITE order
|
||||
* (CREATED → performRestore → RESUMED) and this file shipped with that
|
||||
* code, matching the KDoc. The 2026-04-15 Compose BOM bump flipped the
|
||||
* contract and the overlay started throwing on every destructive-verb
|
||||
* confirmation attempt. Caught by Bailey's on-device voice→SMS test
|
||||
* that same day — see the `BridgeSafetyMgr` stack trace in the session
|
||||
* log. The chip path didn't trigger it because it was never exercised
|
||||
* in the same build + flavor combo; only the confirmation modal path
|
||||
* hit the assertion.
|
||||
*/
|
||||
private class OverlayLifecycleOwner :
|
||||
LifecycleOwner,
|
||||
@@ -261,9 +279,11 @@ private class OverlayLifecycleOwner :
|
||||
get() = savedStateController.savedStateRegistry
|
||||
|
||||
fun start() {
|
||||
// Order matters — see KDoc above.
|
||||
registry.currentState = Lifecycle.State.CREATED
|
||||
// Restore saved state FIRST — must run while currentState is still
|
||||
// INITIALIZED or performAttach() throws. See KDoc above for the
|
||||
// assertion story.
|
||||
savedStateController.performRestore(null)
|
||||
registry.currentState = Lifecycle.State.CREATED
|
||||
registry.currentState = Lifecycle.State.RESUMED
|
||||
}
|
||||
|
||||
|
||||
+102
-2
@@ -336,7 +336,10 @@ class BridgeCommandHandler(
|
||||
}
|
||||
)
|
||||
|
||||
if (!service.isMasterEnabled() && path != "/current_app") {
|
||||
if (!service.isMasterEnabled() &&
|
||||
path != "/current_app" &&
|
||||
path != "/return_to_hermes"
|
||||
) {
|
||||
return respond(
|
||||
requestId, 403,
|
||||
buildJsonObject {
|
||||
@@ -492,6 +495,57 @@ class BridgeCommandHandler(
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
// === PHASE3-return-to-hermes ===
|
||||
// Bring the Hermes Relay app back to foreground. Used by the
|
||||
// server-side agent as the final step of any multi-app task
|
||||
// (e.g. after driving Messages to send an SMS) so the user
|
||||
// sees the agent's reply in-context without manually switching
|
||||
// apps. The phone knows its own package name via service — no
|
||||
// parameter needed, works transparently on both sideload and
|
||||
// googlePlay flavors.
|
||||
//
|
||||
// Allowed even when the master toggle is off: returning focus
|
||||
// to our own app isn't a destructive action, and this tool
|
||||
// should still work if the user flips the toggle mid-session
|
||||
// so the agent can at least wrap up cleanly. Blocklist and
|
||||
// destructive-verb checks don't apply — it's a self-foreground
|
||||
// intent, not a phone-control action.
|
||||
"/return_to_hermes" -> {
|
||||
val selfPkg = service.packageName
|
||||
val intent = runCatching {
|
||||
service.packageManager.getLaunchIntentForPackage(selfPkg)
|
||||
}.getOrNull()
|
||||
if (intent == null) {
|
||||
respond(
|
||||
requestId, 500,
|
||||
buildJsonObject {
|
||||
put("error", "couldn't resolve launch intent for self ($selfPkg)")
|
||||
}
|
||||
)
|
||||
return
|
||||
}
|
||||
intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP)
|
||||
val launchResult = runCatching { service.startActivity(intent) }
|
||||
if (launchResult.isFailure) {
|
||||
respond(
|
||||
requestId, 500,
|
||||
buildJsonObject {
|
||||
put("error", launchResult.exceptionOrNull()?.message
|
||||
?: "startActivity failed for $selfPkg")
|
||||
}
|
||||
)
|
||||
return
|
||||
}
|
||||
respond(
|
||||
requestId, 200,
|
||||
buildJsonObject {
|
||||
put("ok", true)
|
||||
put("package", selfPkg)
|
||||
}
|
||||
)
|
||||
}
|
||||
// === END PHASE3-return-to-hermes ===
|
||||
// === END PHASE3-baseline-handlers ===
|
||||
|
||||
// === PHASE3-event-stream: B1 android_event_stream toggle ===
|
||||
@@ -1230,12 +1284,58 @@ class BridgeCommandHandler(
|
||||
}
|
||||
if (result.data.isEmpty()) put("ok", true)
|
||||
} else {
|
||||
put("error", result.error ?: "unknown error")
|
||||
val err = result.error ?: "unknown error"
|
||||
put("error", err)
|
||||
// M2: structured error code for the LLM tool-calling path.
|
||||
// ActionExecutor returns free-text errors like "Grant contacts
|
||||
// permission in Settings..." which LLMs CAN interpret, but
|
||||
// adding a machine-readable error_code + required_permission
|
||||
// gives the server-side agent a cleaner signal to classify
|
||||
// responses + offer actionable next steps instead of relaying
|
||||
// whatever phrasing the error string happens to use. Free
|
||||
// text stays alongside for LLMs that prefer it.
|
||||
classifyBridgeError(err)?.let { (code, perm) ->
|
||||
put("error_code", code)
|
||||
if (perm != null) put("required_permission", perm)
|
||||
}
|
||||
}
|
||||
}
|
||||
respond(requestId, status, payload)
|
||||
}
|
||||
|
||||
/**
|
||||
* Substring-classify an ActionExecutor error string into a structured
|
||||
* (error_code, required_permission?) pair. Returns null when the error
|
||||
* doesn't match any known category — in that case the free-text
|
||||
* `error` field on the response is the only signal.
|
||||
*
|
||||
* The substring patterns are coupled to
|
||||
* [ActionExecutor.searchContacts] / [ActionExecutor.sendSms] /
|
||||
* [ActionExecutor.makeCall] phrasings. If those messages are changed,
|
||||
* update this classifier too or the `error_code` will degrade to
|
||||
* null and the LLM will fall back to reading the free text.
|
||||
*/
|
||||
private fun classifyBridgeError(err: String): Pair<String, String?>? {
|
||||
val lower = err.lowercase()
|
||||
return when {
|
||||
// searchContacts upfront check + mid-query SecurityException
|
||||
"contacts permission" in lower || "contacts permission revoked" in lower ->
|
||||
"permission_denied" to "android.permission.READ_CONTACTS"
|
||||
// sendSms upfront check
|
||||
"sms permission" in lower ->
|
||||
"permission_denied" to "android.permission.SEND_SMS"
|
||||
// makeCall SecurityException recovery
|
||||
"permission denied despite grant" in lower ->
|
||||
"permission_denied" to "android.permission.CALL_PHONE"
|
||||
// Service-side failures that aren't permission-shaped
|
||||
"not connected" in lower || "service not connected" in lower ->
|
||||
"service_unavailable" to null
|
||||
"user denied destructive action" in lower ->
|
||||
"user_denied" to null
|
||||
else -> null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A4: recycle a list of window roots returned by
|
||||
* [HermesAccessibilityService.snapshotAllWindows]. Matches the same
|
||||
|
||||
@@ -338,7 +338,34 @@ class ChatHandler {
|
||||
// just collected against the reloaded IDs are guaranteed to fire.
|
||||
dispatchedMediaMarkers.clear()
|
||||
|
||||
_messages.value = if (loaded.size > MAX_MESSAGES) loaded.takeLast(MAX_MESSAGES) else loaded
|
||||
// === PHASE3-voice-intents-chathistory ===
|
||||
// Preserve local-only voice-intent trace messages across a reload.
|
||||
// These messages are injected by [appendLocalVoiceIntentTrace] with
|
||||
// IDs prefixed "voice-intent-" and never reach the server-side
|
||||
// session, so a wholesale `_messages.value = loaded` assignment
|
||||
// would wipe them. Bailey hit this 2026-04-15: voice fall-through
|
||||
// ("proceed" → not a recognized intent → chat.sendMessage) triggered
|
||||
// a history reload on stream complete and the previous voice trace
|
||||
// vanished, making it look like "the chat cleared". Server-side
|
||||
// sync (so these traces reach the LLM's session memory too) is
|
||||
// still a v0.4.1 follow-up, but preserving them client-side is
|
||||
// enough to fix the disappearing-scrollback bug today.
|
||||
val preservedVoiceTraces = _messages.value.filter {
|
||||
it.id.startsWith("voice-intent-")
|
||||
}
|
||||
val merged = if (preservedVoiceTraces.isEmpty()) {
|
||||
loaded
|
||||
} else {
|
||||
// Merge by timestamp so voice traces interleave with the
|
||||
// reloaded server messages in chronological order. The voice
|
||||
// trace IDs carry `System.currentTimeMillis()` in their suffix
|
||||
// (see appendLocalVoiceIntentTrace), so ChatMessage.timestamp
|
||||
// is the source of truth here.
|
||||
(loaded + preservedVoiceTraces).sortedBy { it.timestamp }
|
||||
}
|
||||
|
||||
_messages.value = if (merged.size > MAX_MESSAGES) merged.takeLast(MAX_MESSAGES) else merged
|
||||
// === END PHASE3-voice-intents-chathistory ===
|
||||
|
||||
// Now that the reloaded messages are in state, fire callbacks so the
|
||||
// ViewModel can insert LOADING/FAILED attachments via mutateMessage.
|
||||
|
||||
@@ -63,7 +63,10 @@ import androidx.compose.ui.input.pointer.pointerInput
|
||||
import androidx.compose.ui.platform.LocalHapticFeedback
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.style.TextAlign
|
||||
import androidx.compose.ui.text.style.TextOverflow
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.data.ChatMessage
|
||||
import com.hermesandroid.relay.data.MessageRole
|
||||
import com.hermesandroid.relay.ui.LocalSnackbarHost
|
||||
import com.hermesandroid.relay.ui.showHumanError
|
||||
import com.hermesandroid.relay.util.HumanError
|
||||
@@ -91,6 +94,22 @@ fun VoiceModeOverlay(
|
||||
// Nullable so existing call sites compile; when wired, voice errors
|
||||
// surface as global snackbars in addition to the inline banner below.
|
||||
errorEvents: SharedFlow<HumanError>? = null,
|
||||
// === PHASE3-voice-mode-transcript ===
|
||||
// Compact rolling transcript of the last N chat messages — the
|
||||
// caller (ChatScreen) passes `chatViewModel.messages.takeLast(6)`.
|
||||
// Voice mode is voice-first but wasn't giving the user any visibility
|
||||
// into conversation history during a session, so this adds a compact
|
||||
// strip below the sphere that observes the same ChatHandler message
|
||||
// flow the chat tab uses. Includes local-only voice-intent traces
|
||||
// (agentName = "Voice action", id prefix "voice-intent-") rendered
|
||||
// via MarkdownContent so the bold + inline code in traces like
|
||||
// "**Opened Chrome** `com.android.chrome`" shows correctly.
|
||||
//
|
||||
// Default empty-list so existing call sites compile; the empty state
|
||||
// hint ("Tap the mic to speak") still renders when the transcript
|
||||
// is empty.
|
||||
transcriptMessages: List<ChatMessage> = emptyList(),
|
||||
// === END PHASE3-voice-mode-transcript ===
|
||||
) {
|
||||
val surface = MaterialTheme.colorScheme.surface
|
||||
val haptic = LocalHapticFeedback.current
|
||||
@@ -165,23 +184,27 @@ fun VoiceModeOverlay(
|
||||
}
|
||||
}
|
||||
|
||||
// Center column: pinned "you said" chip → sphere → waveform → scrolling response.
|
||||
// Center column: pinned "you said" chip → sphere → waveform → scrolling transcript.
|
||||
//
|
||||
// Layout uses fixed-weight slots (sphere 0.55, response 1f) instead of
|
||||
// SpaceBetween so the sphere/waveform don't drift around as the response
|
||||
// grows. The response area owns its own scroll state with auto-scroll-to-tail
|
||||
// and a top/bottom fade mask for overflow indication.
|
||||
val textScrollState = rememberScrollState()
|
||||
// Layout uses fixed-weight slots (sphere 1.5, transcript 1f) instead
|
||||
// of SpaceBetween so the sphere/waveform don't drift as the transcript
|
||||
// grows. The transcript area owns its own scroll state with auto-
|
||||
// scroll-to-tail and a top/bottom fade mask for overflow indication.
|
||||
//
|
||||
// Transcript content sources:
|
||||
// - transcriptMessages: last N chat messages (includes voice-intent
|
||||
// traces rendered via MarkdownContent)
|
||||
// - uiState.responseText: the in-flight streaming response for the
|
||||
// current turn (a bubble that hasn't yet been finalized into a
|
||||
// ChatMessage by the time the user sees it mid-stream)
|
||||
val transcriptScrollState = rememberScrollState()
|
||||
|
||||
// Auto-scroll to the tail every time the response length changes.
|
||||
// Without this, streaming tokens accumulate below the viewport and
|
||||
// the user has to drag down by hand — that's the "feels behind"
|
||||
// sensation we were chasing. animateScrollTo gives a smooth follow
|
||||
// rather than a snap on every token.
|
||||
LaunchedEffect(uiState.responseText.length) {
|
||||
if (uiState.responseText.isNotEmpty()) {
|
||||
textScrollState.animateScrollTo(textScrollState.maxValue)
|
||||
}
|
||||
// Auto-scroll to the tail whenever: (a) a new message arrives in the
|
||||
// transcript, (b) the streaming responseText grows. Both conditions
|
||||
// fire the same smooth animateScrollTo so the user's eye never has
|
||||
// to chase token churn.
|
||||
LaunchedEffect(transcriptMessages.size, uiState.responseText.length) {
|
||||
transcriptScrollState.animateScrollTo(transcriptScrollState.maxValue)
|
||||
}
|
||||
|
||||
// Vertical fade brush for the scroll area: top + bottom edges fade
|
||||
@@ -268,16 +291,19 @@ fun VoiceModeOverlay(
|
||||
}
|
||||
}
|
||||
|
||||
// Response area. Plain Text — no AnimatedContent — so streaming
|
||||
// tokens accrete in place instead of fade-flickering on every
|
||||
// delta. The empty-state hint keeps a small AnimatedContent
|
||||
// because that's a real discrete swap (Idle → Listening → ...).
|
||||
// Transcript area. Shows the last N chat messages in a compact
|
||||
// rolling list, plus the in-flight streaming responseText for
|
||||
// the current turn. Empty-state hint renders when BOTH the
|
||||
// transcript AND the live response are empty (first launch,
|
||||
// fresh voice session).
|
||||
Box(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.weight(1f, fill = true),
|
||||
) {
|
||||
if (uiState.responseText.isNotBlank()) {
|
||||
val hasTranscript = transcriptMessages.isNotEmpty()
|
||||
val hasLiveResponse = uiState.responseText.isNotBlank()
|
||||
if (hasTranscript || hasLiveResponse) {
|
||||
Column(
|
||||
modifier = Modifier
|
||||
.fillMaxSize()
|
||||
@@ -286,18 +312,22 @@ fun VoiceModeOverlay(
|
||||
drawContent()
|
||||
drawRect(brush = fadeBrush, blendMode = BlendMode.DstIn)
|
||||
}
|
||||
.verticalScroll(textScrollState)
|
||||
.verticalScroll(transcriptScrollState)
|
||||
.padding(vertical = 12.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(10.dp),
|
||||
) {
|
||||
Text(
|
||||
text = uiState.responseText,
|
||||
style = MaterialTheme.typography.titleMedium,
|
||||
color = MaterialTheme.colorScheme.onSurface,
|
||||
// Long paragraph responses read poorly when centered.
|
||||
// Empty-state hint stays Center via the else branch.
|
||||
textAlign = TextAlign.Start,
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
)
|
||||
transcriptMessages.forEach { msg ->
|
||||
CompactTranscriptRow(msg)
|
||||
}
|
||||
// Live streaming response for the current turn —
|
||||
// rendered as a trailing row so it sits below the
|
||||
// committed history. Once the turn completes and
|
||||
// the message is appended to chat history,
|
||||
// responseText clears and this row collapses back
|
||||
// into the transcript list naturally.
|
||||
if (hasLiveResponse) {
|
||||
StreamingResponseRow(uiState.responseText)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
Box(
|
||||
@@ -514,3 +544,110 @@ private fun InteractionMode.label(): String = when (this) {
|
||||
InteractionMode.HoldToTalk -> "Hold to talk"
|
||||
InteractionMode.Continuous -> "Continuous"
|
||||
}
|
||||
|
||||
/**
|
||||
* Compact transcript row for voice mode. Rendering rules:
|
||||
*
|
||||
* - `id.startsWith("voice-intent-")` → voice-action trace, rendered via
|
||||
* [MarkdownContent] unbounded (these are the rich bridge-intent bubbles
|
||||
* like "**Opened Chrome** `com.android.chrome` — exact match", which
|
||||
* need the bold + inline code styling to read well).
|
||||
* - `role == USER` → small italic caption "YOU" + body in bodySmall,
|
||||
* two-line truncation.
|
||||
* - `role == ASSISTANT` → caption "AGENT" + body in bodyMedium,
|
||||
* four-line truncation.
|
||||
* - `role == SYSTEM` → skipped entirely (voice mode is a conversation
|
||||
* surface, not a system-message debug view).
|
||||
*
|
||||
* The caller is responsible for bounding the list length (voice mode
|
||||
* uses `takeLast(6)`).
|
||||
*/
|
||||
@Composable
|
||||
private fun CompactTranscriptRow(message: ChatMessage) {
|
||||
if (message.role == MessageRole.SYSTEM) return
|
||||
|
||||
// A voice-intent trace is a PAIR of messages added by
|
||||
// ChatHandler.appendLocalVoiceIntentTrace — one USER with the raw
|
||||
// utterance ("voice-intent-user-$ts") and one ASSISTANT with the
|
||||
// action description ("voice-intent-action-$ts"). Only the assistant
|
||||
// half should carry the "ACTION" caption + MarkdownContent rendering;
|
||||
// the user half should stay labeled "YOU" so the transcript reads as
|
||||
// a normal user→assistant exchange. Pre-fix we keyed off the id
|
||||
// prefix alone and mislabeled the user half as ACTION (Bailey
|
||||
// 2026-04-15 screenshot: duplicate ACTION row with the raw utterance).
|
||||
val isVoiceActionBubble = message.role == MessageRole.ASSISTANT &&
|
||||
message.id.startsWith("voice-intent-")
|
||||
val caption = when {
|
||||
isVoiceActionBubble -> "ACTION"
|
||||
message.role == MessageRole.USER -> "YOU"
|
||||
else -> "AGENT"
|
||||
}
|
||||
val captionColor = when {
|
||||
isVoiceActionBubble -> MaterialTheme.colorScheme.tertiary
|
||||
message.role == MessageRole.USER -> MaterialTheme.colorScheme.primary
|
||||
else -> MaterialTheme.colorScheme.secondary
|
||||
}
|
||||
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalAlignment = Alignment.Start,
|
||||
) {
|
||||
Text(
|
||||
text = caption,
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = captionColor,
|
||||
)
|
||||
Spacer(Modifier.height(2.dp))
|
||||
when {
|
||||
isVoiceActionBubble -> MarkdownContent(
|
||||
content = message.content,
|
||||
textColor = MaterialTheme.colorScheme.onSurface,
|
||||
)
|
||||
message.role == MessageRole.USER -> Text(
|
||||
text = message.content,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
maxLines = 2,
|
||||
overflow = TextOverflow.Ellipsis,
|
||||
)
|
||||
else -> Text(
|
||||
text = message.content,
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
color = MaterialTheme.colorScheme.onSurface,
|
||||
maxLines = 4,
|
||||
overflow = TextOverflow.Ellipsis,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Row for the in-flight streaming response — `uiState.responseText` that
|
||||
* hasn't yet been committed to chat history. Separate composable so it
|
||||
* can render slightly differently from committed [CompactTranscriptRow]
|
||||
* (larger type, onSurface instead of onSurfaceVariant) matching the
|
||||
* pre-transcript single-line "current response" visual weight, and so
|
||||
* streaming tokens accrete in place without the AnimatedContent flicker
|
||||
* we had before transcript mode.
|
||||
*/
|
||||
@Composable
|
||||
private fun StreamingResponseRow(responseText: String) {
|
||||
Column(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalAlignment = Alignment.Start,
|
||||
) {
|
||||
Text(
|
||||
text = "AGENT",
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.secondary,
|
||||
)
|
||||
Spacer(Modifier.height(2.dp))
|
||||
Text(
|
||||
text = responseText,
|
||||
style = MaterialTheme.typography.titleMedium,
|
||||
color = MaterialTheme.colorScheme.onSurface,
|
||||
textAlign = TextAlign.Start,
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1324,6 +1324,12 @@ fun ChatScreen(
|
||||
// Agent B's overlay collects this flow and renders classified
|
||||
// voice errors (mic capture, STT/TTS failures, relay drops).
|
||||
errorEvents = voiceViewModel.errorEvents,
|
||||
// Voice-first transcript: pass the last N chat messages so
|
||||
// voice mode can show a compact rolling history including
|
||||
// local-only voice-intent traces (agentName="Voice action").
|
||||
// Bounded to 6 to keep voice mode visually focused on sphere
|
||||
// + waveform + mic — the full scroll lives in the chat tab.
|
||||
transcriptMessages = messages.takeLast(6),
|
||||
)
|
||||
}
|
||||
} // end Box
|
||||
|
||||
@@ -487,15 +487,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
// server-side session) so the gateway-side LLM still won't
|
||||
// see prior voice actions in its session memory; that's a
|
||||
// v0.4.1 follow-up tracked in ROADMAP.md.
|
||||
val actionDescription = buildString {
|
||||
append(result.intentLabel)
|
||||
if (result.spokenConfirmation != null) {
|
||||
append(": ")
|
||||
append(result.spokenConfirmation)
|
||||
} else {
|
||||
append(" — done")
|
||||
}
|
||||
}
|
||||
val actionDescription = formatVoiceIntentTrace(result)
|
||||
chatVm.recordVoiceIntent(
|
||||
userText = userText,
|
||||
actionDescription = actionDescription,
|
||||
@@ -531,6 +523,24 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
}
|
||||
// === END PHASE3-voice-intents ===
|
||||
|
||||
// === PHASE3-voice-intents-fallback-visibility ===
|
||||
// Classifier returned NotApplicable — we're about to fall through
|
||||
// to chatVm.sendMessage(). The SSE stream takes 500–1500 ms to
|
||||
// open and start emitting deltas, during which the voice UI would
|
||||
// sit in Idle showing nothing new. Users interpret that as "the
|
||||
// utterance was dropped." Flip to Thinking immediately so the
|
||||
// empty-state hint shows "Thinking..." until tokens arrive and
|
||||
// the stream observer takes over. Also pin the transcribed text
|
||||
// so the user can see what we heard while we wait.
|
||||
_uiState.update {
|
||||
it.copy(
|
||||
state = VoiceState.Thinking,
|
||||
transcribedText = userText,
|
||||
responseText = "",
|
||||
)
|
||||
}
|
||||
// === END PHASE3-voice-intents-fallback-visibility ===
|
||||
|
||||
// Reset sentence buffering state for the new turn.
|
||||
sentenceBuffer = StringBuilder()
|
||||
lastObservedMessageId = null
|
||||
@@ -840,6 +850,137 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
currentTurnJob?.cancel()
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a voice-intent trace for the chat scroll. Uses the structured
|
||||
* [IntentResult.Handled.details] map to produce a richer message than the
|
||||
* pre-v0.4.0 formatter, which could only say "Open App — done" and left
|
||||
* the user wondering *which* app actually launched (see the 2026-04-15
|
||||
* "open Chrome → opens Google app" report).
|
||||
*
|
||||
* Output is markdown so the existing chat bubble renderer picks up bold
|
||||
* labels and inline `code` for package names. Format differs per intent:
|
||||
*
|
||||
* - `Open App` success → **Opened Chrome**\n`com.android.chrome` — exact match
|
||||
* - `Open App` failure → **Open App** — I couldn't find an app called 'foo'
|
||||
* - `Send SMS` awaiting confirmation → **Send SMS — awaiting confirmation**
|
||||
* \nTo: Hannah (+1555...)\nBody: smoke test
|
||||
* - Safe intents without details → bare label
|
||||
*/
|
||||
private fun formatVoiceIntentTrace(result: IntentResult.Handled): String {
|
||||
val d = result.details
|
||||
val errorCode = d["error"]
|
||||
return when (result.intentLabel) {
|
||||
"Open App" -> {
|
||||
val label = d["appLabel"]
|
||||
val pkg = d["packageName"]
|
||||
val tier = d["matchTier"]
|
||||
val requested = d["requestedName"]
|
||||
when {
|
||||
label != null && pkg != null -> buildString {
|
||||
append("**Opened ")
|
||||
append(label)
|
||||
append("**")
|
||||
append('\n')
|
||||
append('`')
|
||||
append(pkg)
|
||||
append('`')
|
||||
if (tier != null) {
|
||||
append(" — ")
|
||||
append(tier)
|
||||
append(" match")
|
||||
}
|
||||
}
|
||||
errorCode == "app_not_found" -> buildString {
|
||||
append("**Open App**")
|
||||
append('\n')
|
||||
append("Couldn't find an app called '")
|
||||
append(requested ?: "?")
|
||||
append("'.")
|
||||
}
|
||||
errorCode == "service_missing" -> buildString {
|
||||
append("**Open App — bridge offline**")
|
||||
append('\n')
|
||||
append("Enable Hermes accessibility in Settings to open apps by voice.")
|
||||
}
|
||||
errorCode == "other_error" -> buildString {
|
||||
append("**Open App — error**")
|
||||
append('\n')
|
||||
append(d["errorMessage"] ?: "unknown error")
|
||||
}
|
||||
else -> "**Open App** — done"
|
||||
}
|
||||
}
|
||||
"Send SMS" -> {
|
||||
val contact = d["contact"]
|
||||
val number = d["resolvedNumber"]
|
||||
val body = d["body"]
|
||||
when {
|
||||
number != null -> buildString {
|
||||
append("**Send SMS — awaiting confirmation**")
|
||||
append('\n')
|
||||
append("To: ")
|
||||
append(contact ?: "?")
|
||||
append(" (")
|
||||
append(number)
|
||||
append(')')
|
||||
if (body != null) {
|
||||
append('\n')
|
||||
append("Body: ")
|
||||
append(body)
|
||||
}
|
||||
}
|
||||
errorCode == "permission_missing_sms" -> buildString {
|
||||
append("**Send SMS — permission needed**")
|
||||
append('\n')
|
||||
append("Grant SMS permission in Settings › Apps › Hermes Relay › Permissions.")
|
||||
}
|
||||
errorCode == "permission_missing_contacts" -> buildString {
|
||||
append("**Send SMS — permission needed**")
|
||||
append('\n')
|
||||
append("Grant Contacts permission to look up '")
|
||||
append(contact ?: "?")
|
||||
append("' in Settings › Apps › Hermes Relay › Permissions.")
|
||||
}
|
||||
errorCode == "service_missing" -> buildString {
|
||||
append("**Send SMS — bridge offline**")
|
||||
append('\n')
|
||||
append("Enable Hermes accessibility in Settings first.")
|
||||
}
|
||||
errorCode == "contact_not_found" -> buildString {
|
||||
append("**Send SMS**")
|
||||
append('\n')
|
||||
append("Couldn't find a contact called '")
|
||||
append(contact ?: "?")
|
||||
append("'.")
|
||||
}
|
||||
errorCode == "contact_no_phone" -> buildString {
|
||||
append("**Send SMS**")
|
||||
append('\n')
|
||||
append(contact ?: "Contact")
|
||||
append(" has no phone number on file.")
|
||||
}
|
||||
errorCode == "other_error" -> buildString {
|
||||
append("**Send SMS — error**")
|
||||
append('\n')
|
||||
append(d["errorMessage"] ?: "unknown error")
|
||||
}
|
||||
else -> "**Send SMS** — dispatched"
|
||||
}
|
||||
}
|
||||
else -> buildString {
|
||||
append("**")
|
||||
append(result.intentLabel)
|
||||
append("**")
|
||||
if (result.spokenConfirmation != null) {
|
||||
append(" — ")
|
||||
append(result.spokenConfirmation)
|
||||
} else {
|
||||
append(" — done")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
@@ -82,6 +82,25 @@ sealed class IntentResult {
|
||||
val intentLabel: String,
|
||||
val spokenConfirmation: String? = null,
|
||||
val requiresConfirmation: Boolean = false,
|
||||
/**
|
||||
* Optional structured details about what the handler resolved. Keys
|
||||
* are handler-specific and documented per intent:
|
||||
*
|
||||
* - `OpenApp` → `appLabel`, `packageName`, `matchTier`
|
||||
* (`matchTier` ∈ {`"exact"`, `"prefix"`, `"contains"`})
|
||||
* - `SendSms` → `contact`, `resolvedNumber`, `body`
|
||||
* - Tap / Scroll / Back / Home → empty for now
|
||||
*
|
||||
* The UI layer (VoiceViewModel chat-trace formatter) renders these
|
||||
* as a human-readable postscript to [intentLabel] so the user can
|
||||
* see exactly which app got launched / which number got used. Empty
|
||||
* map means the UI falls back to the bare [intentLabel].
|
||||
*
|
||||
* Backwards-compat default is empty map so existing callers (the
|
||||
* googlePlay no-op factory, any tests) keep compiling without touching
|
||||
* this field.
|
||||
*/
|
||||
val details: Map<String, String> = emptyMap(),
|
||||
) : IntentResult()
|
||||
|
||||
/** The text is not a phone-control intent. Fall through to chat. */
|
||||
|
||||
+242
-68
@@ -1,7 +1,10 @@
|
||||
package com.hermesandroid.relay.voice
|
||||
|
||||
import android.Manifest
|
||||
import android.content.Intent
|
||||
import android.content.pm.PackageManager
|
||||
import android.util.Log
|
||||
import androidx.core.content.ContextCompat
|
||||
import com.hermesandroid.relay.accessibility.HermesAccessibilityService
|
||||
import com.hermesandroid.relay.network.ChannelMultiplexer
|
||||
import com.hermesandroid.relay.network.models.Envelope
|
||||
@@ -111,22 +114,90 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
// modal in the loop (BridgeCommandHandler still gates /send_sms
|
||||
// through the destructive-verb confirmation).
|
||||
is VoiceIntent.SendSms -> {
|
||||
val resolvedNumber = resolveContactPhone(intent.contact)
|
||||
if (resolvedNumber == null) {
|
||||
Log.i(TAG, "C4 contact resolution failed for '${intent.contact}'")
|
||||
// Pre-flight SMS permission check — if SEND_SMS isn't granted
|
||||
// we'd burn through the 5s confirmation window + safety modal
|
||||
// and then silently fail inside executor.sendSms. Better to
|
||||
// skip dispatch entirely and speak an actionable message. The
|
||||
// LLM tool-calling path hits the same permission check in
|
||||
// ActionExecutor.sendSms (line 1414) — this is just moving the
|
||||
// voice-local check earlier to avoid the dead countdown.
|
||||
val smsPermission = checkSmsPermission()
|
||||
if (smsPermission != null) {
|
||||
IntentResult.Handled(
|
||||
intentLabel = "Send SMS",
|
||||
spokenConfirmation = "I couldn't find a contact called " +
|
||||
"${intent.contact}.",
|
||||
spokenConfirmation = smsPermission,
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"body" to intent.body,
|
||||
"error" to "permission_missing_sms",
|
||||
),
|
||||
)
|
||||
} else {
|
||||
handleDestructive(
|
||||
label = "Send SMS",
|
||||
spokenConfirmation = "About to text ${intent.contact} at " +
|
||||
"$resolvedNumber: ${intent.body}. Say cancel to stop.",
|
||||
envelope = buildSmsEnvelope(intent, resolvedNumber),
|
||||
)
|
||||
when (val resolution = resolveContactPhone(intent.contact)) {
|
||||
is ContactResolution.Found -> handleDestructive(
|
||||
label = "Send SMS",
|
||||
spokenConfirmation = "About to text ${intent.contact} at " +
|
||||
"${resolution.number}: ${intent.body}. Say cancel to stop.",
|
||||
envelope = buildSmsEnvelope(intent, resolution.number),
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"resolvedNumber" to resolution.number,
|
||||
"body" to intent.body,
|
||||
),
|
||||
)
|
||||
is ContactResolution.ServiceMissing -> IntentResult.Handled(
|
||||
intentLabel = "Send SMS",
|
||||
spokenConfirmation = "I can't reach the bridge service. " +
|
||||
"Make sure Hermes accessibility is enabled in Settings.",
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"error" to "service_missing",
|
||||
),
|
||||
)
|
||||
is ContactResolution.PermissionMissing -> IntentResult.Handled(
|
||||
intentLabel = "Send SMS",
|
||||
spokenConfirmation = "I need Contacts permission to look up " +
|
||||
"that number. Tap the Bridge tab to grant it.",
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"error" to "permission_missing_contacts",
|
||||
),
|
||||
)
|
||||
is ContactResolution.NotFound -> IntentResult.Handled(
|
||||
intentLabel = "Send SMS",
|
||||
spokenConfirmation = "I couldn't find a contact called " +
|
||||
"${intent.contact}.",
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"error" to "contact_not_found",
|
||||
),
|
||||
)
|
||||
is ContactResolution.NoPhoneNumber -> IntentResult.Handled(
|
||||
intentLabel = "Send SMS",
|
||||
spokenConfirmation = "${intent.contact} doesn't have a phone " +
|
||||
"number on file.",
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"error" to "contact_no_phone",
|
||||
),
|
||||
)
|
||||
is ContactResolution.OtherError -> IntentResult.Handled(
|
||||
intentLabel = "Send SMS",
|
||||
spokenConfirmation = "I hit an error looking up ${intent.contact}: " +
|
||||
resolution.message,
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"contact" to intent.contact,
|
||||
"error" to "other_error",
|
||||
"errorMessage" to resolution.message,
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
// H5: resolve the human app name → Android package name locally
|
||||
@@ -137,19 +208,53 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
// prefix → contains, all case-insensitive) covers most natural
|
||||
// utterances without needing a maintained alias table.
|
||||
is VoiceIntent.OpenApp -> {
|
||||
val resolvedPackage = resolveAppPackage(intent.appName)
|
||||
if (resolvedPackage == null) {
|
||||
Log.i(TAG, "H5 app resolution failed for '${intent.appName}'")
|
||||
IntentResult.Handled(
|
||||
when (val resolution = resolveAppPackage(intent.appName)) {
|
||||
is AppResolution.Found -> {
|
||||
Log.i(
|
||||
TAG,
|
||||
"H5 resolved '${intent.appName}' → '${resolution.label}' " +
|
||||
"(${resolution.packageName}) via ${resolution.matchTier}",
|
||||
)
|
||||
handleSafe(
|
||||
label = "Open App",
|
||||
envelope = buildOpenAppEnvelope(intent, resolution.packageName),
|
||||
details = mapOf(
|
||||
"requestedName" to intent.appName,
|
||||
"appLabel" to resolution.label,
|
||||
"packageName" to resolution.packageName,
|
||||
"matchTier" to resolution.matchTier,
|
||||
),
|
||||
)
|
||||
}
|
||||
is AppResolution.ServiceMissing -> IntentResult.Handled(
|
||||
intentLabel = "Open App",
|
||||
spokenConfirmation = "I can't reach the bridge service. " +
|
||||
"Make sure Hermes accessibility is enabled in Settings.",
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"requestedName" to intent.appName,
|
||||
"error" to "service_missing",
|
||||
),
|
||||
)
|
||||
is AppResolution.NotFound -> IntentResult.Handled(
|
||||
intentLabel = "Open App",
|
||||
spokenConfirmation = "I couldn't find an app called " +
|
||||
"${intent.appName}.",
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"requestedName" to intent.appName,
|
||||
"error" to "app_not_found",
|
||||
),
|
||||
)
|
||||
} else {
|
||||
handleSafe(
|
||||
label = "Open App",
|
||||
envelope = buildOpenAppEnvelope(intent, resolvedPackage),
|
||||
is AppResolution.OtherError -> IntentResult.Handled(
|
||||
intentLabel = "Open App",
|
||||
spokenConfirmation = resolution.message,
|
||||
requiresConfirmation = false,
|
||||
details = mapOf(
|
||||
"requestedName" to intent.appName,
|
||||
"error" to "other_error",
|
||||
"errorMessage" to resolution.message,
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -157,10 +262,6 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
label = "Tap",
|
||||
envelope = buildTapEnvelope(intent),
|
||||
)
|
||||
is VoiceIntent.Scroll -> handleSafe(
|
||||
label = "Scroll",
|
||||
envelope = buildScrollEnvelope(intent),
|
||||
)
|
||||
VoiceIntent.Back -> handleSafe(
|
||||
label = "Navigate back",
|
||||
envelope = buildPressKeyEnvelope("back"),
|
||||
@@ -189,6 +290,7 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
label: String,
|
||||
spokenConfirmation: String,
|
||||
envelope: Envelope,
|
||||
details: Map<String, String> = emptyMap(),
|
||||
): IntentResult.Handled {
|
||||
pendingJob = ownScope.launch {
|
||||
try {
|
||||
@@ -203,18 +305,21 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
intentLabel = label,
|
||||
spokenConfirmation = spokenConfirmation,
|
||||
requiresConfirmation = true,
|
||||
details = details,
|
||||
)
|
||||
}
|
||||
|
||||
private suspend fun handleSafe(
|
||||
label: String,
|
||||
envelope: Envelope,
|
||||
details: Map<String, String> = emptyMap(),
|
||||
): IntentResult.Handled {
|
||||
dispatch(envelope)
|
||||
return IntentResult.Handled(
|
||||
intentLabel = label,
|
||||
spokenConfirmation = null,
|
||||
requiresConfirmation = false,
|
||||
details = details,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -290,29 +395,63 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* C4 resolver: voice contact name → first phone number on the best
|
||||
* matching contact, via the same `ActionExecutor.searchContacts`
|
||||
* code path that backs the `/search_contacts` bridge route.
|
||||
* Pre-flight SEND_SMS runtime permission check. Returns null if
|
||||
* granted, or a ready-to-speak error message if denied or if the
|
||||
* a11y service isn't connected (no context to query against).
|
||||
*
|
||||
* Why here and not deferred to `ActionExecutor.sendSms`: that inner
|
||||
* check IS what catches real-world failures, but only after the
|
||||
* 5-second confirmation countdown and the safety-modal flow have
|
||||
* already run. Failing that far into the dispatch means the user
|
||||
* sees the preview, hears the countdown, taps Allow on the modal,
|
||||
* and THEN gets silent failure when the SmsManager call returns
|
||||
* "permission denied". Pulling the check forward lets us short-
|
||||
* circuit with an accurate spoken message before any of that fires.
|
||||
*/
|
||||
private fun checkSmsPermission(): String? {
|
||||
val service = HermesAccessibilityService.instance
|
||||
?: return "I can't reach the bridge service. Make sure Hermes " +
|
||||
"accessibility is enabled in Settings."
|
||||
val granted = ContextCompat.checkSelfPermission(
|
||||
service, Manifest.permission.SEND_SMS
|
||||
) == PackageManager.PERMISSION_GRANTED
|
||||
return if (granted) {
|
||||
null
|
||||
} else {
|
||||
"I need SMS permission to send a text. Tap the Bridge tab to grant it."
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* C4 resolver: voice contact name → classified resolution result, via
|
||||
* the same `ActionExecutor.searchContacts` code path that backs the
|
||||
* `/search_contacts` bridge route.
|
||||
*
|
||||
* Pre-v0.4.0 this returned `String?` and collapsed every failure mode
|
||||
* (service missing / permission missing / no match / no phone number)
|
||||
* to `null`. The voice handler then hardcoded "I couldn't find a
|
||||
* contact called X" which was honest only in the last case — users
|
||||
* who hadn't granted READ_CONTACTS heard "not found" and went looking
|
||||
* for a typo instead of the real permission gap. Bailey hit this
|
||||
* 2026-04-15. Returning a [ContactResolution] lets the caller speak
|
||||
* an accurate message per category.
|
||||
*
|
||||
* `ActionExecutor.searchContacts` itself pre-checks the READ_CONTACTS
|
||||
* runtime permission and returns a distinctive error string ("Grant
|
||||
* contacts permission...") when it's missing. We substring-match that
|
||||
* string here to classify — not the cleanest contract but pragmatic,
|
||||
* and it avoids a breaking change to the ActionResult shape that the
|
||||
* LLM tool-calling path also depends on.
|
||||
*
|
||||
* Implementation note: `searchContacts` returns the contact's phones
|
||||
* as a comma-joined string ("`+15551234567, +15559876543`") because
|
||||
* that's the wire shape the C2 tier designed for the agent's prose
|
||||
* output. For the voice flow we only need a single number to dial,
|
||||
* so we split on `,` and take the first non-blank entry. Trimming is
|
||||
* required because the join uses `", "` (comma + space).
|
||||
*
|
||||
* Returns null when:
|
||||
* - the accessibility service is not connected
|
||||
* - searchContacts returns ok=false (e.g. contacts permission
|
||||
* missing — the user gets a spoken "couldn't find" response
|
||||
* and can grant the permission then retry)
|
||||
* - the contacts list is empty (no match for the spoken name)
|
||||
* - the matched contact has no phone numbers stored
|
||||
* as a comma-joined string ("+15551234567, +15559876543"). For voice
|
||||
* we only need a single number to dial, so we split on `,` and take
|
||||
* the first non-blank entry.
|
||||
*/
|
||||
private suspend fun resolveContactPhone(contactName: String): String? {
|
||||
private suspend fun resolveContactPhone(contactName: String): ContactResolution {
|
||||
val service = HermesAccessibilityService.instance ?: run {
|
||||
Log.w(TAG, "C4 resolveContactPhone: a11y service not connected")
|
||||
return null
|
||||
return ContactResolution.ServiceMissing
|
||||
}
|
||||
// ContactsContract queries hit the on-device content provider —
|
||||
// wrap in IO dispatcher so the voice coroutine doesn't block on
|
||||
@@ -321,17 +460,45 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
service.actionExecutor.searchContacts(contactName, limit = 5)
|
||||
}
|
||||
if (!result.ok) {
|
||||
Log.i(TAG, "C4 searchContacts failed: ${result.error}")
|
||||
return null
|
||||
val err = result.error.orEmpty()
|
||||
Log.i(TAG, "C4 searchContacts failed: $err")
|
||||
// ActionExecutor.searchContacts uses specific phrasing for the
|
||||
// permission-denied paths ("Grant contacts permission" for the
|
||||
// upfront check, "Contacts permission revoked" for the mid-query
|
||||
// SecurityException). Either phrase → permission missing.
|
||||
val lower = err.lowercase()
|
||||
return if ("contacts permission" in lower || "permission revoked" in lower) {
|
||||
ContactResolution.PermissionMissing
|
||||
} else {
|
||||
ContactResolution.OtherError(err.ifBlank { "unknown error" })
|
||||
}
|
||||
}
|
||||
@Suppress("UNCHECKED_CAST")
|
||||
val contacts = result.data["contacts"] as? List<Map<String, Any?>>
|
||||
?: return null
|
||||
if (contacts.isEmpty()) return null
|
||||
val phonesField = contacts.first()["phones"] as? String ?: return null
|
||||
return phonesField.split(",")
|
||||
?: return ContactResolution.NotFound
|
||||
if (contacts.isEmpty()) return ContactResolution.NotFound
|
||||
val phonesField = contacts.first()["phones"] as? String
|
||||
?: return ContactResolution.NoPhoneNumber
|
||||
val number = phonesField.split(",")
|
||||
.map { it.trim() }
|
||||
.firstOrNull { it.isNotBlank() }
|
||||
?: return ContactResolution.NoPhoneNumber
|
||||
return ContactResolution.Found(number)
|
||||
}
|
||||
|
||||
/** Result of the C4 contact → phone number lookup. */
|
||||
private sealed class ContactResolution {
|
||||
data class Found(val number: String) : ContactResolution()
|
||||
/** a11y service not connected — whole bridge pipeline is offline. */
|
||||
data object ServiceMissing : ContactResolution()
|
||||
/** READ_CONTACTS runtime permission not granted. */
|
||||
data object PermissionMissing : ContactResolution()
|
||||
/** Query succeeded but no contact matched the spoken name. */
|
||||
data object NotFound : ContactResolution()
|
||||
/** Matched a contact but the record has no stored phone number. */
|
||||
data object NoPhoneNumber : ContactResolution()
|
||||
/** Everything else — surface the raw error so the user gets a hint. */
|
||||
data class OtherError(val message: String) : ContactResolution()
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -358,10 +525,10 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
* same release). Without it Android 11+ silently returns a near-
|
||||
* empty candidate list and every match attempt fails.
|
||||
*/
|
||||
private suspend fun resolveAppPackage(appName: String): String? {
|
||||
private suspend fun resolveAppPackage(appName: String): AppResolution {
|
||||
val service = HermesAccessibilityService.instance ?: run {
|
||||
Log.w(TAG, "H5 resolveAppPackage: a11y service not connected")
|
||||
return null
|
||||
return AppResolution.ServiceMissing
|
||||
}
|
||||
val pm = service.packageManager
|
||||
// PackageManager queries can be slow on devices with many apps
|
||||
@@ -381,21 +548,42 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
}
|
||||
}
|
||||
if (candidates.isEmpty()) {
|
||||
// Distinctive failure mode: PackageManager returned nothing
|
||||
// because the <queries> manifest declaration is missing. The
|
||||
// user can't fix this themselves — it's a build bug — so
|
||||
// report as OtherError rather than NotFound so the spoken
|
||||
// confirmation doesn't blame the user's vocabulary.
|
||||
Log.w(TAG, "H5 candidate list empty — missing <queries> in manifest?")
|
||||
return null
|
||||
return AppResolution.OtherError("Launcher app list empty. This is a build issue — please report it.")
|
||||
}
|
||||
|
||||
val needle = appName.trim().lowercase()
|
||||
// Tier 1: exact label match.
|
||||
candidates.firstOrNull { it.second.lowercase() == needle }
|
||||
?.let { return it.first }
|
||||
?.let { return AppResolution.Found(it.first, it.second, "exact") }
|
||||
// Tier 2: label starts with the needle.
|
||||
candidates.firstOrNull { it.second.lowercase().startsWith(needle) }
|
||||
?.let { return it.first }
|
||||
?.let { return AppResolution.Found(it.first, it.second, "prefix") }
|
||||
// Tier 3: label contains the needle anywhere.
|
||||
candidates.firstOrNull { it.second.lowercase().contains(needle) }
|
||||
?.let { return it.first }
|
||||
return null
|
||||
?.let { return AppResolution.Found(it.first, it.second, "contains") }
|
||||
return AppResolution.NotFound
|
||||
}
|
||||
|
||||
/** Result of the H5 launcher-inventory fuzzy match. */
|
||||
private sealed class AppResolution {
|
||||
data class Found(
|
||||
val packageName: String,
|
||||
val label: String,
|
||||
/** Which matcher tier won — `exact` | `prefix` | `contains`. */
|
||||
val matchTier: String,
|
||||
) : AppResolution()
|
||||
/** a11y service not connected. */
|
||||
data object ServiceMissing : AppResolution()
|
||||
/** PackageManager returned candidates but none matched. */
|
||||
data object NotFound : AppResolution()
|
||||
/** Unexpected condition (e.g. empty candidate list due to manifest bug). */
|
||||
data class OtherError(val message: String) : AppResolution()
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
@@ -472,20 +660,6 @@ internal class RealVoiceBridgeIntentHandler(
|
||||
},
|
||||
)
|
||||
|
||||
private fun buildScrollEnvelope(i: VoiceIntent.Scroll): Envelope = Envelope(
|
||||
channel = "bridge",
|
||||
type = "bridge.command",
|
||||
payload = buildJsonObject {
|
||||
put("request_id", java.util.UUID.randomUUID().toString())
|
||||
put("method", "POST")
|
||||
put("path", "/scroll")
|
||||
put("body", buildJsonObject {
|
||||
put("direction", i.direction.name.lowercase())
|
||||
})
|
||||
put("source", "voice")
|
||||
},
|
||||
)
|
||||
|
||||
private fun buildPressKeyEnvelope(key: String): Envelope = Envelope(
|
||||
channel = "bridge",
|
||||
type = "bridge.command",
|
||||
|
||||
@@ -16,13 +16,21 @@ package com.hermesandroid.relay.voice
|
||||
*
|
||||
* | Intent | Example phrases | Parsed out |
|
||||
* |---|---|---|
|
||||
* | [VoiceIntent.SendSms] | `text Sam I'll be 10 min late` / `send a message to mom saying on my way` / `text my wife hey` | `contact`, `body` |
|
||||
* | [VoiceIntent.SendSms] | `text Sam I'll be 10 min late` / `send a message to mom saying on my way` / `send Hannah a text saying smoke test` / `message Sam saying hi` | `contact`, `body` |
|
||||
* | [VoiceIntent.OpenApp] | `open camera` / `launch maps` / `open the spotify app` / `start gmail` | `appName` |
|
||||
* | [VoiceIntent.Tap] | `tap send` / `press the ok button` / `click on continue` | `target` |
|
||||
* | [VoiceIntent.Scroll] | `scroll down` / `scroll up` / `scroll to the top` / `scroll to the bottom` | `direction` |
|
||||
* | [VoiceIntent.Back] | `go back` / `navigate back` / `back` | — |
|
||||
* | [VoiceIntent.Home] | `press home` / `go home` / `home screen` | — |
|
||||
*
|
||||
* ## Scroll — removed from voice fast-path
|
||||
*
|
||||
* Scroll used to be a voice intent but was removed in v0.4.0: nobody
|
||||
* actually says "scroll down" aloud to their phone in practice, and the
|
||||
* regex was maintenance debt for a near-zero-usage intent. The /scroll
|
||||
* bridge HTTP route and [ActionExecutor.scroll] are still fully functional
|
||||
* — server-side agents calling `android_scroll` via the WSS bridge path
|
||||
* still work. Only the voice classifier shortcut was removed.
|
||||
*
|
||||
* ## Tuning
|
||||
*
|
||||
* Each regex is intentionally simple. False negatives (classifier says
|
||||
@@ -40,11 +48,8 @@ internal sealed class VoiceIntent {
|
||||
data class SendSms(val contact: String, val body: String) : VoiceIntent()
|
||||
data class OpenApp(val appName: String) : VoiceIntent()
|
||||
data class Tap(val target: String) : VoiceIntent()
|
||||
data class Scroll(val direction: ScrollDirection) : VoiceIntent()
|
||||
data object Back : VoiceIntent()
|
||||
data object Home : VoiceIntent()
|
||||
|
||||
enum class ScrollDirection { UP, DOWN, TOP, BOTTOM }
|
||||
}
|
||||
|
||||
internal object VoiceIntentClassifier {
|
||||
@@ -57,17 +62,38 @@ internal object VoiceIntentClassifier {
|
||||
RegexOption.IGNORE_CASE,
|
||||
)
|
||||
|
||||
// text X saying Y | text X: Y | send a message to X saying Y | text X Y
|
||||
// Direct form:
|
||||
// text X saying Y | text X: Y | send a message to X saying Y | message X saying Y
|
||||
// Ordered: the "saying"/":" variants match before the no-separator form
|
||||
// so we don't greedily swallow the body into the contact field.
|
||||
private val SMS_WITH_SEPARATOR = Regex(
|
||||
"^(?:text|send\\s+(?:a\\s+)?(?:message|text|sms)\\s+to)\\s+" +
|
||||
"^(?:text|message|send\\s+(?:a\\s+)?(?:message|text|sms)\\s+to)\\s+" +
|
||||
"(?<contact>[\\w'\\- ]+?)\\s+" +
|
||||
"(?:saying|that|:\\s*|,\\s*)\\s*" +
|
||||
"(?<body>.+)$",
|
||||
RegexOption.IGNORE_CASE,
|
||||
)
|
||||
|
||||
// Indirect-object form:
|
||||
// send X a text saying Y | send X a message that Y | shoot X an sms: Y
|
||||
// English has two equally-natural SMS phrasings — direct ("text Sam hello",
|
||||
// "send a text to Sam") and indirect ("send Sam a text"). Pre-0.4.0 the
|
||||
// classifier only covered direct, so utterances like "Can you send Hannah
|
||||
// a text saying hi?" (filler-stripped to "send Hannah a text saying hi")
|
||||
// fell through to the LLM. Bailey hit this 2026-04-15. This pattern
|
||||
// anchors on the second "text/message/sms" keyword AFTER the contact,
|
||||
// which prevents it from false-matching the direct "send a text to X"
|
||||
// form (no second keyword there). Non-greedy contact capture is safe
|
||||
// because the anchor is mandatory.
|
||||
private val SMS_INDIRECT = Regex(
|
||||
"^(?:send|shoot)\\s+" +
|
||||
"(?<contact>[\\w'\\- ]+?)\\s+" +
|
||||
"(?:a\\s+|an\\s+)(?:text|message|sms)\\s+" +
|
||||
"(?:saying|that|:\\s*|,\\s*)\\s*" +
|
||||
"(?<body>.+)$",
|
||||
RegexOption.IGNORE_CASE,
|
||||
)
|
||||
|
||||
// Fallback: "text <contact> <body>" where <contact> is one or two words.
|
||||
// Keeps the contact parse conservative so "text my wife" without a body
|
||||
// falls through to NotApplicable instead of sending an empty SMS.
|
||||
@@ -88,11 +114,6 @@ internal object VoiceIntentClassifier {
|
||||
RegexOption.IGNORE_CASE,
|
||||
)
|
||||
|
||||
private val SCROLL = Regex(
|
||||
"^scroll\\s+(?:to\\s+the\\s+)?(?<dir>up|down|top|bottom)\\s*$",
|
||||
RegexOption.IGNORE_CASE,
|
||||
)
|
||||
|
||||
private val BACK = Regex(
|
||||
"^(?:go\\s+)?(?:navigate\\s+)?back\\s*$",
|
||||
RegexOption.IGNORE_CASE,
|
||||
@@ -124,6 +145,13 @@ internal object VoiceIntentClassifier {
|
||||
return VoiceIntent.SendSms(contact = contact, body = body)
|
||||
}
|
||||
}
|
||||
SMS_INDIRECT.matchEntire(text)?.let { m ->
|
||||
val contact = m.groups["contact"]?.value?.trim().orEmpty()
|
||||
val body = m.groups["body"]?.value?.trim().orEmpty()
|
||||
if (contact.isNotEmpty() && body.isNotEmpty()) {
|
||||
return VoiceIntent.SendSms(contact = contact, body = body)
|
||||
}
|
||||
}
|
||||
SMS_NO_SEPARATOR.matchEntire(text)?.let { m ->
|
||||
val contact = m.groups["contact"]?.value?.trim().orEmpty()
|
||||
val body = m.groups["body"]?.value?.trim().orEmpty()
|
||||
@@ -139,16 +167,6 @@ internal object VoiceIntentClassifier {
|
||||
val target = m.groups["target"]?.value?.trim().orEmpty()
|
||||
if (target.isNotEmpty()) return VoiceIntent.Tap(target)
|
||||
}
|
||||
SCROLL.matchEntire(text)?.let { m ->
|
||||
val dir = when (m.groups["dir"]?.value?.lowercase()) {
|
||||
"up" -> VoiceIntent.ScrollDirection.UP
|
||||
"down" -> VoiceIntent.ScrollDirection.DOWN
|
||||
"top" -> VoiceIntent.ScrollDirection.TOP
|
||||
"bottom" -> VoiceIntent.ScrollDirection.BOTTOM
|
||||
else -> null
|
||||
}
|
||||
if (dir != null) return VoiceIntent.Scroll(dir)
|
||||
}
|
||||
if (BACK.matchEntire(text) != null) return VoiceIntent.Back
|
||||
if (HOME.matchEntire(text) != null) return VoiceIntent.Home
|
||||
|
||||
|
||||
+175
-17
@@ -46,12 +46,43 @@ def _auth_headers() -> dict:
|
||||
return {}
|
||||
|
||||
def _check_requirements() -> bool:
|
||||
"""Returns True if the relay is running and a phone is connected."""
|
||||
"""Returns True if the relay is running, a phone is connected, and the
|
||||
accessibility service is granted.
|
||||
|
||||
Uses ``/bridge/status`` — NOT ``/ping``. The relay's ``/ping`` is a pure
|
||||
liveness probe returning ``{pong, ts}`` and lacks the
|
||||
``phone_connected`` / ``bridge.accessibility_granted`` fields this
|
||||
function needs. The older code here pointed at ``/ping`` and checked
|
||||
for those fields anyway, which meant ``_check_requirements`` always
|
||||
returned ``False`` and every tool except ``android_setup`` was
|
||||
silently hidden from the gateway's tool pool. Caught 2026-04-15 by
|
||||
Bailey: Victor reported "no android_* tools available" while the
|
||||
Android app's bridge + accessibility were both healthy, and a direct
|
||||
curl against ``/bridge/status`` returned ``phone_connected: true`` +
|
||||
``bridge.accessibility_granted: true``.
|
||||
|
||||
Gating on BOTH ``phone_connected`` AND ``accessibility_granted`` is
|
||||
deliberate: if accessibility is revoked (common after an Android
|
||||
Studio reinstall — Android's ``AccessibilityManagerService`` scrubs
|
||||
enabled services on package replacement), the phone-side
|
||||
``BridgeCommandHandler.dispatch()`` short-circuits with HTTP 503 on
|
||||
every tool. Hiding the tools when a11y is missing keeps the LLM
|
||||
honest about capability instead of letting it confidently dispatch
|
||||
commands that will fail.
|
||||
"""
|
||||
try:
|
||||
r = requests.get(f"{_bridge_url()}/ping", headers=_auth_headers(), timeout=2)
|
||||
r = requests.get(
|
||||
f"{_bridge_url()}/bridge/status",
|
||||
headers=_auth_headers(),
|
||||
timeout=2,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
return data.get("phone_connected", False) or data.get("accessibilityService", False)
|
||||
phone_connected = bool(data.get("phone_connected", False))
|
||||
a11y_granted = bool(
|
||||
data.get("bridge", {}).get("accessibility_granted", False)
|
||||
)
|
||||
return phone_connected and a11y_granted
|
||||
return False
|
||||
except Exception:
|
||||
return False
|
||||
@@ -296,6 +327,73 @@ def android_current_app() -> str:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
# ── Tier C tools: direct contact / SMS / call dispatch ────────────────────────
|
||||
#
|
||||
# These wrap phone-side routes that were already fully implemented
|
||||
# (/search_contacts, /send_sms, /call in BridgeCommandHandler + matching
|
||||
# ActionExecutor methods with runtime permission pre-checks) but had never
|
||||
# been exposed as LLM-callable tools. Pre-0.4.0 the only way for an
|
||||
# agent to send an SMS was to drive the Messages app step-by-step via
|
||||
# open_app + read_screen + tap_text + type — fragile across OEMs and slow.
|
||||
# Direct SmsManager dispatch is safer and matches what the voice fast-path
|
||||
# already does locally.
|
||||
#
|
||||
# The phone handles all the hard parts: runtime permission pre-check,
|
||||
# destructive-verb confirmation modal (user taps Allow/Deny before the
|
||||
# action fires), multi-part SMS segmentation, delivery ack. If the flavor
|
||||
# is googlePlay (not sideload) the phone returns 403 with a clear message
|
||||
# and the agent can degrade to the UI-automation path.
|
||||
|
||||
def android_search_contacts(query: str, limit: int = 20) -> str:
|
||||
"""Search the phone's contact book by name. Returns matching contacts
|
||||
with their phone numbers. Requires READ_CONTACTS permission on phone."""
|
||||
try:
|
||||
data = _post("/search_contacts", {"query": query, "limit": limit})
|
||||
return json.dumps(data)
|
||||
except Exception as e:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
def android_send_sms(to: str, body: str) -> str:
|
||||
"""Send an SMS directly via the phone's SmsManager (not by driving the
|
||||
Messages app UI). `to` must be a phone number — use android_search_contacts
|
||||
first if you only have a name. The phone shows a confirmation modal to
|
||||
the user before the message fires; this call blocks until they tap
|
||||
Allow or Deny (up to the configured confirmation timeout)."""
|
||||
try:
|
||||
data = _post("/send_sms", {"to": to, "body": body})
|
||||
return json.dumps(data)
|
||||
except Exception as e:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
def android_call(number: str) -> str:
|
||||
"""Dial a phone number. With CALL_PHONE granted (sideload) the call is
|
||||
auto-placed; without it the system dialer opens pre-populated and the
|
||||
user taps Call manually. Phone shows a confirmation modal before either
|
||||
mode — blocks until the user reacts."""
|
||||
try:
|
||||
data = _post("/call", {"number": number})
|
||||
return json.dumps(data)
|
||||
except Exception as e:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
def android_return_to_hermes() -> str:
|
||||
"""Bring the Hermes Relay app back to the foreground on the user's
|
||||
phone. Call this as the FINAL step of any multi-app task (sending a
|
||||
text, opening Maps, taking a screenshot from another app) so the user
|
||||
sees your reply in-context without having to manually switch apps.
|
||||
Do NOT call this mid-task — only when you're ready to hand control
|
||||
back. Allowed even when Bridge master toggle is off, so you can
|
||||
always wrap up cleanly."""
|
||||
try:
|
||||
data = _post("/return_to_hermes", {})
|
||||
return json.dumps(data)
|
||||
except Exception as e:
|
||||
return json.dumps({"error": str(e)})
|
||||
|
||||
|
||||
def _get_public_ip() -> str:
|
||||
"""Detect this server's public IP address."""
|
||||
for service in ["https://api.ipify.org", "https://ifconfig.me/ip", "https://icanhazip.com"]:
|
||||
@@ -586,23 +684,83 @@ _SCHEMAS = {
|
||||
"required": ["pairing_code"],
|
||||
},
|
||||
},
|
||||
"android_search_contacts": {
|
||||
"name": "android_search_contacts",
|
||||
"description": "Search the phone's contact book by name. Returns matching contacts with their phone numbers. Use this BEFORE android_send_sms or android_call when the user gives a contact name rather than a phone number. Requires READ_CONTACTS permission on phone.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "Name or partial name to search for (e.g. 'Hannah', 'mom', 'dr. smith').",
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": "Max matches to return. Default 20.",
|
||||
"default": 20,
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
"android_send_sms": {
|
||||
"name": "android_send_sms",
|
||||
"description": "Send an SMS directly via the phone's native SmsManager — does NOT drive the Messages app UI. Much more reliable than open_app + tap_text + type. The `to` argument MUST be a phone number (use android_search_contacts first if you only have a name). The phone shows a confirmation modal before sending; this call blocks until the user taps Allow or Deny. Sideload flavor only — returns 403 on Google Play builds.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"to": {
|
||||
"type": "string",
|
||||
"description": "Phone number (E.164 or local format, e.g. '+15551234567' or '555-123-4567').",
|
||||
},
|
||||
"body": {
|
||||
"type": "string",
|
||||
"description": "Message body text. Multi-part messages (>160 chars) are segmented automatically.",
|
||||
},
|
||||
},
|
||||
"required": ["to", "body"],
|
||||
},
|
||||
},
|
||||
"android_call": {
|
||||
"name": "android_call",
|
||||
"description": "Place a phone call. With CALL_PHONE granted (sideload flavor) the call is auto-dialed; otherwise the system dialer opens pre-populated and the user taps Call manually. Phone shows a confirmation modal before either mode and blocks until the user reacts. Use android_search_contacts first if you only have a name.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"number": {
|
||||
"type": "string",
|
||||
"description": "Phone number to dial (E.164 or local format).",
|
||||
},
|
||||
},
|
||||
"required": ["number"],
|
||||
},
|
||||
},
|
||||
"android_return_to_hermes": {
|
||||
"name": "android_return_to_hermes",
|
||||
"description": "Bring the Hermes Relay app on the phone back to the foreground. Call this as the FINAL step of any phone-control task that opened or brought focus to another app (Messages, Maps, Chrome, etc.) so the user sees your reply in-context without manually switching apps. Do NOT call mid-task — only when you're ready to hand control back. Allowed even when the Bridge master toggle is disabled.",
|
||||
"parameters": {"type": "object", "properties": {}, "required": []},
|
||||
},
|
||||
}
|
||||
|
||||
# ── Tool handlers map ──────────────────────────────────────────────────────────
|
||||
|
||||
_HANDLERS = {
|
||||
"android_ping": lambda args, **kw: android_ping(),
|
||||
"android_read_screen": lambda args, **kw: android_read_screen(**args),
|
||||
"android_tap": lambda args, **kw: android_tap(**args),
|
||||
"android_tap_text": lambda args, **kw: android_tap_text(**args),
|
||||
"android_type": lambda args, **kw: android_type(**args),
|
||||
"android_swipe": lambda args, **kw: android_swipe(**args),
|
||||
"android_open_app": lambda args, **kw: android_open_app(**args),
|
||||
"android_press_key": lambda args, **kw: android_press_key(**args),
|
||||
"android_screenshot": lambda args, **kw: android_screenshot(),
|
||||
"android_scroll": lambda args, **kw: android_scroll(**args),
|
||||
"android_wait": lambda args, **kw: android_wait(**args),
|
||||
"android_get_apps": lambda args, **kw: android_get_apps(),
|
||||
"android_current_app": lambda args, **kw: android_current_app(),
|
||||
"android_setup": lambda args, **kw: android_setup(**args),
|
||||
"android_ping": lambda args, **kw: android_ping(),
|
||||
"android_read_screen": lambda args, **kw: android_read_screen(**args),
|
||||
"android_tap": lambda args, **kw: android_tap(**args),
|
||||
"android_tap_text": lambda args, **kw: android_tap_text(**args),
|
||||
"android_type": lambda args, **kw: android_type(**args),
|
||||
"android_swipe": lambda args, **kw: android_swipe(**args),
|
||||
"android_open_app": lambda args, **kw: android_open_app(**args),
|
||||
"android_press_key": lambda args, **kw: android_press_key(**args),
|
||||
"android_screenshot": lambda args, **kw: android_screenshot(),
|
||||
"android_scroll": lambda args, **kw: android_scroll(**args),
|
||||
"android_wait": lambda args, **kw: android_wait(**args),
|
||||
"android_get_apps": lambda args, **kw: android_get_apps(),
|
||||
"android_current_app": lambda args, **kw: android_current_app(),
|
||||
"android_setup": lambda args, **kw: android_setup(**args),
|
||||
"android_search_contacts": lambda args, **kw: android_search_contacts(**args),
|
||||
"android_send_sms": lambda args, **kw: android_send_sms(**args),
|
||||
"android_call": lambda args, **kw: android_call(**args),
|
||||
"android_return_to_hermes": lambda args, **kw: android_return_to_hermes(),
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user