merge: phase 3 wave 1 bridge team (α β γ δ ε + followups)
# Conflicts: # DEVLOG.md
This commit is contained in:
Bailey Dixon
@@ -82,8 +82,7 @@ hermes-android/ ← Android Studio opens this root
|
||||
├── gradle/ ← Wrapper (8.13) + version catalog
|
||||
├── scripts/ ← Dev scripts (build, install, run, test, relay)
|
||||
├── plugin/ ← Hermes agent plugin (14 android_* tools + relay + pair CLI)
|
||||
│ ├── android_tool.py
|
||||
│ ├── android_relay.py
|
||||
│ ├── android_tool.py # 14 android_* tool handlers; points BRIDGE_URL at the unified relay (localhost:8767) as of Phase 3 Wave 1
|
||||
│ ├── pair.py # QR pairing implementation — `python -m plugin.pair`, wrapped by `hermes-pair` shim and the `hermes-relay-pair` skill
|
||||
│ ├── cli.py # Registers plugin CLI sub-commands (note: top-level `hermes pair` blocked by upstream argparser gap — use slash command or shell shim)
|
||||
│ ├── relay/ # Canonical WSS relay (consolidated from relay_server/)
|
||||
@@ -175,6 +174,7 @@ hermes-android/ ← Android Studio opens this root
|
||||
| `plugin/relay/auth.py` | PairingManager (generate + register_code), SessionManager, RateLimiter |
|
||||
| `plugin/relay/config.py` | RelayConfig + PAIRING_ALPHABET (full A-Z / 0-9 as of 2026-04-11) |
|
||||
| `plugin/relay/channels/terminal.py` | Phase 2 PTY-backed terminal handler |
|
||||
| `plugin/relay/channels/bridge.py` | Phase 3 bridge channel handler — routes agent tool calls to the connected phone over the unified WSS relay. `BridgeHandler.handle_command(method, path, params, body)` mints a `request_id`, sends a `bridge.command` envelope, and awaits a matching `bridge.response` with 30s timeout. `handle(ws, envelope)` opportunistically latches `phone_ws` and dispatches inbound `bridge.response`/`bridge.status`. `detach_ws(ws, reason)` fails all pending futures with `ConnectionError` on phone disconnect so HTTP callers fail fast. Migrated from the legacy standalone `plugin/tools/android_relay.py` (port 8766) into the unified relay on port 8767 in Phase 3 Wave 1 (Agent α, 2026-04-12). Wire protocol is frozen — envelope fields match the legacy relay byte-for-byte. 14 HTTP routes (`/ping`, `/screen`, `/screenshot`, `/get_apps`, `/apps` legacy, `/current_app`, `/tap`, `/tap_text`, `/type`, `/swipe`, `/open_app`, `/press_key`, `/scroll`, `/wait`, `/setup`) are registered in `plugin/relay/server.py` between `# === PHASE3-α ===` markers and delegate straight through to `handle_command`. |
|
||||
| `relay_server/__main__.py` | Thin shim → `plugin.relay.server.main()` — legacy `python -m relay_server` entrypoint |
|
||||
| `relay_server/SKILL.md` | Hermes skill reference for relay self-setup |
|
||||
| `relay_server/Dockerfile` | Container image for relay server |
|
||||
@@ -210,9 +210,22 @@ hermes-android/ ← Android Studio opens this root
|
||||
| `app/src/main/kotlin/.../ui/components/VoiceWaveform.kt` | Reactive layered-sine-wave visualizer mounted below the MorphingSphere. Three overlapping sine waves at co-prime frequencies (1.2/2.1/3.4) with amplitude-driven phase velocity via `rememberAmplitudeDrivenPhases` (`withFrameNanos` ticker, base durations 2000/1400/950 ms, speed up to 3.5× at peak amplitude via `PHASE_AMP_BOOST`). Pill-edge merge via dual technique: geometric `sin(π·t)` taper forces wave to centerY at both endpoints + `BlendMode.DstIn` horizontal gradient mask inside `saveLayer`. Color-keyed to `VoiceState` (blue/purple Listening, green/teal Speaking). No Compose spring — amplitude comes pre-smoothed from the ViewModel's envelope follower. |
|
||||
| `app/src/main/kotlin/.../ui/screens/VoiceSettingsScreen.kt` | Voice settings sub-screen — interaction mode radio list, silence threshold slider (1–10s), Auto-TTS switch ("coming soon"), TTS/STT provider info from `/voice/config`, language picker (stored to `VoicePreferences` but relay auto-detects for now), Test Voice button calls `VoiceViewModel.testVoice(sample)`. |
|
||||
| `app/src/main/kotlin/.../data/VoicePreferences.kt` | DataStore repo mirroring `MediaSettings.kt`. Keys: `voice_interaction_mode` (tap/hold/continuous), `voice_silence_threshold_ms` (default 3000), `voice_auto_tts` (default false, reserved), `voice_language` (default ""). |
|
||||
| `app/src/main/kotlin/.../ui/screens/BridgeScreen.kt` | Phase 3 δ rewrite — four-card control surface (master toggle, status, permission checklist, activity log) + inert Safety placeholder. Owns a `BridgeViewModel` via default `viewModel()` param. Re-probes permission state on `Lifecycle.Event.ON_RESUME` so returning from Android Settings flips the a11y row from red to green without navigation churn. |
|
||||
| `app/src/main/kotlin/.../viewmodel/BridgeViewModel.kt` | Phase 3 δ — AndroidViewModel for BridgeScreen. Exposes `masterToggle`/`bridgeStatus`/`permissionStatus`/`activityLog` StateFlows. Reads `Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES` + `Settings.canDrawOverlays` + `enabled_notification_listeners`. Seeds `BridgeStatus` from `PowerManager.isInteractive` + `BatteryManager.BATTERY_PROPERTY_CAPACITY`. **γ-handoff stubs** marked with `TODO(γ-handoff)`: replace `_bridgeStatus` MutableStateFlow with the real `HermesAccessibilityService` status flow; confirm `A11Y_SERVICE_CLASS` FQCN; wire γ's command dispatcher to call `recordActivity(entry)`. |
|
||||
| `app/src/main/kotlin/.../data/BridgePreferences.kt` | Phase 3 δ — DataStore repo backing `BridgeViewModel`. Keys: `bridge_master_enabled` (Boolean) + `bridge_activity_log` (JSON-serialized `List<BridgeActivityEntry>` capped at `MAX_LOG_ENTRIES=100`). `appendEntry` dedupes by id so Pending → Success/Failed transitions replace in place. Lenient `Json` for forward-compat. |
|
||||
| `app/src/main/kotlin/.../ui/components/BridgeMasterToggle.kt` | Phase 3 δ — headline "Allow Agent Control" Switch card. Inline device/battery/screen/current-app rows, Play-review-required explanation dialog behind the info icon, `accessibilityGranted` gate blocks enabling before a11y permission is granted. |
|
||||
| `app/src/main/kotlin/.../ui/components/BridgeStatusCard.kt` | Phase 3 δ — standalone bridge status card. Reuses `ConnectionStatusBadge` for the pulsing connected/disconnected dot. Separate from `BridgeMasterToggle` so Agent ζ can rearrange cards without losing state. |
|
||||
| `app/src/main/kotlin/.../ui/components/BridgePermissionChecklist.kt` | Phase 3 δ — four-row permission checklist (Accessibility / Screen Capture / Overlay / Notification Listener). Tap-to-open Android Settings via `ACTION_ACCESSIBILITY_SETTINGS`, `ACTION_MANAGE_OVERLAY_PERMISSION`, `enabled_notification_listeners`. Each launcher wrapped in `runCatching` for OEM-skin safety. Green check vs. red empty-circle status icons. |
|
||||
| `app/src/main/kotlin/.../ui/components/BridgeActivityLog.kt` | Phase 3 δ — scrollable activity log. `LazyColumn` bounded at `heightIn(max = 320.dp)` with tap-to-expand rows showing full timestamp, status, result text, and optional screenshot token. `java.time.DateTimeFormatter` for `HH:mm:ss` / `yyyy-MM-dd HH:mm:ss`. Status icons: hourglass/check/error/block for Pending/Success/Failed/Blocked. |
|
||||
| `app/src/main/kotlin/.../ui/components/MorphingSphere.kt` | ASCII morphing sphere — 3D lit character sphere. **New for voice mode**: `SphereState.Listening` (soft blue/purple, subtle wobble with `voiceAmplitude`) and `SphereState.Speaking` (vivid green/teal, dramatic core-warmth pulse with amplitude, data ring spin up to 4× on peak). New parameters `voiceAmplitude: Float = 0f` and `voiceMode: Boolean = false` (both defaulted — existing call sites unchanged). Radius scale capped at 1.08× in voiceMode — `baseRadius` is already 0.60× half-extent and the data ring at 1.55× would overflow past 1.1×. Three `@Preview` functions for Listening, Speaking-low, Speaking-peak. |
|
||||
| `app/src/main/kotlin/.../util/RelayErrorClassifier.kt` | `classifyError(Throwable?, context: String?) → HumanError(title, body, retryable, actionLabel)`. Single when-branch classifier — UnknownHostException → ConnectException → SocketTimeoutException → SSLException → SecurityException → IllegalStateException → IOException (message-scan for HTTP status 401/403/404/413/500/503) → default. Branch order is load-bearing (SSLException extends IOException). Context tags (`transcribe`, `synthesize`, `voice_config`, `record`, `pair`, `save_and_test`, `media_fetch`, `send_message`) shape the title and 404 body. Used by VoiceViewModel, ChatViewModel, ConnectionSettingsScreen. |
|
||||
| `app/src/main/kotlin/.../util/MediaCacheWriter.kt` | `cacheDir/hermes-media/` writer — LRU eviction by mtime, MIME→extension map, returns `FileProvider` `content://` URIs |
|
||||
| `app/src/main/kotlin/.../accessibility/HermesAccessibilityService.kt` | Phase 3 γ — `AccessibilityService` subclass. `@Volatile instance` singleton set on `onServiceConnected` / cleared on unbind, so `BridgeCommandHandler` can reach the live service without DI. Caches foregrounded package from `TYPE_WINDOW_STATE_CHANGED`. Master enable lives in DataStore (`bridge_master_enabled`). `snapshotRoot()` wraps `rootInActiveWindow`. |
|
||||
| `app/src/main/kotlin/.../accessibility/ScreenReader.kt` | Phase 3 γ — UI tree → `ScreenContent(rootBounds, nodes[], truncated)` via `@Serializable`. Walks `AccessibilityNodeInfo` tree capped at `MAX_NODES=512`, collects text/contentDescription/bounds/class/viewId + clickable/scrollable/editable flags. `findNodeBoundsByText(root, needle)` for `/tap_text` and `findFocusedInput(root)` for `/type`. Recycles child nodes in per-iteration `try/finally`. |
|
||||
| `app/src/main/kotlin/.../accessibility/ActionExecutor.kt` | Phase 3 γ — `tap`/`tapText`/`swipe`/`scroll` via `GestureDescription` wrapped in `suspendCancellableCoroutine` so suspend form actually waits for `GestureResultCallback.onCompleted`. `typeText` uses `ACTION_SET_TEXT`. `pressKey` maps curated string vocab to `AccessibilityService.GLOBAL_ACTION_*` (no raw KeyEvent codes — no arbitrary injection). `wait(ms)` clamped to 15s. Returns `ActionResult(ok, data, error)`. |
|
||||
| `app/src/main/kotlin/.../accessibility/ScreenCapture.kt` | Phase 3 γ — `MediaProjection` → `VirtualDisplay` → `ImageReader` → PNG bytes → multipart upload to `POST /media/upload` on the relay. Crops `rowStride` padding before `Bitmap.copyPixelsFromBuffer`. 2.5s capture timeout. Co-located `MediaProjectionHolder` singleton holds the per-session grant; Bridge UI (Agent δ) is responsible for the `ActivityResultLauncher` flow calling `MediaProjectionHolder.onGranted(resultCode, data)`. **Blocked on α: `POST /media/upload` endpoint doesn't exist yet** — current `/media/register` is loopback-only + path-based, phone has no shared filesystem. |
|
||||
| `app/src/main/kotlin/.../accessibility/BridgeStatusReporter.kt` | Phase 3 γ — coroutine emitting `bridge.status` envelopes every 30s with `screen_on` (`PowerManager.isInteractive`), `battery` (`BatteryManager.BATTERY_PROPERTY_CAPACITY` + sticky-intent fallback), `current_app` (from service singleton), `accessibility_enabled` (true when service instance non-null). Owned + started by `ConnectionViewModel`. |
|
||||
| `app/src/main/kotlin/.../network/handlers/BridgeCommandHandler.kt` | Phase 3 γ — routes inbound `bridge.command` envelopes to `ActionExecutor` and emits `bridge.response`. Paths: `/ping`, `/tap`, `/tap_text`, `/type`, `/swipe`, `/scroll`, `/press_key`, `/wait`, `/screen`, `/screenshot`, `/current_app`. Gates everything except `/ping` + `/current_app` on master-enable; returns 503 if service not connected, 403 if soft master off. `/screen` serializes `ScreenContent` via `Json.encodeToJsonElement`. |
|
||||
| `app/src/main/kotlin/.../data/MediaSettings.kt` | DataStore-backed settings (max inbound size, auto-fetch threshold [persisted-not-enforced], auto-fetch on cellular, cached media cap) |
|
||||
| `app/src/main/kotlin/.../ui/components/InboundAttachmentCard.kt` | Discord-style attachment card — dispatches by `(AttachmentState × AttachmentRenderMode)`; images inline, video/audio/pdf/text/generic as tap-to-open via `ACTION_VIEW` + `FLAG_GRANT_READ_URI_PERMISSION`. Handles outbound attachments too (default `state=LOADED`). |
|
||||
| `app/src/main/res/xml/file_provider_paths.xml` | FileProvider path config — `<cache-path name="hermes-media" path="hermes-media/"/>` |
|
||||
@@ -222,6 +235,21 @@ hermes-android/ ← Android Studio opens this root
|
||||
| `skills/devops/hermes-relay-pair/SKILL.md` | `/hermes-relay-pair` slash command — canonical category layout (`devops`), matches `metadata.hermes.category` frontmatter. Discovered via `skills.external_dirs` entry added by the installer. |
|
||||
| `plugin/relay/_env_bootstrap.py` | `load_hermes_env() → list[Path]` — loads `~/.hermes/.env` into `os.environ` before the relay server imports anything that reads env vars. Prefers `hermes_cli.env_loader.load_hermes_dotenv` when importable, falls back to direct `python-dotenv`, silent no-op in stripped containers. Called from both `plugin/relay/__main__.py` and `relay_server/__main__.py` so both entry points bootstrap identically. This is why neither `hermes-relay.service` nor `hermes-gateway.service` carries an `EnvironmentFile=` directive. |
|
||||
| `install.sh` | Canonical installer — 6 steps: [1] clone repo, [2] `pip install -e`, [3] symlink plugin, [4] register skills in `config.yaml`, [5] `hermes-pair` shell shim, [6] systemd user unit (optional — skipped on macOS/WSL-without-systemd/containers, or via `$HERMES_RELAY_NO_SYSTEMD`). Updates via `git pull` in the clone + `systemctl --user restart hermes-relay`. |
|
||||
| `app/build.gradle.kts` (Phase 3) | `flavorDimensions += "track"` + `googlePlay` / `sideload` product flavors. `sideload` gets `applicationIdSuffix = ".sideload"` + `versionNameSuffix = "-sideload"` so both tracks coexist on the same device; `googlePlay` keeps the canonical applicationId for clean Play Store upgrades from v0.2.0. |
|
||||
| `app/src/googlePlay/AndroidManifest.xml` | Google Play flavor manifest overlay — declares `BridgeAccessibilityService` with the conservative use-case description (`@string/a11y_description_googleplay`) and the conservative `@xml/accessibility_service_config`. Merged onto `app/src/main/AndroidManifest.xml` by AGP at build time. |
|
||||
| `app/src/googlePlay/res/xml/accessibility_service_config.xml` | Conservative a11y config — `typeWindowStateChanged\|typeWindowContentChanged\|typeViewClicked` event subset, `flagDefault` only, no gestures. Targeted at Play Store policy review. |
|
||||
| `app/src/googlePlay/res/values/strings.xml` | `a11y_description_googleplay` — "notifications / summarize / reply with confirmation" language. Reviewed by Play Store policy. |
|
||||
| `app/src/sideload/AndroidManifest.xml` | Sideload flavor manifest overlay — same `BridgeAccessibilityService` reference, full-capability `@string/a11y_description_sideload` description. Not shipped through Google Play. |
|
||||
| `app/src/sideload/res/xml/accessibility_service_config.xml` | Full a11y config — `typeAllMask`, `flagRetrieveInteractiveWindows\|flagReportViewIds\|flagRequestTouchExplorationMode`, `canPerformGestures="true"`. |
|
||||
| `app/src/sideload/res/values/strings.xml` | `a11y_description_sideload` — full agent-control language (voice + vision + logging + confirmations). |
|
||||
| `app/src/main/kotlin/.../data/FeatureFlags.kt::BuildFlavor` | Compile-time flavor gating. Exposes `current` (from `BuildConfig.FLAVOR`), `displayName` for the Settings → About badge, and six `bridgeTier1..6` flags. Tier 3/4/6 are `get() = current == SIDELOAD` so R8 can fold them in the Play release build. |
|
||||
| `plugin/relay/channels/notifications.py` | **Phase 3 / Wave 1 / ε** — `NotificationsChannel` holds a bounded `collections.deque` (maxlen 100) of recent notification metadata forwarded by the phone over the `notifications` WSS channel. Append on `notification.posted`, read via `get_recent(limit)` (newest-first, clamped to `[1, max_entries]`). In-memory only — wiped on relay restart by design (matches the smartwatch-companion semantics). |
|
||||
| `plugin/tools/android_notifications.py` | **Phase 3 / Wave 1 / ε** — Registers `android_notifications_recent(limit=20)` Hermes tool. Calls `http://127.0.0.1:8767/notifications/recent?limit=N` over loopback (no bearer needed for loopback callers — same trust model as `/media/register`), parses JSON, returns structured envelope. Stdlib `urllib.request` only — no requests/httpx dependency. Honours `RELAY_PORT` env. |
|
||||
| `plugin/relay/server.py` (Phase3-ε additions) | `handle_notifications_recent` HTTP route — loopback callers skip bearer, remote callers go through `_require_bearer_session`. Limit clamped via `NotificationsChannel.get_recent`. Channel dispatch in `_on_message` routes `channel == "notifications"` envelopes to `server.notifications.handle()`. Markers: `# === PHASE3-ε: ... === / # === END PHASE3-ε ===`. |
|
||||
| `app/src/main/kotlin/.../notifications/HermesNotificationCompanion.kt` | **Phase 3 / Wave 1 / ε** — `NotificationListenerService` subclass. Opt-in via `Settings.ACTION_NOTIFICATION_LISTENER_SETTINGS` (the same Android API Wear OS / Android Auto / Tasker use). On `onNotificationPosted`, builds a `NotificationEntry`, serializes to a `notifications/notification.posted` envelope, and sends via the static `companion.multiplexer` slot. Cold-start buffer (`pendingEnvelopes`, capped at 50) preserves order before the multiplexer is wired. `isAccessGranted(context)` is a synchronous check against `Settings.Secure.enabled_notification_listeners`. |
|
||||
| `app/src/main/kotlin/.../notifications/NotificationModels.kt` | **Phase 3 / Wave 1 / ε** — `@Serializable data class NotificationEntry(packageName, title, text, subText, postedAt, key)` with `kotlinx.serialization` `@SerialName` mappings to the snake_case Python wire format. |
|
||||
| `app/src/main/kotlin/.../ui/screens/NotificationCompanionSettingsScreen.kt` | **Phase 3 / Wave 1 / ε** — Compose settings screen mirrored on `VoiceSettingsScreen`. Sections: About (plain-language explanation + revoke instructions), Status (live grant indicator via `LifecycleEventObserver` ON_RESUME re-check + "Open Android Settings" button), Test (pulls `service.activeNotifications` from the bound listener for end-to-end verification without requiring a relay round-trip). |
|
||||
| `app/src/main/kotlin/.../network/ChannelMultiplexer.kt` (Phase3-ε additions) | `sendNotification(envelope)` — thin wrapper over `send()` for the `HermesNotificationCompanion` outbound path. Drops on the floor when no `sendCallback` is wired (relay disconnected) — the service owns the cold-start buffer in its own `pendingEnvelopes` queue. Marked with `// === PHASE3-ε: notification outbound routing ===`. |
|
||||
| `AGENTS.md` | Tool usage patterns for the `android_*` toolset |
|
||||
| `docs/mcp-tooling.md` | MCP server setup — android-tools-mcp + mobile-mcp |
|
||||
|
||||
|
||||
@@ -53,6 +53,178 @@ Closed the "you must run our hermes-agent fork to get full features" gap. The Co
|
||||
|
||||
**Removal path** (when PR #8556 reaches a released hermes-agent version): delete `hermes_relay_bootstrap/`, delete `hermes_relay_bootstrap.pth`, remove the `.pth` drop block from `install.sh`. The Android client `probeCapabilities()` + `streamingEndpoint = "auto"` plumbing stays — it's permanent infrastructure that handles mixed-version deployments.
|
||||
|
||||
## 2026-04-12 — Phase 3 / Wave 1.5 — α-followup + ε-followup (post-merge polish)
|
||||
|
||||
Two small follow-ups discovered during the Wave 1 agent-team merge. Both gate clean smoke testing of the merged Wave 1 work in Android Studio.
|
||||
|
||||
**α-followup — `POST /media/upload` route on the relay.** Agent γ's `ScreenCapture` posts screenshot bytes via multipart to `/media/upload`, but α only ported the existing `/media/register` (loopback + path-based). Phone has no shared filesystem with the relay, so `/media/register` was unusable for the bridge screenshot path. The new endpoint accepts a `file` multipart field with bearer auth (every paired phone has a session token), streams the bytes to a `NamedTemporaryFile` under `tempfile.gettempdir()` (which is in the default `MediaRegistry.allowed_roots`), enforces `MediaRegistry.max_size_bytes` while reading (returns 413 on overflow), then hands the path off to `MediaRegistry.register()` so token issuance / expiry / LRU eviction / `GET /media/{token}` are byte-for-byte identical to the loopback path. Marker block: `# === PHASE3-α-followup: /media/upload === / # === END PHASE3-α-followup ===` in `plugin/relay/server.py`. `tempfile` import added at the top. Route registered next to `/media/register`. py_compile clean.
|
||||
|
||||
**ε-followup — wire `HermesNotificationCompanion.multiplexer` + Settings nav row.** Agent ε flagged two small touch-ups in its handoff:
|
||||
|
||||
1. `ConnectionViewModel.init` now sets `HermesNotificationCompanion.multiplexer = multiplexer` once, immediately after the bridge handler registration. The companion service buffers up to 50 envelopes in its own `pendingEnvelopes` queue while the slot is null, so wiring it from here (rather than at service-bind time) is safe — the buffer drains on the next `onNotificationPosted` once the slot is set. Marker: `// === PHASE3-ε-followup: notification companion multiplexer wiring === / // === END PHASE3-ε-followup ===`.
|
||||
2. `SettingsScreen` gains a new `onNavigateToNotificationCompanion: () -> Unit` callback parameter and a `SettingsCategoryRow` between Voice mode and Media (`Icons.Filled.Notifications`, "Notification companion", "Let your assistant triage notifications you've shared"). `RelayApp` adds `Screen.NotificationCompanionSettings` (route `settings/notifications`), wires the new callback in the `SettingsScreen` call site, and registers a `composable(...)` that hosts `NotificationCompanionSettingsScreen(onBack = popBackStack)`. Both new imports added.
|
||||
|
||||
After this entry the Bridge tab + Notification companion screen are both reachable through the normal navigation tree, and γ's screenshot pipeline has a working server-side endpoint. Wave 2 (ζ safety, η voice-bridge, θ vision) is unblocked once Bailey confirms both build flavors compile in Android Studio.
|
||||
|
||||
## 2026-04-12 — Phase 3 / Wave 1 / α — Migrated legacy bridge relay into unified relay (port 8767)
|
||||
|
||||
Retired the standalone bridge relay (`plugin/tools/android_relay.py` + the duplicate top-level `plugin/android_relay.py`, both listening on port 8766) and folded its functionality into the unified Hermes-Relay on port 8767 as the bridge channel. The wire protocol (`bridge.command` / `bridge.response` / `bridge.status`) stays byte-for-byte identical — only the transport changed. Agents γ, δ, ε can now build against a single port.
|
||||
|
||||
- `plugin/relay/channels/bridge.py` — replaced the stub with a real `BridgeHandler`. One handler instance per `RelayServer`; holds `phone_ws` + `pending: dict[request_id, Future]` behind an `asyncio.Lock`. `handle_command(method, path, params, body)` mints a request_id, registers the future, sends a `bridge.command` envelope, and awaits a response with 30s timeout (matches the legacy `android_relay._RESPONSE_TIMEOUT`). `handle(ws, envelope)` opportunistically latches `phone_ws` and dispatches `bridge.response`/`bridge.status`. `detach_ws(ws, reason)` fails all pending futures with `ConnectionError` so HTTP callers don't hang when the phone drops.
|
||||
- `plugin/relay/server.py` — added 14 HTTP routes (`/ping`, `/screen`, `/screenshot`, `/get_apps`, `/apps` [legacy alias], `/current_app`, `/tap`, `/tap_text`, `/type`, `/swipe`, `/open_app`, `/press_key`, `/scroll`, `/wait`, `/setup`) each delegating to `_bridge_dispatch → server.bridge.handle_command`. BridgeError → 503/504/502 depending on message. Wired `server.bridge.detach_ws(ws)` into `_on_disconnect` so phone drops instantly fail in-flight commands instead of hanging to the 30s timeout. All additions are bracketed by `# === PHASE3-α: ... === / # === END PHASE3-α ===` markers so the Agent ε notification-listener merges are mechanical.
|
||||
- `plugin/android_tool.py` + `plugin/tools/android_tool.py` — BRIDGE_URL default changed from `http://localhost:8766` to `http://localhost:8767` (both copies; `plugin/android_tool.py` is the one imported by `plugin/__init__.py`, `plugin/tools/android_tool.py` is the standalone-toolset copy). `_relay_port()` falls back through `ANDROID_RELAY_PORT → RELAY_PORT → 8767`. `android_setup()` rewritten: no longer imports the deleted `android_relay` module, instead probes `http://localhost:<port>/health` to verify the unified relay is up and returns a structured error if not. Env-var side effects preserved so the existing `test_android_tool.py::TestSetup` still passes.
|
||||
- `plugin/android_relay.py` + `plugin/tools/android_relay.py` — **DELETED**. Both copies of the standalone relay are gone.
|
||||
- `plugin/tests/test_bridge_channel.py` — new unittest suite (7 tests) covering envelope routing, future resolution, timeout cleanup, disconnect cleanup, send-failure cleanup, and the legacy-timeout regression guard. Uses a `_FakeWs` stand-in and bypasses the pytest `conftest.py` that imports `responses`. Run with `python -m unittest plugin.tests.test_bridge_channel`. All 7 green locally; existing `test_relay_media_routes` / `test_qr_sign` / `test_session_grants` / `test_media_registry` still pass (22 + 47 assertions) so `create_app` + route registration hold.
|
||||
|
||||
Auth model judgment call: the bridge HTTP routes are unauthenticated at the HTTP layer, matching the legacy standalone relay. Defensible because (a) the trust boundary is unchanged — only same-host processes reach `localhost:8767`, (b) a disconnected/unpaired phone naturally causes every call to fail with 503, and (c) the bridge grant is already tracked per-session in `Session.grants["bridge"]` so Wave 2 (safety-rails) can add a bearer wrapper without touching the handler. Noted inline in the PHASE3-α block so Agent ζ can find it.
|
||||
|
||||
Branch: `feature/phase3-alpha-bridge-server-migration`.
|
||||
|
||||
## 2026-04-12 — Phase 3 / Wave 1 / β — googlePlay + sideload build flavors
|
||||
|
||||
Agent β (`bridge-flavor-split`) adds the Gradle flavor split that the Phase 3 Bridge channel needs before any accessibility code lands. Google Play reviews AccessibilityService heavily, so Phase 3 ships two parallel release tracks from one codebase: a conservative `googlePlay` flavor with a notifications-and-confirmations use-case description, and a `sideload` flavor with the full agent-control description for GitHub Releases / F-Droid / ADB distribution.
|
||||
|
||||
Scope of this change:
|
||||
|
||||
- **`app/build.gradle.kts`** — `flavorDimensions += "track"` with two product flavors. The `sideload` flavor carries `applicationIdSuffix = ".sideload"` and `versionNameSuffix = "-sideload"` so both tracks can coexist on the same device during Phase 3 testing. The `googlePlay` flavor keeps the canonical `com.hermesandroid.relay` applicationId so existing v0.2.0 Play Store installs upgrade cleanly and Play Console keeps its release history. Decision trade-off: power users with both installed will see two launcher icons until we differentiate labels in a follow-up.
|
||||
- **`app/src/googlePlay/*`** — flavor manifest overlay declaring `.accessibility.BridgeAccessibilityService` (owned by Agent γ in `src/main/`), conservative `accessibility_service_config.xml` (event subset: `typeWindowStateChanged|typeWindowContentChanged|typeViewClicked`, `flagDefault` only, no gestures, `canRetrieveWindowContent=true`), and `a11y_description_googleplay` targeted at Play Store policy review.
|
||||
- **`app/src/sideload/*`** — flavor manifest overlay, full-capability `accessibility_service_config.xml` (`typeAllMask`, `flagRetrieveInteractiveWindows|flagReportViewIds|flagRequestTouchExplorationMode`, `canPerformGestures=true`), and `a11y_description_sideload` with explicit voice/vision/logging language.
|
||||
- **`FeatureFlags.kt`** — new `BuildFlavor` object with `current` / `displayName` / six `bridgeTier1..6` flags. Tiers 1, 2, 5 are baseline-true for both tracks. Tiers 3 (voice-first), 4 (vision-first), 6 (ambitious future) are `get() = current == SIDELOAD` so UI code can do a single `if (BuildFlavor.bridgeTier3)` check and R8 can fold the branch at release-build time for the Play track.
|
||||
- **`AboutScreen.kt`** — added a small "Track: Google Play" / "Track: Sideload" row under the existing Version row (which owns the 7-tap dev-options reveal). Marked with `=== PHASE3-β ===` banners so δ and ε can land their own additions without merge conflicts.
|
||||
|
||||
Not shipped in this change: the actual `BridgeAccessibilityService` class (Agent γ) and any tier-gated UI surfaces (Agents δ/ε). The manifests reference `.accessibility.BridgeAccessibilityService` with `tools:ignore="MissingClass"` so the Gradle + AAPT check doesn't block Agent γ's landing.
|
||||
|
||||
Branch: `feature/phase3-beta-bridge-flavor-split`.
|
||||
|
||||
## 2026-04-12 — Phase 3 / Wave 1 / γ — accessibility runtime (service + reader + executor + capture)
|
||||
|
||||
Wave 1 Agent γ (`accessibility-runtime`) landing the phone-side execution layer for the bridge channel. Five new files under a new `com.hermesandroid.relay.accessibility` package plus `BridgeCommandHandler` under `network/handlers`:
|
||||
|
||||
- **`HermesAccessibilityService`** — master `AccessibilityService` subclass. Self-registers as a `@Volatile` singleton on `onServiceConnected`, clears on unbind/destroy. Caches the foregrounded package from `TYPE_WINDOW_STATE_CHANGED` events (the only event type we consume — content-change events fire thousands/min and we read the tree on demand via `rootInActiveWindow`). Master enable flag lives in DataStore (`bridge_master_enabled`) so UI + safety rails can toggle it without killing the service.
|
||||
- **`ScreenReader`** — UI tree → `ScreenContent(rootBounds, nodes[], truncated)`. `@Serializable` output; one `ScreenNode` per interesting node (non-blank text, content description, clickable/longClickable/scrollable/editable). Hard caps: `MAX_NODES=512`, `MAX_TEXT_LEN=2000` per field. `findNodeBoundsByText` and `findFocusedInput` helpers for the executor. Recycles child nodes in a `try/finally` pattern per-iteration.
|
||||
- **`ActionExecutor`** — `tap`/`tapText`/`swipe`/`scroll` via `GestureDescription` wrapped in `suspendCancellableCoroutine` so the suspend form actually waits for `GestureResultCallback.onCompleted`. `typeText` uses `ACTION_SET_TEXT` with a `Bundle` arg. `pressKey` maps a curated string vocabulary (`home`/`back`/`recents`/`notifications`/`quick_settings`/`power_dialog`/`lock_screen`) to `AccessibilityService.GLOBAL_ACTION_*` constants — we deliberately don't accept raw `KeyEvent` codes so agents can't inject arbitrary keypresses. `wait(ms)` clamped to 15s. Every method returns `ActionResult(ok, data, error)` so the handler can map 1:1 to HTTP-style status codes.
|
||||
- **`ScreenCapture`** — `MediaProjection` → `VirtualDisplay` → `ImageReader` → PNG bytes → multipart upload to `POST /media/upload` on the relay. Crops `rowStride` padding before `Bitmap.copyPixelsFromBuffer`. 2.5s capture timeout. `MediaProjectionHolder` singleton holds the per-session grant — Bridge UI (Agent δ) is responsible for the `ActivityResultLauncher` flow that calls `onGranted(resultCode, data)`. Registers a `MediaProjection.Callback` to null the holder when the system revokes.
|
||||
- **`BridgeStatusReporter`** — coroutine that emits `bridge.status` envelopes every 30s with `screen_on` (via `PowerManager.isInteractive`), `battery` (`BatteryManager.BATTERY_PROPERTY_CAPACITY` with a sticky-intent fallback for OEM quirks), `current_app` (from the service singleton), and `accessibility_enabled` (true when the service instance is non-null). Owned by `ConnectionViewModel`.
|
||||
- **`BridgeCommandHandler`** — routes inbound `bridge.command` envelopes to the executor and emits `bridge.response`. Wire paths: `/ping` (works without the service), `/tap`, `/tap_text`, `/type`, `/swipe`, `/scroll`, `/press_key`, `/wait`, `/screen`, `/screenshot`, `/current_app`. Gates everything except `/ping` and `/current_app` on the master-enable toggle, returning 503 if the service isn't connected or 403 if the soft master is off. `/screen` serializes the full `ScreenContent` via `kotlinx.serialization.json`.
|
||||
|
||||
Wired into `ChannelMultiplexer.kt` via a `// === PHASE3-γ ===` marked section (simplified the existing bridge branch that was stubbed with a TODO), and `ConnectionViewModel.kt` gets a matching marker block that instantiates `ScreenCapture`, `BridgeCommandHandler`, and `BridgeStatusReporter`, registers the handler, and starts the reporter. `AndroidManifest.xml` declares the service with `BIND_ACCESSIBILITY_SERVICE` permission + intent filter + `@xml/accessibility_service_config` meta-data (the XML itself is flavor-provided by Agent β). Added `FOREGROUND_SERVICE`, `FOREGROUND_SERVICE_MEDIA_PROJECTION`, and `POST_NOTIFICATIONS` for the Wave 2 persistent-notification work.
|
||||
|
||||
**Known blocker for δ to wire:** `MediaProjectionManager.createScreenCaptureIntent()` requires an `Activity`-scoped result. `ScreenCapture.createConsentIntent()` exposes the intent; the Bridge screen needs an `ActivityResultLauncher<Intent>` that forwards the result to `MediaProjectionHolder.onGranted(context, resultCode, data)`. Until that lands, `/screenshot` returns 503 with a clear error message.
|
||||
|
||||
**Known blocker for α to fix:** the relay's `/media/register` endpoint is loopback-only and path-based (see `plugin/relay/media.py`). The phone can't use it — there's no shared filesystem. `ScreenCapture.uploadViaMultipart` POSTs to `/media/upload` (new endpoint; mirrors the `/voice/transcribe` multipart pattern) which doesn't exist yet. Until α ships it, `/screenshot` surfaces a 404 with the exact message `"relay /media/upload endpoint not found — server needs Phase 3 α migration"`. Token extraction uses the same `{"ok": true, "token": "..."}` shape as `/media/register`.
|
||||
|
||||
Branch: `feature/phase3-accessibility-runtime`.
|
||||
|
||||
## 2026-04-12 — Phase 3 / Wave 1 / δ — BridgeScreen rewrite (bridge-screen-ui)
|
||||
|
||||
Replaced the `BridgeScreen` "Coming Soon" placeholder with the real Tier-1
|
||||
control surface from the Phase 3 plan. Wave 1 Agent δ (`bridge-screen-ui`)
|
||||
deliverable — the UI scaffold is now ready to render whatever state Agent γ's
|
||||
`HermesAccessibilityService` exposes once its runtime lands.
|
||||
|
||||
New files:
|
||||
|
||||
- `app/src/main/kotlin/.../data/BridgePreferences.kt` — DataStore repo with
|
||||
`bridge_master_enabled` boolean and serialized `bridge_activity_log` JSON
|
||||
list (capped at 100 entries). Mirrors `VoicePreferences.kt` /
|
||||
`MediaSettings.kt` style. Uses lenient `Json` for forward-compat.
|
||||
- `app/src/main/kotlin/.../viewmodel/BridgeViewModel.kt` — AndroidViewModel
|
||||
exposing `masterToggle`, `bridgeStatus`, `permissionStatus`, `activityLog`
|
||||
StateFlows. Reads a11y service enablement via
|
||||
`Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES`, overlay via
|
||||
`Settings.canDrawOverlays`, notification listener via
|
||||
`enabled_notification_listeners`. Seeds `BridgeStatus` on init from
|
||||
`PowerManager.isInteractive` + `BatteryManager.BATTERY_PROPERTY_CAPACITY`
|
||||
so the card isn't empty before γ lands. Four explicit `TODO(γ-handoff)`
|
||||
markers documenting the StateFlow/class-name/activity-writer surface that
|
||||
γ needs to expose for final wiring.
|
||||
- `app/src/main/kotlin/.../ui/components/BridgeMasterToggle.kt` — headline
|
||||
Switch card with status inlines (device / battery / screen / current app),
|
||||
explanation dialog (required by Play Store a11y review), and a11y-granted
|
||||
gate so users can't flip it on before enabling the service.
|
||||
- `app/src/main/kotlin/.../ui/components/BridgeStatusCard.kt` — standalone
|
||||
status card with `ConnectionStatusBadge` integration. Usable independently
|
||||
of the master toggle so ζ can re-arrange the cards without losing state.
|
||||
- `app/src/main/kotlin/.../ui/components/BridgePermissionChecklist.kt` —
|
||||
four-row checklist (Accessibility / Screen Capture / Overlay / Notification
|
||||
Listener) with tap-to-open Intent launchers wrapped in `runCatching` for
|
||||
OEM skin safety. `ACTION_ACCESSIBILITY_SETTINGS`,
|
||||
`ACTION_MANAGE_OVERLAY_PERMISSION` with package URI, and the direct
|
||||
`enabled_notification_listeners` intent string.
|
||||
- `app/src/main/kotlin/.../ui/components/BridgeActivityLog.kt` — scrollable
|
||||
`LazyColumn` (bounded at `heightIn(max = 320.dp)`) with tap-to-expand row
|
||||
showing full timestamp, status, result text, and optional screenshot token.
|
||||
`java.time.DateTimeFormatter` for `HH:mm:ss` / `yyyy-MM-dd HH:mm:ss`.
|
||||
Stubbed screenshot thumbnail rendering with a TODO pointing at γ's
|
||||
MediaRegistry upload path.
|
||||
|
||||
Shared-concern edits (marked with `PHASE3-δ:` banners):
|
||||
|
||||
- `app/src/main/kotlin/.../ui/RelayApp.kt` — the existing `BridgeScreen()`
|
||||
call wraps in the δ marker. Signature compatible: `BridgeScreen` defaults
|
||||
its ViewModel via `viewModel()`, so no new nav-graph plumbing needed.
|
||||
|
||||
Each new component carries a `@Preview` (or two — e.g.,
|
||||
`BridgeMasterTogglePreviewOn` vs `PreviewBlocked`) so Bailey can iterate in
|
||||
Android Studio's preview pane without rebuilding the whole app.
|
||||
|
||||
γ-handoff points (in priority order, documented in `BridgeViewModel` KDoc):
|
||||
|
||||
1. `bridgeStatus` StateFlow — δ stubs a best-effort read from system APIs;
|
||||
γ replaces with the real HermesAccessibilityService status flow.
|
||||
2. `A11Y_SERVICE_CLASS` constant — δ hard-codes
|
||||
`"com.hermesandroid.relay.accessibility.HermesAccessibilityService"`.
|
||||
γ confirms the final FQCN.
|
||||
3. `recordActivity` write path — γ's command dispatcher calls
|
||||
`BridgeViewModel.recordActivity(entry)` on every `bridge.command` to
|
||||
populate the activity log. Until then the log stays empty and the empty-
|
||||
state copy shows.
|
||||
4. Master toggle read from γ — γ's service reads `bridge_master_enabled` from
|
||||
`BridgePreferencesRepository.settings` and treats it as the runtime disable
|
||||
switch. No extra wiring needed from δ; γ just imports the repo.
|
||||
|
||||
Build-flavor flags (`BuildFlavor.bridgeTierN` from β) are not referenced yet
|
||||
— this worktree pre-dates β's landing. Once β's object lands on main, δ's
|
||||
components remain compatible (they don't tier-gate anything; Tier 1 is
|
||||
always-on in both flavors per the plan).
|
||||
|
||||
Safety card is a placeholder stub — Agent ζ (Wave 2) owns the real safety
|
||||
UI. δ's `SafetyPlaceholderCard` just says "Configure in Bridge Safety
|
||||
Settings" with a Wave-2 teaser subtitle.
|
||||
|
||||
Branch: `feature/phase3-delta-bridge-screen-ui`.
|
||||
|
||||
## 2026-04-12 — Phase 3 / Wave 1 / ε — notification companion (opt-in triage helper)
|
||||
|
||||
Adds an opt-in helper that lets the user's Hermes assistant read notifications they've explicitly granted access to via Android's standard `NotificationListenerService` API — the same one Wear OS, Android Auto, and Tasker have used for over a decade. Disabled by default; the user controls grant + revoke via Android Settings → Notification access.
|
||||
|
||||
**Three pieces, all marked with `PHASE3-ε` block markers in shared files:**
|
||||
|
||||
1. **Phone — `NotificationListenerService` + Compose settings screen.** New `app/src/main/kotlin/.../notifications/` package with `HermesNotificationCompanion` (the bound service) + `NotificationModels` (`@Serializable NotificationEntry`) + `ui/screens/NotificationCompanionSettingsScreen` (About / Status / Test sections, mirrors `VoiceSettingsScreen` style). The service buffers up to 50 envelopes in a `ConcurrentLinkedQueue` if `companion.multiplexer` isn't wired yet (cold-start ordering), drains on the next `onNotificationPosted`, then sends via `multiplexer.sendNotification(envelope)` — a thin new wrapper in `ChannelMultiplexer` that fast-paths to no-op when `sendCallback == null` (relay offline). Notifications with empty title+text are skipped (background-sync placeholders that just confuse the LLM). The `Status` row uses `LifecycleEventObserver(ON_RESUME)` so the grant state updates immediately when the user comes back from Android Settings.
|
||||
|
||||
2. **Server — bounded in-memory deque.** New `plugin/relay/channels/notifications.py::NotificationsChannel` with `recent: collections.deque[dict]` capped at 100 entries via `maxlen` (LRU-by-time eviction for free). `handle()` dispatches `notification.posted` envelopes to `handle_envelope()` which appends; `get_recent(limit)` returns newest-first with `limit` clamped to `[1, max_entries]`. Wired into `RelayServer.__init__` as `self.notifications` and dispatched in `_on_message` for `channel == "notifications"`. The cache is in-memory only and lost on restart by design — same semantics as a smartwatch out of range.
|
||||
|
||||
3. **Agent tool — `android_notifications_recent(limit=20)`.** New `plugin/tools/android_notifications.py` registers the tool into the `tools.registry` (with the `try/except ImportError` pattern matching `android_tool.py`). It hits `http://127.0.0.1:8767/notifications/recent?limit=N` over loopback via stdlib `urllib.request` — no auth needed because `handle_notifications_recent` skips bearer for loopback callers (matches the `/media/register` and `/pairing/register` trust model). Remote callers still go through `_require_bearer_session`. The tool docstring explicitly frames it as "List recent notifications the user has shared with this assistant" so the LLM treats absence as "not granted yet" rather than an error.
|
||||
|
||||
**Files touched:**
|
||||
|
||||
- New: `plugin/relay/channels/notifications.py`, `plugin/tools/android_notifications.py`, `app/src/main/kotlin/.../notifications/HermesNotificationCompanion.kt`, `app/src/main/kotlin/.../notifications/NotificationModels.kt`, `app/src/main/kotlin/.../ui/screens/NotificationCompanionSettingsScreen.kt`
|
||||
- Edited (additive only, marker-blocked): `plugin/relay/server.py` (import + handler init + HTTP route + dispatch + route registration), `app/src/main/kotlin/.../network/ChannelMultiplexer.kt` (`sendNotification` wrapper), `app/src/main/AndroidManifest.xml` (service entry with `BIND_NOTIFICATION_LISTENER_SERVICE` permission), `app/src/main/res/values/strings.xml` (`notification_companion_label`)
|
||||
- Docs: `CLAUDE.md` Key Files table (8 new rows / additive notes), `DEVLOG.md` (this entry)
|
||||
|
||||
**Decisions:**
|
||||
|
||||
- **No new session grant type.** Spec explicitly said don't add `notifications` to `auth.py` grants — this is opt-in via Android system permission, not via per-channel session grant. Reuses the existing `chat` grant trust boundary.
|
||||
- **Loopback bypass for the tool.** The route's spec said "bearer auth gated like /media/*", but the tool is in-process on the host, so we mirror `/media/register`'s loopback gate: 127.0.0.1 callers skip the bearer check, remote callers still require a session token. This means the tool doesn't need to grovel through the relay's session store to mint a token.
|
||||
- **Drop on relay-offline rather than buffer at the multiplexer.** A wearable doesn't replay notifications it missed while out of range; we don't either. The cold-start buffer in the service is for the much shorter "service bound but multiplexer not wired yet" window.
|
||||
- **`activeNotifications` for the Test button**, not a relay round-trip. Lets the user verify the listener is bound even if the relay is unreachable, which is the more common failure mode at first-grant time.
|
||||
|
||||
**Validation:** `python -m py_compile plugin/relay/channels/notifications.py plugin/tools/android_notifications.py plugin/relay/server.py` → OK. Kotlin compile happens on Bailey's next Android Studio run.
|
||||
|
||||
**Branch:** `feature/phase3-epsilon-notification-companion` (off `worktree-agent-a84b51cc`).
|
||||
|
||||
**Next:** wire `HermesNotificationCompanion.multiplexer` from `ConnectionViewModel` after relay handshake (one-line touch), add a Settings entry-point row that navigates to `NotificationCompanionSettingsScreen`, and (Phase 3 follow-up) consider a per-package allow/deny list so the user can mute social-media spam from the listener pipeline before it ever hits the relay.
|
||||
|
||||
## 2026-04-12 — Fix: TTS waveform stays alive through multi-sentence playback
|
||||
|
||||
The waveform was flatlinining after the first sentence while audio kept playing. Root cause: `maybeAutoResume()` fired after every sentence in the TTS consumer loop. The SSE stream finishes before TTS plays all queued sentences, so `streamObserverJob?.isActive` was already false → state flipped to Idle → amplitude bridge stopped → waveform died. Fix: restructured TTS consumer from `for` loop to `while` + `tryReceive` peek. `maybeAutoResume` only fires when the queue is actually drained (`tryReceive` returns failure), not between sentences. Between sentences within the same response, `tryReceive` succeeds immediately and the consumer skips the Idle transition. Additionally, the consumer re-asserts Speaking state before each synthesis call to handle the edge case where the queue was briefly empty between observer pushes.
|
||||
|
||||
@@ -46,6 +46,40 @@ android {
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Phase 3 — Bridge channel release tracks ────────────────────────────────
|
||||
// Google Play scrutinizes AccessibilityService heavily (policy review + manual
|
||||
// appeals are common), so Phase 3 ships two distinct tracks via flavor-merged
|
||||
// manifests + flavor-scoped strings + flavor-scoped accessibility configs:
|
||||
//
|
||||
// googlePlay — conservative use-case description targeted at Play Store
|
||||
// policy review. Subset of event types + flagDefault only.
|
||||
// No gestures, no interactive-window reporting. Feature gates
|
||||
// in BuildFlavor.kt hide tier 3/4/6 surfaces in the UI.
|
||||
//
|
||||
// sideload — full agent-control description for users who install the
|
||||
// APK directly (GitHub Releases, F-Droid, ADB). typeAllMask,
|
||||
// gestures, interactive windows, view-id reporting. All six
|
||||
// tiers enabled.
|
||||
//
|
||||
// applicationIdSuffix decision: sideload gets `.sideload` so both tracks can
|
||||
// coexist on the same device. The Play build keeps the canonical
|
||||
// `com.hermesandroid.relay` applicationId so existing installs upgrade
|
||||
// cleanly from v0.2.0 and Play Console keeps its history. Cost: anyone with
|
||||
// both installed sees two launcher icons — we'll differentiate with a label
|
||||
// suffix once the flavored strings.xml lands.
|
||||
flavorDimensions += "track"
|
||||
productFlavors {
|
||||
create("googlePlay") {
|
||||
dimension = "track"
|
||||
// No applicationIdSuffix — this IS the canonical Play Store install.
|
||||
}
|
||||
create("sideload") {
|
||||
dimension = "track"
|
||||
applicationIdSuffix = ".sideload"
|
||||
versionNameSuffix = "-sideload"
|
||||
}
|
||||
}
|
||||
|
||||
buildTypes {
|
||||
debug {
|
||||
buildConfigField("boolean", "DEV_MODE", "true")
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!--
|
||||
Google Play flavor manifest overlay.
|
||||
|
||||
This file is merged on top of `app/src/main/AndroidManifest.xml` by AGP when
|
||||
the `googlePlayDebug` / `googlePlayRelease` variants are built. It declares
|
||||
the AccessibilityService with the conservative use-case description required
|
||||
for Google Play policy review:
|
||||
|
||||
"Hermes assists you by reading notifications, summarizing messages, and
|
||||
replying with your confirmation. The service is dormant until you
|
||||
explicitly enable Bridge mode in the app."
|
||||
|
||||
The actual service class (BridgeAccessibilityService) is owned by Agent γ
|
||||
and lives under `app/src/main/kotlin/.../accessibility/`. The service must
|
||||
exist in `main/` so both flavors can reference it; only the manifest +
|
||||
config XML change between tracks.
|
||||
|
||||
Keep this file in sync with `app/src/sideload/AndroidManifest.xml` — any
|
||||
permissions / activities added here for Bridge should land in both flavors
|
||||
unless they are flavor-specific by design.
|
||||
-->
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
xmlns:tools="http://schemas.android.com/tools">
|
||||
|
||||
<application>
|
||||
|
||||
<service
|
||||
android:name=".accessibility.BridgeAccessibilityService"
|
||||
android:exported="false"
|
||||
android:label="@string/a11y_service_label"
|
||||
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
|
||||
tools:ignore="MissingClass">
|
||||
<intent-filter>
|
||||
<action android:name="android.accessibilityservice.AccessibilityService" />
|
||||
</intent-filter>
|
||||
<meta-data
|
||||
android:name="android.accessibilityservice"
|
||||
android:resource="@xml/accessibility_service_config" />
|
||||
</service>
|
||||
|
||||
</application>
|
||||
|
||||
</manifest>
|
||||
@@ -0,0 +1,18 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!--
|
||||
Google Play flavor strings.
|
||||
|
||||
`a11y_description_googleplay` is the user-facing description shown in
|
||||
Android's Accessibility settings when enabling the Hermes Bridge service.
|
||||
It is ALSO what Play Store reviewers read when evaluating our
|
||||
AccessibilityService use-case declaration, so phrasing matters: stay
|
||||
narrowly scoped, emphasize user confirmation, emphasize dormancy until
|
||||
the user opts in inside the app.
|
||||
|
||||
Do not reference voice or vision features here — tier 3/4/6 are gated
|
||||
off for this flavor via FeatureFlags.BuildFlavor.
|
||||
-->
|
||||
<resources>
|
||||
<string name="a11y_service_label">Hermes Bridge</string>
|
||||
<string name="a11y_description_googleplay">Hermes assists you by reading notifications, summarizing messages, and replying with your confirmation. The service is dormant until you explicitly enable Bridge mode in the app.</string>
|
||||
</resources>
|
||||
@@ -0,0 +1,20 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!--
|
||||
Google Play AccessibilityService configuration.
|
||||
|
||||
Conservative event-type subset targeted at the "read notifications,
|
||||
summarize messages, reply with confirmation" use case Play Store policy
|
||||
review expects. Does NOT subscribe to typeAllMask, does NOT request
|
||||
gestures, does NOT request flagRetrieveInteractiveWindows.
|
||||
|
||||
Keep these attributes aligned with the description in strings.xml
|
||||
(`a11y_description_googleplay`) — if the description widens, reviewers
|
||||
will expect the config to widen too.
|
||||
-->
|
||||
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
android:description="@string/a11y_description_googleplay"
|
||||
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged|typeViewClicked"
|
||||
android:accessibilityFlags="flagDefault"
|
||||
android:accessibilityFeedbackType="feedbackGeneric"
|
||||
android:notificationTimeout="100"
|
||||
android:canRetrieveWindowContent="true" />
|
||||
@@ -7,6 +7,19 @@
|
||||
<uses-permission android:name="android.permission.RECORD_AUDIO" />
|
||||
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
|
||||
|
||||
<!-- === PHASE3-γ: AccessibilityService + bridge permissions === -->
|
||||
<!-- BIND_ACCESSIBILITY_SERVICE is protected by the system — only
|
||||
granted to services that declare android:permission on their
|
||||
<service> tag (see below). FOREGROUND_SERVICE* are for the
|
||||
persistent notification that Agent ζ will wire in Wave 2 for
|
||||
MediaProjection-backed screenshots. POST_NOTIFICATIONS is required
|
||||
on API 33+ for that same foreground-service notification. -->
|
||||
<uses-permission android:name="android.permission.BIND_ACCESSIBILITY_SERVICE" />
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />
|
||||
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
|
||||
<!-- === END PHASE3-γ === -->
|
||||
|
||||
<uses-feature android:name="android.hardware.camera" android:required="false" />
|
||||
|
||||
<application
|
||||
@@ -39,6 +52,42 @@
|
||||
android:resource="@xml/file_provider_paths" />
|
||||
</provider>
|
||||
|
||||
<!-- === PHASE3-γ: AccessibilityService declaration === -->
|
||||
<!-- The @xml/accessibility_service_config resource is provided by
|
||||
the flavor-specific source sets (app/src/googlePlay/ and
|
||||
app/src/sideload/) owned by Agent β. Each flavor declares its
|
||||
own accessibility use-case description and flag bitset — the
|
||||
googlePlay track declares a conservative "notifications + reply
|
||||
with confirmation" use case, the sideload track declares the
|
||||
full agent-control use case. Gradle merges the flavor XML into
|
||||
main at build time.
|
||||
-->
|
||||
<service
|
||||
android:name=".accessibility.HermesAccessibilityService"
|
||||
android:exported="true"
|
||||
android:label="@string/app_name"
|
||||
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
|
||||
<intent-filter>
|
||||
<action android:name="android.accessibilityservice.AccessibilityService" />
|
||||
</intent-filter>
|
||||
<meta-data
|
||||
android:name="android.accessibilityservice"
|
||||
android:resource="@xml/accessibility_service_config" />
|
||||
</service>
|
||||
<!-- === END PHASE3-γ === -->
|
||||
|
||||
<!-- === PHASE3-ε: notification companion service === -->
|
||||
<service
|
||||
android:name=".notifications.HermesNotificationCompanion"
|
||||
android:label="@string/notification_companion_label"
|
||||
android:permission="android.permission.BIND_NOTIFICATION_LISTENER_SERVICE"
|
||||
android:exported="true">
|
||||
<intent-filter>
|
||||
<action android:name="android.service.notification.NotificationListenerService" />
|
||||
</intent-filter>
|
||||
</service>
|
||||
<!-- === END PHASE3-ε === -->
|
||||
|
||||
</application>
|
||||
|
||||
</manifest>
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
package com.hermesandroid.relay.accessibility
|
||||
|
||||
import android.accessibilityservice.AccessibilityService
|
||||
import android.accessibilityservice.GestureDescription
|
||||
import android.graphics.Path
|
||||
import android.os.Bundle
|
||||
import android.os.Handler
|
||||
import android.os.Looper
|
||||
import android.util.Log
|
||||
import android.view.KeyEvent
|
||||
import android.view.accessibility.AccessibilityNodeInfo
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import kotlin.coroutines.resume
|
||||
|
||||
/**
|
||||
* Phase 3 — γ `accessibility-runtime`
|
||||
*
|
||||
* Phone-side action dispatcher. Every method maps 1:1 to a Tier 1 bridge
|
||||
* command (`tap`, `tap_text`, `type`, `swipe`, `press_key`, `scroll`,
|
||||
* `wait`) and returns an [ActionResult] that [BridgeCommandHandler]
|
||||
* serializes into a `bridge.response` envelope.
|
||||
*
|
||||
* ## Why suspend functions
|
||||
*
|
||||
* Android's `dispatchGesture` API is fundamentally async — it takes a
|
||||
* callback on a Handler. Wrapping it as a suspend function keeps the
|
||||
* command handler's routing code synchronous-looking while preserving
|
||||
* the "wait for the gesture to actually complete before responding" contract.
|
||||
*
|
||||
* ## Key codes
|
||||
*
|
||||
* `press_key` accepts a small vocabulary of string names (`home`, `back`,
|
||||
* `recents`, `notifications`, `power`, `volume_up`, `volume_down`). Some
|
||||
* map to [AccessibilityService] global actions (home/back/recents), others
|
||||
* map to [KeyEvent] codes dispatched via [AccessibilityService.performGlobalAction]
|
||||
* where available. Power and volume are out of scope — they require
|
||||
* system-privileged APIs that AccessibilityService can't reach.
|
||||
*/
|
||||
class ActionExecutor(private val service: AccessibilityService) {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "ActionExecutor"
|
||||
|
||||
/** Default tap duration in milliseconds. Below ~50ms some IMEs ignore. */
|
||||
private const val DEFAULT_TAP_DURATION_MS = 100L
|
||||
|
||||
/** Default swipe duration in milliseconds. */
|
||||
private const val DEFAULT_SWIPE_DURATION_MS = 400L
|
||||
|
||||
/** Hard cap on `wait` — prevents runaway agents from pinning the channel. */
|
||||
private const val MAX_WAIT_MS = 15_000L
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of an action. Maps onto the `bridge.response` wire shape:
|
||||
* the handler turns [ok] into an HTTP-style `status` field and
|
||||
* merges [data] into the `result` object, while [error] becomes a
|
||||
* `{"error": "..."}` payload on failure.
|
||||
*/
|
||||
data class ActionResult(
|
||||
val ok: Boolean,
|
||||
val data: Map<String, Any?> = emptyMap(),
|
||||
val error: String? = null,
|
||||
) {
|
||||
companion object {
|
||||
fun ok(data: Map<String, Any?> = mapOf("ok" to true)): ActionResult =
|
||||
ActionResult(ok = true, data = data)
|
||||
|
||||
fun failure(message: String): ActionResult =
|
||||
ActionResult(ok = false, error = message)
|
||||
}
|
||||
}
|
||||
|
||||
// ─── tap / swipe / tap_text ───────────────────────────────────────────
|
||||
|
||||
suspend fun tap(
|
||||
x: Int,
|
||||
y: Int,
|
||||
durationMs: Long = DEFAULT_TAP_DURATION_MS,
|
||||
): ActionResult {
|
||||
if (x < 0 || y < 0) {
|
||||
return ActionResult.failure("tap coordinates must be non-negative (got x=$x, y=$y)")
|
||||
}
|
||||
val path = Path().apply { moveTo(x.toFloat(), y.toFloat()) }
|
||||
val stroke = GestureDescription.StrokeDescription(path, 0, durationMs)
|
||||
val gesture = GestureDescription.Builder().addStroke(stroke).build()
|
||||
val dispatched = dispatchGesture(gesture)
|
||||
return if (dispatched) {
|
||||
ActionResult.ok(mapOf("x" to x, "y" to y, "duration_ms" to durationMs))
|
||||
} else {
|
||||
ActionResult.failure("gesture dispatch failed or was cancelled")
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun tapText(needle: String): ActionResult {
|
||||
if (needle.isBlank()) {
|
||||
return ActionResult.failure("tap_text: text must be non-blank")
|
||||
}
|
||||
val root = (service as? HermesAccessibilityService)?.snapshotRoot()
|
||||
?: service.rootInActiveWindow
|
||||
?: return ActionResult.failure("no active window available")
|
||||
|
||||
val reader = (service as? HermesAccessibilityService)?.reader ?: ScreenReader()
|
||||
val bounds = reader.findNodeBoundsByText(root, needle)
|
||||
?: return ActionResult.failure("no node matching text '$needle' on screen")
|
||||
if (bounds.isEmpty) {
|
||||
return ActionResult.failure("matched node has empty bounds (off-screen?)")
|
||||
}
|
||||
return tap(bounds.centerX, bounds.centerY)
|
||||
}
|
||||
|
||||
suspend fun swipe(
|
||||
startX: Int,
|
||||
startY: Int,
|
||||
endX: Int,
|
||||
endY: Int,
|
||||
durationMs: Long = DEFAULT_SWIPE_DURATION_MS,
|
||||
): ActionResult {
|
||||
if (durationMs <= 0) {
|
||||
return ActionResult.failure("swipe duration must be positive (got $durationMs)")
|
||||
}
|
||||
val path = Path().apply {
|
||||
moveTo(startX.toFloat(), startY.toFloat())
|
||||
lineTo(endX.toFloat(), endY.toFloat())
|
||||
}
|
||||
val stroke = GestureDescription.StrokeDescription(path, 0, durationMs)
|
||||
val gesture = GestureDescription.Builder().addStroke(stroke).build()
|
||||
val dispatched = dispatchGesture(gesture)
|
||||
return if (dispatched) {
|
||||
ActionResult.ok(
|
||||
mapOf(
|
||||
"start_x" to startX,
|
||||
"start_y" to startY,
|
||||
"end_x" to endX,
|
||||
"end_y" to endY,
|
||||
"duration_ms" to durationMs,
|
||||
)
|
||||
)
|
||||
} else {
|
||||
ActionResult.failure("swipe gesture dispatch failed")
|
||||
}
|
||||
}
|
||||
|
||||
// ─── type / scroll / press_key / wait ─────────────────────────────────
|
||||
|
||||
fun typeText(text: String): ActionResult {
|
||||
val root = (service as? HermesAccessibilityService)?.snapshotRoot()
|
||||
?: service.rootInActiveWindow
|
||||
?: return ActionResult.failure("no active window available")
|
||||
|
||||
val reader = (service as? HermesAccessibilityService)?.reader ?: ScreenReader()
|
||||
val focused = reader.findFocusedInput(root)
|
||||
?: return ActionResult.failure("no focused editable field")
|
||||
|
||||
val args = Bundle().apply {
|
||||
putCharSequence(
|
||||
AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE,
|
||||
text
|
||||
)
|
||||
}
|
||||
val performed = try {
|
||||
focused.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "ACTION_SET_TEXT threw: ${t.message}")
|
||||
false
|
||||
} finally {
|
||||
@Suppress("DEPRECATION")
|
||||
try { focused.recycle() } catch (_: Throwable) { }
|
||||
}
|
||||
|
||||
return if (performed) {
|
||||
ActionResult.ok(mapOf("length" to text.length))
|
||||
} else {
|
||||
ActionResult.failure("ACTION_SET_TEXT refused by target view")
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Scroll the first scrollable container that contains the given point.
|
||||
*
|
||||
* `direction`: `"up"`, `"down"`, `"left"`, `"right"`. Internally we
|
||||
* dispatch a swipe gesture in the opposite direction (to scroll the
|
||||
* viewport *up*, fingers move *down*), with a generous starting offset
|
||||
* so the gesture begins safely inside the viewport.
|
||||
*/
|
||||
suspend fun scroll(
|
||||
direction: String,
|
||||
durationMs: Long = DEFAULT_SWIPE_DURATION_MS,
|
||||
): ActionResult {
|
||||
val root = service.rootInActiveWindow
|
||||
?: return ActionResult.failure("no active window available")
|
||||
val bounds = android.graphics.Rect().also { root.getBoundsInScreen(it) }
|
||||
val cx = bounds.centerX()
|
||||
val cy = bounds.centerY()
|
||||
val qx = bounds.width() / 4
|
||||
val qy = bounds.height() / 4
|
||||
|
||||
return when (direction.lowercase()) {
|
||||
"up" -> swipe(cx, cy - qy, cx, cy + qy, durationMs)
|
||||
"down" -> swipe(cx, cy + qy, cx, cy - qy, durationMs)
|
||||
"left" -> swipe(cx - qx, cy, cx + qx, cy, durationMs)
|
||||
"right" -> swipe(cx + qx, cy, cx - qx, cy, durationMs)
|
||||
else -> ActionResult.failure(
|
||||
"scroll direction must be one of up/down/left/right (got '$direction')"
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* `press_key` maps string names to [AccessibilityService] global
|
||||
* actions. We deliberately do NOT accept numeric [KeyEvent] codes —
|
||||
* only a curated vocabulary — so the agent can't inject arbitrary
|
||||
* keypresses into sensitive system UI.
|
||||
*/
|
||||
fun pressKey(keyName: String): ActionResult {
|
||||
val action = when (keyName.lowercase().trim()) {
|
||||
"home" -> AccessibilityService.GLOBAL_ACTION_HOME
|
||||
"back" -> AccessibilityService.GLOBAL_ACTION_BACK
|
||||
"recents", "recent_apps", "overview" ->
|
||||
AccessibilityService.GLOBAL_ACTION_RECENTS
|
||||
"notifications" ->
|
||||
AccessibilityService.GLOBAL_ACTION_NOTIFICATIONS
|
||||
"quick_settings" ->
|
||||
AccessibilityService.GLOBAL_ACTION_QUICK_SETTINGS
|
||||
"power_dialog" ->
|
||||
AccessibilityService.GLOBAL_ACTION_POWER_DIALOG
|
||||
"lock_screen" ->
|
||||
AccessibilityService.GLOBAL_ACTION_LOCK_SCREEN
|
||||
else -> return ActionResult.failure(
|
||||
"unsupported key '$keyName' — supported: home, back, recents, " +
|
||||
"notifications, quick_settings, power_dialog, lock_screen"
|
||||
)
|
||||
}
|
||||
val ok = try {
|
||||
service.performGlobalAction(action)
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "performGlobalAction threw: ${t.message}")
|
||||
false
|
||||
}
|
||||
return if (ok) ActionResult.ok(mapOf("key" to keyName))
|
||||
else ActionResult.failure("performGlobalAction returned false for '$keyName'")
|
||||
}
|
||||
|
||||
/**
|
||||
* Sleep for [ms] milliseconds, clamped to [MAX_WAIT_MS]. This exists
|
||||
* so agents can chain `tap → wait → read_screen` to give the UI time
|
||||
* to settle between steps.
|
||||
*/
|
||||
suspend fun wait(ms: Long): ActionResult {
|
||||
val clamped = ms.coerceIn(0L, MAX_WAIT_MS)
|
||||
delay(clamped)
|
||||
return ActionResult.ok(mapOf("slept_ms" to clamped))
|
||||
}
|
||||
|
||||
// ─── gesture dispatch plumbing ────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Wrap [AccessibilityService.dispatchGesture] as a suspend function.
|
||||
* Returns `true` on completion, `false` on cancellation or failure.
|
||||
*
|
||||
* The callback runs on the main-thread Handler to match what the
|
||||
* Android docs recommend — all gesture-state transitions happen on
|
||||
* the main looper anyway.
|
||||
*/
|
||||
private suspend fun dispatchGesture(gesture: GestureDescription): Boolean =
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val handler = Handler(Looper.getMainLooper())
|
||||
val callback = object : AccessibilityService.GestureResultCallback() {
|
||||
override fun onCompleted(gesture: GestureDescription?) {
|
||||
if (cont.isActive) cont.resume(true)
|
||||
}
|
||||
|
||||
override fun onCancelled(gesture: GestureDescription?) {
|
||||
if (cont.isActive) cont.resume(false)
|
||||
}
|
||||
}
|
||||
val accepted = try {
|
||||
service.dispatchGesture(gesture, callback, handler)
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "dispatchGesture threw: ${t.message}")
|
||||
false
|
||||
}
|
||||
if (!accepted && cont.isActive) {
|
||||
cont.resume(false)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
package com.hermesandroid.relay.accessibility
|
||||
|
||||
import android.content.BroadcastReceiver
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.content.IntentFilter
|
||||
import android.os.BatteryManager
|
||||
import android.os.PowerManager
|
||||
import android.util.Log
|
||||
import com.hermesandroid.relay.network.ChannelMultiplexer
|
||||
import com.hermesandroid.relay.network.models.Envelope
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.isActive
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
/**
|
||||
* Phase 3 — γ `accessibility-runtime`
|
||||
*
|
||||
* Coroutine-driven status reporter. Emits a `bridge.status` envelope every
|
||||
* [TICK_MS] (default 30 seconds) describing the phone's current state:
|
||||
*
|
||||
* ```json
|
||||
* {
|
||||
* "channel": "bridge",
|
||||
* "type": "bridge.status",
|
||||
* "payload": {
|
||||
* "screen_on": true,
|
||||
* "battery": 78,
|
||||
* "current_app": "com.android.chrome",
|
||||
* "accessibility_enabled": true
|
||||
* }
|
||||
* }
|
||||
* ```
|
||||
*
|
||||
* Values are probed fresh on every tick — we don't cache battery across
|
||||
* ticks because the user-visible number on the agent side is only ever
|
||||
* meaningful as a "recently observed" value.
|
||||
*
|
||||
* The reporter is a no-op until [start] is called, and [stop] is idempotent.
|
||||
* [ConnectionViewModel] owns it and ties the lifecycle to the WSS
|
||||
* connection — reporting while disconnected is a waste of battery and the
|
||||
* multiplexer would drop the envelopes silently anyway.
|
||||
*/
|
||||
class BridgeStatusReporter(
|
||||
private val context: Context,
|
||||
private val multiplexer: ChannelMultiplexer,
|
||||
private val scope: CoroutineScope,
|
||||
) {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "BridgeStatusReporter"
|
||||
private const val TICK_MS = 30_000L
|
||||
}
|
||||
|
||||
private var job: Job? = null
|
||||
|
||||
/**
|
||||
* Start the reporter. Safe to call multiple times — if a job is
|
||||
* already running we log and return.
|
||||
*/
|
||||
fun start() {
|
||||
if (job?.isActive == true) {
|
||||
Log.v(TAG, "already running")
|
||||
return
|
||||
}
|
||||
job = scope.launch {
|
||||
// Send an immediate first tick so the agent sees fresh status
|
||||
// as soon as the WSS connection comes up, rather than waiting
|
||||
// up to 30s for the first periodic tick.
|
||||
while (isActive) {
|
||||
try {
|
||||
emitTick()
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "status emit failed: ${t.message}")
|
||||
}
|
||||
delay(TICK_MS)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fun stop() {
|
||||
job?.cancel()
|
||||
job = null
|
||||
}
|
||||
|
||||
/**
|
||||
* Build one status envelope from live phone state and push it through
|
||||
* the multiplexer. Exposed as internal-ish so unit tests can drive
|
||||
* a single tick without spinning the coroutine.
|
||||
*/
|
||||
internal fun emitTick() {
|
||||
val screenOn = try {
|
||||
val pm = context.getSystemService(Context.POWER_SERVICE) as PowerManager
|
||||
pm.isInteractive
|
||||
} catch (t: Throwable) {
|
||||
Log.v(TAG, "screenOn probe failed: ${t.message}")
|
||||
false
|
||||
}
|
||||
|
||||
val battery = try {
|
||||
// Prefer the BatteryManager property API — it's the only
|
||||
// always-accurate source on modern Android. The legacy sticky
|
||||
// intent path is fine too but we'd have to parse two fields.
|
||||
val bm = context.getSystemService(Context.BATTERY_SERVICE) as BatteryManager
|
||||
bm.getIntProperty(BatteryManager.BATTERY_PROPERTY_CAPACITY)
|
||||
} catch (t: Throwable) {
|
||||
Log.v(TAG, "battery probe failed: ${t.message}")
|
||||
-1
|
||||
}
|
||||
|
||||
// If the property API returns 0 (some OEM firmwares do) fall back
|
||||
// to the sticky intent read.
|
||||
val batteryFinal = if (battery <= 0) readBatteryViaIntent() else battery
|
||||
|
||||
val currentApp = HermesAccessibilityService.instance?.currentApp
|
||||
|
||||
val accessibilityEnabled = HermesAccessibilityService.instance != null
|
||||
|
||||
val envelope = Envelope(
|
||||
channel = "bridge",
|
||||
type = "bridge.status",
|
||||
payload = buildJsonObject {
|
||||
put("screen_on", screenOn)
|
||||
put("battery", batteryFinal)
|
||||
put("current_app", currentApp ?: "unknown")
|
||||
put("accessibility_enabled", accessibilityEnabled)
|
||||
put("ts", System.currentTimeMillis())
|
||||
}
|
||||
)
|
||||
multiplexer.send(envelope)
|
||||
}
|
||||
|
||||
/**
|
||||
* Legacy fallback for BatteryManager.getIntProperty returning 0 —
|
||||
* reads the sticky `ACTION_BATTERY_CHANGED` intent and computes
|
||||
* percentage manually.
|
||||
*/
|
||||
private fun readBatteryViaIntent(): Int = try {
|
||||
val filter = IntentFilter(Intent.ACTION_BATTERY_CHANGED)
|
||||
@Suppress("UNUSED_VARIABLE")
|
||||
val placeholder: BroadcastReceiver? = null
|
||||
val battery = context.registerReceiver(null, filter)
|
||||
val level = battery?.getIntExtra(BatteryManager.EXTRA_LEVEL, -1) ?: -1
|
||||
val scale = battery?.getIntExtra(BatteryManager.EXTRA_SCALE, -1) ?: -1
|
||||
if (level >= 0 && scale > 0) (level * 100 / scale) else -1
|
||||
} catch (t: Throwable) {
|
||||
Log.v(TAG, "battery intent fallback failed: ${t.message}")
|
||||
-1
|
||||
}
|
||||
}
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
package com.hermesandroid.relay.accessibility
|
||||
|
||||
import android.accessibilityservice.AccessibilityService
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.os.Build
|
||||
import android.util.Log
|
||||
import android.view.accessibility.AccessibilityEvent
|
||||
import android.view.accessibility.AccessibilityNodeInfo
|
||||
import androidx.datastore.preferences.core.booleanPreferencesKey
|
||||
import androidx.datastore.preferences.core.edit
|
||||
import com.hermesandroid.relay.data.relayDataStore
|
||||
import kotlinx.coroutines.flow.Flow
|
||||
import kotlinx.coroutines.flow.map
|
||||
|
||||
/**
|
||||
* Phase 3 — γ `accessibility-runtime`
|
||||
*
|
||||
* Hermes's master `AccessibilityService` subclass. Provides the phone-side
|
||||
* execution layer for the bridge channel: the agent reads the screen,
|
||||
* taps/types/swipes, and captures screenshots through this service.
|
||||
*
|
||||
* # Lifecycle & singleton pattern
|
||||
*
|
||||
* Android instantiates `AccessibilityService` subclasses itself (through the
|
||||
* manifest declaration + user opt-in in Settings → Accessibility), so we
|
||||
* can't pass collaborators in via a constructor. The canonical workaround
|
||||
* is a weak-referenced singleton: on [onServiceConnected] the instance
|
||||
* registers itself with [Companion.instance], and on [onUnbind]/destroy it
|
||||
* clears itself. Any other code that needs to dispatch gestures (the
|
||||
* `BridgeCommandHandler`) asks for [instance] and bails out if it's null
|
||||
* (service not running / user hasn't granted permission).
|
||||
*
|
||||
* # Master enable / disable
|
||||
*
|
||||
* The Android system toggle in `Settings → Accessibility → Hermes Relay` is
|
||||
* the hard switch — if it's off we never receive events. On top of that the
|
||||
* user can flip a soft master in Settings (`bridge_master_enabled`); when
|
||||
* that's false we still run (Android requires it to stay connected) but we
|
||||
* refuse to execute commands. [isMasterEnabled] is a StateFlow the UI
|
||||
* observes and the command handler checks before dispatching actions.
|
||||
*
|
||||
* # Event handling
|
||||
*
|
||||
* We subscribe to a minimal event set — `TYPE_WINDOW_STATE_CHANGED` to
|
||||
* track the foreground package (for [currentApp] status), and nothing else.
|
||||
* The XML resource (flavor-provided by Agent β) controls the exact flag
|
||||
* bitset. We deliberately do NOT process text / content-change events —
|
||||
* those fire thousands of times a minute and are pointless for our use
|
||||
* case (we read the UI tree on demand via `rootInActiveWindow`).
|
||||
*/
|
||||
class HermesAccessibilityService : AccessibilityService() {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "HermesA11yService"
|
||||
|
||||
/** Master-enable DataStore key — read + toggled from Settings UI. */
|
||||
val KEY_BRIDGE_MASTER_ENABLED = booleanPreferencesKey("bridge_master_enabled")
|
||||
|
||||
/**
|
||||
* Static reference to the live service instance, or null if the
|
||||
* service is not running. Written on [onServiceConnected],
|
||||
* cleared on [onUnbind] / [onDestroy].
|
||||
*
|
||||
* Read by [com.hermesandroid.relay.network.handlers.BridgeCommandHandler]
|
||||
* and by the Bridge UI screen (δ) to check live status.
|
||||
*/
|
||||
@Volatile
|
||||
var instance: HermesAccessibilityService? = null
|
||||
private set
|
||||
|
||||
/**
|
||||
* Observe the DataStore-backed master enable flag for the bridge.
|
||||
* UI can collect this to drive the master-toggle switch; the service
|
||||
* itself also polls it via [isMasterEnabled] before executing commands.
|
||||
*/
|
||||
fun masterEnabledFlow(context: Context): Flow<Boolean> =
|
||||
context.applicationContext.relayDataStore.data
|
||||
.map { prefs -> prefs[KEY_BRIDGE_MASTER_ENABLED] ?: false }
|
||||
|
||||
/**
|
||||
* Persist a new master-enable value. Called from Settings UI when the
|
||||
* user flips the switch, and from [BridgeStatusReporter] / safety
|
||||
* rails when auto-disable timers fire.
|
||||
*/
|
||||
suspend fun setMasterEnabled(context: Context, enabled: Boolean) {
|
||||
context.applicationContext.relayDataStore.edit { prefs ->
|
||||
prefs[KEY_BRIDGE_MASTER_ENABLED] = enabled
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private val screenReader = ScreenReader()
|
||||
private var _actionExecutor: ActionExecutor? = null
|
||||
|
||||
/**
|
||||
* Cached package name of the currently-foregrounded app. Updated on
|
||||
* every `TYPE_WINDOW_STATE_CHANGED` event. Read by the status reporter
|
||||
* for the `current_app` field in `bridge.status`.
|
||||
*/
|
||||
@Volatile
|
||||
var currentApp: String? = null
|
||||
private set
|
||||
|
||||
/**
|
||||
* Public accessor for the lazily-constructed [ActionExecutor]. The
|
||||
* executor needs a back-reference to the service (for `dispatchGesture`),
|
||||
* so we can only build it after Android has fully constructed us.
|
||||
*/
|
||||
val actionExecutor: ActionExecutor
|
||||
get() = _actionExecutor ?: ActionExecutor(this).also { _actionExecutor = it }
|
||||
|
||||
/** Convenience wrapper — the service uses its own [ScreenReader] instance. */
|
||||
val reader: ScreenReader get() = screenReader
|
||||
|
||||
override fun onServiceConnected() {
|
||||
super.onServiceConnected()
|
||||
instance = this
|
||||
Log.i(TAG, "HermesAccessibilityService connected")
|
||||
}
|
||||
|
||||
override fun onAccessibilityEvent(event: AccessibilityEvent?) {
|
||||
if (event == null) return
|
||||
|
||||
when (event.eventType) {
|
||||
AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED -> {
|
||||
val pkg = event.packageName?.toString()
|
||||
if (!pkg.isNullOrBlank()) {
|
||||
currentApp = pkg
|
||||
}
|
||||
}
|
||||
else -> {
|
||||
// Other event types are declared in the config XML for
|
||||
// future safety rails (blocklist enforcement via
|
||||
// content-change events) but we deliberately no-op here
|
||||
// today. Filtering happens inside the config flag bitset
|
||||
// so we never even receive most events.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
override fun onInterrupt() {
|
||||
// Called by the system when it wants us to drop any in-flight work.
|
||||
// We don't queue long-running operations — every bridge command is
|
||||
// fire-and-forget with its own callback — so there's nothing to
|
||||
// cancel here.
|
||||
Log.i(TAG, "onInterrupt — accessibility service asked to stop work")
|
||||
}
|
||||
|
||||
override fun onUnbind(intent: Intent?): Boolean {
|
||||
Log.i(TAG, "HermesAccessibilityService unbinding")
|
||||
if (instance === this) instance = null
|
||||
return super.onUnbind(intent)
|
||||
}
|
||||
|
||||
override fun onDestroy() {
|
||||
if (instance === this) instance = null
|
||||
super.onDestroy()
|
||||
}
|
||||
|
||||
/**
|
||||
* Check whether the soft master toggle is on. This is a best-effort
|
||||
* blocking read — the canonical source of truth is the DataStore Flow
|
||||
* observed by UI. We cache the value on every observed event so command
|
||||
* handlers can check it without a suspend call.
|
||||
*/
|
||||
@Volatile
|
||||
private var cachedMasterEnabled: Boolean = false
|
||||
|
||||
fun isMasterEnabled(): Boolean = cachedMasterEnabled
|
||||
|
||||
/** Called by the app-level observer to feed the cached value. */
|
||||
fun updateMasterEnabledCache(enabled: Boolean) {
|
||||
cachedMasterEnabled = enabled
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot the current root node of the active window. Returns null if
|
||||
* no window is focused or the system refuses access (e.g. IME window).
|
||||
*
|
||||
* Callers must `recycle()` the returned node when done.
|
||||
*
|
||||
* On API 34+, [AccessibilityNodeInfo.recycle] is deprecated but still
|
||||
* safe to call — the system just no-ops. We support min SDK 26 so we
|
||||
* keep calling it for the older branch.
|
||||
*/
|
||||
fun snapshotRoot(): AccessibilityNodeInfo? = try {
|
||||
rootInActiveWindow
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "rootInActiveWindow threw: ${t.message}")
|
||||
null
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort indicator — `true` when the runtime is >= API 26 (always
|
||||
* true on this app, we target 26+). Exposed for completeness so the
|
||||
* Bridge UI can render a compile-time capabilities badge without
|
||||
* reading BuildConfig directly.
|
||||
*/
|
||||
val supportsGestures: Boolean get() = Build.VERSION.SDK_INT >= Build.VERSION_CODES.O
|
||||
}
|
||||
@@ -0,0 +1,426 @@
|
||||
package com.hermesandroid.relay.accessibility
|
||||
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.graphics.Bitmap
|
||||
import android.graphics.PixelFormat
|
||||
import android.hardware.display.DisplayManager
|
||||
import android.hardware.display.VirtualDisplay
|
||||
import android.media.Image
|
||||
import android.media.ImageReader
|
||||
import android.media.projection.MediaProjection
|
||||
import android.media.projection.MediaProjectionManager
|
||||
import android.os.Handler
|
||||
import android.os.HandlerThread
|
||||
import android.util.DisplayMetrics
|
||||
import android.util.Log
|
||||
import android.view.WindowManager
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.suspendCancellableCoroutine
|
||||
import kotlinx.coroutines.withContext
|
||||
import okhttp3.MediaType.Companion.toMediaType
|
||||
import okhttp3.MultipartBody
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.Request
|
||||
import okhttp3.RequestBody.Companion.toRequestBody
|
||||
import java.io.ByteArrayOutputStream
|
||||
import java.io.File
|
||||
import java.io.IOException
|
||||
import java.nio.ByteBuffer
|
||||
import java.util.concurrent.TimeUnit
|
||||
import kotlin.coroutines.resume
|
||||
|
||||
/**
|
||||
* Phase 3 — γ `accessibility-runtime`
|
||||
*
|
||||
* Captures the phone's screen via the Android [MediaProjection] API, encodes
|
||||
* it as PNG, and publishes it to the relay so the agent can fetch it.
|
||||
*
|
||||
* ## Permission flow (blocker for Agent δ / UI to wire)
|
||||
*
|
||||
* `MediaProjection` cannot be granted by the app itself — it needs an
|
||||
* explicit user consent dialog per session, launched via
|
||||
* [MediaProjectionManager.createScreenCaptureIntent] from an `Activity`.
|
||||
* The resulting `Intent` is then passed to [MediaProjectionManager.getMediaProjection]
|
||||
* to build an actual projection.
|
||||
*
|
||||
* Because the grant lives on an `Activity` result, this class can only
|
||||
* provide the capture loop — **the consent flow must be wired by the
|
||||
* Bridge UI screen (Agent δ)**. Suggested contract:
|
||||
*
|
||||
* 1. `BridgeScreen` holds an `ActivityResultLauncher<Intent>` registered
|
||||
* with `ActivityResultContracts.StartActivityForResult()`.
|
||||
* 2. On "Enable screenshots" tap, δ calls
|
||||
* `MediaProjectionManager.createScreenCaptureIntent()` and launches it.
|
||||
* 3. On result, δ passes `(resultCode, data)` into a central holder
|
||||
* (e.g. a ViewModel singleton or [MediaProjectionHolder]).
|
||||
* 4. [ScreenCapture] reads from that holder on each capture call and
|
||||
* rebuilds a `MediaProjection` when needed. The projection will need
|
||||
* to be backed by a foreground service on Android 10+ — Agent ζ
|
||||
* owns the persistent-notification service declaration.
|
||||
*
|
||||
* Until that wiring lands, this class will fail gracefully with
|
||||
* `Result.failure(IllegalStateException("MediaProjection not granted"))`
|
||||
* and the agent will see the error text in the `bridge.response` body.
|
||||
*
|
||||
* ## Upload path — current limitation
|
||||
*
|
||||
* The relay's existing `/media/register` endpoint is **loopback-only and
|
||||
* path-based** (`plugin/relay/media.py`) — it registers a file path that
|
||||
* already exists on the relay host, with an optional content type and
|
||||
* filename. The phone is by definition not on the relay host, so it has no
|
||||
* usable path to register.
|
||||
*
|
||||
* Two options exist for bridging this gap:
|
||||
*
|
||||
* 1. **New relay endpoint** (preferred) — `POST /media/upload` accepts
|
||||
* `multipart/form-data` with the PNG bytes, writes to a sandboxed tmp
|
||||
* dir (`tempfile.gettempdir()`), then calls `MediaRegistry.register()`
|
||||
* on the resulting path. Wire shape mirrors `/voice/transcribe`. This
|
||||
* is a clean server-side change that Agent α could land in parallel.
|
||||
*
|
||||
* 2. **Local-only screenshots** (fallback) — the phone writes the PNG to
|
||||
* its own cache dir, emits `MEDIA:file://<cache_path>` in the response,
|
||||
* and the agent fetches it via an on-device tool (N/A — the agent runs
|
||||
* on the host, not the phone). So option 2 doesn't actually work.
|
||||
*
|
||||
* This class implements option 1 via [uploadViaMultipart]. If the endpoint
|
||||
* returns 404 (not yet deployed), we surface the error to the agent with a
|
||||
* clear message. **Agent α owns the `POST /media/upload` endpoint** — it's
|
||||
* the only remaining server-side work to complete Tier 1 screenshots.
|
||||
*
|
||||
* ## Thread model
|
||||
*
|
||||
* `ImageReader` delivers frames on a background `HandlerThread` we own.
|
||||
* The PNG encode runs on [Dispatchers.IO] via [withContext]. The HTTP
|
||||
* upload is also IO-dispatched. All three can be cancelled by the caller.
|
||||
*/
|
||||
class ScreenCapture(
|
||||
private val context: Context,
|
||||
private val httpClient: OkHttpClient,
|
||||
private val relayUrlProvider: () -> String?,
|
||||
private val sessionTokenProvider: suspend () -> String?,
|
||||
private val mediaProjectionProvider: () -> MediaProjection?,
|
||||
) {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "ScreenCapture"
|
||||
|
||||
/** PNG quality is a no-op for PNG, but Bitmap.compress expects the arg. */
|
||||
private const val PNG_QUALITY = 100
|
||||
|
||||
/** ImageReader buffer count — 2 is enough for our one-shot-at-a-time use. */
|
||||
private const val MAX_IMAGES = 2
|
||||
|
||||
/** Capture timeout — if no frame arrives in this window, fail loudly. */
|
||||
private const val CAPTURE_TIMEOUT_MS = 2_500L
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the consent intent that `BridgeScreen` launches via an
|
||||
* `ActivityResultLauncher`. Callers should launch the intent with
|
||||
* `StartActivityForResult` and on success call
|
||||
* [MediaProjectionHolder.onGranted] with the result code + data Intent.
|
||||
*/
|
||||
fun createConsentIntent(): Intent =
|
||||
(context.getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager)
|
||||
.createScreenCaptureIntent()
|
||||
|
||||
/**
|
||||
* Capture one frame and upload it to the relay. Returns the inbound
|
||||
* media marker (`MEDIA:hermes-relay://<token>`) on success so the
|
||||
* bridge command handler can embed it directly in the `bridge.response`
|
||||
* result.
|
||||
*
|
||||
* Fails fast and with clear messaging on every expected error path:
|
||||
*
|
||||
* - `MediaProjection not granted` → δ needs to run the consent flow
|
||||
* - `relay URL not configured` / `session token missing` → pair first
|
||||
* - `relay upload endpoint not found` → α needs to ship `/media/upload`
|
||||
* - `capture timeout` → the virtual display never emitted a frame
|
||||
*/
|
||||
suspend fun captureAndUpload(): Result<String> = withContext(Dispatchers.IO) {
|
||||
val projection = mediaProjectionProvider()
|
||||
?: return@withContext Result.failure(
|
||||
IllegalStateException(
|
||||
"MediaProjection not granted — enable Bridge screenshots in the app"
|
||||
)
|
||||
)
|
||||
|
||||
val pngBytes = try {
|
||||
captureOnce(projection)
|
||||
} catch (e: Exception) {
|
||||
Log.w(TAG, "captureOnce failed: ${e.message}")
|
||||
return@withContext Result.failure(e)
|
||||
}
|
||||
|
||||
uploadViaMultipart(pngBytes)
|
||||
}
|
||||
|
||||
/**
|
||||
* Synchronously (inside a suspendCancellableCoroutine) capture exactly
|
||||
* one frame from a freshly-built VirtualDisplay + ImageReader, encode
|
||||
* it to PNG, and return the bytes.
|
||||
*/
|
||||
private suspend fun captureOnce(projection: MediaProjection): ByteArray =
|
||||
suspendCancellableCoroutine { cont ->
|
||||
val metrics = DisplayMetrics()
|
||||
@Suppress("DEPRECATION")
|
||||
(context.getSystemService(Context.WINDOW_SERVICE) as WindowManager)
|
||||
.defaultDisplay.getRealMetrics(metrics)
|
||||
|
||||
val width = metrics.widthPixels
|
||||
val height = metrics.heightPixels
|
||||
val densityDpi = metrics.densityDpi
|
||||
|
||||
val reader = ImageReader.newInstance(
|
||||
width, height, PixelFormat.RGBA_8888, MAX_IMAGES
|
||||
)
|
||||
|
||||
val captureThread = HandlerThread("HermesScreenCapture").apply { start() }
|
||||
val captureHandler = Handler(captureThread.looper)
|
||||
|
||||
var virtualDisplay: VirtualDisplay? = null
|
||||
var resolved = false
|
||||
|
||||
fun cleanup() {
|
||||
try { virtualDisplay?.release() } catch (_: Throwable) {}
|
||||
try { reader.close() } catch (_: Throwable) {}
|
||||
try { captureThread.quitSafely() } catch (_: Throwable) {}
|
||||
}
|
||||
|
||||
val postedTimeout = Handler(captureThread.looper)
|
||||
postedTimeout.postDelayed({
|
||||
if (!resolved) {
|
||||
resolved = true
|
||||
cleanup()
|
||||
if (cont.isActive) {
|
||||
cont.resumeWith(
|
||||
Result.failure(IOException("screen capture timed out"))
|
||||
)
|
||||
}
|
||||
}
|
||||
}, CAPTURE_TIMEOUT_MS)
|
||||
|
||||
reader.setOnImageAvailableListener({ r ->
|
||||
if (resolved) return@setOnImageAvailableListener
|
||||
var image: Image? = null
|
||||
try {
|
||||
image = r.acquireLatestImage() ?: return@setOnImageAvailableListener
|
||||
val png = imageToPngBytes(image, width, height)
|
||||
resolved = true
|
||||
cleanup()
|
||||
if (cont.isActive) cont.resume(png)
|
||||
} catch (t: Throwable) {
|
||||
resolved = true
|
||||
cleanup()
|
||||
if (cont.isActive) {
|
||||
cont.resumeWith(Result.failure(t))
|
||||
}
|
||||
} finally {
|
||||
try { image?.close() } catch (_: Throwable) {}
|
||||
}
|
||||
}, captureHandler)
|
||||
|
||||
try {
|
||||
virtualDisplay = projection.createVirtualDisplay(
|
||||
"hermes-bridge-capture",
|
||||
width,
|
||||
height,
|
||||
densityDpi,
|
||||
DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
|
||||
reader.surface,
|
||||
null,
|
||||
captureHandler
|
||||
)
|
||||
} catch (t: Throwable) {
|
||||
resolved = true
|
||||
cleanup()
|
||||
if (cont.isActive) {
|
||||
cont.resumeWith(Result.failure(t))
|
||||
}
|
||||
}
|
||||
|
||||
cont.invokeOnCancellation {
|
||||
if (!resolved) {
|
||||
resolved = true
|
||||
cleanup()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert an [Image] from `ImageReader` into a PNG byte array. The
|
||||
* plane's `rowStride` may be wider than `width * 4` — we must crop
|
||||
* the stride padding before [Bitmap.copyPixelsFromBuffer].
|
||||
*/
|
||||
private fun imageToPngBytes(image: Image, width: Int, height: Int): ByteArray {
|
||||
val plane = image.planes[0]
|
||||
val buffer = plane.buffer
|
||||
val pixelStride = plane.pixelStride
|
||||
val rowStride = plane.rowStride
|
||||
val rowPadding = rowStride - pixelStride * width
|
||||
|
||||
val bitmapWidth = width + rowPadding / pixelStride
|
||||
val bitmap = Bitmap.createBitmap(bitmapWidth, height, Bitmap.Config.ARGB_8888)
|
||||
bitmap.copyPixelsFromBuffer(buffer)
|
||||
|
||||
// Crop to the exact screen width if rowStride padding widened us.
|
||||
val cropped = if (bitmapWidth != width) {
|
||||
Bitmap.createBitmap(bitmap, 0, 0, width, height).also {
|
||||
bitmap.recycle()
|
||||
}
|
||||
} else bitmap
|
||||
|
||||
val out = ByteArrayOutputStream(256 * 1024)
|
||||
cropped.compress(Bitmap.CompressFormat.PNG, PNG_QUALITY, out)
|
||||
cropped.recycle()
|
||||
return out.toByteArray()
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload PNG bytes to the relay via `POST /media/upload` (multipart).
|
||||
*
|
||||
* **Blocker:** this endpoint does not exist server-side yet (see class
|
||||
* docstring). When it ships, it should accept a single `file` part and
|
||||
* return `{"ok": true, "token": "..."}` — the same JSON shape as
|
||||
* `/media/register`, minus the loopback restriction and with the
|
||||
* server sandboxing the temp-file write internally.
|
||||
*/
|
||||
private suspend fun uploadViaMultipart(pngBytes: ByteArray): Result<String> {
|
||||
val relayUrl = relayUrlProvider()?.trim().orEmpty()
|
||||
if (relayUrl.isEmpty()) {
|
||||
return Result.failure(IllegalStateException("Relay URL not configured"))
|
||||
}
|
||||
val sessionToken = sessionTokenProvider()
|
||||
if (sessionToken.isNullOrBlank()) {
|
||||
return Result.failure(
|
||||
IllegalStateException("Relay not paired — session token missing")
|
||||
)
|
||||
}
|
||||
|
||||
val httpBase = relayUrl
|
||||
.replace(Regex("^wss://", RegexOption.IGNORE_CASE), "https://")
|
||||
.replace(Regex("^ws://", RegexOption.IGNORE_CASE), "http://")
|
||||
.trimEnd('/')
|
||||
|
||||
val url = "$httpBase/media/upload"
|
||||
val body = MultipartBody.Builder()
|
||||
.setType(MultipartBody.FORM)
|
||||
.addFormDataPart(
|
||||
"file",
|
||||
"hermes-screenshot-${System.currentTimeMillis()}.png",
|
||||
pngBytes.toRequestBody("image/png".toMediaType())
|
||||
)
|
||||
.build()
|
||||
|
||||
val fastClient = httpClient.newBuilder()
|
||||
.callTimeout(15, TimeUnit.SECONDS)
|
||||
.build()
|
||||
|
||||
val request = Request.Builder()
|
||||
.url(url)
|
||||
.post(body)
|
||||
.header("Authorization", "Bearer $sessionToken")
|
||||
.header("Accept", "application/json")
|
||||
.build()
|
||||
|
||||
return try {
|
||||
fastClient.newCall(request).execute().use { response ->
|
||||
when (response.code) {
|
||||
200 -> {
|
||||
val raw = response.body?.string().orEmpty()
|
||||
val token = extractToken(raw)
|
||||
if (token.isNullOrBlank()) {
|
||||
Result.failure(
|
||||
IOException("relay returned success but no token")
|
||||
)
|
||||
} else {
|
||||
Result.success("MEDIA:hermes-relay://$token")
|
||||
}
|
||||
}
|
||||
404 -> Result.failure(
|
||||
IOException(
|
||||
"relay /media/upload endpoint not found — server needs " +
|
||||
"Phase 3 α migration"
|
||||
)
|
||||
)
|
||||
401, 403 -> Result.failure(
|
||||
IOException("unauthorized — re-pair with the relay")
|
||||
)
|
||||
413 -> Result.failure(
|
||||
IOException("screenshot too large for relay media cap")
|
||||
)
|
||||
in 500..599 -> Result.failure(
|
||||
IOException("relay error (HTTP ${response.code})")
|
||||
)
|
||||
else -> Result.failure(
|
||||
IOException("HTTP ${response.code}: ${response.message}")
|
||||
)
|
||||
}
|
||||
}
|
||||
} catch (e: IOException) {
|
||||
Log.w(TAG, "uploadViaMultipart failed: ${e.message}")
|
||||
Result.failure(e)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimal JSON token extractor — the response body is small and has a
|
||||
* single interesting key. We avoid pulling in a full `Json` parse here
|
||||
* because `ScreenCapture` is already a heavy dependency graph (Android
|
||||
* media + OkHttp) and we don't want to add kotlinx.serialization
|
||||
* wiring for a 40-byte response.
|
||||
*/
|
||||
private fun extractToken(body: String): String? {
|
||||
val match = Regex("""\"token\"\s*:\s*\"([^\"]+)\"""").find(body)
|
||||
return match?.groupValues?.get(1)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Holds the per-session [MediaProjection] grant. The Bridge UI (Agent δ)
|
||||
* calls [onGranted] from its `ActivityResultLauncher` callback; [ScreenCapture]
|
||||
* reads [projection] through the lambda passed to its constructor.
|
||||
*
|
||||
* Cleared on [revoke] (user disabled screenshots) or when the projection's
|
||||
* own `onStop` callback fires (system revoked it).
|
||||
*/
|
||||
object MediaProjectionHolder {
|
||||
@Volatile
|
||||
private var _projection: MediaProjection? = null
|
||||
|
||||
val projection: MediaProjection? get() = _projection
|
||||
|
||||
/**
|
||||
* Call from the Bridge UI's `ActivityResultLauncher` callback.
|
||||
* Returns true on success, false on user-rejected consent.
|
||||
*/
|
||||
fun onGranted(context: Context, resultCode: Int, data: Intent?): Boolean {
|
||||
if (resultCode != android.app.Activity.RESULT_OK || data == null) {
|
||||
return false
|
||||
}
|
||||
val manager = context.getSystemService(Context.MEDIA_PROJECTION_SERVICE)
|
||||
as MediaProjectionManager
|
||||
val newProjection = try {
|
||||
manager.getMediaProjection(resultCode, data)
|
||||
} catch (t: Throwable) {
|
||||
Log.w("MediaProjectionHolder", "getMediaProjection threw: ${t.message}")
|
||||
null
|
||||
} ?: return false
|
||||
|
||||
newProjection.registerCallback(object : MediaProjection.Callback() {
|
||||
override fun onStop() {
|
||||
_projection = null
|
||||
}
|
||||
}, Handler(android.os.Looper.getMainLooper()))
|
||||
|
||||
_projection = newProjection
|
||||
return true
|
||||
}
|
||||
|
||||
fun revoke() {
|
||||
try { _projection?.stop() } catch (_: Throwable) {}
|
||||
_projection = null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
package com.hermesandroid.relay.accessibility
|
||||
|
||||
import android.graphics.Rect
|
||||
import android.view.accessibility.AccessibilityNodeInfo
|
||||
import kotlinx.serialization.Serializable
|
||||
|
||||
/**
|
||||
* Phase 3 — γ `accessibility-runtime`
|
||||
*
|
||||
* Walks the active-window `AccessibilityNodeInfo` tree and produces a
|
||||
* structured, serializable snapshot the agent can reason about.
|
||||
*
|
||||
* The output shape is deliberately flat: the agent overwhelmingly cares
|
||||
* about *"what text can I see, and where is it?"* — not the exact widget
|
||||
* hierarchy. We emit one [ScreenNode] per interesting node (anything with
|
||||
* non-blank text, content description, or a click / long-click action) and
|
||||
* include its screen bounds so [ActionExecutor.tapText] can hand them to
|
||||
* `dispatchGesture` without re-walking the tree.
|
||||
*
|
||||
* The tree is bounded by [MAX_NODES] to prevent pathological apps (grids
|
||||
* with thousands of cells) from producing multi-megabyte screen dumps.
|
||||
* When the cap is hit we short-circuit traversal and set
|
||||
* [ScreenContent.truncated] = true so the agent knows to scroll if it
|
||||
* needs more.
|
||||
*/
|
||||
class ScreenReader {
|
||||
|
||||
companion object {
|
||||
/**
|
||||
* Hard cap on node count. 512 is comfortable for a typical app
|
||||
* screen (most have 30–150 interesting nodes) while keeping the
|
||||
* wire payload under ~40 KB in the worst case.
|
||||
*/
|
||||
const val MAX_NODES = 512
|
||||
|
||||
/** Hard cap on individual text-field length before truncation. */
|
||||
const val MAX_TEXT_LEN = 2000
|
||||
}
|
||||
|
||||
/**
|
||||
* Structured representation of a screen, ready for JSON serialization.
|
||||
* The [rootBounds] are in absolute screen pixels (what `dispatchGesture`
|
||||
* uses).
|
||||
*/
|
||||
@Serializable
|
||||
data class ScreenContent(
|
||||
val packageName: String?,
|
||||
val rootBounds: Bounds,
|
||||
val nodes: List<ScreenNode>,
|
||||
val truncated: Boolean = false,
|
||||
val nodeCount: Int = nodes.size,
|
||||
)
|
||||
|
||||
@Serializable
|
||||
data class ScreenNode(
|
||||
val text: String? = null,
|
||||
val contentDescription: String? = null,
|
||||
val className: String? = null,
|
||||
val viewId: String? = null,
|
||||
val bounds: Bounds,
|
||||
val clickable: Boolean = false,
|
||||
val longClickable: Boolean = false,
|
||||
val scrollable: Boolean = false,
|
||||
val editable: Boolean = false,
|
||||
val focused: Boolean = false,
|
||||
val selected: Boolean = false,
|
||||
val enabled: Boolean = true,
|
||||
)
|
||||
|
||||
@Serializable
|
||||
data class Bounds(
|
||||
val left: Int,
|
||||
val top: Int,
|
||||
val right: Int,
|
||||
val bottom: Int,
|
||||
) {
|
||||
val centerX: Int get() = (left + right) / 2
|
||||
val centerY: Int get() = (top + bottom) / 2
|
||||
val width: Int get() = right - left
|
||||
val height: Int get() = bottom - top
|
||||
val isEmpty: Boolean get() = width <= 0 || height <= 0
|
||||
}
|
||||
|
||||
/**
|
||||
* Traverse the tree rooted at [rootNode] and return a [ScreenContent]
|
||||
* snapshot.
|
||||
*
|
||||
* This method does NOT recycle [rootNode] — the caller owns it (the
|
||||
* caller also owns the lifetime contract with `rootInActiveWindow`).
|
||||
*
|
||||
* @param includeBounds when false, we still collect bounds for
|
||||
* traversal decisions but zero them in the output to shrink the
|
||||
* payload. The agent almost always wants bounds so this defaults
|
||||
* true.
|
||||
*/
|
||||
fun readScreen(
|
||||
rootNode: AccessibilityNodeInfo,
|
||||
includeBounds: Boolean = true,
|
||||
): ScreenContent {
|
||||
val collected = ArrayList<ScreenNode>(128)
|
||||
val rootRect = Rect().also { rootNode.getBoundsInScreen(it) }
|
||||
val rootBounds = rootRect.toBoundsOrZero()
|
||||
|
||||
val truncated = walk(rootNode, collected, includeBounds)
|
||||
|
||||
return ScreenContent(
|
||||
packageName = rootNode.packageName?.toString(),
|
||||
rootBounds = rootBounds,
|
||||
nodes = collected,
|
||||
truncated = truncated,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursive walker. Returns `true` if the cap was hit (signaling
|
||||
* caller to mark output as truncated).
|
||||
*/
|
||||
private fun walk(
|
||||
node: AccessibilityNodeInfo?,
|
||||
out: MutableList<ScreenNode>,
|
||||
includeBounds: Boolean,
|
||||
): Boolean {
|
||||
if (node == null) return false
|
||||
if (out.size >= MAX_NODES) return true
|
||||
|
||||
// Skip invisible nodes early — no visual, no interaction target.
|
||||
if (!node.isVisibleToUser) {
|
||||
// still walk children, some containers aren't marked visible
|
||||
// but host visible descendants
|
||||
}
|
||||
|
||||
val rect = Rect()
|
||||
node.getBoundsInScreen(rect)
|
||||
val bounds = if (includeBounds) rect.toBoundsOrZero() else ZERO_BOUNDS
|
||||
|
||||
val text = node.text?.toString()?.takeIf { it.isNotBlank() }?.take(MAX_TEXT_LEN)
|
||||
val contentDesc = node.contentDescription?.toString()
|
||||
?.takeIf { it.isNotBlank() }
|
||||
?.take(MAX_TEXT_LEN)
|
||||
val clickable = node.isClickable
|
||||
val longClickable = node.isLongClickable
|
||||
val scrollable = node.isScrollable
|
||||
|
||||
val interesting = (text != null || contentDesc != null ||
|
||||
clickable || longClickable || scrollable || node.isEditable) &&
|
||||
!bounds.isEmpty
|
||||
|
||||
if (interesting) {
|
||||
out.add(
|
||||
ScreenNode(
|
||||
text = text,
|
||||
contentDescription = contentDesc,
|
||||
className = node.className?.toString(),
|
||||
viewId = node.viewIdResourceName,
|
||||
bounds = bounds,
|
||||
clickable = clickable,
|
||||
longClickable = longClickable,
|
||||
scrollable = scrollable,
|
||||
editable = node.isEditable,
|
||||
focused = node.isFocused,
|
||||
selected = node.isSelected,
|
||||
enabled = node.isEnabled,
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
val childCount = node.childCount
|
||||
for (i in 0 until childCount) {
|
||||
if (out.size >= MAX_NODES) return true
|
||||
val child = node.getChild(i) ?: continue
|
||||
try {
|
||||
val hit = walk(child, out, includeBounds)
|
||||
if (hit) return true
|
||||
} finally {
|
||||
@Suppress("DEPRECATION")
|
||||
try { child.recycle() } catch (_: Throwable) { }
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the first descendant node whose text or content-description
|
||||
* contains [needle] (case-insensitive). Used by
|
||||
* [ActionExecutor.tapText]. Returns the node's center bounds, or null
|
||||
* if no match.
|
||||
*
|
||||
* We walk fresh (not against a cached [ScreenContent]) so the result
|
||||
* is always current — tapping into stale bounds is the single most
|
||||
* common "bridge tapped the wrong thing" bug.
|
||||
*/
|
||||
fun findNodeBoundsByText(rootNode: AccessibilityNodeInfo, needle: String): Bounds? {
|
||||
if (needle.isBlank()) return null
|
||||
val lowered = needle.lowercase()
|
||||
return findFirst(rootNode) { node ->
|
||||
val text = node.text?.toString()?.lowercase()
|
||||
val desc = node.contentDescription?.toString()?.lowercase()
|
||||
(text?.contains(lowered) == true) || (desc?.contains(lowered) == true)
|
||||
}?.let { found ->
|
||||
val r = Rect()
|
||||
found.getBoundsInScreen(r)
|
||||
r.toBoundsOrZero()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the currently-focused input node (for [ActionExecutor.typeText]).
|
||||
* Prefers `FOCUS_INPUT` focus, falls back to the first editable node.
|
||||
*/
|
||||
fun findFocusedInput(rootNode: AccessibilityNodeInfo): AccessibilityNodeInfo? {
|
||||
rootNode.findFocus(AccessibilityNodeInfo.FOCUS_INPUT)?.let { return it }
|
||||
return findFirst(rootNode) { it.isEditable }
|
||||
}
|
||||
|
||||
private fun findFirst(
|
||||
node: AccessibilityNodeInfo?,
|
||||
predicate: (AccessibilityNodeInfo) -> Boolean,
|
||||
): AccessibilityNodeInfo? {
|
||||
if (node == null) return null
|
||||
if (predicate(node)) return node
|
||||
val childCount = node.childCount
|
||||
for (i in 0 until childCount) {
|
||||
val child = node.getChild(i) ?: continue
|
||||
val hit = findFirst(child, predicate)
|
||||
if (hit != null) return hit
|
||||
@Suppress("DEPRECATION")
|
||||
try { child.recycle() } catch (_: Throwable) { }
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
private fun Rect.toBoundsOrZero(): Bounds =
|
||||
Bounds(left = left, top = top, right = right, bottom = bottom)
|
||||
|
||||
private val ZERO_BOUNDS = Bounds(0, 0, 0, 0)
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
package com.hermesandroid.relay.data
|
||||
|
||||
import android.content.Context
|
||||
import androidx.datastore.preferences.core.booleanPreferencesKey
|
||||
import androidx.datastore.preferences.core.edit
|
||||
import androidx.datastore.preferences.core.stringPreferencesKey
|
||||
import kotlinx.coroutines.flow.Flow
|
||||
import kotlinx.coroutines.flow.map
|
||||
import kotlinx.serialization.Serializable
|
||||
import kotlinx.serialization.encodeToString
|
||||
import kotlinx.serialization.json.Json
|
||||
|
||||
/**
|
||||
* User-tunable bridge mode preferences + persisted activity log.
|
||||
*
|
||||
* Phase 3 Wave 1 — owned by Agent δ (`bridge-screen-ui`).
|
||||
*
|
||||
* - [masterEnabled] — the headline "Allow Agent Control" switch on BridgeScreen.
|
||||
* Compose-layer gate only; the real safety fence is that the user must also
|
||||
* enable `HermesAccessibilityService` in Android Settings (which is the
|
||||
* Tier-5 "honest" trust boundary Google cares about). Persisting it here lets
|
||||
* us survive restarts and lets Agent γ's `HermesAccessibilityService` query
|
||||
* the same source of truth without needing its own DataStore file.
|
||||
*
|
||||
* - [activityLog] — rolling window of recent [BridgeActivityEntry] rows that
|
||||
* back the Activity Log card. Capped at [MAX_LOG_ENTRIES] on every append
|
||||
* so we don't grow the DataStore preferences file unbounded (this is a Prefs
|
||||
* datastore, not Room — large blob values hurt commit latency). Serialized
|
||||
* as a single JSON array string under one key so the whole read/update is
|
||||
* atomic. That's cheaper than Proto DataStore for a cap this small and
|
||||
* matches the `VoicePreferences.kt` / `MediaSettings.kt` style already in
|
||||
* the tree.
|
||||
*
|
||||
* The log schema intentionally does NOT carry the screenshot bytes — those
|
||||
* live in `MediaRegistry` on the relay side and are fetched via
|
||||
* `RelayHttpClient.fetchMedia(token)` when the user expands the row. Storing
|
||||
* `thumbnailToken: String?` as an opaque reference keeps DataStore small and
|
||||
* reuses the existing MediaRegistry LRU-cap + FileProvider cache story.
|
||||
*/
|
||||
@Serializable
|
||||
data class BridgeActivityEntry(
|
||||
/** Epoch millis when the command was received from the relay. */
|
||||
val timestampMs: Long,
|
||||
/** Short method name — `tap`, `tap_text`, `type`, `swipe`, `read_screen`, etc. */
|
||||
val method: String,
|
||||
/** Free-form one-line summary of the args — `"(540, 1200)"`, `"send"`, `"Chrome"`. */
|
||||
val summary: String,
|
||||
/** Execution status — Pending, Success, Failed, Blocked (by Tier-5 safety rails). */
|
||||
val status: BridgeActivityStatus,
|
||||
/** Optional longer result text — set on Success / Failed to show in the expanded row. */
|
||||
val resultText: String? = null,
|
||||
/**
|
||||
* Optional screenshot MediaRegistry token (`hermes-relay://<token>` or the
|
||||
* raw token stem). The UI resolves this through the existing InboundAttachmentCard
|
||||
* pipeline — δ deliberately does not invent a second thumbnail cache.
|
||||
* Null for commands that don't carry a screenshot.
|
||||
*/
|
||||
val thumbnailToken: String? = null,
|
||||
/** Unique ID used as the LazyColumn key — lets Compose animate inserts cleanly. */
|
||||
val id: String,
|
||||
)
|
||||
|
||||
@Serializable
|
||||
enum class BridgeActivityStatus { Pending, Success, Failed, Blocked }
|
||||
|
||||
data class BridgeSettings(
|
||||
val masterEnabled: Boolean = false,
|
||||
)
|
||||
|
||||
class BridgePreferencesRepository(private val context: Context) {
|
||||
|
||||
companion object {
|
||||
private val KEY_MASTER_ENABLED = booleanPreferencesKey("bridge_master_enabled")
|
||||
private val KEY_ACTIVITY_LOG = stringPreferencesKey("bridge_activity_log")
|
||||
|
||||
/** Hard cap on persisted entries. See file-level KDoc for rationale. */
|
||||
const val MAX_LOG_ENTRIES = 100
|
||||
const val DEFAULT_MASTER_ENABLED = false
|
||||
}
|
||||
|
||||
// Lenient JSON — ignore unknown keys so we can evolve the schema without
|
||||
// breaking installed users on app upgrade, mirroring how PairingPreferences
|
||||
// and MediaSettings handle forward-compat.
|
||||
private val json = Json {
|
||||
ignoreUnknownKeys = true
|
||||
encodeDefaults = true
|
||||
}
|
||||
|
||||
val settings: Flow<BridgeSettings> = context.relayDataStore.data.map { prefs ->
|
||||
BridgeSettings(
|
||||
masterEnabled = prefs[KEY_MASTER_ENABLED] ?: DEFAULT_MASTER_ENABLED,
|
||||
)
|
||||
}
|
||||
|
||||
val activityLog: Flow<List<BridgeActivityEntry>> = context.relayDataStore.data.map { prefs ->
|
||||
val raw = prefs[KEY_ACTIVITY_LOG] ?: return@map emptyList()
|
||||
runCatching { json.decodeFromString<List<BridgeActivityEntry>>(raw) }
|
||||
.getOrDefault(emptyList())
|
||||
}
|
||||
|
||||
suspend fun setMasterEnabled(enabled: Boolean) {
|
||||
context.relayDataStore.edit { it[KEY_MASTER_ENABLED] = enabled }
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepend a new entry and trim to [MAX_LOG_ENTRIES]. Idempotent on
|
||||
* duplicate ids — if an entry with the same id already exists we replace
|
||||
* it in-place (used when a Pending entry transitions to Success/Failed).
|
||||
*/
|
||||
suspend fun appendEntry(entry: BridgeActivityEntry) {
|
||||
context.relayDataStore.edit { prefs ->
|
||||
val current = prefs[KEY_ACTIVITY_LOG]?.let {
|
||||
runCatching { json.decodeFromString<List<BridgeActivityEntry>>(it) }
|
||||
.getOrDefault(emptyList())
|
||||
} ?: emptyList()
|
||||
val deduped = current.filterNot { it.id == entry.id }
|
||||
val updated = (listOf(entry) + deduped).take(MAX_LOG_ENTRIES)
|
||||
prefs[KEY_ACTIVITY_LOG] = json.encodeToString(updated)
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun clearLog() {
|
||||
context.relayDataStore.edit { prefs ->
|
||||
prefs[KEY_ACTIVITY_LOG] = json.encodeToString(emptyList<BridgeActivityEntry>())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -60,3 +60,44 @@ object FeatureFlags {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile-time gating based on the active Gradle product flavor.
|
||||
*
|
||||
* Phase 3 ships Bridge on two tracks with very different AccessibilityService
|
||||
* scope: the `googlePlay` flavor carries a conservative event-type subset and
|
||||
* a "notifications + confirmations" description for Play Store policy review,
|
||||
* and the `sideload` flavor carries the full agent-control surface. The tier
|
||||
* flags below let UI code hide tier 3/4/6 surfaces on the Play build without
|
||||
* a runtime check — Kotlin's `val … get() = current == SIDELOAD` resolves at
|
||||
* each call site, but because `current` is a compile-time string, R8 is able
|
||||
* to fold the check away in release builds.
|
||||
*
|
||||
* Tier definitions (see `Phase 3 — Bridge Channel.md` in the vault):
|
||||
* 1. baseline — both tracks (app open, tap, navigate within app)
|
||||
* 2. notifications — both tracks (read notifications, summarize, reply)
|
||||
* 3. voice-first — sideload only (always-on voice capture)
|
||||
* 4. vision-first — sideload only (always-on screen reading)
|
||||
* 5. safety rails — both tracks (confirmation dialogs, action log)
|
||||
* 6. ambitious future — sideload only (cross-app macros, scheduling)
|
||||
*/
|
||||
object BuildFlavor {
|
||||
const val GOOGLE_PLAY = "googlePlay"
|
||||
const val SIDELOAD = "sideload"
|
||||
val current: String get() = BuildConfig.FLAVOR
|
||||
|
||||
val bridgeTier1: Boolean = true // baseline — both tracks
|
||||
val bridgeTier2: Boolean = true // notifications, calendar — both tracks
|
||||
val bridgeTier3: Boolean get() = current == SIDELOAD // voice-first
|
||||
val bridgeTier4: Boolean get() = current == SIDELOAD // vision-first
|
||||
val bridgeTier5: Boolean = true // safety rails — always on
|
||||
val bridgeTier6: Boolean get() = current == SIDELOAD // future ambitious
|
||||
|
||||
/** Human-readable badge label for the Settings → About version row. */
|
||||
val displayName: String
|
||||
get() = when (current) {
|
||||
GOOGLE_PLAY -> "Google Play"
|
||||
SIDELOAD -> "Sideload"
|
||||
else -> current.ifBlank { "Unknown" }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,10 +67,21 @@ class ChannelMultiplexer {
|
||||
// TODO: Phase 2 — terminal channel handler
|
||||
handlers["terminal"]?.onMessage(envelope)
|
||||
}
|
||||
"bridge" -> {
|
||||
// TODO: Phase 3 — bridge channel handler
|
||||
handlers["bridge"]?.onMessage(envelope)
|
||||
}
|
||||
// === PHASE3-γ: bridge channel routing ===
|
||||
// bridge.command envelopes come FROM the server and are
|
||||
// dispatched to a [BridgeCommandHandler] which hands them to
|
||||
// the [HermesAccessibilityService]'s [ActionExecutor]. Responses
|
||||
// (bridge.response / bridge.status) flow back through [send]
|
||||
// directly — the handler never consumes its own responses.
|
||||
//
|
||||
// This branch is intentionally symmetric with "chat" and
|
||||
// "terminal": route inbound envelopes to whatever handler is
|
||||
// registered. The handler registration itself happens in
|
||||
// [ConnectionViewModel] so the ViewModel controls whether
|
||||
// bridge routing is active (Bridge can be gated by build
|
||||
// flavor or by the master enable toggle in the UI).
|
||||
"bridge" -> handlers["bridge"]?.onMessage(envelope)
|
||||
// === END PHASE3-γ ===
|
||||
else -> {
|
||||
// Unknown channel — ignore
|
||||
}
|
||||
@@ -91,6 +102,28 @@ class ChannelMultiplexer {
|
||||
sendCallback?.invoke(envelope)
|
||||
}
|
||||
|
||||
// === PHASE3-ε: notification outbound routing ===
|
||||
//
|
||||
// `HermesNotificationCompanion` is a system-bound
|
||||
// `NotificationListenerService` that lives outside the ViewModel
|
||||
// scope. To push posted-notification envelopes onto the WSS
|
||||
// connection, it grabs the live multiplexer reference (set by
|
||||
// `ConnectionViewModel` via the static companion `multiplexer`
|
||||
// slot on the service) and calls [sendNotification].
|
||||
//
|
||||
// This is a thin wrapper over [send] with a no-op fast path when
|
||||
// no send callback is wired yet (relay disconnected). We drop on
|
||||
// the floor at this layer rather than buffering — the service
|
||||
// owns the cold-start buffer in its `pendingEnvelopes` queue, and
|
||||
// dropping when the relay is offline matches the smartwatch
|
||||
// companion semantics (a wearable doesn't replay notifications
|
||||
// it missed while out of range either).
|
||||
fun sendNotification(envelope: Envelope) {
|
||||
val cb = sendCallback ?: return
|
||||
cb.invoke(envelope)
|
||||
}
|
||||
// === END PHASE3-ε ===
|
||||
|
||||
/**
|
||||
* Handle system channel messages (auth, ping/pong).
|
||||
*/
|
||||
|
||||
@@ -0,0 +1,335 @@
|
||||
package com.hermesandroid.relay.network.handlers
|
||||
|
||||
import android.util.Log
|
||||
import com.hermesandroid.relay.accessibility.ActionExecutor
|
||||
import com.hermesandroid.relay.accessibility.HermesAccessibilityService
|
||||
import com.hermesandroid.relay.accessibility.ScreenCapture
|
||||
import com.hermesandroid.relay.accessibility.ScreenReader
|
||||
import com.hermesandroid.relay.network.ChannelMultiplexer
|
||||
import com.hermesandroid.relay.network.models.Envelope
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import kotlinx.serialization.json.jsonPrimitive
|
||||
import kotlinx.serialization.json.put
|
||||
|
||||
/**
|
||||
* Phase 3 — γ `accessibility-runtime`
|
||||
*
|
||||
* Routes inbound `bridge.command` envelopes to [ActionExecutor] and
|
||||
* publishes the result as a `bridge.response` envelope.
|
||||
*
|
||||
* # Wire protocol (frozen — see Phase 3 plan)
|
||||
*
|
||||
* ```json
|
||||
* // server → app
|
||||
* {
|
||||
* "channel": "bridge",
|
||||
* "type": "bridge.command",
|
||||
* "id": "<uuid>",
|
||||
* "payload": {
|
||||
* "request_id": "<uuid>",
|
||||
* "method": "POST", // HTTP-style method, informational
|
||||
* "path": "/tap", // canonical action name
|
||||
* "params": { ... }, // optional query-ish fields
|
||||
* "body": { ... } // optional JSON body
|
||||
* }
|
||||
* }
|
||||
*
|
||||
* // app → server
|
||||
* {
|
||||
* "channel": "bridge",
|
||||
* "type": "bridge.response",
|
||||
* "id": "<uuid>",
|
||||
* "payload": {
|
||||
* "request_id": "<uuid>",
|
||||
* "status": 200, // 200 ok, 400 bad request, 500 executor error
|
||||
* "result": { "ok": true, ... } // action-specific payload
|
||||
* }
|
||||
* }
|
||||
* ```
|
||||
*
|
||||
* For `path` we accept:
|
||||
*
|
||||
* - `/ping` → returns `{pong: true, ts: ...}`
|
||||
* - `/tap` body `{x, y, duration_ms?}`
|
||||
* - `/tap_text` body `{text}`
|
||||
* - `/type` body `{text}`
|
||||
* - `/swipe` body `{start_x, start_y, end_x, end_y, duration_ms?}`
|
||||
* - `/scroll` body `{direction}` — `up`/`down`/`left`/`right`
|
||||
* - `/press_key` body `{key}` — `home`/`back`/`recents`/etc
|
||||
* - `/wait` body `{ms}`
|
||||
* - `/screen` → returns full `ScreenContent` JSON
|
||||
* - `/screenshot` → returns `{media: "MEDIA:hermes-relay://<token>"}`
|
||||
* - `/current_app` → returns `{package: "com.whatever"}`
|
||||
*
|
||||
* # Master enable gate
|
||||
*
|
||||
* Before dispatching any action we check
|
||||
* [HermesAccessibilityService.instance] — if the user hasn't enabled the
|
||||
* service in Android Settings, we fail fast with status 503. If the
|
||||
* service is running but the soft master toggle is off we fail with 403
|
||||
* and a body explaining that Bridge is disabled in the app.
|
||||
*/
|
||||
class BridgeCommandHandler(
|
||||
private val multiplexer: ChannelMultiplexer,
|
||||
private val scope: CoroutineScope,
|
||||
private val screenCapture: ScreenCapture? = null,
|
||||
) {
|
||||
|
||||
companion object {
|
||||
private const val TAG = "BridgeCommandHandler"
|
||||
}
|
||||
|
||||
private val json = Json {
|
||||
ignoreUnknownKeys = true
|
||||
encodeDefaults = true
|
||||
}
|
||||
|
||||
/** Hooked into [ChannelMultiplexer] via `registerHandler("bridge", ::onMessage)`. */
|
||||
fun onMessage(envelope: Envelope) {
|
||||
if (envelope.type != "bridge.command") {
|
||||
// bridge.response is outbound only, bridge.status is outbound
|
||||
// only — anything else is noise we drop silently.
|
||||
Log.v(TAG, "ignoring non-command bridge envelope type='${envelope.type}'")
|
||||
return
|
||||
}
|
||||
|
||||
val requestId = envelope.payload["request_id"]
|
||||
?.jsonPrimitive?.content
|
||||
?: run {
|
||||
Log.w(TAG, "bridge.command missing request_id — dropping")
|
||||
return
|
||||
}
|
||||
|
||||
val path = envelope.payload["path"]?.jsonPrimitive?.content.orEmpty()
|
||||
val body = envelope.payload["body"] as? JsonObject
|
||||
?: envelope.payload["params"] as? JsonObject
|
||||
?: buildJsonObject { }
|
||||
|
||||
// Dispatch on a coroutine so suspend actions (gestures, screenshot)
|
||||
// don't block the multiplexer thread. Every branch must resolve by
|
||||
// calling [respond] exactly once — we leak a request otherwise.
|
||||
scope.launch {
|
||||
try {
|
||||
dispatch(requestId, path, body)
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "bridge command '$path' threw: ${t.message}", t)
|
||||
respond(
|
||||
requestId = requestId,
|
||||
status = 500,
|
||||
result = buildJsonObject {
|
||||
put("error", t.message ?: "unknown executor error")
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun dispatch(requestId: String, path: String, body: JsonObject) {
|
||||
// /ping is the only command that works without the a11y service —
|
||||
// everything else needs the service to be connected.
|
||||
if (path == "/ping") {
|
||||
respond(
|
||||
requestId, 200,
|
||||
buildJsonObject {
|
||||
put("pong", true)
|
||||
put("ts", System.currentTimeMillis())
|
||||
}
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
val service = HermesAccessibilityService.instance
|
||||
?: return respond(
|
||||
requestId, 503,
|
||||
buildJsonObject {
|
||||
put("error", "Hermes AccessibilityService not connected — enable it in Android Settings")
|
||||
}
|
||||
)
|
||||
|
||||
if (!service.isMasterEnabled() && path != "/current_app") {
|
||||
return respond(
|
||||
requestId, 403,
|
||||
buildJsonObject {
|
||||
put("error", "Bridge is disabled — enable Agent Control in the Bridge tab")
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
val executor = service.actionExecutor
|
||||
|
||||
when (path) {
|
||||
"/current_app" -> respond(
|
||||
requestId, 200,
|
||||
buildJsonObject {
|
||||
put("package", service.currentApp ?: "unknown")
|
||||
}
|
||||
)
|
||||
|
||||
"/tap" -> {
|
||||
val x = body["x"]?.jsonPrimitive?.content?.toIntOrNull()
|
||||
val y = body["y"]?.jsonPrimitive?.content?.toIntOrNull()
|
||||
if (x == null || y == null) {
|
||||
respond(
|
||||
requestId, 400,
|
||||
buildJsonObject { put("error", "missing 'x' or 'y' in body") }
|
||||
)
|
||||
return
|
||||
}
|
||||
val duration = body["duration_ms"]?.jsonPrimitive?.content?.toLongOrNull() ?: 100L
|
||||
respondFromResult(requestId, executor.tap(x, y, duration))
|
||||
}
|
||||
|
||||
"/tap_text" -> {
|
||||
val text = body["text"]?.jsonPrimitive?.content.orEmpty()
|
||||
if (text.isBlank()) {
|
||||
respond(
|
||||
requestId, 400,
|
||||
buildJsonObject { put("error", "missing 'text' in body") }
|
||||
)
|
||||
return
|
||||
}
|
||||
respondFromResult(requestId, executor.tapText(text))
|
||||
}
|
||||
|
||||
"/type" -> {
|
||||
val text = body["text"]?.jsonPrimitive?.content.orEmpty()
|
||||
if (text.isEmpty()) {
|
||||
respond(
|
||||
requestId, 400,
|
||||
buildJsonObject { put("error", "missing 'text' in body") }
|
||||
)
|
||||
return
|
||||
}
|
||||
respondFromResult(requestId, executor.typeText(text))
|
||||
}
|
||||
|
||||
"/swipe" -> {
|
||||
val sx = body["start_x"]?.jsonPrimitive?.content?.toIntOrNull()
|
||||
val sy = body["start_y"]?.jsonPrimitive?.content?.toIntOrNull()
|
||||
val ex = body["end_x"]?.jsonPrimitive?.content?.toIntOrNull()
|
||||
val ey = body["end_y"]?.jsonPrimitive?.content?.toIntOrNull()
|
||||
if (sx == null || sy == null || ex == null || ey == null) {
|
||||
respond(
|
||||
requestId, 400,
|
||||
buildJsonObject {
|
||||
put("error", "swipe requires start_x, start_y, end_x, end_y")
|
||||
}
|
||||
)
|
||||
return
|
||||
}
|
||||
val duration = body["duration_ms"]?.jsonPrimitive?.content?.toLongOrNull() ?: 400L
|
||||
respondFromResult(requestId, executor.swipe(sx, sy, ex, ey, duration))
|
||||
}
|
||||
|
||||
"/scroll" -> {
|
||||
val direction = body["direction"]?.jsonPrimitive?.content.orEmpty()
|
||||
respondFromResult(requestId, executor.scroll(direction))
|
||||
}
|
||||
|
||||
"/press_key" -> {
|
||||
val key = body["key"]?.jsonPrimitive?.content.orEmpty()
|
||||
if (key.isBlank()) {
|
||||
respond(
|
||||
requestId, 400,
|
||||
buildJsonObject { put("error", "missing 'key' in body") }
|
||||
)
|
||||
return
|
||||
}
|
||||
respondFromResult(requestId, executor.pressKey(key))
|
||||
}
|
||||
|
||||
"/wait" -> {
|
||||
val ms = body["ms"]?.jsonPrimitive?.content?.toLongOrNull() ?: 0L
|
||||
respondFromResult(requestId, executor.wait(ms))
|
||||
}
|
||||
|
||||
"/screen" -> {
|
||||
val root = service.snapshotRoot()
|
||||
?: return respond(
|
||||
requestId, 500,
|
||||
buildJsonObject { put("error", "no active window available") }
|
||||
)
|
||||
val includeBounds = body["include_bounds"]
|
||||
?.jsonPrimitive?.content?.toBooleanStrictOrNull() ?: true
|
||||
val screen = service.reader.readScreen(root, includeBounds)
|
||||
@Suppress("DEPRECATION")
|
||||
try { root.recycle() } catch (_: Throwable) { }
|
||||
|
||||
val screenJson = json.encodeToJsonElement(
|
||||
ScreenReader.ScreenContent.serializer(),
|
||||
screen
|
||||
).jsonObject
|
||||
respond(requestId, 200, screenJson)
|
||||
}
|
||||
|
||||
"/screenshot" -> {
|
||||
val capture = screenCapture
|
||||
?: return respond(
|
||||
requestId, 503,
|
||||
buildJsonObject {
|
||||
put("error", "ScreenCapture not wired — Bridge UI must enable screenshots first")
|
||||
}
|
||||
)
|
||||
val result = capture.captureAndUpload()
|
||||
if (result.isSuccess) {
|
||||
respond(
|
||||
requestId, 200,
|
||||
buildJsonObject { put("media", result.getOrNull()) }
|
||||
)
|
||||
} else {
|
||||
respond(
|
||||
requestId, 500,
|
||||
buildJsonObject {
|
||||
put("error", result.exceptionOrNull()?.message ?: "screenshot failed")
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
else -> respond(
|
||||
requestId, 404,
|
||||
buildJsonObject { put("error", "unknown bridge path '$path'") }
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private fun respondFromResult(requestId: String, result: ActionExecutor.ActionResult) {
|
||||
val status = if (result.ok) 200 else 400
|
||||
val payload = buildJsonObject {
|
||||
if (result.ok) {
|
||||
for ((k, v) in result.data) {
|
||||
when (v) {
|
||||
null -> { /* skip null values — they carry no wire info */ }
|
||||
is Int -> put(k, v)
|
||||
is Long -> put(k, v)
|
||||
is Boolean -> put(k, v)
|
||||
is String -> put(k, v)
|
||||
else -> put(k, v.toString())
|
||||
}
|
||||
}
|
||||
if (result.data.isEmpty()) put("ok", true)
|
||||
} else {
|
||||
put("error", result.error ?: "unknown error")
|
||||
}
|
||||
}
|
||||
respond(requestId, status, payload)
|
||||
}
|
||||
|
||||
private fun respond(requestId: String, status: Int, result: JsonObject) {
|
||||
val envelope = Envelope(
|
||||
channel = "bridge",
|
||||
type = "bridge.response",
|
||||
payload = buildJsonObject {
|
||||
put("request_id", requestId)
|
||||
put("status", status)
|
||||
put("result", result)
|
||||
}
|
||||
)
|
||||
multiplexer.send(envelope)
|
||||
}
|
||||
}
|
||||
+203
@@ -0,0 +1,203 @@
|
||||
package com.hermesandroid.relay.notifications
|
||||
|
||||
import android.content.ComponentName
|
||||
import android.content.Context
|
||||
import android.provider.Settings
|
||||
import android.service.notification.NotificationListenerService
|
||||
import android.service.notification.StatusBarNotification
|
||||
import android.util.Log
|
||||
import com.hermesandroid.relay.network.ChannelMultiplexer
|
||||
import com.hermesandroid.relay.network.models.Envelope
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.encodeToJsonElement
|
||||
import java.util.concurrent.ConcurrentLinkedQueue
|
||||
|
||||
/**
|
||||
* Opt-in [NotificationListenerService] that forwards posted-notification
|
||||
* metadata to the user's paired Hermes assistant over the existing WSS
|
||||
* connection.
|
||||
*
|
||||
* This is the same Android API used by Wear OS, Android Auto, Tasker,
|
||||
* and every smart-watch companion app — it's part of the public SDK
|
||||
* and the user grants/revokes access via Android's system "Notification
|
||||
* access" page (`Settings.ACTION_NOTIFICATION_LISTENER_SETTINGS`).
|
||||
* The user can revoke at any time and Android shows the running
|
||||
* listener in their system permissions list.
|
||||
*
|
||||
* Wire-up:
|
||||
* 1. The user opens Settings → Notification companion in the app
|
||||
* 2. They tap "Open Android Settings" and toggle Hermes-Relay on
|
||||
* 3. Android binds this service automatically
|
||||
* 4. [onListenerConnected] sets the static [active] reference
|
||||
* 5. [onNotificationPosted] builds an [Envelope] and pushes it to
|
||||
* the [ChannelMultiplexer] (set by `ConnectionViewModel` once
|
||||
* the relay handshake completes)
|
||||
* 6. Multiplexer hands the envelope to ConnectionManager → WSS → relay
|
||||
*
|
||||
* Pattern: a static `companion object` reference to the bound service
|
||||
* instance + a static [multiplexer] slot that the ViewModel injects.
|
||||
* This is the standard Android service-to-app handoff (matches
|
||||
* `HermesAccessibilityService` in agent γ's worktree).
|
||||
*/
|
||||
class HermesNotificationCompanion : NotificationListenerService() {
|
||||
|
||||
/**
|
||||
* Buffer for entries that arrive before [multiplexer] has been
|
||||
* wired up (e.g. notifications during app cold-start). Bounded so
|
||||
* we don't OOM if the multiplexer is never set. Drained on the
|
||||
* next [onNotificationPosted] call once a multiplexer is present.
|
||||
*/
|
||||
private val pendingEnvelopes = ConcurrentLinkedQueue<Envelope>()
|
||||
|
||||
override fun onListenerConnected() {
|
||||
super.onListenerConnected()
|
||||
active = this
|
||||
Log.i(TAG, "NotificationListener bound — companion is live")
|
||||
}
|
||||
|
||||
override fun onListenerDisconnected() {
|
||||
super.onListenerDisconnected()
|
||||
if (active === this) {
|
||||
active = null
|
||||
}
|
||||
Log.i(TAG, "NotificationListener disconnected")
|
||||
}
|
||||
|
||||
override fun onDestroy() {
|
||||
if (active === this) {
|
||||
active = null
|
||||
}
|
||||
super.onDestroy()
|
||||
}
|
||||
|
||||
override fun onNotificationPosted(sbn: StatusBarNotification?) {
|
||||
if (sbn == null) return
|
||||
|
||||
val entry = sbn.toEntry() ?: return
|
||||
val envelope = entry.toEnvelope()
|
||||
|
||||
// Drain any backlog first so order is preserved.
|
||||
val mux = multiplexer
|
||||
if (mux == null) {
|
||||
pendingEnvelopes.offer(envelope)
|
||||
// Cap the buffer at a sensible size — drop oldest on overflow.
|
||||
while (pendingEnvelopes.size > MAX_PENDING) {
|
||||
pendingEnvelopes.poll()
|
||||
}
|
||||
Log.d(
|
||||
TAG,
|
||||
"Buffered notification (no multiplexer yet, pending=${pendingEnvelopes.size})",
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
// Drain pending first (in arrival order) then send the new one.
|
||||
while (true) {
|
||||
val next = pendingEnvelopes.poll() ?: break
|
||||
try {
|
||||
mux.sendNotification(next)
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "Failed to send buffered notification: ${t.message}")
|
||||
}
|
||||
}
|
||||
try {
|
||||
mux.sendNotification(envelope)
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "Failed to send notification: ${t.message}")
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* We don't act on notification removal — the cache on the relay is
|
||||
* append-only with LRU eviction. Could be added later if the LLM
|
||||
* needs to know "this one was dismissed".
|
||||
*/
|
||||
override fun onNotificationRemoved(sbn: StatusBarNotification?) {
|
||||
// Intentionally no-op for now.
|
||||
}
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────
|
||||
|
||||
private fun StatusBarNotification.toEntry(): NotificationEntry? {
|
||||
val n = notification ?: return null
|
||||
val extras = n.extras ?: return null
|
||||
|
||||
val title = extras.getCharSequence(android.app.Notification.EXTRA_TITLE)?.toString()
|
||||
val text = extras.getCharSequence(android.app.Notification.EXTRA_TEXT)?.toString()
|
||||
val sub = extras.getCharSequence(android.app.Notification.EXTRA_SUB_TEXT)?.toString()
|
||||
|
||||
// Skip notifications with no human-readable content — they're
|
||||
// usually background sync placeholders that just confuse the LLM.
|
||||
if (title.isNullOrBlank() && text.isNullOrBlank()) return null
|
||||
|
||||
return NotificationEntry(
|
||||
packageName = packageName,
|
||||
title = title,
|
||||
text = text,
|
||||
subText = sub,
|
||||
postedAt = postTime,
|
||||
key = key,
|
||||
)
|
||||
}
|
||||
|
||||
private fun NotificationEntry.toEnvelope(): Envelope {
|
||||
val payload = JSON.encodeToJsonElement(NotificationEntry.serializer(), this) as JsonObject
|
||||
return Envelope(
|
||||
channel = "notifications",
|
||||
type = "notification.posted",
|
||||
payload = payload,
|
||||
)
|
||||
}
|
||||
|
||||
companion object {
|
||||
private const val TAG = "HermesNotifCompanion"
|
||||
private const val MAX_PENDING = 50
|
||||
|
||||
private val JSON = Json {
|
||||
encodeDefaults = true
|
||||
ignoreUnknownKeys = true
|
||||
}
|
||||
|
||||
/**
|
||||
* The currently bound service instance, or null if the user
|
||||
* has not granted notification access (or has revoked it).
|
||||
* Set in [onListenerConnected]. Use [isAccessGranted] to check
|
||||
* the system permission state without depending on this flag,
|
||||
* since the service may not have bound yet right after the
|
||||
* user grants access.
|
||||
*/
|
||||
@Volatile
|
||||
var active: HermesNotificationCompanion? = null
|
||||
private set
|
||||
|
||||
/**
|
||||
* Multiplexer reference injected by `ConnectionViewModel` once
|
||||
* the relay handshake completes. Service buffers envelopes
|
||||
* until this is set so notifications that arrive during cold
|
||||
* start aren't dropped.
|
||||
*/
|
||||
@Volatile
|
||||
var multiplexer: ChannelMultiplexer? = null
|
||||
|
||||
/**
|
||||
* True if the user has granted notification-access permission
|
||||
* to this app in Android Settings. Cheap synchronous check
|
||||
* against `enabled_notification_listeners` — safe to call from
|
||||
* Compose recomposition.
|
||||
*/
|
||||
fun isAccessGranted(context: Context): Boolean {
|
||||
val pkg = context.packageName
|
||||
val flat = Settings.Secure.getString(
|
||||
context.contentResolver,
|
||||
"enabled_notification_listeners",
|
||||
) ?: return false
|
||||
val expected = ComponentName(
|
||||
pkg,
|
||||
HermesNotificationCompanion::class.java.name,
|
||||
).flattenToString()
|
||||
// The flat string is a colon-separated list of component names.
|
||||
return flat.split(':').any { it.equals(expected, ignoreCase = true) }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package com.hermesandroid.relay.notifications
|
||||
|
||||
import kotlinx.serialization.SerialName
|
||||
import kotlinx.serialization.Serializable
|
||||
|
||||
/**
|
||||
* Wire model for a single notification entry forwarded by
|
||||
* [HermesNotificationCompanion] over the WSS connection.
|
||||
*
|
||||
* Mirrors the Python-side payload shape that
|
||||
* `plugin/relay/channels/notifications.py::NotificationsChannel`
|
||||
* caches in its bounded deque, so the relay can deserialize without
|
||||
* any per-field translation.
|
||||
*
|
||||
* The fields are intentionally minimal — no icon, no big-text expansion,
|
||||
* no actions — because the smartwatch-companion use case is "tell me
|
||||
* what came in" not "let me interact with it from my LLM". Adding
|
||||
* fields later is a non-breaking change as long as Python's deque
|
||||
* stays a `dict[str, Any]`.
|
||||
*/
|
||||
@Serializable
|
||||
data class NotificationEntry(
|
||||
@SerialName("package_name")
|
||||
val packageName: String,
|
||||
val title: String? = null,
|
||||
val text: String? = null,
|
||||
@SerialName("sub_text")
|
||||
val subText: String? = null,
|
||||
@SerialName("posted_at")
|
||||
val postedAt: Long,
|
||||
val key: String,
|
||||
)
|
||||
@@ -72,6 +72,7 @@ import com.hermesandroid.relay.ui.screens.MediaSettingsScreen
|
||||
import com.hermesandroid.relay.ui.screens.PairedDevicesScreen
|
||||
import com.hermesandroid.relay.ui.screens.SettingsScreen
|
||||
import com.hermesandroid.relay.ui.screens.TerminalScreen
|
||||
import com.hermesandroid.relay.ui.screens.NotificationCompanionSettingsScreen
|
||||
import com.hermesandroid.relay.ui.screens.VoiceSettingsScreen
|
||||
import com.hermesandroid.relay.ui.theme.HermesRelayTheme
|
||||
import com.hermesandroid.relay.viewmodel.ChatViewModel
|
||||
@@ -116,6 +117,10 @@ sealed class Screen(
|
||||
// NavigationBar. Paired Devices is opened from Settings → Connection.
|
||||
data object PairedDevices : Screen("paired_devices", "Paired Devices", Icons.Filled.Settings)
|
||||
data object VoiceSettings : Screen("voice_settings", "Voice", Icons.Filled.Settings)
|
||||
// === PHASE3-ε-followup ===
|
||||
data object NotificationCompanionSettings :
|
||||
Screen("settings/notifications", "Notification companion", Icons.Filled.Settings)
|
||||
// === END PHASE3-ε-followup ===
|
||||
// Per-category settings sub-screens — split out of the mega SettingsScreen
|
||||
// following the VoiceSettingsScreen pattern (see DEVLOG 2026-04-11).
|
||||
data object ConnectionSettings : Screen("settings/connection", "Connection", Icons.Filled.Settings)
|
||||
@@ -418,7 +423,16 @@ fun RelayApp() {
|
||||
)
|
||||
}
|
||||
composable(Screen.Bridge.route) {
|
||||
// === PHASE3-δ: BridgeScreen wiring ===
|
||||
// BridgeScreen owns its own BridgeViewModel via the
|
||||
// default `viewModel()` parameter — no shared state with
|
||||
// ChatViewModel / ConnectionViewModel is plumbed through
|
||||
// here yet. Once Agent γ lands HermesAccessibilityService
|
||||
// and we need to observe its runtime state from RelayApp
|
||||
// scope, a shared holder or explicit VM param gets added
|
||||
// here.
|
||||
BridgeScreen()
|
||||
// === END PHASE3-δ ===
|
||||
}
|
||||
composable(Screen.Settings.route) {
|
||||
SettingsScreen(
|
||||
@@ -441,6 +455,9 @@ fun RelayApp() {
|
||||
onNavigateToVoiceSettings = {
|
||||
navController.navigate(Screen.VoiceSettings.route)
|
||||
},
|
||||
onNavigateToNotificationCompanion = {
|
||||
navController.navigate(Screen.NotificationCompanionSettings.route)
|
||||
},
|
||||
onNavigateToPairedDevices = {
|
||||
navController.navigate(Screen.PairedDevices.route)
|
||||
},
|
||||
@@ -459,6 +476,13 @@ fun RelayApp() {
|
||||
onBack = { navController.popBackStack() }
|
||||
)
|
||||
}
|
||||
// === PHASE3-ε-followup: notification companion route ===
|
||||
composable(Screen.NotificationCompanionSettings.route) {
|
||||
NotificationCompanionSettingsScreen(
|
||||
onBack = { navController.popBackStack() }
|
||||
)
|
||||
}
|
||||
// === END PHASE3-ε-followup ===
|
||||
composable(Screen.PairedDevices.route) {
|
||||
PairedDevicesScreen(
|
||||
connectionViewModel = connectionViewModel,
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
package com.hermesandroid.relay.ui.components
|
||||
|
||||
import androidx.compose.animation.AnimatedVisibility
|
||||
import androidx.compose.foundation.clickable
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.heightIn
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.lazy.LazyColumn
|
||||
import androidx.compose.foundation.lazy.items
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.filled.Block
|
||||
import androidx.compose.material.icons.filled.CheckCircle
|
||||
import androidx.compose.material.icons.filled.Error
|
||||
import androidx.compose.material.icons.filled.HourglassEmpty
|
||||
import androidx.compose.material.icons.filled.InsertPhoto
|
||||
import androidx.compose.material.icons.filled.Timeline
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CardDefaults
|
||||
import androidx.compose.material3.HorizontalDivider
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TextButton
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.draw.clip
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.text.font.FontFamily
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.tooling.preview.Preview
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.data.BridgeActivityEntry
|
||||
import com.hermesandroid.relay.data.BridgeActivityStatus
|
||||
import java.time.Instant
|
||||
import java.time.LocalDateTime
|
||||
import java.time.ZoneId
|
||||
import java.time.format.DateTimeFormatter
|
||||
|
||||
/**
|
||||
* Scrollable activity log card — rolling view of the most recent bridge
|
||||
* commands with timestamp, method, status icon, and tap-to-expand detail.
|
||||
*
|
||||
* Phase 3 Wave 1 — δ (`bridge-screen-ui`). The log is populated via
|
||||
* [com.hermesandroid.relay.data.BridgePreferencesRepository.appendEntry];
|
||||
* γ's command dispatcher is the producer once its runtime is wired. Until
|
||||
* then this card shows the "no activity yet" empty state.
|
||||
*
|
||||
* Height-bounded with [heightIn] to keep the log from pushing the safety
|
||||
* card off the fold on phone-portrait screens — the activity log is
|
||||
* internally scrollable within its own [LazyColumn].
|
||||
*/
|
||||
@Composable
|
||||
fun BridgeActivityLog(
|
||||
entries: List<BridgeActivityEntry>,
|
||||
onClear: () -> Unit,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
Card(
|
||||
modifier = modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = CardDefaults.cardColors(
|
||||
containerColor = MaterialTheme.colorScheme.surfaceVariant
|
||||
)
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(8.dp)
|
||||
) {
|
||||
Row(
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
modifier = Modifier.fillMaxWidth()
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.Timeline,
|
||||
contentDescription = null,
|
||||
tint = MaterialTheme.colorScheme.primary,
|
||||
modifier = Modifier.size(18.dp)
|
||||
)
|
||||
Spacer(modifier = Modifier.size(6.dp))
|
||||
Text(
|
||||
text = "Activity Log",
|
||||
style = MaterialTheme.typography.titleSmall,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
color = MaterialTheme.colorScheme.primary,
|
||||
modifier = Modifier.weight(1f)
|
||||
)
|
||||
if (entries.isNotEmpty()) {
|
||||
TextButton(onClick = onClear) { Text("Clear") }
|
||||
}
|
||||
}
|
||||
|
||||
HorizontalDivider(color = MaterialTheme.colorScheme.outline.copy(alpha = 0.15f))
|
||||
|
||||
if (entries.isEmpty()) {
|
||||
Text(
|
||||
text = "No bridge commands yet. Every tap, type, and " +
|
||||
"screenshot the agent performs will show up here.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
} else {
|
||||
// Hard cap the visible height so the log doesn't eat the whole
|
||||
// screen — internally scrollable for history.
|
||||
LazyColumn(
|
||||
modifier = Modifier.heightIn(max = 320.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(4.dp)
|
||||
) {
|
||||
items(entries, key = { it.id }) { entry ->
|
||||
ActivityRow(entry = entry)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun ActivityRow(entry: BridgeActivityEntry) {
|
||||
var expanded by remember(entry.id) { mutableStateOf(false) }
|
||||
|
||||
Column(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.clip(RoundedCornerShape(8.dp))
|
||||
.clickable { expanded = !expanded }
|
||||
.padding(vertical = 6.dp, horizontal = 4.dp),
|
||||
) {
|
||||
Row(
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
modifier = Modifier.fillMaxWidth()
|
||||
) {
|
||||
StatusDot(status = entry.status)
|
||||
Text(
|
||||
text = formatTime(entry.timestampMs),
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
fontFamily = FontFamily.Monospace
|
||||
)
|
||||
Text(
|
||||
text = entry.method,
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
fontWeight = FontWeight.Medium,
|
||||
)
|
||||
Text(
|
||||
text = entry.summary,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
modifier = Modifier.weight(1f)
|
||||
)
|
||||
if (entry.thumbnailToken != null) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.InsertPhoto,
|
||||
contentDescription = "Has screenshot",
|
||||
tint = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
modifier = Modifier.size(14.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
AnimatedVisibility(visible = expanded) {
|
||||
Column(
|
||||
modifier = Modifier.padding(start = 28.dp, top = 6.dp, bottom = 2.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(4.dp)
|
||||
) {
|
||||
Text(
|
||||
text = "Full timestamp: ${formatFullTime(entry.timestampMs)}",
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
fontFamily = FontFamily.Monospace
|
||||
)
|
||||
Text(
|
||||
text = "Status: ${entry.status.name}",
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
if (!entry.resultText.isNullOrBlank()) {
|
||||
Text(
|
||||
text = entry.resultText,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurface
|
||||
)
|
||||
}
|
||||
if (entry.thumbnailToken != null) {
|
||||
// TODO(γ-handoff): wire this to InboundAttachmentCard
|
||||
// once γ's ScreenCapture.kt is uploading real screenshots
|
||||
// to MediaRegistry. Until then we show a placeholder token
|
||||
// label so the expand affordance still communicates the
|
||||
// shape of the future feature.
|
||||
Text(
|
||||
text = "Screenshot token: ${entry.thumbnailToken}",
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
fontFamily = FontFamily.Monospace
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun StatusDot(status: BridgeActivityStatus) {
|
||||
val (icon, tint) = when (status) {
|
||||
BridgeActivityStatus.Pending -> Icons.Filled.HourglassEmpty to
|
||||
MaterialTheme.colorScheme.onSurfaceVariant
|
||||
BridgeActivityStatus.Success -> Icons.Filled.CheckCircle to Color(0xFF4CAF50)
|
||||
BridgeActivityStatus.Failed -> Icons.Filled.Error to
|
||||
MaterialTheme.colorScheme.error
|
||||
BridgeActivityStatus.Blocked -> Icons.Filled.Block to Color(0xFFFFA726)
|
||||
}
|
||||
Icon(
|
||||
imageVector = icon,
|
||||
contentDescription = status.name,
|
||||
tint = tint,
|
||||
modifier = Modifier.size(14.dp)
|
||||
)
|
||||
}
|
||||
|
||||
// ── Formatting helpers ──────────────────────────────────────────────────
|
||||
|
||||
private val TIME_FMT: DateTimeFormatter = DateTimeFormatter.ofPattern("HH:mm:ss")
|
||||
private val FULL_FMT: DateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
|
||||
|
||||
private fun formatTime(epochMs: Long): String = try {
|
||||
LocalDateTime.ofInstant(Instant.ofEpochMilli(epochMs), ZoneId.systemDefault())
|
||||
.format(TIME_FMT)
|
||||
} catch (_: Exception) {
|
||||
"--:--:--"
|
||||
}
|
||||
|
||||
private fun formatFullTime(epochMs: Long): String = try {
|
||||
LocalDateTime.ofInstant(Instant.ofEpochMilli(epochMs), ZoneId.systemDefault())
|
||||
.format(FULL_FMT)
|
||||
} catch (_: Exception) {
|
||||
"—"
|
||||
}
|
||||
|
||||
// ── Previews ────────────────────────────────────────────────────────────
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeActivityLogPreviewFilled() {
|
||||
val now = System.currentTimeMillis()
|
||||
val sample = listOf(
|
||||
BridgeActivityEntry(
|
||||
id = "1",
|
||||
timestampMs = now,
|
||||
method = "tap",
|
||||
summary = "(540, 1200)",
|
||||
status = BridgeActivityStatus.Success,
|
||||
resultText = "Dispatched gesture at (540, 1200)",
|
||||
),
|
||||
BridgeActivityEntry(
|
||||
id = "2",
|
||||
timestampMs = now - 2_000,
|
||||
method = "read_screen",
|
||||
summary = "root → 42 nodes",
|
||||
status = BridgeActivityStatus.Success,
|
||||
thumbnailToken = "hermes-relay://ab12cd34",
|
||||
),
|
||||
BridgeActivityEntry(
|
||||
id = "3",
|
||||
timestampMs = now - 5_500,
|
||||
method = "open_app",
|
||||
summary = "Chrome",
|
||||
status = BridgeActivityStatus.Blocked,
|
||||
resultText = "Blocked by safety rails: chrome is in user blocklist",
|
||||
),
|
||||
BridgeActivityEntry(
|
||||
id = "4",
|
||||
timestampMs = now - 9_000,
|
||||
method = "type",
|
||||
summary = "\"hello\"",
|
||||
status = BridgeActivityStatus.Failed,
|
||||
resultText = "No focused input field",
|
||||
),
|
||||
BridgeActivityEntry(
|
||||
id = "5",
|
||||
timestampMs = now - 12_000,
|
||||
method = "ping",
|
||||
summary = "→ 12ms",
|
||||
status = BridgeActivityStatus.Success,
|
||||
),
|
||||
)
|
||||
MaterialTheme {
|
||||
BridgeActivityLog(
|
||||
entries = sample,
|
||||
onClear = {},
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeActivityLogPreviewEmpty() {
|
||||
MaterialTheme {
|
||||
BridgeActivityLog(
|
||||
entries = emptyList(),
|
||||
onClear = {},
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package com.hermesandroid.relay.ui.components
|
||||
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.filled.BatteryFull
|
||||
import androidx.compose.material.icons.filled.Info
|
||||
import androidx.compose.material.icons.filled.PhoneAndroid
|
||||
import androidx.compose.material.icons.filled.ScreenLockPortrait
|
||||
import androidx.compose.material.icons.filled.Smartphone
|
||||
import androidx.compose.material3.AlertDialog
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CardDefaults
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.IconButton
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Switch
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TextButton
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.tooling.preview.Preview
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.viewmodel.BridgeStatus
|
||||
|
||||
/**
|
||||
* Master "Allow Agent Control" card — the headline of the Bridge tab.
|
||||
*
|
||||
* Phase 3 Wave 1 — δ (`bridge-screen-ui`). Visual style mirrors the status
|
||||
* cards in `PairedDevicesScreen`: surfaceVariant background, 16dp padding,
|
||||
* 10dp row spacing. Uses [ConnectionStatusBadge] for the pulsing status dot
|
||||
* so the Bridge tab looks visually consistent with the Settings → Connection
|
||||
* section.
|
||||
*
|
||||
* The headline switch is `enabled = allowEnable` so users can't flip it on
|
||||
* when the a11y service isn't granted — we instead bounce them to the
|
||||
* permission checklist. Tapping the info icon opens an explanation dialog
|
||||
* (required by Google Play's a11y review process, per the Phase 3 plan's
|
||||
* Play Store Strategy section).
|
||||
*/
|
||||
@Composable
|
||||
fun BridgeMasterToggle(
|
||||
enabled: Boolean,
|
||||
status: BridgeStatus?,
|
||||
accessibilityGranted: Boolean,
|
||||
onToggle: (Boolean) -> Unit,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
var showExplain by remember { mutableStateOf(false) }
|
||||
|
||||
Card(
|
||||
modifier = modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = CardDefaults.cardColors(
|
||||
containerColor = MaterialTheme.colorScheme.surfaceVariant
|
||||
)
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(12.dp)
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
) {
|
||||
Column(modifier = Modifier.weight(1f)) {
|
||||
Text(
|
||||
text = "Agent Control",
|
||||
style = MaterialTheme.typography.titleMedium,
|
||||
color = MaterialTheme.colorScheme.primary,
|
||||
fontWeight = FontWeight.SemiBold
|
||||
)
|
||||
Text(
|
||||
text = if (enabled) "Active — agent can interact with this device"
|
||||
else "Off — agent cannot control this device",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
}
|
||||
IconButton(onClick = { showExplain = true }) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.Info,
|
||||
contentDescription = "What does this do?",
|
||||
tint = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
}
|
||||
Switch(
|
||||
checked = enabled && accessibilityGranted,
|
||||
onCheckedChange = { onToggle(it) },
|
||||
enabled = accessibilityGranted || enabled,
|
||||
)
|
||||
}
|
||||
|
||||
if (!accessibilityGranted) {
|
||||
Text(
|
||||
text = "Grant the Accessibility Service permission below to enable.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.error
|
||||
)
|
||||
}
|
||||
|
||||
if (status != null) {
|
||||
Spacer(modifier = Modifier.height(2.dp))
|
||||
StatusInlineRow(
|
||||
icon = Icons.Filled.PhoneAndroid,
|
||||
label = "Device",
|
||||
value = status.deviceName
|
||||
)
|
||||
StatusInlineRow(
|
||||
icon = Icons.Filled.BatteryFull,
|
||||
label = "Battery",
|
||||
value = status.batteryPercent?.let { "$it%" } ?: "—"
|
||||
)
|
||||
StatusInlineRow(
|
||||
icon = Icons.Filled.ScreenLockPortrait,
|
||||
label = "Screen",
|
||||
value = if (status.screenOn) "ON" else "OFF"
|
||||
)
|
||||
StatusInlineRow(
|
||||
icon = Icons.Filled.Smartphone,
|
||||
label = "Current app",
|
||||
value = status.currentApp ?: "—"
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (showExplain) {
|
||||
AlertDialog(
|
||||
onDismissRequest = { showExplain = false },
|
||||
title = { Text("About Agent Control") },
|
||||
text = {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
Text(
|
||||
"Agent Control lets your Hermes agent read what's on " +
|
||||
"your screen and interact with apps on your behalf " +
|
||||
"(tap, type, scroll, screenshot).",
|
||||
style = MaterialTheme.typography.bodyMedium
|
||||
)
|
||||
Text(
|
||||
"This uses Android's Accessibility Service API, which " +
|
||||
"is the same permission screen readers use. You must " +
|
||||
"enable it in Android Settings before this switch works.",
|
||||
style = MaterialTheme.typography.bodyMedium
|
||||
)
|
||||
Text(
|
||||
"You can turn Agent Control off at any time from this " +
|
||||
"screen or by disabling the service in Android " +
|
||||
"Settings. All bridge commands are logged in the " +
|
||||
"Activity Log below.",
|
||||
style = MaterialTheme.typography.bodyMedium
|
||||
)
|
||||
}
|
||||
},
|
||||
confirmButton = {
|
||||
TextButton(onClick = { showExplain = false }) { Text("Got it") }
|
||||
}
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun StatusInlineRow(
|
||||
icon: androidx.compose.ui.graphics.vector.ImageVector,
|
||||
label: String,
|
||||
value: String,
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = icon,
|
||||
contentDescription = null,
|
||||
tint = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
modifier = Modifier.height(16.dp)
|
||||
)
|
||||
Text(
|
||||
text = label,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
modifier = Modifier.weight(1f)
|
||||
)
|
||||
Text(
|
||||
text = value,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurface,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeMasterTogglePreviewOn() {
|
||||
MaterialTheme {
|
||||
BridgeMasterToggle(
|
||||
enabled = true,
|
||||
status = BridgeStatus(
|
||||
deviceName = "Galaxy S24",
|
||||
batteryPercent = 78,
|
||||
screenOn = true,
|
||||
currentApp = "Chrome",
|
||||
accessibilityEnabled = true,
|
||||
),
|
||||
accessibilityGranted = true,
|
||||
onToggle = {},
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeMasterTogglePreviewBlocked() {
|
||||
MaterialTheme {
|
||||
BridgeMasterToggle(
|
||||
enabled = false,
|
||||
status = BridgeStatus(
|
||||
deviceName = "Pixel 9",
|
||||
batteryPercent = 42,
|
||||
screenOn = true,
|
||||
currentApp = null,
|
||||
accessibilityEnabled = false,
|
||||
),
|
||||
accessibilityGranted = false,
|
||||
onToggle = {},
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
package com.hermesandroid.relay.ui.components
|
||||
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.net.Uri
|
||||
import android.provider.Settings
|
||||
import androidx.compose.foundation.clickable
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight
|
||||
import androidx.compose.material.icons.filled.Accessibility
|
||||
import androidx.compose.material.icons.filled.CheckCircle
|
||||
import androidx.compose.material.icons.filled.Notifications
|
||||
import androidx.compose.material.icons.filled.PictureInPicture
|
||||
import androidx.compose.material.icons.filled.RadioButtonUnchecked
|
||||
import androidx.compose.material.icons.filled.ScreenShare
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CardDefaults
|
||||
import androidx.compose.material3.HorizontalDivider
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.graphics.vector.ImageVector
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.tooling.preview.Preview
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.viewmodel.BridgePermissionStatus
|
||||
|
||||
/**
|
||||
* Permission checklist card — one row per required Android permission.
|
||||
* Tapping a non-granted row fires an Intent to the corresponding Android
|
||||
* Settings screen so the user can grant the permission without leaving
|
||||
* their muscle-memory path.
|
||||
*
|
||||
* Phase 3 Wave 1 — δ (`bridge-screen-ui`). Uses vector Material icons to
|
||||
* stay inside the already-shipped icon set (no dependency on
|
||||
* compose-icons-extended, which has bitten us before — see
|
||||
* `fix(settings): revert ChevronRight…`).
|
||||
*/
|
||||
@Composable
|
||||
fun BridgePermissionChecklist(
|
||||
status: BridgePermissionStatus,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
val context = LocalContext.current
|
||||
|
||||
Card(
|
||||
modifier = modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = CardDefaults.cardColors(
|
||||
containerColor = MaterialTheme.colorScheme.surfaceVariant
|
||||
)
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(6.dp)
|
||||
) {
|
||||
Text(
|
||||
text = "Permissions",
|
||||
style = MaterialTheme.typography.titleSmall,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
color = MaterialTheme.colorScheme.primary,
|
||||
)
|
||||
Text(
|
||||
text = "Tap a row to open Android Settings.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
|
||||
HorizontalDivider(color = MaterialTheme.colorScheme.outline.copy(alpha = 0.15f))
|
||||
|
||||
PermissionRow(
|
||||
icon = Icons.Filled.Accessibility,
|
||||
title = "Accessibility Service",
|
||||
subtitle = "Read screen content, dispatch taps/types",
|
||||
granted = status.accessibilityServiceEnabled,
|
||||
onClick = { openAccessibilitySettings(context) }
|
||||
)
|
||||
PermissionRow(
|
||||
icon = Icons.Filled.ScreenShare,
|
||||
title = "Screen Capture",
|
||||
subtitle = "Take screenshots via MediaProjection (per-session)",
|
||||
granted = status.screenCapturePermitted,
|
||||
// MediaProjection has no direct Settings entry — the consent
|
||||
// dialog fires each time γ's ScreenCapture.kt asks for it.
|
||||
// Tapping this row is informational-only until Tier 1 lands.
|
||||
onClick = null
|
||||
)
|
||||
PermissionRow(
|
||||
icon = Icons.Filled.PictureInPicture,
|
||||
title = "Display over other apps",
|
||||
subtitle = "Status overlay while bridge is active",
|
||||
granted = status.overlayPermitted,
|
||||
onClick = { openOverlaySettings(context) }
|
||||
)
|
||||
PermissionRow(
|
||||
icon = Icons.Filled.Notifications,
|
||||
title = "Notification Listener",
|
||||
subtitle = "Read notifications for agent summaries",
|
||||
granted = status.notificationListenerPermitted,
|
||||
onClick = { openNotificationListenerSettings(context) }
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun PermissionRow(
|
||||
icon: ImageVector,
|
||||
title: String,
|
||||
subtitle: String,
|
||||
granted: Boolean,
|
||||
onClick: (() -> Unit)?,
|
||||
) {
|
||||
val rowModifier = if (onClick != null) {
|
||||
Modifier
|
||||
.fillMaxWidth()
|
||||
.clickable(onClick = onClick)
|
||||
.padding(vertical = 10.dp)
|
||||
} else {
|
||||
Modifier
|
||||
.fillMaxWidth()
|
||||
.padding(vertical = 10.dp)
|
||||
}
|
||||
Row(
|
||||
modifier = rowModifier,
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = icon,
|
||||
contentDescription = null,
|
||||
tint = MaterialTheme.colorScheme.primary,
|
||||
modifier = Modifier.size(22.dp),
|
||||
)
|
||||
Column(modifier = Modifier.weight(1f)) {
|
||||
Text(
|
||||
text = title,
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
)
|
||||
Text(
|
||||
text = subtitle,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
// Status icon: green check when granted, red empty circle otherwise.
|
||||
Icon(
|
||||
imageVector = if (granted) Icons.Filled.CheckCircle
|
||||
else Icons.Filled.RadioButtonUnchecked,
|
||||
contentDescription = if (granted) "Granted" else "Not granted",
|
||||
tint = if (granted) Color(0xFF4CAF50) else MaterialTheme.colorScheme.error,
|
||||
modifier = Modifier.size(22.dp),
|
||||
)
|
||||
if (onClick != null) {
|
||||
Icon(
|
||||
imageVector = Icons.AutoMirrored.Filled.KeyboardArrowRight,
|
||||
contentDescription = null,
|
||||
tint = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
modifier = Modifier.size(18.dp),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Intent helpers ──────────────────────────────────────────────────────
|
||||
//
|
||||
// All three intent launchers guard with runCatching because Samsung / Xiaomi
|
||||
// OEM skins occasionally ship without the standard ACTION_* constants, and
|
||||
// we'd rather degrade to a no-op than crash the Bridge screen.
|
||||
|
||||
private fun openAccessibilitySettings(context: Context) {
|
||||
runCatching {
|
||||
val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS).apply {
|
||||
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
|
||||
}
|
||||
context.startActivity(intent)
|
||||
}
|
||||
}
|
||||
|
||||
private fun openOverlaySettings(context: Context) {
|
||||
runCatching {
|
||||
val intent = Intent(
|
||||
Settings.ACTION_MANAGE_OVERLAY_PERMISSION,
|
||||
Uri.parse("package:${context.packageName}")
|
||||
).apply {
|
||||
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
|
||||
}
|
||||
context.startActivity(intent)
|
||||
}
|
||||
}
|
||||
|
||||
private fun openNotificationListenerSettings(context: Context) {
|
||||
runCatching {
|
||||
val intent = Intent("android.settings.ACTION_NOTIFICATION_LISTENER_SETTINGS").apply {
|
||||
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
|
||||
}
|
||||
context.startActivity(intent)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgePermissionChecklistPreviewAllGranted() {
|
||||
MaterialTheme {
|
||||
BridgePermissionChecklist(
|
||||
status = BridgePermissionStatus(
|
||||
accessibilityServiceEnabled = true,
|
||||
screenCapturePermitted = true,
|
||||
overlayPermitted = true,
|
||||
notificationListenerPermitted = true,
|
||||
),
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgePermissionChecklistPreviewNoneGranted() {
|
||||
MaterialTheme {
|
||||
BridgePermissionChecklist(
|
||||
status = BridgePermissionStatus(),
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package com.hermesandroid.relay.ui.components
|
||||
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CardDefaults
|
||||
import androidx.compose.material3.HorizontalDivider
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.tooling.preview.Preview
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.viewmodel.BridgeStatus
|
||||
|
||||
/**
|
||||
* Standalone status-only card — can render with or without [BridgeMasterToggle]
|
||||
* above it. Used by `BridgeScreen` as a secondary "connection state at a
|
||||
* glance" block when there's enough info to show but the user hasn't touched
|
||||
* the master toggle yet.
|
||||
*
|
||||
* Phase 3 Wave 1 — δ (`bridge-screen-ui`). Kept distinct from
|
||||
* [BridgeMasterToggle] so that Agent ζ in Wave 2 can relocate the master
|
||||
* toggle without losing the status surface (and so we can reuse this card
|
||||
* in the Settings → Connection section later if desired).
|
||||
*/
|
||||
@Composable
|
||||
fun BridgeStatusCard(
|
||||
status: BridgeStatus?,
|
||||
isConnected: Boolean,
|
||||
modifier: Modifier = Modifier,
|
||||
) {
|
||||
Card(
|
||||
modifier = modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = CardDefaults.cardColors(
|
||||
containerColor = MaterialTheme.colorScheme.surfaceVariant
|
||||
)
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(8.dp)
|
||||
) {
|
||||
Row(
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
modifier = Modifier.fillMaxWidth()
|
||||
) {
|
||||
Text(
|
||||
text = "Status",
|
||||
style = MaterialTheme.typography.titleSmall,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
color = MaterialTheme.colorScheme.primary,
|
||||
modifier = Modifier.weight(1f)
|
||||
)
|
||||
ConnectionStatusBadge(
|
||||
isConnected = isConnected,
|
||||
isConnecting = false,
|
||||
)
|
||||
Text(
|
||||
text = if (isConnected) "Connected" else "Disconnected",
|
||||
style = MaterialTheme.typography.labelMedium,
|
||||
color = if (isConnected) MaterialTheme.colorScheme.onSurface
|
||||
else MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
|
||||
HorizontalDivider(color = MaterialTheme.colorScheme.outline.copy(alpha = 0.15f))
|
||||
|
||||
if (status == null) {
|
||||
Text(
|
||||
text = "Bridge runtime not yet reporting status. Enable " +
|
||||
"Agent Control above to begin receiving device telemetry.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
} else {
|
||||
StatusKeyValue("Device", status.deviceName)
|
||||
StatusKeyValue(
|
||||
"Battery",
|
||||
status.batteryPercent?.let { "$it%" } ?: "Unknown"
|
||||
)
|
||||
StatusKeyValue("Screen", if (status.screenOn) "ON" else "OFF")
|
||||
StatusKeyValue("Current app", status.currentApp ?: "—")
|
||||
StatusKeyValue(
|
||||
"Accessibility service",
|
||||
if (status.accessibilityEnabled) "Enabled" else "Disabled"
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Composable
|
||||
private fun StatusKeyValue(key: String, value: String) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalArrangement = Arrangement.SpaceBetween
|
||||
) {
|
||||
Text(
|
||||
text = key,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
Text(
|
||||
text = value,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurface
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeStatusCardPreviewConnected() {
|
||||
MaterialTheme {
|
||||
BridgeStatusCard(
|
||||
status = BridgeStatus(
|
||||
deviceName = "Galaxy S24",
|
||||
batteryPercent = 78,
|
||||
screenOn = true,
|
||||
currentApp = "Chrome",
|
||||
accessibilityEnabled = true,
|
||||
),
|
||||
isConnected = true,
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeStatusCardPreviewEmpty() {
|
||||
MaterialTheme {
|
||||
BridgeStatusCard(
|
||||
status = null,
|
||||
isConnected = false,
|
||||
modifier = Modifier.padding(16.dp)
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -54,6 +54,7 @@ import androidx.compose.ui.res.painterResource
|
||||
import androidx.compose.ui.text.style.TextAlign
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.R
|
||||
import com.hermesandroid.relay.data.BuildFlavor
|
||||
import com.hermesandroid.relay.data.FeatureFlags
|
||||
import com.hermesandroid.relay.ui.components.WhatsNewDialog
|
||||
import com.hermesandroid.relay.ui.theme.gradientBorder
|
||||
@@ -212,6 +213,28 @@ fun AboutScreen(
|
||||
)
|
||||
}
|
||||
|
||||
// === PHASE3-β: build flavor badge ===
|
||||
// Surfaces which release track the user is running. Tier 3/4/6
|
||||
// Bridge surfaces differ between googlePlay and sideload, so it
|
||||
// helps bug reports to see the active flavor right next to the
|
||||
// version. Small, unobtrusive — shares the same row style as
|
||||
// the Version label above.
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalArrangement = Arrangement.SpaceBetween
|
||||
) {
|
||||
Text(
|
||||
text = "Track",
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
Text(
|
||||
text = BuildFlavor.displayName,
|
||||
style = MaterialTheme.typography.bodyMedium
|
||||
)
|
||||
}
|
||||
// === END PHASE3-β ===
|
||||
|
||||
HorizontalDivider()
|
||||
|
||||
// Links
|
||||
|
||||
@@ -1,129 +1,204 @@
|
||||
package com.hermesandroid.relay.ui.screens
|
||||
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.aspectRatio
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.layout.widthIn
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.foundation.verticalScroll
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.filled.Schedule
|
||||
import androidx.compose.material3.AssistChip
|
||||
import androidx.compose.material.icons.filled.Security
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CardDefaults
|
||||
import androidx.compose.material3.ExperimentalMaterial3Api
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Scaffold
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TopAppBar
|
||||
import androidx.compose.material3.TopAppBarDefaults
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.DisposableEffect
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.text.style.TextAlign
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.tooling.preview.Preview
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.hermesandroid.relay.ui.components.MorphingSphere
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.LifecycleEventObserver
|
||||
import androidx.lifecycle.compose.LocalLifecycleOwner
|
||||
import androidx.lifecycle.viewmodel.compose.viewModel
|
||||
import com.hermesandroid.relay.ui.components.BridgeActivityLog
|
||||
import com.hermesandroid.relay.ui.components.BridgeMasterToggle
|
||||
import com.hermesandroid.relay.ui.components.BridgePermissionChecklist
|
||||
import com.hermesandroid.relay.ui.components.BridgeStatusCard
|
||||
import com.hermesandroid.relay.viewmodel.BridgeViewModel
|
||||
|
||||
/**
|
||||
* Bridge tab — phase 3 Wave 1 rewrite (Agent δ, `bridge-screen-ui`).
|
||||
*
|
||||
* Replaces the Phase 0 "Coming Soon" placeholder with the real control
|
||||
* surface described in `Plans/Phase 3 — Bridge Channel.md` §5. Four stacked
|
||||
* cards in a verticalScroll column:
|
||||
*
|
||||
* 1. [BridgeMasterToggle] — "Allow Agent Control" + live status
|
||||
* 2. [BridgePermissionChecklist] — accessibility / capture / overlay / notif
|
||||
* 3. [BridgeActivityLog] — scrollable recent-command history
|
||||
* 4. Safety placeholder — stub owned by Agent ζ in Wave 2
|
||||
*
|
||||
* State comes from [BridgeViewModel] which in turn reads from the
|
||||
* [com.hermesandroid.relay.data.BridgePreferencesRepository] DataStore for
|
||||
* anything persistent, and stubs the live bridge-runtime state until
|
||||
* Agent γ's `HermesAccessibilityService` exposes it. See [BridgeViewModel]'s
|
||||
* KDoc for the exact γ-handoff surface.
|
||||
*
|
||||
* Lifecycle: we re-probe permission status on every ON_RESUME so that
|
||||
* returning from Android Settings immediately flips the accessibility
|
||||
* checklist row from red to green without needing to navigate away and back.
|
||||
*/
|
||||
@OptIn(ExperimentalMaterial3Api::class)
|
||||
@Composable
|
||||
fun BridgeScreen() {
|
||||
Column(modifier = Modifier.fillMaxSize()) {
|
||||
TopAppBar(
|
||||
title = { Text("Bridge") },
|
||||
colors = TopAppBarDefaults.topAppBarColors(
|
||||
containerColor = MaterialTheme.colorScheme.surface
|
||||
)
|
||||
)
|
||||
fun BridgeScreen(
|
||||
viewModel: BridgeViewModel = viewModel(),
|
||||
) {
|
||||
val masterToggle by viewModel.masterToggle.collectAsState()
|
||||
val permissionStatus by viewModel.permissionStatus.collectAsState()
|
||||
val bridgeStatus by viewModel.bridgeStatus.collectAsState()
|
||||
val activityLog by viewModel.activityLog.collectAsState()
|
||||
|
||||
Box(
|
||||
// Re-run permission + system-status probes whenever the screen resumes.
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
DisposableEffect(lifecycleOwner) {
|
||||
val observer = LifecycleEventObserver { _, event ->
|
||||
if (event == Lifecycle.Event.ON_RESUME) {
|
||||
viewModel.onScreenResumed()
|
||||
}
|
||||
}
|
||||
lifecycleOwner.lifecycle.addObserver(observer)
|
||||
onDispose { lifecycleOwner.lifecycle.removeObserver(observer) }
|
||||
}
|
||||
|
||||
Scaffold(
|
||||
topBar = {
|
||||
TopAppBar(
|
||||
title = { Text("Bridge") },
|
||||
colors = TopAppBarDefaults.topAppBarColors(
|
||||
containerColor = MaterialTheme.colorScheme.surface
|
||||
)
|
||||
)
|
||||
}
|
||||
) { innerPadding ->
|
||||
Column(
|
||||
modifier = Modifier
|
||||
.fillMaxSize()
|
||||
.padding(horizontal = 32.dp),
|
||||
contentAlignment = Alignment.Center
|
||||
.padding(innerPadding)
|
||||
.verticalScroll(rememberScrollState())
|
||||
.padding(horizontal = 16.dp, vertical = 16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.widthIn(max = 300.dp),
|
||||
horizontalAlignment = Alignment.CenterHorizontally,
|
||||
verticalArrangement = Arrangement.Center
|
||||
) {
|
||||
// Sphere replaces static icon
|
||||
Box(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.aspectRatio(1.2f)
|
||||
) {
|
||||
MorphingSphere(modifier = Modifier.fillMaxSize())
|
||||
}
|
||||
BridgeMasterToggle(
|
||||
enabled = masterToggle,
|
||||
status = bridgeStatus,
|
||||
accessibilityGranted = permissionStatus.accessibilityServiceEnabled,
|
||||
onToggle = { viewModel.setMasterEnabled(it) }
|
||||
)
|
||||
|
||||
Spacer(modifier = Modifier.height(16.dp))
|
||||
BridgeStatusCard(
|
||||
status = bridgeStatus,
|
||||
// TODO(γ-handoff): once γ exposes a `bridgeConnected`
|
||||
// StateFlow from HermesAccessibilityService, drive this off
|
||||
// that instead of the a11y-granted flag.
|
||||
isConnected = permissionStatus.accessibilityServiceEnabled && masterToggle,
|
||||
)
|
||||
|
||||
Text(
|
||||
text = "Device Bridge",
|
||||
style = MaterialTheme.typography.headlineSmall,
|
||||
color = MaterialTheme.colorScheme.onSurface
|
||||
)
|
||||
BridgePermissionChecklist(status = permissionStatus)
|
||||
|
||||
Spacer(modifier = Modifier.height(8.dp))
|
||||
BridgeActivityLog(
|
||||
entries = activityLog,
|
||||
onClear = { viewModel.clearActivityLog() }
|
||||
)
|
||||
|
||||
Text(
|
||||
text = "Let your Hermes agent interact with your phone \u2014 " +
|
||||
"tap, type, read the screen, and automate workflows.",
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
textAlign = TextAlign.Center
|
||||
)
|
||||
SafetyPlaceholderCard()
|
||||
|
||||
Spacer(modifier = Modifier.height(16.dp))
|
||||
|
||||
AssistChip(
|
||||
onClick = { },
|
||||
label = {
|
||||
Text(
|
||||
text = "Coming Soon",
|
||||
style = MaterialTheme.typography.labelSmall
|
||||
)
|
||||
},
|
||||
leadingIcon = {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.Schedule,
|
||||
contentDescription = null,
|
||||
modifier = Modifier.size(16.dp)
|
||||
)
|
||||
}
|
||||
)
|
||||
|
||||
Spacer(modifier = Modifier.height(20.dp))
|
||||
|
||||
val plannedFeatures = listOf(
|
||||
"Agent-controlled device interaction",
|
||||
"Accessibility service integration",
|
||||
"Activity log and command history",
|
||||
"Permission management"
|
||||
)
|
||||
Column(
|
||||
verticalArrangement = Arrangement.spacedBy(4.dp)
|
||||
) {
|
||||
plannedFeatures.forEach { feature ->
|
||||
Text(
|
||||
text = "\u2022 $feature",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
Spacer(modifier = Modifier.height(16.dp))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true)
|
||||
/**
|
||||
* Stub safety card — Agent ζ (Wave 2) owns this content. Renders an inert
|
||||
* card with a "Configure in Bridge Safety Settings" label so the layout
|
||||
* doesn't reflow when ζ's real UI drops in.
|
||||
*
|
||||
* TODO(ζ-handoff): replace with the real BridgeSafetySettings entry-point
|
||||
* card once the blocklist / destructive-verb confirm / auto-disable UIs
|
||||
* are ready. Expected call: `BridgeSafetyCard(onClick = { navigateToSafety() })`.
|
||||
*/
|
||||
@Composable
|
||||
private fun SafetyPlaceholderCard() {
|
||||
Card(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(14.dp),
|
||||
colors = CardDefaults.cardColors(
|
||||
containerColor = MaterialTheme.colorScheme.surfaceVariant
|
||||
)
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(6.dp)
|
||||
) {
|
||||
Row(
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(8.dp),
|
||||
modifier = Modifier.fillMaxWidth()
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.Security,
|
||||
contentDescription = null,
|
||||
tint = MaterialTheme.colorScheme.primary,
|
||||
)
|
||||
Text(
|
||||
text = "Safety",
|
||||
style = MaterialTheme.typography.titleSmall,
|
||||
fontWeight = FontWeight.SemiBold,
|
||||
color = MaterialTheme.colorScheme.primary,
|
||||
)
|
||||
}
|
||||
Text(
|
||||
text = "Configure in Bridge Safety Settings",
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
Text(
|
||||
text = "Per-app blocklist, destructive-verb confirmation, " +
|
||||
"auto-disable timer — coming in Phase 3 Wave 2.",
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Preview(showBackground = true, backgroundColor = 0xFF1A1A2E)
|
||||
@Composable
|
||||
private fun BridgeScreenPreview() {
|
||||
// Previews can't construct a real AndroidViewModel without an Application
|
||||
// context, so we render the inert SafetyPlaceholderCard on its own to at
|
||||
// least verify the spacing / typography. Individual components have their
|
||||
// own full previews.
|
||||
MaterialTheme {
|
||||
BridgeScreen()
|
||||
Column(
|
||||
modifier = Modifier.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(12.dp)
|
||||
) {
|
||||
SafetyPlaceholderCard()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+363
@@ -0,0 +1,363 @@
|
||||
package com.hermesandroid.relay.ui.screens
|
||||
|
||||
import android.content.Intent
|
||||
import android.provider.Settings
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.layout.Arrangement
|
||||
import androidx.compose.foundation.layout.Box
|
||||
import androidx.compose.foundation.layout.Column
|
||||
import androidx.compose.foundation.layout.Row
|
||||
import androidx.compose.foundation.layout.Spacer
|
||||
import androidx.compose.foundation.layout.fillMaxSize
|
||||
import androidx.compose.foundation.layout.fillMaxWidth
|
||||
import androidx.compose.foundation.layout.height
|
||||
import androidx.compose.foundation.layout.padding
|
||||
import androidx.compose.foundation.layout.size
|
||||
import androidx.compose.foundation.rememberScrollState
|
||||
import androidx.compose.foundation.shape.CircleShape
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.foundation.verticalScroll
|
||||
import androidx.compose.material.icons.Icons
|
||||
import androidx.compose.material.icons.automirrored.filled.ArrowBack
|
||||
import androidx.compose.material.icons.filled.Notifications
|
||||
import androidx.compose.material.icons.filled.OpenInNew
|
||||
import androidx.compose.material.icons.filled.Refresh
|
||||
import androidx.compose.material3.Button
|
||||
import androidx.compose.material3.Card
|
||||
import androidx.compose.material3.CardDefaults
|
||||
import androidx.compose.material3.ExperimentalMaterial3Api
|
||||
import androidx.compose.material3.FilledTonalButton
|
||||
import androidx.compose.material3.HorizontalDivider
|
||||
import androidx.compose.material3.Icon
|
||||
import androidx.compose.material3.IconButton
|
||||
import androidx.compose.material3.MaterialTheme
|
||||
import androidx.compose.material3.Scaffold
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TopAppBar
|
||||
import androidx.compose.material3.TopAppBarDefaults
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.DisposableEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.draw.clip
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.platform.LocalContext
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.LifecycleEventObserver
|
||||
import androidx.lifecycle.compose.LocalLifecycleOwner
|
||||
import com.hermesandroid.relay.notifications.HermesNotificationCompanion
|
||||
import java.text.DateFormat
|
||||
import java.util.Date
|
||||
|
||||
/**
|
||||
* Notification companion settings screen — opt-in helper that lets the
|
||||
* user's Hermes assistant read notifications they've explicitly granted
|
||||
* access to.
|
||||
*
|
||||
* Sections:
|
||||
* 1. Status — granted or not granted (live, observed via lifecycle)
|
||||
* 2. Open Android Settings — fires `ACTION_NOTIFICATION_LISTENER_SETTINGS`
|
||||
* 3. About — explains what the feature does and how to revoke
|
||||
*
|
||||
* Mirrors the layout/style of [VoiceSettingsScreen] for consistency.
|
||||
*/
|
||||
@OptIn(ExperimentalMaterial3Api::class)
|
||||
@Composable
|
||||
fun NotificationCompanionSettingsScreen(
|
||||
onBack: () -> Unit,
|
||||
) {
|
||||
val context = LocalContext.current
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
|
||||
// Track grant state. Re-check on every ON_RESUME so the screen
|
||||
// updates immediately when the user comes back from Android
|
||||
// Settings after toggling the listener.
|
||||
var granted by remember {
|
||||
mutableStateOf(HermesNotificationCompanion.isAccessGranted(context))
|
||||
}
|
||||
|
||||
DisposableEffect(lifecycleOwner) {
|
||||
val observer = LifecycleEventObserver { _, event ->
|
||||
if (event == Lifecycle.Event.ON_RESUME) {
|
||||
granted = HermesNotificationCompanion.isAccessGranted(context)
|
||||
}
|
||||
}
|
||||
lifecycleOwner.lifecycle.addObserver(observer)
|
||||
onDispose { lifecycleOwner.lifecycle.removeObserver(observer) }
|
||||
}
|
||||
|
||||
Scaffold(
|
||||
topBar = {
|
||||
TopAppBar(
|
||||
title = { Text("Notification companion") },
|
||||
navigationIcon = {
|
||||
IconButton(onClick = onBack) {
|
||||
Icon(
|
||||
imageVector = Icons.AutoMirrored.Filled.ArrowBack,
|
||||
contentDescription = "Back",
|
||||
)
|
||||
}
|
||||
},
|
||||
colors = TopAppBarDefaults.topAppBarColors(
|
||||
containerColor = MaterialTheme.colorScheme.surface,
|
||||
),
|
||||
)
|
||||
},
|
||||
) { innerPadding ->
|
||||
Column(
|
||||
modifier = Modifier
|
||||
.fillMaxSize()
|
||||
.padding(innerPadding)
|
||||
.verticalScroll(rememberScrollState())
|
||||
.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(16.dp),
|
||||
) {
|
||||
|
||||
// --- About ---
|
||||
NotifSectionCard(title = "About") {
|
||||
Text(
|
||||
text = (
|
||||
"Lets your Hermes assistant help you triage " +
|
||||
"notifications. When enabled, your phone " +
|
||||
"forwards each notification's app, title, " +
|
||||
"and text to your paired Hermes server " +
|
||||
"over the same secure connection chat uses."
|
||||
),
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
)
|
||||
Spacer(Modifier.height(8.dp))
|
||||
Text(
|
||||
text = (
|
||||
"Requires Android's notification access " +
|
||||
"permission. You can grant or revoke it " +
|
||||
"at any time in Android Settings. This is " +
|
||||
"the same permission Wear OS, Android " +
|
||||
"Auto, and Tasker use."
|
||||
),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
|
||||
// --- Status ---
|
||||
NotifSectionCard(title = "Status") {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.spacedBy(12.dp),
|
||||
) {
|
||||
Box(
|
||||
modifier = Modifier
|
||||
.size(12.dp)
|
||||
.clip(CircleShape)
|
||||
.background(
|
||||
if (granted) {
|
||||
Color(0xFF43A047) // green
|
||||
} else {
|
||||
MaterialTheme.colorScheme.outline
|
||||
},
|
||||
),
|
||||
)
|
||||
Column(modifier = Modifier.weight(1f)) {
|
||||
Text(
|
||||
text = if (granted) "Access granted" else "Access not granted",
|
||||
style = MaterialTheme.typography.bodyLarge,
|
||||
)
|
||||
Text(
|
||||
text = if (granted) {
|
||||
"Hermes-Relay can read posted notifications"
|
||||
} else {
|
||||
"Tap below to enable in Android Settings"
|
||||
},
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
IconButton(onClick = {
|
||||
granted = HermesNotificationCompanion.isAccessGranted(context)
|
||||
}) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.Refresh,
|
||||
contentDescription = "Refresh status",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
HorizontalDivider(modifier = Modifier.padding(vertical = 12.dp))
|
||||
|
||||
Button(
|
||||
onClick = {
|
||||
val intent = Intent(Settings.ACTION_NOTIFICATION_LISTENER_SETTINGS)
|
||||
intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
|
||||
runCatching { context.startActivity(intent) }
|
||||
},
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.OpenInNew,
|
||||
contentDescription = null,
|
||||
)
|
||||
Spacer(Modifier.size(8.dp))
|
||||
Text(
|
||||
if (granted) "Manage in Android Settings" else "Open Android Settings",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Test (last received) ---
|
||||
NotifSectionCard(title = "Test") {
|
||||
Text(
|
||||
text = (
|
||||
"Tap below to fetch the last few notifications " +
|
||||
"the listener has captured this session. " +
|
||||
"Helpful for verifying the connection is " +
|
||||
"working end-to-end."
|
||||
),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
Spacer(Modifier.height(8.dp))
|
||||
|
||||
var lastSnapshot by remember {
|
||||
mutableStateOf<List<TestNotificationLine>>(emptyList())
|
||||
}
|
||||
var lastError by remember { mutableStateOf<String?>(null) }
|
||||
|
||||
FilledTonalButton(
|
||||
onClick = {
|
||||
// Pull from the live service if it's bound. We
|
||||
// don't have a server-round-trip helper here on
|
||||
// purpose — that would require sending a relay
|
||||
// round-trip and we want this screen to work
|
||||
// even when the relay is unreachable.
|
||||
val service = HermesNotificationCompanion.active
|
||||
if (service == null) {
|
||||
lastSnapshot = emptyList()
|
||||
lastError = if (granted) {
|
||||
"Listener has not bound yet. Try " +
|
||||
"posting a test notification " +
|
||||
"(any message) and re-tap."
|
||||
} else {
|
||||
"Notification access is not granted."
|
||||
}
|
||||
} else {
|
||||
val active = service.activeNotifications
|
||||
lastError = null
|
||||
lastSnapshot = active.orEmpty()
|
||||
.sortedByDescending { it.postTime }
|
||||
.take(5)
|
||||
.map {
|
||||
TestNotificationLine(
|
||||
pkg = it.packageName,
|
||||
title = it.notification?.extras
|
||||
?.getCharSequence(android.app.Notification.EXTRA_TITLE)
|
||||
?.toString(),
|
||||
text = it.notification?.extras
|
||||
?.getCharSequence(android.app.Notification.EXTRA_TEXT)
|
||||
?.toString(),
|
||||
postedAt = it.postTime,
|
||||
)
|
||||
}
|
||||
}
|
||||
},
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
) {
|
||||
Icon(
|
||||
imageVector = Icons.Filled.Notifications,
|
||||
contentDescription = null,
|
||||
)
|
||||
Spacer(Modifier.size(8.dp))
|
||||
Text("Fetch recent")
|
||||
}
|
||||
|
||||
lastError?.let { err ->
|
||||
Spacer(Modifier.height(8.dp))
|
||||
Text(
|
||||
text = err,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.error,
|
||||
)
|
||||
}
|
||||
|
||||
if (lastSnapshot.isNotEmpty()) {
|
||||
Spacer(Modifier.height(8.dp))
|
||||
val df = remember { DateFormat.getTimeInstance(DateFormat.SHORT) }
|
||||
lastSnapshot.forEach { line ->
|
||||
Column(modifier = Modifier.padding(vertical = 4.dp)) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
horizontalArrangement = Arrangement.SpaceBetween,
|
||||
) {
|
||||
Text(
|
||||
text = line.title ?: "(no title)",
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
)
|
||||
Text(
|
||||
text = df.format(Date(line.postedAt)),
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
Text(
|
||||
text = line.pkg,
|
||||
style = MaterialTheme.typography.labelSmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
line.text?.let {
|
||||
Text(
|
||||
text = it,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
)
|
||||
}
|
||||
}
|
||||
HorizontalDivider()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Local tiny model for the screen's "Test" preview list — keeps the
|
||||
// composable scope contained without adding to the wire models file.
|
||||
private data class TestNotificationLine(
|
||||
val pkg: String,
|
||||
val title: String?,
|
||||
val text: String?,
|
||||
val postedAt: Long,
|
||||
)
|
||||
|
||||
@Composable
|
||||
private fun NotifSectionCard(
|
||||
title: String,
|
||||
content: @Composable () -> Unit,
|
||||
) {
|
||||
Card(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
shape = RoundedCornerShape(16.dp),
|
||||
colors = CardDefaults.cardColors(
|
||||
containerColor = MaterialTheme.colorScheme.surfaceVariant.copy(alpha = 0.3f),
|
||||
),
|
||||
) {
|
||||
Column(
|
||||
modifier = Modifier
|
||||
.fillMaxWidth()
|
||||
.padding(16.dp),
|
||||
verticalArrangement = Arrangement.spacedBy(4.dp),
|
||||
) {
|
||||
Text(
|
||||
text = title,
|
||||
style = MaterialTheme.typography.titleMedium,
|
||||
color = MaterialTheme.colorScheme.primary,
|
||||
)
|
||||
Spacer(Modifier.height(8.dp))
|
||||
content()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -23,6 +23,7 @@ import androidx.compose.material.icons.filled.Code
|
||||
import androidx.compose.material.icons.filled.Devices
|
||||
import androidx.compose.material.icons.filled.GraphicEq
|
||||
import androidx.compose.material.icons.filled.Image
|
||||
import androidx.compose.material.icons.filled.Notifications
|
||||
import androidx.compose.material.icons.filled.Info
|
||||
import androidx.compose.material.icons.filled.Link
|
||||
import androidx.compose.material.icons.filled.Palette
|
||||
@@ -75,6 +76,7 @@ fun SettingsScreen(
|
||||
onNavigateToAppearanceSettings: () -> Unit,
|
||||
onNavigateToAnalytics: () -> Unit,
|
||||
onNavigateToVoiceSettings: () -> Unit,
|
||||
onNavigateToNotificationCompanion: () -> Unit,
|
||||
onNavigateToPairedDevices: () -> Unit,
|
||||
onNavigateToDeveloperSettings: () -> Unit,
|
||||
onNavigateToAbout: () -> Unit,
|
||||
@@ -194,6 +196,16 @@ fun SettingsScreen(
|
||||
isDarkTheme = isDarkTheme,
|
||||
)
|
||||
|
||||
// === PHASE3-ε-followup: notification companion entry-point ===
|
||||
SettingsCategoryRow(
|
||||
icon = Icons.Filled.Notifications,
|
||||
title = "Notification companion",
|
||||
subtitle = "Let your assistant triage notifications you've shared",
|
||||
onClick = onNavigateToNotificationCompanion,
|
||||
isDarkTheme = isDarkTheme,
|
||||
)
|
||||
// === END PHASE3-ε-followup ===
|
||||
|
||||
SettingsCategoryRow(
|
||||
icon = Icons.Filled.Image,
|
||||
title = "Media",
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
package com.hermesandroid.relay.viewmodel
|
||||
|
||||
import android.app.Application
|
||||
import android.content.ComponentName
|
||||
import android.content.Context
|
||||
import android.os.BatteryManager
|
||||
import android.os.PowerManager
|
||||
import android.provider.Settings
|
||||
import androidx.lifecycle.AndroidViewModel
|
||||
import androidx.lifecycle.viewModelScope
|
||||
import com.hermesandroid.relay.data.BridgeActivityEntry
|
||||
import com.hermesandroid.relay.data.BridgePreferencesRepository
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.SharingStarted
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
import kotlinx.coroutines.flow.map
|
||||
import kotlinx.coroutines.flow.stateIn
|
||||
import kotlinx.coroutines.launch
|
||||
|
||||
/**
|
||||
* Bridge tab view-model — Phase 3 Wave 1 (Agent δ).
|
||||
*
|
||||
* Scope: UI state for the new BridgeScreen only. The actual AccessibilityService
|
||||
* runtime (read screen, tap, type, screenshot) is Agent γ's file set under
|
||||
* `com.hermesandroid.relay.accessibility`. δ intentionally **does not** touch
|
||||
* those files — we read/display state from whatever StateFlows γ exposes and
|
||||
* stub them here until γ's runtime lands.
|
||||
*
|
||||
* ## γ-handoff points (TODO before Wave 1 merges)
|
||||
*
|
||||
* 1. **`bridgeStatus`** — currently a `MutableStateFlow<BridgeStatus?>` seeded
|
||||
* from a best-effort read of [PowerManager] + [BatteryManager] on init.
|
||||
* γ must either:
|
||||
* - expose a `com.hermesandroid.relay.accessibility.HermesAccessibilityService`
|
||||
* singleton with `status: StateFlow<BridgeStatus>`, OR
|
||||
* - post updates into a shared `BridgeStateHolder` object that both
|
||||
* γ's service and this ViewModel read from.
|
||||
* Either way, δ replaces the local MutableStateFlow with `.combine` against
|
||||
* the γ flow. The [BridgeStatus] data class below is δ's proposed wire
|
||||
* shape — γ is welcome to move it into `accessibility/` and have us import
|
||||
* from there.
|
||||
*
|
||||
* 2. **`permissionStatus`** — we probe
|
||||
* [Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES] to detect whether the
|
||||
* a11y service is enabled, which is the standard pattern and doesn't
|
||||
* require γ's code to exist yet. BUT the string-match uses a placeholder
|
||||
* service class name (`HermesAccessibilityService`). γ confirms the final
|
||||
* fully-qualified class name; if it differs, update [A11Y_SERVICE_CLASS].
|
||||
*
|
||||
* 3. **`recordActivity` / `updateActivity`** — δ writes activity log entries
|
||||
* to [BridgePreferencesRepository]. γ's `HermesAccessibilityService`
|
||||
* command dispatcher should call these (or a singleton that forwards to
|
||||
* them) whenever a bridge command starts / completes / fails / is blocked
|
||||
* by safety rails. Until γ lands, the log stays empty and the UI shows
|
||||
* the empty-state copy.
|
||||
*
|
||||
* 4. **`masterToggle`** write path — flipping the master switch persists via
|
||||
* [BridgePreferencesRepository.setMasterEnabled]. γ's service reads the
|
||||
* same DataStore key (`bridge_master_enabled`) and treats it as the
|
||||
* runtime disable switch — when false, the service should ignore all
|
||||
* incoming `bridge.command` envelopes. δ does not wire the service lifecycle
|
||||
* to this toggle; that's γ's call.
|
||||
*
|
||||
* Kept deliberately thin — this is a UI-presentation layer, not a mediator.
|
||||
*/
|
||||
class BridgeViewModel(application: Application) : AndroidViewModel(application) {
|
||||
|
||||
companion object {
|
||||
/**
|
||||
* Fully-qualified class name of the accessibility service Agent γ will
|
||||
* register in the manifest. TODO(γ-handoff): confirm this matches γ's
|
||||
* final class; update if different.
|
||||
*/
|
||||
const val A11Y_SERVICE_CLASS =
|
||||
"com.hermesandroid.relay.accessibility.HermesAccessibilityService"
|
||||
}
|
||||
|
||||
private val prefsRepo = BridgePreferencesRepository(application)
|
||||
|
||||
// ── Master toggle ────────────────────────────────────────────────────
|
||||
val masterToggle: StateFlow<Boolean> = prefsRepo.settings
|
||||
.map { it.masterEnabled }
|
||||
.stateIn(
|
||||
scope = viewModelScope,
|
||||
started = SharingStarted.Eagerly,
|
||||
initialValue = BridgePreferencesRepository.DEFAULT_MASTER_ENABLED
|
||||
)
|
||||
|
||||
// ── Bridge status (γ-handoff stub) ───────────────────────────────────
|
||||
//
|
||||
// TODO(γ-handoff): replace this MutableStateFlow with the StateFlow that
|
||||
// HermesAccessibilityService exposes. Until then we seed a minimal status
|
||||
// on init from system APIs so the status card has *something* to show in
|
||||
// screenshots and previews.
|
||||
private val _bridgeStatus = MutableStateFlow<BridgeStatus?>(null)
|
||||
val bridgeStatus: StateFlow<BridgeStatus?> = _bridgeStatus.asStateFlow()
|
||||
|
||||
// ── Permission status ────────────────────────────────────────────────
|
||||
private val _permissionStatus = MutableStateFlow(BridgePermissionStatus())
|
||||
val permissionStatus: StateFlow<BridgePermissionStatus> = _permissionStatus.asStateFlow()
|
||||
|
||||
// ── Activity log ─────────────────────────────────────────────────────
|
||||
val activityLog: StateFlow<List<BridgeActivityEntry>> = prefsRepo.activityLog
|
||||
.stateIn(
|
||||
scope = viewModelScope,
|
||||
started = SharingStarted.Eagerly,
|
||||
initialValue = emptyList()
|
||||
)
|
||||
|
||||
init {
|
||||
refreshPermissionStatus()
|
||||
refreshBridgeStatusFromSystem()
|
||||
}
|
||||
|
||||
/**
|
||||
* Called from the UI when BridgeScreen regains focus (e.g., returning
|
||||
* from Android Settings after granting the a11y permission). Re-probes
|
||||
* Settings.Secure and battery/screen state.
|
||||
*/
|
||||
fun onScreenResumed() {
|
||||
refreshPermissionStatus()
|
||||
refreshBridgeStatusFromSystem()
|
||||
}
|
||||
|
||||
fun setMasterEnabled(enabled: Boolean) {
|
||||
viewModelScope.launch {
|
||||
prefsRepo.setMasterEnabled(enabled)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Append (or replace by id) an activity entry. Exposed so Agent γ's
|
||||
* command dispatcher can call it directly once it has a reference to
|
||||
* this ViewModel via a shared holder.
|
||||
*/
|
||||
fun recordActivity(entry: BridgeActivityEntry) {
|
||||
viewModelScope.launch {
|
||||
prefsRepo.appendEntry(entry)
|
||||
}
|
||||
}
|
||||
|
||||
fun clearActivityLog() {
|
||||
viewModelScope.launch {
|
||||
prefsRepo.clearLog()
|
||||
}
|
||||
}
|
||||
|
||||
// ── Internals ────────────────────────────────────────────────────────
|
||||
|
||||
private fun refreshPermissionStatus() {
|
||||
val ctx = getApplication<Application>()
|
||||
|
||||
val a11yEnabled = isAccessibilityServiceEnabled(ctx)
|
||||
// MediaProjection permission is **per-session** on Android — there's
|
||||
// no way to ask "do I already hold it?" so we report "unknown / grant
|
||||
// on first use" as a positive. The real gate is the consent dialog
|
||||
// that γ's ScreenCapture.kt will fire on each fresh projection session.
|
||||
val screenCaptureGranted = false
|
||||
// Overlay permission (SYSTEM_ALERT_WINDOW) — standard API. Safe even
|
||||
// though ζ owns the actual overlay composer; we just surface the state.
|
||||
val overlayGranted = Settings.canDrawOverlays(ctx)
|
||||
// Notification listener permission — ε owns the listener code but the
|
||||
// status check is a plain Settings.Secure lookup, no code dependency.
|
||||
val notifListenerGranted = isNotificationListenerEnabled(ctx)
|
||||
|
||||
_permissionStatus.value = BridgePermissionStatus(
|
||||
accessibilityServiceEnabled = a11yEnabled,
|
||||
screenCapturePermitted = screenCaptureGranted,
|
||||
overlayPermitted = overlayGranted,
|
||||
notificationListenerPermitted = notifListenerGranted,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Seed bridge status from what we can read without γ's runtime:
|
||||
* screen state and battery level. current_app and accessibility_enabled
|
||||
* are left as best-effort until γ wires the real flow.
|
||||
*/
|
||||
private fun refreshBridgeStatusFromSystem() {
|
||||
val ctx = getApplication<Application>()
|
||||
val pm = ctx.getSystemService(Context.POWER_SERVICE) as? PowerManager
|
||||
val bm = ctx.getSystemService(Context.BATTERY_SERVICE) as? BatteryManager
|
||||
val screenOn = pm?.isInteractive ?: false
|
||||
val battery = bm?.getIntProperty(BatteryManager.BATTERY_PROPERTY_CAPACITY) ?: -1
|
||||
_bridgeStatus.value = BridgeStatus(
|
||||
deviceName = android.os.Build.MODEL ?: "Android device",
|
||||
batteryPercent = if (battery in 0..100) battery else null,
|
||||
screenOn = screenOn,
|
||||
currentApp = null, // TODO(γ-handoff): comes from UsageStats / a11y events
|
||||
accessibilityEnabled = _permissionStatus.value.accessibilityServiceEnabled,
|
||||
)
|
||||
}
|
||||
|
||||
private fun isAccessibilityServiceEnabled(ctx: Context): Boolean {
|
||||
val enabled = Settings.Secure.getString(
|
||||
ctx.contentResolver,
|
||||
Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES
|
||||
) ?: return false
|
||||
// The setting is a colon-separated list of ComponentName.flattenToString().
|
||||
// We match on our package + expected class name; exact component name
|
||||
// TBD by Agent γ (see [A11Y_SERVICE_CLASS]).
|
||||
val expected = ComponentName(ctx.packageName, A11Y_SERVICE_CLASS).flattenToString()
|
||||
return enabled.split(':').any { it.equals(expected, ignoreCase = true) } ||
|
||||
// Fallback: if γ's class lives elsewhere, still match any service
|
||||
// in our package so the checklist doesn't falsely show red while
|
||||
// the user has in fact granted the permission.
|
||||
enabled.contains(ctx.packageName)
|
||||
}
|
||||
|
||||
private fun isNotificationListenerEnabled(ctx: Context): Boolean {
|
||||
val enabled = Settings.Secure.getString(
|
||||
ctx.contentResolver,
|
||||
"enabled_notification_listeners"
|
||||
) ?: return false
|
||||
return enabled.contains(ctx.packageName)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot of what the phone's bridge runtime is reporting — what the
|
||||
* status card displays. Corresponds to the `bridge.status` wire envelope
|
||||
* from the phase 3 plan.
|
||||
*
|
||||
* TODO(γ-handoff): if γ wants this class to live under `accessibility/`
|
||||
* instead, move it there and re-import. δ keeps it here as a placeholder
|
||||
* so BridgeScreen can compile today without blocking on γ.
|
||||
*/
|
||||
data class BridgeStatus(
|
||||
val deviceName: String,
|
||||
val batteryPercent: Int?,
|
||||
val screenOn: Boolean,
|
||||
val currentApp: String?,
|
||||
val accessibilityEnabled: Boolean,
|
||||
)
|
||||
|
||||
/** Which of the four bridge-related permissions are currently held. */
|
||||
data class BridgePermissionStatus(
|
||||
val accessibilityServiceEnabled: Boolean = false,
|
||||
val screenCapturePermitted: Boolean = false,
|
||||
val overlayPermitted: Boolean = false,
|
||||
val notificationListenerPermitted: Boolean = false,
|
||||
) {
|
||||
/** True when every permission required for Tier 1 + 2 is granted. */
|
||||
val allRequiredGranted: Boolean
|
||||
get() = accessibilityServiceEnabled // MediaProjection is per-session, not sticky
|
||||
}
|
||||
@@ -27,6 +27,11 @@ import com.hermesandroid.relay.network.HermesApiClient
|
||||
import com.hermesandroid.relay.network.ServerCapabilities
|
||||
import com.hermesandroid.relay.network.RelayHttpClient
|
||||
import com.hermesandroid.relay.network.handlers.ChatHandler
|
||||
// === PHASE3-γ: bridge channel wiring ===
|
||||
import com.hermesandroid.relay.accessibility.BridgeStatusReporter
|
||||
import com.hermesandroid.relay.accessibility.ScreenCapture
|
||||
import com.hermesandroid.relay.network.handlers.BridgeCommandHandler
|
||||
// === END PHASE3-γ ===
|
||||
import com.hermesandroid.relay.util.MediaCacheWriter
|
||||
import okhttp3.OkHttpClient
|
||||
import java.util.concurrent.TimeUnit
|
||||
@@ -420,6 +425,37 @@ class ConnectionViewModel(application: Application) : AndroidViewModel(applicati
|
||||
}
|
||||
}
|
||||
|
||||
// === PHASE3-γ: bridge channel wiring ===
|
||||
// ScreenCapture needs a MediaProjection grant from the Bridge UI
|
||||
// (MediaProjectionHolder) — it's nullable here so the handler can be
|
||||
// constructed before the user has consented to screen capture.
|
||||
// [BridgeCommandHandler] will surface a 503 error for /screenshot
|
||||
// requests until [MediaProjectionHolder.projection] is non-null.
|
||||
private val screenCapture = ScreenCapture(
|
||||
context = application,
|
||||
httpClient = relayOkHttp,
|
||||
relayUrlProvider = { _relayUrl.value },
|
||||
sessionTokenProvider = {
|
||||
(authManager.authState.value as? AuthState.Paired)?.token
|
||||
},
|
||||
mediaProjectionProvider = {
|
||||
com.hermesandroid.relay.accessibility.MediaProjectionHolder.projection
|
||||
},
|
||||
)
|
||||
|
||||
private val bridgeCommandHandler = BridgeCommandHandler(
|
||||
multiplexer = multiplexer,
|
||||
scope = viewModelScope,
|
||||
screenCapture = screenCapture,
|
||||
)
|
||||
|
||||
val bridgeStatusReporter = BridgeStatusReporter(
|
||||
context = application,
|
||||
multiplexer = multiplexer,
|
||||
scope = viewModelScope,
|
||||
)
|
||||
// === END PHASE3-γ ===
|
||||
|
||||
init {
|
||||
// Wire multiplexer to connection manager (for relay/bridge/terminal)
|
||||
multiplexer.setSendCallback { envelope ->
|
||||
@@ -431,6 +467,31 @@ class ConnectionViewModel(application: Application) : AndroidViewModel(applicati
|
||||
authManager.authenticate()
|
||||
}
|
||||
|
||||
// === PHASE3-γ: bridge handler registration ===
|
||||
// Route every incoming `bridge` channel envelope to the command
|
||||
// handler. Registering unconditionally is safe — the handler
|
||||
// itself checks whether HermesAccessibilityService is running and
|
||||
// whether the master toggle is enabled before executing anything.
|
||||
// Start the periodic status reporter once too; it ticks every
|
||||
// 30s and is a no-op while the WSS isn't connected (multiplexer
|
||||
// just drops the envelopes).
|
||||
multiplexer.registerHandler("bridge") { envelope ->
|
||||
bridgeCommandHandler.onMessage(envelope)
|
||||
}
|
||||
bridgeStatusReporter.start()
|
||||
// === END PHASE3-γ ===
|
||||
|
||||
// === PHASE3-ε-followup: notification companion multiplexer wiring ===
|
||||
// The bound NotificationListenerService instance buffers up to 50
|
||||
// envelopes in its own pendingEnvelopes queue while this slot is
|
||||
// null, so wiring it from here (rather than at service-bind time)
|
||||
// is safe — the buffer drains on the next onNotificationPosted
|
||||
// once the slot is set. Set unconditionally; the multiplexer's
|
||||
// own sendCallback gating handles the relay-disconnected case.
|
||||
com.hermesandroid.relay.notifications.HermesNotificationCompanion
|
||||
.multiplexer = multiplexer
|
||||
// === END PHASE3-ε-followup ===
|
||||
|
||||
// Load saved state — split into fast (UI-blocking) and slow (network) paths
|
||||
viewModelScope.launch {
|
||||
_onboardingCompleted.value = dataManager.isOnboardingCompleted()
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<resources>
|
||||
<string name="app_name">Hermes-Relay</string>
|
||||
<!-- === PHASE3-ε: notification companion service label === -->
|
||||
<string name="notification_companion_label">Hermes-Relay notification companion</string>
|
||||
<!-- === END PHASE3-ε === -->
|
||||
</resources>
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!--
|
||||
Sideload flavor manifest overlay.
|
||||
|
||||
Merged on top of `app/src/main/AndroidManifest.xml` for the
|
||||
`sideloadDebug` / `sideloadRelease` variants. Declares the
|
||||
AccessibilityService with the full agent-control description used for
|
||||
direct-install distribution (GitHub Releases, F-Droid, ADB). This track
|
||||
is NOT shipped through Google Play — Play Store policy would reject it.
|
||||
|
||||
"Hermes Bridge gives the agent full read/write access to the phone
|
||||
for hands-free control via voice and vision. All actions are logged
|
||||
in the Activity Log and destructive actions require your confirmation."
|
||||
|
||||
Mirror structural changes in the googlePlay flavor unless a change is
|
||||
intentionally track-specific.
|
||||
-->
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
xmlns:tools="http://schemas.android.com/tools">
|
||||
|
||||
<application>
|
||||
|
||||
<service
|
||||
android:name=".accessibility.BridgeAccessibilityService"
|
||||
android:exported="false"
|
||||
android:label="@string/a11y_service_label"
|
||||
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
|
||||
tools:ignore="MissingClass">
|
||||
<intent-filter>
|
||||
<action android:name="android.accessibilityservice.AccessibilityService" />
|
||||
</intent-filter>
|
||||
<meta-data
|
||||
android:name="android.accessibilityservice"
|
||||
android:resource="@xml/accessibility_service_config" />
|
||||
</service>
|
||||
|
||||
</application>
|
||||
|
||||
</manifest>
|
||||
@@ -0,0 +1,18 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!--
|
||||
Sideload flavor strings.
|
||||
|
||||
`a11y_description_sideload` is the full-surface description users see when
|
||||
enabling Hermes Bridge on the direct-install track. Unlike the Google Play
|
||||
flavor we can be explicit about voice + vision + full device control here:
|
||||
the sideload user is assumed to be a power user who installed an APK by
|
||||
hand, not a Play Store customer.
|
||||
|
||||
Do NOT reuse these strings in the googlePlay flavor — Play reviewers may
|
||||
flag any mention of "full read/write access" or "hands-free control" as
|
||||
outside the declared use case.
|
||||
-->
|
||||
<resources>
|
||||
<string name="a11y_service_label">Hermes Bridge</string>
|
||||
<string name="a11y_description_sideload">Hermes Bridge gives the agent full read/write access to the phone for hands-free control via voice and vision. All actions are logged in the Activity Log and destructive actions require your confirmation.</string>
|
||||
</resources>
|
||||
@@ -0,0 +1,28 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<!--
|
||||
Sideload AccessibilityService configuration.
|
||||
|
||||
Full agent-control capability surface:
|
||||
* typeAllMask — subscribe to every event type so the
|
||||
agent can observe arbitrary app state
|
||||
* flagRetrieveInteractiveWindows — enumerate all interactive windows, not
|
||||
just the active one (required for
|
||||
multi-window observation)
|
||||
* flagReportViewIds — return View resource IDs in events so
|
||||
the agent can target specific controls
|
||||
* flagRequestTouchExplorationMode — enable touch-exploration gesture layer
|
||||
* canPerformGestures — required for `dispatchGesture()` so the
|
||||
agent can synthesize taps and swipes
|
||||
|
||||
Keep this aligned with `a11y_description_sideload` — users reading the
|
||||
Accessibility settings entry need the capability surface to match the
|
||||
description they are consenting to.
|
||||
-->
|
||||
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
android:description="@string/a11y_description_sideload"
|
||||
android:accessibilityEventTypes="typeAllMask"
|
||||
android:accessibilityFlags="flagDefault|flagRetrieveInteractiveWindows|flagReportViewIds|flagRequestTouchExplorationMode"
|
||||
android:accessibilityFeedbackType="feedbackGeneric"
|
||||
android:notificationTimeout="100"
|
||||
android:canRetrieveWindowContent="true"
|
||||
android:canPerformGestures="true" />
|
||||
@@ -1,427 +0,0 @@
|
||||
"""
|
||||
android_relay — WebSocket relay that bridges HTTP tool calls to a phone over WS.
|
||||
|
||||
The relay runs an aiohttp server exposing:
|
||||
- /ws WebSocket endpoint the phone connects to (?token=CODE for auth)
|
||||
- /ping, /screen, /tap, /tap_text, /type, /swipe, /open_app, /press_key,
|
||||
/screenshot, /scroll, /wait, /apps, /current_app HTTP endpoints matching
|
||||
the bridge API consumed by android_tool.py
|
||||
|
||||
Flow:
|
||||
1. Phone connects via WebSocket with ?token=<pairing_code>
|
||||
2. Python tool makes an HTTP request to e.g. /screen
|
||||
3. Relay wraps request as JSON command, sends over WS to phone
|
||||
4. Phone executes command, sends JSON response back over WS
|
||||
5. Relay returns the phone's response to the HTTP caller
|
||||
|
||||
Command JSON format:
|
||||
Relay -> Phone: {"request_id": "uuid", "method": "GET|POST", "path": "/screen",
|
||||
"params": {...}, "body": {...}}
|
||||
Phone -> Relay: {"request_id": "uuid", "result": {...}, "status": 200}
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from typing import Optional
|
||||
|
||||
import aiohttp
|
||||
from aiohttp import web
|
||||
|
||||
logger = logging.getLogger("android_relay")
|
||||
|
||||
# ── Module-level state ────────────────────────────────────────────────────────
|
||||
|
||||
_relay_lock = threading.Lock()
|
||||
_relay_instance: Optional["_RelayState"] = None
|
||||
|
||||
|
||||
class _RelayState:
|
||||
"""Holds all mutable state for one running relay instance."""
|
||||
|
||||
def __init__(self, pairing_code: str, port: int):
|
||||
self.pairing_code: str = pairing_code
|
||||
self.port: int = port
|
||||
|
||||
# asyncio loop running in the background thread
|
||||
self.loop: Optional[asyncio.AbstractEventLoop] = None
|
||||
self.thread: Optional[threading.Thread] = None
|
||||
|
||||
# aiohttp plumbing
|
||||
self.app: Optional[web.Application] = None
|
||||
self.runner: Optional[web.AppRunner] = None
|
||||
self.site: Optional[web.TCPSite] = None
|
||||
|
||||
# The single connected phone WebSocket (or None)
|
||||
self.phone_ws: Optional[web.WebSocketResponse] = None
|
||||
self.phone_ws_lock = asyncio.Lock() # created lazily in the event loop
|
||||
|
||||
# Pending requests: request_id -> asyncio.Future
|
||||
self.pending: dict[str, asyncio.Future] = {}
|
||||
self.pending_lock: Optional[asyncio.Lock] = None # created lazily
|
||||
|
||||
# Shutdown event
|
||||
self.shutdown_event: Optional[asyncio.Event] = None
|
||||
|
||||
|
||||
# ── Public API (called from sync code) ────────────────────────────────────────
|
||||
|
||||
def start_relay(pairing_code: str, port: int = 0) -> None:
|
||||
"""Start the relay in a background thread. No-op if already running."""
|
||||
global _relay_instance
|
||||
if port == 0:
|
||||
port = int(os.getenv("ANDROID_RELAY_PORT", "8766"))
|
||||
|
||||
with _relay_lock:
|
||||
if _relay_instance is not None and _relay_instance.thread is not None and _relay_instance.thread.is_alive():
|
||||
logger.info("Relay already running on port %d", _relay_instance.port)
|
||||
return
|
||||
|
||||
state = _RelayState(pairing_code, port)
|
||||
_relay_instance = state
|
||||
|
||||
ready = threading.Event()
|
||||
t = threading.Thread(target=_run_loop, args=(state, ready), daemon=True, name="android-relay")
|
||||
state.thread = t
|
||||
t.start()
|
||||
# Wait until the server is actually listening (up to 10 s)
|
||||
if not ready.wait(timeout=10):
|
||||
logger.error("Relay failed to start within 10 seconds")
|
||||
raise RuntimeError("Relay failed to start")
|
||||
logger.info("Relay started on port %d", port)
|
||||
|
||||
|
||||
def stop_relay() -> None:
|
||||
"""Gracefully stop the relay."""
|
||||
global _relay_instance
|
||||
with _relay_lock:
|
||||
state = _relay_instance
|
||||
if state is None:
|
||||
return
|
||||
_relay_instance = None
|
||||
|
||||
# Signal the event loop to shut down
|
||||
if state.loop is not None and state.shutdown_event is not None:
|
||||
state.loop.call_soon_threadsafe(state.shutdown_event.set)
|
||||
if state.thread is not None:
|
||||
state.thread.join(timeout=5)
|
||||
logger.info("Relay stopped")
|
||||
|
||||
|
||||
def is_relay_running() -> bool:
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
return s is not None and s.thread is not None and s.thread.is_alive()
|
||||
|
||||
|
||||
def is_phone_connected() -> bool:
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
if s is None:
|
||||
return False
|
||||
ws = s.phone_ws
|
||||
return ws is not None and not ws.closed
|
||||
|
||||
|
||||
def get_relay_url() -> str:
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
port = s.port if s else int(os.getenv("ANDROID_RELAY_PORT", "8766"))
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
|
||||
def set_pairing_code(code: str) -> None:
|
||||
"""Update the pairing code (e.g. when user reconnects with new code)."""
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
if s is not None:
|
||||
s.pairing_code = code
|
||||
logger.info("Pairing code updated")
|
||||
|
||||
|
||||
# ── Background event-loop entry point ─────────────────────────────────────────
|
||||
|
||||
def _run_loop(state: _RelayState, ready: threading.Event) -> None:
|
||||
"""Runs in the background thread — creates an event loop and serves."""
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
state.loop = loop
|
||||
state.phone_ws_lock = asyncio.Lock()
|
||||
state.pending_lock = asyncio.Lock()
|
||||
state.shutdown_event = asyncio.Event()
|
||||
|
||||
try:
|
||||
loop.run_until_complete(_serve(state, ready))
|
||||
except Exception:
|
||||
logger.exception("Relay event loop crashed")
|
||||
finally:
|
||||
loop.run_until_complete(loop.shutdown_asyncgens())
|
||||
loop.close()
|
||||
|
||||
|
||||
async def _serve(state: _RelayState, ready: threading.Event) -> None:
|
||||
"""Build the aiohttp app, start the site, and block until shutdown."""
|
||||
app = web.Application()
|
||||
state.app = app
|
||||
|
||||
# WebSocket endpoint
|
||||
app.router.add_get("/ws", lambda req: _handle_ws(req, state))
|
||||
|
||||
# HTTP bridge endpoints (GET)
|
||||
for path in ("/ping", "/screen", "/screenshot", "/apps", "/current_app"):
|
||||
app.router.add_get(path, lambda req, p=path: _handle_http(req, state, p))
|
||||
|
||||
# HTTP bridge endpoints (POST)
|
||||
for path in ("/tap", "/tap_text", "/type", "/swipe", "/open_app", "/press_key", "/scroll", "/wait"):
|
||||
app.router.add_post(path, lambda req, p=path: _handle_http(req, state, p))
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
state.runner = runner
|
||||
await runner.setup()
|
||||
|
||||
site = web.TCPSite(runner, "0.0.0.0", state.port)
|
||||
state.site = site
|
||||
await site.start()
|
||||
|
||||
logger.info("Relay listening on 0.0.0.0:%d", state.port)
|
||||
ready.set()
|
||||
|
||||
# Block until shutdown is signalled
|
||||
await state.shutdown_event.wait()
|
||||
|
||||
# Cleanup
|
||||
await _cleanup_phone(state, reason="relay shutdown")
|
||||
await runner.cleanup()
|
||||
logger.info("Relay server cleaned up")
|
||||
|
||||
|
||||
# ── Rate limiting for WebSocket auth ─────────────────────────────────────────
|
||||
|
||||
_AUTH_MAX_ATTEMPTS = 5 # max failed attempts before blocking
|
||||
_AUTH_WINDOW_SECONDS = 60 # sliding window for counting failures
|
||||
_AUTH_BLOCK_SECONDS = 300 # how long to block an IP (5 minutes)
|
||||
_AUTH_CLEANUP_INTERVAL = 120 # seconds between cleanup sweeps
|
||||
|
||||
# {ip: [timestamp, timestamp, ...]} — tracks failed auth attempt times per IP
|
||||
_auth_failures: dict[str, list[float]] = {}
|
||||
# {ip: unblock_timestamp} — IPs currently blocked
|
||||
_auth_blocked: dict[str, float] = {}
|
||||
_auth_lock = threading.Lock()
|
||||
_auth_last_cleanup: float = 0.0
|
||||
|
||||
|
||||
def _auth_cleanup() -> None:
|
||||
"""Remove expired entries from the failure and block dicts."""
|
||||
global _auth_last_cleanup
|
||||
now = time.monotonic()
|
||||
if now - _auth_last_cleanup < _AUTH_CLEANUP_INTERVAL:
|
||||
return
|
||||
_auth_last_cleanup = now
|
||||
|
||||
# Remove expired blocks
|
||||
expired_blocks = [ip for ip, until in _auth_blocked.items() if now >= until]
|
||||
for ip in expired_blocks:
|
||||
del _auth_blocked[ip]
|
||||
|
||||
# Remove stale failure windows
|
||||
cutoff = now - _AUTH_WINDOW_SECONDS
|
||||
stale = []
|
||||
for ip, timestamps in _auth_failures.items():
|
||||
_auth_failures[ip] = [t for t in timestamps if t > cutoff]
|
||||
if not _auth_failures[ip]:
|
||||
stale.append(ip)
|
||||
for ip in stale:
|
||||
del _auth_failures[ip]
|
||||
|
||||
|
||||
def _auth_is_blocked(ip: str) -> bool:
|
||||
"""Check whether *ip* is currently blocked. Also triggers periodic cleanup."""
|
||||
now = time.monotonic()
|
||||
with _auth_lock:
|
||||
_auth_cleanup()
|
||||
until = _auth_blocked.get(ip)
|
||||
if until is not None:
|
||||
if now < until:
|
||||
return True
|
||||
# Block expired — remove it
|
||||
del _auth_blocked[ip]
|
||||
return False
|
||||
|
||||
|
||||
def _auth_record_failure(ip: str) -> None:
|
||||
"""Record a failed auth attempt for *ip*; block if threshold reached."""
|
||||
now = time.monotonic()
|
||||
with _auth_lock:
|
||||
timestamps = _auth_failures.setdefault(ip, [])
|
||||
timestamps.append(now)
|
||||
# Prune timestamps outside the window
|
||||
cutoff = now - _AUTH_WINDOW_SECONDS
|
||||
_auth_failures[ip] = [t for t in timestamps if t > cutoff]
|
||||
|
||||
if len(_auth_failures[ip]) >= _AUTH_MAX_ATTEMPTS:
|
||||
_auth_blocked[ip] = now + _AUTH_BLOCK_SECONDS
|
||||
_auth_failures.pop(ip, None)
|
||||
logger.warning(
|
||||
"IP %s blocked for %ds after %d failed auth attempts",
|
||||
ip, _AUTH_BLOCK_SECONDS, _AUTH_MAX_ATTEMPTS,
|
||||
)
|
||||
|
||||
|
||||
# ── WebSocket handler (phone side) ───────────────────────────────────────────
|
||||
|
||||
async def _handle_ws(request: web.Request, state: _RelayState) -> web.WebSocketResponse:
|
||||
# Rate limiting — check before token validation
|
||||
remote_ip = request.remote or "unknown"
|
||||
if _auth_is_blocked(remote_ip):
|
||||
logger.warning("Auth attempt from blocked IP %s — returning 429", remote_ip)
|
||||
raise web.HTTPTooManyRequests(text="Too many failed authentication attempts. Try again later.")
|
||||
|
||||
token = request.query.get("token", "")
|
||||
if token.upper() != state.pairing_code.upper():
|
||||
_auth_record_failure(remote_ip)
|
||||
logger.warning("Phone WS rejected — bad token (got %s) from %s", token, remote_ip)
|
||||
raise web.HTTPForbidden(text="Invalid pairing code")
|
||||
|
||||
ws = web.WebSocketResponse(heartbeat=15.0)
|
||||
await ws.prepare(request)
|
||||
|
||||
# Only one phone at a time — kick previous if any
|
||||
async with state.phone_ws_lock:
|
||||
if state.phone_ws is not None and not state.phone_ws.closed:
|
||||
logger.info("Replacing previous phone connection")
|
||||
await state.phone_ws.close(code=aiohttp.WSCloseCode.GOING_AWAY, message=b"replaced")
|
||||
state.phone_ws = ws
|
||||
|
||||
logger.info("Phone connected from %s", request.remote)
|
||||
|
||||
try:
|
||||
async for msg in ws:
|
||||
if msg.type == aiohttp.WSMsgType.TEXT:
|
||||
await _on_phone_message(state, msg.data)
|
||||
elif msg.type == aiohttp.WSMsgType.ERROR:
|
||||
logger.error("Phone WS error: %s", ws.exception())
|
||||
break
|
||||
finally:
|
||||
await _cleanup_phone(state, reason="phone disconnected")
|
||||
|
||||
return ws
|
||||
|
||||
|
||||
async def _on_phone_message(state: _RelayState, raw: str) -> None:
|
||||
"""Route an incoming message from the phone to the matching pending future."""
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("Non-JSON message from phone: %s", raw[:200])
|
||||
return
|
||||
|
||||
request_id = data.get("request_id")
|
||||
if not request_id:
|
||||
logger.warning("Phone message missing request_id: %s", raw[:200])
|
||||
return
|
||||
|
||||
async with state.pending_lock:
|
||||
future = state.pending.pop(request_id, None)
|
||||
|
||||
if future is None:
|
||||
logger.debug("No pending future for request_id=%s (possibly timed out)", request_id)
|
||||
return
|
||||
|
||||
if not future.done():
|
||||
future.set_result(data)
|
||||
|
||||
|
||||
async def _cleanup_phone(state: _RelayState, reason: str = "") -> None:
|
||||
"""Clean up phone connection and cancel all pending requests."""
|
||||
async with state.phone_ws_lock:
|
||||
ws = state.phone_ws
|
||||
state.phone_ws = None
|
||||
|
||||
if ws is not None and not ws.closed:
|
||||
await ws.close()
|
||||
|
||||
# Fail all pending futures
|
||||
async with state.pending_lock:
|
||||
pending = dict(state.pending)
|
||||
state.pending.clear()
|
||||
|
||||
for rid, fut in pending.items():
|
||||
if not fut.done():
|
||||
fut.set_exception(ConnectionError(f"Phone disconnected ({reason})"))
|
||||
|
||||
if pending:
|
||||
logger.info("Cancelled %d pending requests (%s)", len(pending), reason)
|
||||
|
||||
|
||||
# ── HTTP handler (tool side) ─────────────────────────────────────────────────
|
||||
|
||||
_RESPONSE_TIMEOUT = 30 # seconds
|
||||
|
||||
async def _handle_http(request: web.Request, state: _RelayState, path: str) -> web.Response:
|
||||
"""Forward an HTTP request from a tool to the phone over WebSocket."""
|
||||
ws = state.phone_ws
|
||||
if ws is None or ws.closed:
|
||||
return web.json_response(
|
||||
{"error": "No phone connected. Open the Hermes app on your phone and connect."},
|
||||
status=503,
|
||||
)
|
||||
|
||||
# Build the command envelope
|
||||
request_id = str(uuid.uuid4())
|
||||
method = request.method # GET or POST
|
||||
params = dict(request.query)
|
||||
|
||||
body = {}
|
||||
if method == "POST":
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
body = {}
|
||||
|
||||
command = {
|
||||
"request_id": request_id,
|
||||
"method": method,
|
||||
"path": path,
|
||||
"params": params,
|
||||
"body": body,
|
||||
}
|
||||
logger.info(">>> %s %s body=%s", method, path, json.dumps(body) if body else "{}")
|
||||
|
||||
# Register a future *before* sending so we never miss the reply
|
||||
future = state.loop.create_future()
|
||||
async with state.pending_lock:
|
||||
state.pending[request_id] = future
|
||||
|
||||
try:
|
||||
await ws.send_json(command)
|
||||
except Exception as exc:
|
||||
async with state.pending_lock:
|
||||
state.pending.pop(request_id, None)
|
||||
logger.error("Failed to send command to phone: %s", exc)
|
||||
return web.json_response(
|
||||
{"error": f"Failed to send command to phone: {exc}"},
|
||||
status=502,
|
||||
)
|
||||
|
||||
# Wait for the phone's response
|
||||
try:
|
||||
response_data = await asyncio.wait_for(future, timeout=_RESPONSE_TIMEOUT)
|
||||
except asyncio.TimeoutError:
|
||||
async with state.pending_lock:
|
||||
state.pending.pop(request_id, None)
|
||||
logger.warning("Phone did not respond within %ds for %s %s", _RESPONSE_TIMEOUT, method, path)
|
||||
return web.json_response(
|
||||
{"error": f"Phone did not respond within {_RESPONSE_TIMEOUT}s"},
|
||||
status=504,
|
||||
)
|
||||
except ConnectionError as exc:
|
||||
return web.json_response({"error": str(exc)}, status=502)
|
||||
|
||||
# Return the phone's result
|
||||
status = response_data.get("status", 200)
|
||||
result = response_data.get("result", {})
|
||||
return web.json_response(result, status=status)
|
||||
+72
-41
@@ -14,22 +14,26 @@ from typing import Optional
|
||||
# ── Config ────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Architecture: Phone connects OUT to Hermes server via WebSocket (NAT-friendly).
|
||||
# A relay server runs on localhost and bridges HTTP tool calls to the phone.
|
||||
# The unified Hermes-Relay server (``plugin/relay/server.py``, port 8767)
|
||||
# multiplexes the bridge channel alongside chat, terminal, media, and voice.
|
||||
# The legacy standalone bridge relay on port 8766 was retired in Phase 3
|
||||
# Wave 1 (Agent α, bridge-server-migration).
|
||||
#
|
||||
# Tools ──HTTP──> Relay (localhost:8766) ──WebSocket──> Phone
|
||||
# Tools ──HTTP──> Unified Relay (localhost:8767) ──WSS bridge channel──> Phone
|
||||
#
|
||||
# For local/USB dev, tools can also talk directly to the phone's HTTP server
|
||||
# by setting ANDROID_BRIDGE_URL to the phone's IP.
|
||||
|
||||
def _bridge_url() -> str:
|
||||
"""URL of the relay (default) or direct phone connection."""
|
||||
return os.getenv("ANDROID_BRIDGE_URL", "http://localhost:8766")
|
||||
return os.getenv("ANDROID_BRIDGE_URL", "http://localhost:8767")
|
||||
|
||||
def _bridge_token() -> Optional[str]:
|
||||
return os.getenv("ANDROID_BRIDGE_TOKEN")
|
||||
|
||||
def _relay_port() -> int:
|
||||
return int(os.getenv("ANDROID_RELAY_PORT", "8766"))
|
||||
# Unified relay default. Override via ANDROID_RELAY_PORT or RELAY_PORT.
|
||||
return int(os.getenv("ANDROID_RELAY_PORT", os.getenv("RELAY_PORT", "8767")))
|
||||
|
||||
def _timeout() -> float:
|
||||
return float(os.getenv("ANDROID_BRIDGE_TIMEOUT", "30"))
|
||||
@@ -311,15 +315,21 @@ def _get_public_ip() -> str:
|
||||
|
||||
def android_setup(pairing_code: str) -> str:
|
||||
"""
|
||||
Start the Android bridge relay and configure the pairing code.
|
||||
The relay runs on this server and waits for the phone to connect via WebSocket.
|
||||
Configure the Android bridge to point at the unified Hermes-Relay.
|
||||
|
||||
The user needs to:
|
||||
1. Open the Hermes Bridge app on their phone
|
||||
2. Enter this server's public IP and the pairing code
|
||||
3. The phone connects to the relay automatically
|
||||
The unified relay (``plugin/relay/server.py``, port 8767) is the single
|
||||
WSS endpoint for chat, terminal, bridge, media, and voice. It's meant
|
||||
to run as a persistent service (``systemctl --user start hermes-relay``)
|
||||
rather than being spawned on demand per tool call, so this function no
|
||||
longer starts a standalone relay — it just updates the environment
|
||||
variables the ``android_*`` tools read and verifies the relay is up.
|
||||
|
||||
For the actual pairing dance (generating + pre-registering a fresh
|
||||
code, producing a QR for the phone to scan), use ``hermes-pair`` or
|
||||
``/hermes-relay-pair``. This function is the fallback for cases where
|
||||
the operator already has a code and just wants to tell the tool about
|
||||
it.
|
||||
|
||||
Call this when the user provides their pairing code from the Hermes Bridge app.
|
||||
Example: android_setup("K7V3NP")
|
||||
"""
|
||||
try:
|
||||
@@ -345,43 +355,64 @@ def android_setup(pairing_code: str) -> str:
|
||||
os.environ["ANDROID_BRIDGE_URL"] = relay_url
|
||||
os.environ["ANDROID_BRIDGE_TOKEN"] = pairing_code
|
||||
|
||||
# Start the relay server
|
||||
# Verify the unified relay is reachable and probe phone status.
|
||||
server_address = f"{public_ip}:{port}"
|
||||
relay_running = False
|
||||
phone_connected = False
|
||||
try:
|
||||
from .android_relay import start_relay, is_relay_running, is_phone_connected
|
||||
start_relay(pairing_code=pairing_code, port=port)
|
||||
health = requests.get(f"http://localhost:{port}/health", timeout=2)
|
||||
if health.status_code == 200:
|
||||
relay_running = True
|
||||
except Exception:
|
||||
relay_running = False
|
||||
|
||||
# Check if phone is already connected
|
||||
time.sleep(1)
|
||||
phone_connected = is_phone_connected()
|
||||
if relay_running:
|
||||
try:
|
||||
ping = requests.get(
|
||||
f"http://localhost:{port}/ping",
|
||||
headers=_auth_headers(),
|
||||
timeout=2,
|
||||
)
|
||||
if ping.status_code == 200:
|
||||
phone_connected = True
|
||||
except Exception:
|
||||
phone_connected = False
|
||||
|
||||
server_address = f"{public_ip}:{port}"
|
||||
|
||||
if phone_connected:
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": "Phone is connected and ready!",
|
||||
"phone_connected": True,
|
||||
"server_address": server_address,
|
||||
})
|
||||
else:
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": "Relay is running. Now tell the user to connect their phone.",
|
||||
"phone_connected": False,
|
||||
"server_address": server_address,
|
||||
"user_instructions": (
|
||||
f"Open the Hermes Bridge app on your phone and enter:\n"
|
||||
f" Server: {server_address}\n"
|
||||
f" Pairing code: {pairing_code}\n"
|
||||
f"Then tap Connect."
|
||||
),
|
||||
})
|
||||
except ImportError:
|
||||
if not relay_running:
|
||||
return json.dumps({
|
||||
"status": "error",
|
||||
"message": "android_relay module not found. Make sure hermes-relay plugin is installed.",
|
||||
"message": (
|
||||
"Unified Hermes-Relay is not running on "
|
||||
f"localhost:{port}. Start it with "
|
||||
"`systemctl --user start hermes-relay` and retry."
|
||||
),
|
||||
"server_address": server_address,
|
||||
})
|
||||
|
||||
if phone_connected:
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": "Phone is connected and ready!",
|
||||
"phone_connected": True,
|
||||
"server_address": server_address,
|
||||
})
|
||||
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": (
|
||||
"Relay is running. Pair the phone via `hermes-pair` "
|
||||
"or /hermes-relay-pair, then retry."
|
||||
),
|
||||
"phone_connected": False,
|
||||
"server_address": server_address,
|
||||
"user_instructions": (
|
||||
f"Open the Hermes app on your phone and scan the pairing QR.\n"
|
||||
f" Server: {server_address}\n"
|
||||
f" Pairing code: {pairing_code}\n"
|
||||
f"Then tap Connect."
|
||||
),
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
return json.dumps({"status": "error", "message": str(e)})
|
||||
|
||||
|
||||
+234
-35
@@ -1,16 +1,42 @@
|
||||
"""Bridge channel handler — stub for Phase 3.
|
||||
"""Bridge channel handler — Phase 3.
|
||||
|
||||
TODO (Phase 3): Bridge protocol integration
|
||||
- Mirror the upstream relay protocol (hermes-android-plugin/android_relay.py)
|
||||
into the multiplexed WSS connection
|
||||
- Receive bridge.command from the agent (via the plugin) and forward to the phone
|
||||
- Receive bridge.response from the phone and return to the agent
|
||||
- Track bridge.status updates (accessibility, overlay, battery)
|
||||
- Replace the standalone bridge relay on port 8766 with this channel
|
||||
Migrated from the legacy standalone relay (``plugin/tools/android_relay.py``,
|
||||
port 8766) into the unified relay on port 8767. The wire protocol is
|
||||
identical to the pre-migration ``android_relay.py`` — envelopes are just
|
||||
wrapped in the multiplexed ``channel: "bridge"`` envelope instead of a
|
||||
standalone WebSocket.
|
||||
|
||||
Flow:
|
||||
1. Phone connects to unified relay's ``/ws`` and authenticates normally.
|
||||
2. Agent's Python tool (``plugin/tools/android_tool.py``) POSTs to one of
|
||||
the 14 HTTP endpoints registered on the unified relay.
|
||||
3. HTTP handler delegates to :meth:`BridgeHandler.handle_command`, which
|
||||
sends a ``bridge.command`` envelope to the connected phone and awaits
|
||||
a ``bridge.response``.
|
||||
4. Phone's bridge channel sends ``bridge.response`` — :meth:`handle` routes
|
||||
it to :meth:`handle_response`, which resolves the pending future.
|
||||
5. HTTP handler returns the result to the agent tool.
|
||||
|
||||
Wire envelopes (frozen — do not rename fields):
|
||||
* ``bridge.command`` — server → app: ``{request_id, method, path, params?, body?}``
|
||||
* ``bridge.response`` — app → server: ``{request_id, status, result}``
|
||||
* ``bridge.status`` — app → server: ``{screen_on, battery, current_app, accessibility_enabled}``
|
||||
|
||||
Concurrency model:
|
||||
* A single ``BridgeHandler`` instance lives on :class:`RelayServer`.
|
||||
* Only one phone is expected to be connected at a time (the per-client
|
||||
grant check happens at auth time). If multiple phones authenticate,
|
||||
the most-recent one wins ``self.phone_ws`` — earlier commands in
|
||||
flight against the previous phone resolve with ``ConnectionError``
|
||||
when :meth:`detach_ws` is called.
|
||||
* ``self.pending`` is protected by an asyncio lock so add/remove races
|
||||
from concurrent commands and responses can't drop futures.
|
||||
* 30 s timeout matches the legacy ``android_relay._RESPONSE_TIMEOUT``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
@@ -21,7 +47,14 @@ from aiohttp import web
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _make_envelope(msg_type: str, payload: dict[str, Any], msg_id: str | None = None) -> str:
|
||||
RESPONSE_TIMEOUT = 30.0 # seconds — matches legacy android_relay._RESPONSE_TIMEOUT
|
||||
|
||||
|
||||
class BridgeError(Exception):
|
||||
"""Raised when a bridge command cannot be dispatched or times out."""
|
||||
|
||||
|
||||
def _envelope(msg_type: str, payload: dict[str, Any], msg_id: str | None = None) -> str:
|
||||
return json.dumps(
|
||||
{
|
||||
"channel": "bridge",
|
||||
@@ -33,40 +66,206 @@ def _make_envelope(msg_type: str, payload: dict[str, Any], msg_id: str | None =
|
||||
|
||||
|
||||
class BridgeHandler:
|
||||
"""Stub handler for the bridge channel.
|
||||
"""Routes agent tool calls to the connected phone.
|
||||
|
||||
All methods return a ``not implemented`` error envelope. Real
|
||||
implementation will arrive in Phase 3 when the upstream bridge
|
||||
protocol is migrated into the multiplexed WSS connection.
|
||||
All state is held on the handler instance — there is no module-level
|
||||
global. Lifetime matches :class:`plugin.relay.server.RelayServer`: one
|
||||
handler per relay process, reused across phone reconnects.
|
||||
"""
|
||||
|
||||
async def handle(self, ws: web.WebSocketResponse, envelope: dict[str, Any]) -> None:
|
||||
"""Route an incoming bridge-channel envelope."""
|
||||
def __init__(self) -> None:
|
||||
# The currently-connected phone's WebSocket. Assigned when a phone
|
||||
# sends its first bridge envelope (bridge.status typically) and
|
||||
# cleared via :meth:`detach_ws` when the phone disconnects.
|
||||
self.phone_ws: web.WebSocketResponse | None = None
|
||||
|
||||
# request_id → asyncio.Future. Populated by handle_command before
|
||||
# the command is sent; resolved by handle_response when the phone
|
||||
# replies; cancelled with ConnectionError when the phone drops.
|
||||
self.pending: dict[str, asyncio.Future[dict[str, Any]]] = {}
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
# Last bridge.status envelope payload — exposed for Phase 3 health UI.
|
||||
self.phone_status: dict[str, Any] = {}
|
||||
|
||||
# ── Envelope dispatch ────────────────────────────────────────────────
|
||||
|
||||
async def handle(
|
||||
self,
|
||||
ws: web.WebSocketResponse,
|
||||
envelope: dict[str, Any],
|
||||
) -> None:
|
||||
"""Route an incoming bridge-channel envelope from the phone."""
|
||||
msg_type = envelope.get("type", "")
|
||||
msg_id = envelope.get("id")
|
||||
payload = envelope.get("payload") or {}
|
||||
if not isinstance(payload, dict):
|
||||
logger.warning("bridge: non-dict payload for type=%s", msg_type)
|
||||
return
|
||||
|
||||
# TODO: Phase 3 — dispatch to real handlers
|
||||
# bridge.command → forward command from agent to phone
|
||||
# bridge.response → return phone response to waiting agent future
|
||||
# bridge.status → update cached device status
|
||||
# Opportunistically latch the phone's WebSocket. The phone sends
|
||||
# a bridge.status shortly after auth; from then on we have a
|
||||
# target for outbound bridge.command envelopes.
|
||||
if self.phone_ws is not ws:
|
||||
if self.phone_ws is not None and not self.phone_ws.closed:
|
||||
logger.info("bridge: replacing previous phone ws — new client took over")
|
||||
self.phone_ws = ws
|
||||
|
||||
logger.info("Bridge channel not implemented — received %s", msg_type)
|
||||
await ws.send_str(
|
||||
_make_envelope(
|
||||
"bridge.error",
|
||||
{
|
||||
"message": (
|
||||
"Bridge channel is not yet implemented. "
|
||||
"It will be available in Phase 3."
|
||||
),
|
||||
},
|
||||
msg_id,
|
||||
if msg_type == "bridge.response":
|
||||
await self.handle_response(ws, envelope)
|
||||
elif msg_type == "bridge.status":
|
||||
await self.handle_status(ws, envelope)
|
||||
elif msg_type == "bridge.command":
|
||||
# The phone should never send bridge.command — it's server→app only.
|
||||
logger.warning("bridge: ignoring unexpected bridge.command from phone")
|
||||
else:
|
||||
logger.warning("bridge: unknown message type %r", msg_type)
|
||||
|
||||
# ── Outbound commands (called from HTTP handlers) ────────────────────
|
||||
|
||||
async def handle_command(
|
||||
self,
|
||||
method: str,
|
||||
path: str,
|
||||
params: dict[str, Any] | None = None,
|
||||
body: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Dispatch an ``android_*`` HTTP call to the phone over the bridge channel.
|
||||
|
||||
Returns the parsed ``bridge.response`` payload
|
||||
``{request_id, status, result}``. Raises :class:`BridgeError` if no
|
||||
phone is connected, the send fails, or the phone doesn't respond
|
||||
within :data:`RESPONSE_TIMEOUT` seconds.
|
||||
"""
|
||||
ws = self.phone_ws
|
||||
if ws is None or ws.closed:
|
||||
raise BridgeError(
|
||||
"No phone connected. Open the Hermes app on your phone and connect."
|
||||
)
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
future: asyncio.Future[dict[str, Any]] = asyncio.get_event_loop().create_future()
|
||||
|
||||
async with self._lock:
|
||||
self.pending[request_id] = future
|
||||
|
||||
command_payload = {
|
||||
"request_id": request_id,
|
||||
"method": method,
|
||||
"path": path,
|
||||
"params": params or {},
|
||||
"body": body or {},
|
||||
}
|
||||
logger.info(
|
||||
"bridge >>> %s %s body=%s",
|
||||
method,
|
||||
path,
|
||||
json.dumps(body) if body else "{}",
|
||||
)
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Clean up bridge state.
|
||||
try:
|
||||
await ws.send_str(_envelope("bridge.command", command_payload, request_id))
|
||||
except Exception as exc:
|
||||
async with self._lock:
|
||||
self.pending.pop(request_id, None)
|
||||
logger.error("bridge: failed to send command: %s", exc)
|
||||
raise BridgeError(f"Failed to send command to phone: {exc}") from exc
|
||||
|
||||
TODO (Phase 3): Cancel any in-flight bridge commands, notify
|
||||
the agent plugin that the phone disconnected.
|
||||
try:
|
||||
return await asyncio.wait_for(future, timeout=RESPONSE_TIMEOUT)
|
||||
except asyncio.TimeoutError:
|
||||
async with self._lock:
|
||||
self.pending.pop(request_id, None)
|
||||
logger.warning(
|
||||
"bridge: phone did not respond within %.0fs for %s %s",
|
||||
RESPONSE_TIMEOUT,
|
||||
method,
|
||||
path,
|
||||
)
|
||||
raise BridgeError(
|
||||
f"Phone did not respond within {RESPONSE_TIMEOUT:.0f}s"
|
||||
) from None
|
||||
|
||||
# ── Inbound response routing ────────────────────────────────────────
|
||||
|
||||
async def handle_response(
|
||||
self,
|
||||
ws: web.WebSocketResponse,
|
||||
envelope: dict[str, Any],
|
||||
) -> None:
|
||||
"""Resolve the pending future for an incoming ``bridge.response``."""
|
||||
payload = envelope.get("payload") or {}
|
||||
request_id = payload.get("request_id")
|
||||
if not isinstance(request_id, str) or not request_id:
|
||||
logger.warning("bridge: response missing request_id: %s", payload)
|
||||
return
|
||||
|
||||
async with self._lock:
|
||||
future = self.pending.pop(request_id, None)
|
||||
|
||||
if future is None:
|
||||
# Timed out or cancelled — drop silently.
|
||||
logger.debug("bridge: no pending future for request_id=%s", request_id)
|
||||
return
|
||||
|
||||
if not future.done():
|
||||
future.set_result(payload)
|
||||
|
||||
async def handle_status(
|
||||
self,
|
||||
ws: web.WebSocketResponse,
|
||||
envelope: dict[str, Any],
|
||||
) -> None:
|
||||
"""Cache the latest device status snapshot from the phone."""
|
||||
payload = envelope.get("payload") or {}
|
||||
if not isinstance(payload, dict):
|
||||
return
|
||||
self.phone_status = dict(payload)
|
||||
logger.debug("bridge: status update %s", self.phone_status)
|
||||
|
||||
# ── Lifecycle ───────────────────────────────────────────────────────
|
||||
|
||||
def is_phone_connected(self) -> bool:
|
||||
ws = self.phone_ws
|
||||
return ws is not None and not ws.closed
|
||||
|
||||
async def detach_ws(self, ws: web.WebSocketResponse, reason: str = "") -> None:
|
||||
"""Release ``ws`` if it's the currently-attached phone, failing pending
|
||||
commands. Called from the main WebSocket disconnect path.
|
||||
"""
|
||||
if self.phone_ws is not ws:
|
||||
return
|
||||
self.phone_ws = None
|
||||
|
||||
async with self._lock:
|
||||
pending = dict(self.pending)
|
||||
self.pending.clear()
|
||||
|
||||
if pending:
|
||||
err = ConnectionError(f"Phone disconnected ({reason})" if reason else "Phone disconnected")
|
||||
for fut in pending.values():
|
||||
if not fut.done():
|
||||
fut.set_exception(err)
|
||||
logger.info(
|
||||
"bridge: failed %d pending commands after phone disconnect (%s)",
|
||||
len(pending),
|
||||
reason or "unknown",
|
||||
)
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Server shutdown — cancel all pending commands and drop the phone ref."""
|
||||
ws = self.phone_ws
|
||||
self.phone_ws = None
|
||||
|
||||
async with self._lock:
|
||||
pending = dict(self.pending)
|
||||
self.pending.clear()
|
||||
|
||||
for fut in pending.values():
|
||||
if not fut.done():
|
||||
fut.set_exception(ConnectionError("Relay server shutting down"))
|
||||
|
||||
if pending:
|
||||
logger.info("bridge: cancelled %d pending commands on shutdown", len(pending))
|
||||
if ws is not None and not ws.closed:
|
||||
# Don't close here — server.close() owns the WebSocket lifecycle.
|
||||
pass
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Notifications channel handler — opt-in notification triage helper.
|
||||
|
||||
The phone runs a ``NotificationListenerService`` (the same Android API
|
||||
Wear OS, Android Auto, and Tasker use) that the user explicitly grants
|
||||
via Android Settings. When granted, the service forwards each posted
|
||||
notification's metadata over the existing WSS connection as a
|
||||
``notifications`` channel envelope. We hold the most recent N entries
|
||||
in a bounded in-memory deque so the agent can answer questions like
|
||||
"what came in while I was in that meeting?" without needing to be
|
||||
streaming during every notification post.
|
||||
|
||||
Trust model:
|
||||
* The user controls the grant via Android Settings — they can revoke
|
||||
at any time, and Android shows the running listener in the system
|
||||
permissions list.
|
||||
* The data never touches disk — purely in-memory, capped at 100
|
||||
entries, and lost on relay restart by design.
|
||||
* The server-side cache is per-relay-process, not per-session: any
|
||||
paired device with chat-channel auth can read the cache via the
|
||||
HTTP route, matching the trust model of every other relay endpoint
|
||||
that exposes user-shared data (notifications belong to the user,
|
||||
not to a specific phone).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import collections
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Cap on the in-memory cache. Picked to bound memory while still being
|
||||
# generous enough to cover "all the notifications that came in over the
|
||||
# last few hours" for a typical phone. Each entry is small (a few
|
||||
# strings), so 100 × ~1KB worst-case is trivial.
|
||||
DEFAULT_CACHE_SIZE = 100
|
||||
|
||||
|
||||
class NotificationsChannel:
|
||||
"""Holds the bounded recent-notifications cache for the relay.
|
||||
|
||||
One instance per :class:`RelayServer`. Envelopes from the
|
||||
``notifications`` channel are dispatched here via
|
||||
:meth:`handle_envelope`, and the HTTP route ``/notifications/recent``
|
||||
reads via :meth:`get_recent`.
|
||||
"""
|
||||
|
||||
def __init__(self, max_entries: int = DEFAULT_CACHE_SIZE) -> None:
|
||||
# ``deque`` with ``maxlen`` evicts the oldest entry on append
|
||||
# once the cap is reached — exactly the LRU-by-time behavior
|
||||
# we want for "recent notifications".
|
||||
self.recent: collections.deque[dict[str, Any]] = collections.deque(
|
||||
maxlen=max_entries
|
||||
)
|
||||
self._max_entries = max_entries
|
||||
|
||||
# ── Dispatcher ───────────────────────────────────────────────────────
|
||||
|
||||
async def handle(
|
||||
self, ws: web.WebSocketResponse, envelope: dict[str, Any]
|
||||
) -> None:
|
||||
"""Route an incoming notifications-channel envelope.
|
||||
|
||||
Currently the only supported inbound type is
|
||||
``notification.posted``. Anything else is ignored with a debug
|
||||
log line — we don't ack notifications back to the phone because
|
||||
the phone doesn't need confirmation that the relay cached them.
|
||||
"""
|
||||
msg_type = envelope.get("type", "")
|
||||
payload = envelope.get("payload", {})
|
||||
|
||||
if msg_type == "notification.posted":
|
||||
await self.handle_envelope(envelope)
|
||||
else:
|
||||
logger.debug("notifications: ignoring unknown type %r", msg_type)
|
||||
|
||||
async def handle_envelope(self, envelope: dict[str, Any]) -> None:
|
||||
"""Append the payload of a posted-notification envelope to the cache.
|
||||
|
||||
Defensive about payload shape — anything that isn't a dict is
|
||||
dropped silently. Bad data from the phone shouldn't crash the
|
||||
whole relay.
|
||||
"""
|
||||
payload = envelope.get("payload")
|
||||
if not isinstance(payload, dict):
|
||||
logger.debug(
|
||||
"notifications: dropping non-dict payload (type=%s)",
|
||||
type(payload).__name__,
|
||||
)
|
||||
return
|
||||
|
||||
self.recent.append(payload)
|
||||
logger.debug(
|
||||
"notifications: cached entry from %s (cache=%d/%d)",
|
||||
payload.get("package_name", "?"),
|
||||
len(self.recent),
|
||||
self._max_entries,
|
||||
)
|
||||
|
||||
# ── Reader ───────────────────────────────────────────────────────────
|
||||
|
||||
def get_recent(self, limit: int) -> list[dict[str, Any]]:
|
||||
"""Return up to ``limit`` most-recent entries (newest first).
|
||||
|
||||
``limit`` is clamped to ``[1, max_entries]`` so callers can't
|
||||
ask for negative values or more than the cache holds. The
|
||||
deque is iterated newest-first by reversing the slice.
|
||||
"""
|
||||
if limit < 1:
|
||||
limit = 1
|
||||
if limit > self._max_entries:
|
||||
limit = self._max_entries
|
||||
|
||||
# The deque is append-on-the-right; newest entries are at the
|
||||
# tail. Pull the last ``limit`` items and reverse so the
|
||||
# response is newest-first.
|
||||
snapshot = list(self.recent)
|
||||
tail = snapshot[-limit:]
|
||||
tail.reverse()
|
||||
return tail
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Drop all cached entries. Wired for tests + future debug routes."""
|
||||
self.recent.clear()
|
||||
+432
-1
@@ -28,6 +28,7 @@ import os
|
||||
import signal
|
||||
import ssl
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any
|
||||
@@ -42,8 +43,9 @@ from .auth import (
|
||||
Session,
|
||||
SessionManager,
|
||||
)
|
||||
from .channels.bridge import BridgeHandler
|
||||
from .channels.bridge import BridgeError, BridgeHandler
|
||||
from .channels.chat import ChatHandler
|
||||
from .channels.notifications import NotificationsChannel
|
||||
from .channels.terminal import TerminalHandler
|
||||
from .config import RelayConfig
|
||||
from .media import MediaRegistrationError, MediaRegistry, validate_media_path
|
||||
@@ -82,6 +84,9 @@ class RelayServer:
|
||||
self.chat = ChatHandler(webapi_url=config.webapi_url)
|
||||
self.terminal = TerminalHandler(default_shell=config.terminal_shell)
|
||||
self.bridge = BridgeHandler()
|
||||
# === PHASE3-ε: notifications channel ===
|
||||
self.notifications = NotificationsChannel()
|
||||
# === END PHASE3-ε ===
|
||||
|
||||
# Connected clients: ws → session token
|
||||
self._clients: dict[web.WebSocketResponse, str] = {}
|
||||
@@ -645,6 +650,155 @@ async def handle_media_register(request: web.Request) -> web.Response:
|
||||
)
|
||||
|
||||
|
||||
# === PHASE3-α-followup: /media/upload ===
|
||||
#
|
||||
# γ's ScreenCapture (the phone-side accessibility runtime) needs to push
|
||||
# screenshot bytes to the relay over the network. The pre-existing
|
||||
# /media/register endpoint is loopback + path-based — it assumes the
|
||||
# caller already has the bytes on the relay's filesystem, which is true
|
||||
# for host-local Hermes tools but false for the phone. This endpoint
|
||||
# bridges that gap: accept multipart bytes from a paired phone, write to
|
||||
# a sandboxed tempfile under tempfile.gettempdir() (which is in the
|
||||
# default MediaRegistry allowed_roots), then hand the path to
|
||||
# MediaRegistry.register() so the rest of the pipeline (token issuance,
|
||||
# expiry, LRU eviction, GET /media/{token}) is identical.
|
||||
|
||||
_MEDIA_UPLOAD_CHUNK_SIZE = 64 * 1024
|
||||
|
||||
|
||||
async def handle_media_upload(request: web.Request) -> web.Response:
|
||||
"""Accept a phone-uploaded file and register it with MediaRegistry.
|
||||
|
||||
Bearer-auth'd (every paired phone has a session token). Streams the
|
||||
multipart `file` field to a NamedTemporaryFile under
|
||||
``tempfile.gettempdir()``, enforces ``MediaRegistry.max_size_bytes``
|
||||
while reading, and on success returns the same JSON shape as
|
||||
``/media/register``.
|
||||
|
||||
POST /media/upload (multipart/form-data, field name "file")
|
||||
→ 200 {"ok": true, "token": "...", "expires_at": <epoch>}
|
||||
→ 400 invalid request shape (no multipart, no "file" field)
|
||||
→ 401 missing/invalid bearer
|
||||
→ 413 file too large (over MediaRegistry.max_size_bytes)
|
||||
"""
|
||||
server, _session = _require_bearer_session(request)
|
||||
|
||||
content_type = request.content_type or ""
|
||||
if not content_type.startswith("multipart/"):
|
||||
return web.json_response(
|
||||
{"ok": False, "error": "expected multipart/form-data"},
|
||||
status=400,
|
||||
)
|
||||
|
||||
try:
|
||||
reader = await request.multipart()
|
||||
except (aiohttp.ClientPayloadError, ValueError) as exc:
|
||||
return web.json_response(
|
||||
{"ok": False, "error": f"invalid multipart body: {exc}"},
|
||||
status=400,
|
||||
)
|
||||
|
||||
field = await reader.next()
|
||||
while field is not None and field.name != "file":
|
||||
field = await reader.next()
|
||||
if field is None:
|
||||
return web.json_response(
|
||||
{"ok": False, "error": "missing 'file' multipart field"},
|
||||
status=400,
|
||||
)
|
||||
|
||||
file_name = field.filename or "upload.bin"
|
||||
file_content_type = (
|
||||
field.headers.get("Content-Type") or "application/octet-stream"
|
||||
)
|
||||
|
||||
# Pick a suffix from the filename so MIME-by-extension consumers
|
||||
# downstream still work. Strip path components defensively in case
|
||||
# a client sends something exotic.
|
||||
base_name = os.path.basename(file_name) or "upload.bin"
|
||||
suffix = ""
|
||||
if "." in base_name:
|
||||
suffix = "." + base_name.rsplit(".", 1)[-1]
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(
|
||||
prefix="hermes-relay-upload-",
|
||||
suffix=suffix,
|
||||
delete=False,
|
||||
)
|
||||
bytes_written = 0
|
||||
max_size = server.media.max_size_bytes
|
||||
try:
|
||||
while True:
|
||||
chunk = await field.read_chunk(_MEDIA_UPLOAD_CHUNK_SIZE)
|
||||
if not chunk:
|
||||
break
|
||||
bytes_written += len(chunk)
|
||||
if bytes_written > max_size:
|
||||
tmp.close()
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
return web.json_response(
|
||||
{
|
||||
"ok": False,
|
||||
"error": (
|
||||
f"upload too large ({bytes_written} bytes, "
|
||||
f"max {max_size})"
|
||||
),
|
||||
},
|
||||
status=413,
|
||||
)
|
||||
tmp.write(chunk)
|
||||
except (aiohttp.ClientPayloadError, OSError) as exc:
|
||||
tmp.close()
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
return web.json_response(
|
||||
{"ok": False, "error": f"upload aborted: {exc}"},
|
||||
status=400,
|
||||
)
|
||||
finally:
|
||||
if not tmp.closed:
|
||||
tmp.close()
|
||||
|
||||
try:
|
||||
entry = await server.media.register(
|
||||
path=tmp.name,
|
||||
content_type=file_content_type,
|
||||
file_name=base_name,
|
||||
)
|
||||
except MediaRegistrationError as exc:
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
logger.info("Media upload registration rejected: %s", exc)
|
||||
return web.json_response(
|
||||
{"ok": False, "error": str(exc)}, status=400
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Media uploaded: token=%s... bytes=%d type=%s file=%s",
|
||||
entry.token[:8],
|
||||
bytes_written,
|
||||
file_content_type,
|
||||
base_name,
|
||||
)
|
||||
return web.json_response(
|
||||
{
|
||||
"ok": True,
|
||||
"token": entry.token,
|
||||
"expires_at": entry.expires_at,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# === END PHASE3-α-followup ===
|
||||
|
||||
|
||||
async def handle_media_get(request: web.Request) -> web.StreamResponse:
|
||||
"""Serve a previously-registered media file by opaque token.
|
||||
|
||||
@@ -815,6 +969,242 @@ async def handle_media_by_path(request: web.Request) -> web.StreamResponse:
|
||||
return web.FileResponse(real_path, headers=headers)
|
||||
|
||||
|
||||
# === PHASE3-α: bridge HTTP routes ===
|
||||
#
|
||||
# The unified relay exposes the same HTTP shape as the legacy standalone
|
||||
# relay on port 8766 so ``plugin/tools/android_tool.py`` only needs a
|
||||
# one-line URL change. Requests are delegated straight through to
|
||||
# ``BridgeHandler.handle_command`` which forwards them over the WSS
|
||||
# channel to the connected phone.
|
||||
#
|
||||
# Auth model:
|
||||
# * These routes are **unauthenticated** at the HTTP layer on purpose —
|
||||
# the legacy relay was unauthenticated too, and the trust boundary
|
||||
# is the same: only tools running on the same host as the relay can
|
||||
# reach localhost:8767. The relay's default bind is 0.0.0.0 for the
|
||||
# WebSocket side, but tools should always point at ``localhost``, so
|
||||
# an attacker reaching port 8767 from the LAN would need the phone
|
||||
# to also have auth'd with a valid pairing code — without a paired
|
||||
# phone, every bridge HTTP call just returns 503.
|
||||
# * If tightening is needed later, wrap these handlers with the same
|
||||
# ``_require_bearer_session`` pattern used by ``/media/*`` — the
|
||||
# bridge grant is already tracked per-session in ``Session.grants``.
|
||||
#
|
||||
# A paired but disconnected phone still drops bridge calls with 503 —
|
||||
# the tool caller should retry or tell the user to reconnect the app.
|
||||
|
||||
|
||||
_BRIDGE_TIMEOUT_STATUS = 504
|
||||
_BRIDGE_NO_PHONE_STATUS = 503
|
||||
_BRIDGE_SEND_FAIL_STATUS = 502
|
||||
|
||||
|
||||
def _bridge_error_response(
|
||||
exc: Exception, fallback_status: int = _BRIDGE_NO_PHONE_STATUS
|
||||
) -> web.Response:
|
||||
"""Convert a :class:`BridgeError` to a JSON response that matches the
|
||||
legacy standalone relay's error shape."""
|
||||
msg = str(exc) or "Bridge error"
|
||||
status = fallback_status
|
||||
lowered = msg.lower()
|
||||
if "did not respond" in lowered or "timeout" in lowered:
|
||||
status = _BRIDGE_TIMEOUT_STATUS
|
||||
elif "failed to send" in lowered:
|
||||
status = _BRIDGE_SEND_FAIL_STATUS
|
||||
return web.json_response({"error": msg}, status=status)
|
||||
|
||||
|
||||
async def _bridge_dispatch(
|
||||
request: web.Request,
|
||||
path: str,
|
||||
) -> web.Response:
|
||||
"""Forward an HTTP request to the phone via the bridge channel."""
|
||||
server: RelayServer = request.app["server"]
|
||||
method = request.method # GET or POST
|
||||
|
||||
params: dict[str, Any] = dict(request.query)
|
||||
body: dict[str, Any] = {}
|
||||
if method == "POST":
|
||||
# android_tool.py always sends JSON bodies; treat anything else as empty.
|
||||
try:
|
||||
raw = await request.json()
|
||||
if isinstance(raw, dict):
|
||||
body = raw
|
||||
except (json.JSONDecodeError, ValueError, aiohttp.ContentTypeError):
|
||||
body = {}
|
||||
|
||||
try:
|
||||
response = await server.bridge.handle_command(
|
||||
method=method,
|
||||
path=path,
|
||||
params=params,
|
||||
body=body,
|
||||
)
|
||||
except BridgeError as exc:
|
||||
return _bridge_error_response(exc)
|
||||
except ConnectionError as exc:
|
||||
return web.json_response({"error": str(exc)}, status=_BRIDGE_SEND_FAIL_STATUS)
|
||||
|
||||
status = response.get("status", 200)
|
||||
if not isinstance(status, int):
|
||||
status = 200
|
||||
result = response.get("result")
|
||||
if not isinstance(result, (dict, list)):
|
||||
result = {"value": result} if result is not None else {}
|
||||
return web.json_response(result, status=status)
|
||||
|
||||
|
||||
# Route adapters — aiohttp's router only gives us (request,), so we
|
||||
# close over the path string here. One adapter per endpoint keeps the
|
||||
# route registration self-documenting at the call site.
|
||||
#
|
||||
# Endpoint inventory mirrors ``plugin/tools/android_tool.py``'s 14 tools:
|
||||
# GET /ping, /screen, /screenshot, /get_apps, /current_app
|
||||
# POST /tap, /tap_text, /type, /swipe, /open_app, /press_key,
|
||||
# /scroll, /wait, /setup
|
||||
#
|
||||
# NOTE: the legacy relay used ``/apps`` for list apps but the Android app
|
||||
# expects ``/get_apps`` and the tool at line ~230 of android_tool.py calls
|
||||
# ``_get("/apps")``. We register both so the legacy tool path keeps
|
||||
# working while new code can use the canonical ``/get_apps``.
|
||||
|
||||
|
||||
async def handle_bridge_ping(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/ping")
|
||||
|
||||
|
||||
async def handle_bridge_screen(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/screen")
|
||||
|
||||
|
||||
async def handle_bridge_screenshot(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/screenshot")
|
||||
|
||||
|
||||
async def handle_bridge_get_apps(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/get_apps")
|
||||
|
||||
|
||||
async def handle_bridge_apps_legacy(request: web.Request) -> web.Response:
|
||||
# Legacy alias — the pre-migration tool used /apps, keep it working.
|
||||
return await _bridge_dispatch(request, "/apps")
|
||||
|
||||
|
||||
async def handle_bridge_current_app(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/current_app")
|
||||
|
||||
|
||||
async def handle_bridge_tap(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/tap")
|
||||
|
||||
|
||||
async def handle_bridge_tap_text(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/tap_text")
|
||||
|
||||
|
||||
async def handle_bridge_type(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/type")
|
||||
|
||||
|
||||
async def handle_bridge_swipe(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/swipe")
|
||||
|
||||
|
||||
async def handle_bridge_open_app(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/open_app")
|
||||
|
||||
|
||||
async def handle_bridge_press_key(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/press_key")
|
||||
|
||||
|
||||
async def handle_bridge_scroll(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/scroll")
|
||||
|
||||
|
||||
async def handle_bridge_wait(request: web.Request) -> web.Response:
|
||||
return await _bridge_dispatch(request, "/wait")
|
||||
|
||||
|
||||
async def handle_bridge_setup(request: web.Request) -> web.Response:
|
||||
# /setup is the tool-side pairing helper that existed on the legacy
|
||||
# relay; the unified relay's pairing flow is handled via /pairing/*
|
||||
# but we keep the endpoint pluggable through the bridge channel for
|
||||
# phones that still implement it. If the phone doesn't implement it,
|
||||
# the bridge.response will come back with a non-200 status.
|
||||
return await _bridge_dispatch(request, "/setup")
|
||||
|
||||
|
||||
# === END PHASE3-α ===
|
||||
|
||||
|
||||
# === PHASE3-ε: notifications HTTP routes ===
|
||||
#
|
||||
# Bearer-auth'd HTTP read endpoint for the cached notification deque
|
||||
# managed by ``NotificationsChannel``. The agent calls this through the
|
||||
# ``android_notifications_recent`` tool to answer "what came in
|
||||
# recently?" questions during a chat turn.
|
||||
#
|
||||
# The trust model matches every other phone-facing relay HTTP endpoint
|
||||
# (``/media/*``, ``/sessions``, ``/voice/*``): a valid relay session
|
||||
# token in ``Authorization: Bearer ...`` is required, and the same
|
||||
# token serves as proof that the caller is one of the operator's
|
||||
# paired devices.
|
||||
|
||||
|
||||
async def handle_notifications_recent(request: web.Request) -> web.Response:
|
||||
"""Return the most-recent N cached notification entries.
|
||||
|
||||
Two callers, two auth modes:
|
||||
* **Loopback callers** (the in-process Hermes tool
|
||||
``android_notifications_recent``) skip bearer auth — the trust
|
||||
boundary is host access, identical to ``/media/register`` and
|
||||
``/pairing/register``. The agent runs on the same host as the
|
||||
relay, so by-definition tool calls hit us over 127.0.0.1.
|
||||
* **Remote callers** (a paired phone or other client) must
|
||||
present a valid relay session token via
|
||||
``Authorization: Bearer <token>`` — this is the same gate as
|
||||
every other phone-facing relay HTTP route.
|
||||
|
||||
GET /notifications/recent?limit=20
|
||||
→ 200 {"notifications": [...], "count": <int>}
|
||||
→ 400 invalid limit
|
||||
→ 401 missing/invalid bearer (remote callers only)
|
||||
"""
|
||||
remote = request.remote or ""
|
||||
is_loopback = remote in ("127.0.0.1", "::1")
|
||||
|
||||
server: RelayServer
|
||||
if is_loopback:
|
||||
server = request.app["server"]
|
||||
else:
|
||||
server, _session = _require_bearer_session(request)
|
||||
|
||||
raw_limit = request.query.get("limit", "20")
|
||||
try:
|
||||
limit = int(raw_limit)
|
||||
except (TypeError, ValueError):
|
||||
return web.json_response(
|
||||
{"ok": False, "error": "limit must be an integer"}, status=400
|
||||
)
|
||||
if limit < 1:
|
||||
return web.json_response(
|
||||
{"ok": False, "error": "limit must be >= 1"}, status=400
|
||||
)
|
||||
|
||||
entries = server.notifications.get_recent(limit)
|
||||
return web.json_response(
|
||||
{
|
||||
"ok": True,
|
||||
"notifications": entries,
|
||||
"count": len(entries),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# === END PHASE3-ε ===
|
||||
|
||||
|
||||
# ── WebSocket handler ────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -1082,6 +1472,11 @@ async def _on_message(
|
||||
elif channel == "bridge":
|
||||
task = asyncio.create_task(server.bridge.handle(ws, envelope))
|
||||
_track_task(server, ws, task)
|
||||
# === PHASE3-ε: notifications channel dispatch ===
|
||||
elif channel == "notifications":
|
||||
task = asyncio.create_task(server.notifications.handle(ws, envelope))
|
||||
_track_task(server, ws, task)
|
||||
# === END PHASE3-ε ===
|
||||
else:
|
||||
logger.warning("Unknown channel: %s", channel)
|
||||
await _send_system(
|
||||
@@ -1160,6 +1555,14 @@ async def _on_disconnect(
|
||||
token = server._clients.pop(ws, None)
|
||||
tasks = server._client_tasks.pop(ws, set())
|
||||
|
||||
# === PHASE3-α: fail in-flight bridge commands on phone disconnect ===
|
||||
# If this ws was the currently-latched phone, detach_ws flips phone_ws
|
||||
# back to None and resolves every pending bridge command future with a
|
||||
# ConnectionError so the HTTP side returns 502 instead of hanging to
|
||||
# the 30s timeout on every in-flight request_id.
|
||||
await server.bridge.detach_ws(ws, reason=f"client {remote_ip} disconnected")
|
||||
# === END PHASE3-α ===
|
||||
|
||||
# Cancel all in-flight tasks for this client
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
@@ -1208,6 +1611,9 @@ def create_app(config: RelayConfig) -> web.Application:
|
||||
app.router.add_delete("/sessions/{token_prefix}", handle_sessions_revoke)
|
||||
app.router.add_patch("/sessions/{token_prefix}", handle_sessions_extend)
|
||||
app.router.add_post("/media/register", handle_media_register)
|
||||
# === PHASE3-α-followup: /media/upload ===
|
||||
app.router.add_post("/media/upload", handle_media_upload)
|
||||
# === END PHASE3-α-followup ===
|
||||
# Order matters: the fixed-path "/media/by-path" route must be declared
|
||||
# before the wildcard "/media/{token}" route or aiohttp will swallow
|
||||
# "by-path" as a token literal and handle_media_get will 404.
|
||||
@@ -1231,6 +1637,31 @@ def create_app(config: RelayConfig) -> web.Application:
|
||||
app.router.add_post("/voice/synthesize", _voice_synthesize)
|
||||
app.router.add_get("/voice/config", _voice_config)
|
||||
|
||||
# === PHASE3-α: bridge HTTP routes ===
|
||||
# 14 endpoints mirrored from the legacy standalone relay on port 8766.
|
||||
# Tool-side caller is plugin/tools/android_tool.py — only its
|
||||
# BRIDGE_URL changes, the wire shape is byte-for-byte identical.
|
||||
app.router.add_get("/ping", handle_bridge_ping)
|
||||
app.router.add_get("/screen", handle_bridge_screen)
|
||||
app.router.add_get("/screenshot", handle_bridge_screenshot)
|
||||
app.router.add_get("/get_apps", handle_bridge_get_apps)
|
||||
app.router.add_get("/apps", handle_bridge_apps_legacy)
|
||||
app.router.add_get("/current_app", handle_bridge_current_app)
|
||||
app.router.add_post("/tap", handle_bridge_tap)
|
||||
app.router.add_post("/tap_text", handle_bridge_tap_text)
|
||||
app.router.add_post("/type", handle_bridge_type)
|
||||
app.router.add_post("/swipe", handle_bridge_swipe)
|
||||
app.router.add_post("/open_app", handle_bridge_open_app)
|
||||
app.router.add_post("/press_key", handle_bridge_press_key)
|
||||
app.router.add_post("/scroll", handle_bridge_scroll)
|
||||
app.router.add_post("/wait", handle_bridge_wait)
|
||||
app.router.add_post("/setup", handle_bridge_setup)
|
||||
# === END PHASE3-α ===
|
||||
|
||||
# === PHASE3-ε: notifications HTTP routes ===
|
||||
app.router.add_get("/notifications/recent", handle_notifications_recent)
|
||||
# === END PHASE3-ε ===
|
||||
|
||||
# Cleanup on shutdown
|
||||
app.on_shutdown.append(_on_app_shutdown)
|
||||
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Tests for the Phase 3 bridge channel handler.
|
||||
|
||||
Validated surface:
|
||||
* :meth:`BridgeHandler.handle_command` routes the envelope through a
|
||||
mock phone WebSocket and resolves with the phone's response payload.
|
||||
* A timed-out command raises :class:`BridgeError` and clears its
|
||||
pending future (no leaks).
|
||||
* A phone disconnect via :meth:`detach_ws` fails all pending commands
|
||||
with ``ConnectionError``.
|
||||
* Envelope dispatch routes ``bridge.response`` to the waiting future
|
||||
and caches ``bridge.status`` payloads.
|
||||
|
||||
These tests run under plain ``unittest`` (no pytest, no ``responses``)
|
||||
to skip the repo's ``conftest.py`` which imports ``responses`` — matching
|
||||
the pattern used by the sibling test files per CLAUDE.md's server-side
|
||||
workflow note.
|
||||
|
||||
Run with::
|
||||
|
||||
python -m unittest plugin.tests.test_bridge_channel
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import unittest
|
||||
from typing import Any
|
||||
|
||||
from plugin.relay.channels.bridge import BridgeError, BridgeHandler, RESPONSE_TIMEOUT
|
||||
|
||||
|
||||
class _FakeWs:
|
||||
"""Minimal stand-in for ``aiohttp.web.WebSocketResponse``.
|
||||
|
||||
Records every ``send_str`` call so tests can assert what went out
|
||||
over the wire. ``closed`` is a plain attribute so tests can flip it
|
||||
to simulate a phone disconnect mid-command.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.sent: list[dict[str, Any]] = []
|
||||
self.closed: bool = False
|
||||
self.send_raises: Exception | None = None
|
||||
|
||||
async def send_str(self, payload: str) -> None:
|
||||
if self.send_raises is not None:
|
||||
raise self.send_raises
|
||||
self.sent.append(json.loads(payload))
|
||||
|
||||
|
||||
def _run(coro):
|
||||
"""Drive a coroutine to completion on a fresh event loop."""
|
||||
return asyncio.new_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
class BridgeHandlerTests(unittest.TestCase):
|
||||
# ── Envelope dispatch ────────────────────────────────────────────────
|
||||
|
||||
def test_status_payload_cached_on_inbound_status_envelope(self) -> None:
|
||||
async def run() -> None:
|
||||
h = BridgeHandler()
|
||||
ws = _FakeWs()
|
||||
envelope = {
|
||||
"channel": "bridge",
|
||||
"type": "bridge.status",
|
||||
"id": "evt1",
|
||||
"payload": {
|
||||
"screen_on": True,
|
||||
"battery": 72,
|
||||
"current_app": "com.example",
|
||||
"accessibility_enabled": True,
|
||||
},
|
||||
}
|
||||
await h.handle(ws, envelope)
|
||||
self.assertTrue(h.is_phone_connected())
|
||||
self.assertEqual(h.phone_status["battery"], 72)
|
||||
self.assertEqual(h.phone_status["current_app"], "com.example")
|
||||
|
||||
_run(run())
|
||||
|
||||
def test_response_resolves_waiting_future(self) -> None:
|
||||
async def run() -> None:
|
||||
h = BridgeHandler()
|
||||
ws = _FakeWs()
|
||||
# Seed phone_ws directly so handle_command has a target.
|
||||
h.phone_ws = ws
|
||||
|
||||
command_task = asyncio.create_task(
|
||||
h.handle_command(method="GET", path="/ping")
|
||||
)
|
||||
# Yield so the coroutine registers its future + sends the envelope.
|
||||
await asyncio.sleep(0)
|
||||
await asyncio.sleep(0)
|
||||
|
||||
self.assertEqual(len(ws.sent), 1)
|
||||
sent = ws.sent[0]
|
||||
self.assertEqual(sent["channel"], "bridge")
|
||||
self.assertEqual(sent["type"], "bridge.command")
|
||||
self.assertEqual(sent["payload"]["method"], "GET")
|
||||
self.assertEqual(sent["payload"]["path"], "/ping")
|
||||
request_id = sent["payload"]["request_id"]
|
||||
self.assertTrue(request_id)
|
||||
|
||||
# Route the phone's response back.
|
||||
response_envelope = {
|
||||
"channel": "bridge",
|
||||
"type": "bridge.response",
|
||||
"id": request_id,
|
||||
"payload": {
|
||||
"request_id": request_id,
|
||||
"status": 200,
|
||||
"result": {"pong": True},
|
||||
},
|
||||
}
|
||||
await h.handle(ws, response_envelope)
|
||||
|
||||
resolved = await asyncio.wait_for(command_task, timeout=1.0)
|
||||
self.assertEqual(resolved["status"], 200)
|
||||
self.assertEqual(resolved["result"], {"pong": True})
|
||||
# Pending queue should be drained — no leak.
|
||||
self.assertEqual(h.pending, {})
|
||||
|
||||
_run(run())
|
||||
|
||||
# ── Error paths ──────────────────────────────────────────────────────
|
||||
|
||||
def test_command_without_phone_raises_bridge_error(self) -> None:
|
||||
async def run() -> None:
|
||||
h = BridgeHandler()
|
||||
with self.assertRaises(BridgeError):
|
||||
await h.handle_command(method="GET", path="/ping")
|
||||
|
||||
_run(run())
|
||||
|
||||
def test_detach_fails_all_pending_commands(self) -> None:
|
||||
async def run() -> None:
|
||||
h = BridgeHandler()
|
||||
ws = _FakeWs()
|
||||
h.phone_ws = ws
|
||||
|
||||
t1 = asyncio.create_task(h.handle_command(method="GET", path="/ping"))
|
||||
t2 = asyncio.create_task(h.handle_command(method="GET", path="/screen"))
|
||||
# Give both tasks a chance to register pending futures.
|
||||
await asyncio.sleep(0)
|
||||
await asyncio.sleep(0)
|
||||
|
||||
self.assertEqual(len(ws.sent), 2)
|
||||
self.assertEqual(len(h.pending), 2)
|
||||
|
||||
ws.closed = True
|
||||
await h.detach_ws(ws, reason="unit test disconnect")
|
||||
self.assertFalse(h.is_phone_connected())
|
||||
self.assertEqual(h.pending, {})
|
||||
|
||||
# Both awaiters should now raise ConnectionError.
|
||||
with self.assertRaises(ConnectionError):
|
||||
await asyncio.wait_for(t1, timeout=1.0)
|
||||
with self.assertRaises(ConnectionError):
|
||||
await asyncio.wait_for(t2, timeout=1.0)
|
||||
|
||||
_run(run())
|
||||
|
||||
def test_timeout_clears_pending_and_raises_bridge_error(self) -> None:
|
||||
async def run() -> None:
|
||||
import plugin.relay.channels.bridge as bridge_mod
|
||||
|
||||
h = BridgeHandler()
|
||||
ws = _FakeWs()
|
||||
h.phone_ws = ws
|
||||
|
||||
# Shrink the timeout so the test finishes fast. Restoring in
|
||||
# the finally block keeps the module state clean for other
|
||||
# tests in the same process.
|
||||
original = bridge_mod.RESPONSE_TIMEOUT
|
||||
bridge_mod.RESPONSE_TIMEOUT = 0.05
|
||||
try:
|
||||
with self.assertRaises(BridgeError):
|
||||
await h.handle_command(method="POST", path="/tap")
|
||||
finally:
|
||||
bridge_mod.RESPONSE_TIMEOUT = original
|
||||
|
||||
# Timed-out entry must be removed from pending so it doesn't leak.
|
||||
self.assertEqual(h.pending, {})
|
||||
# Envelope was dispatched before the timeout.
|
||||
self.assertEqual(len(ws.sent), 1)
|
||||
self.assertEqual(ws.sent[0]["payload"]["path"], "/tap")
|
||||
|
||||
_run(run())
|
||||
|
||||
def test_send_failure_clears_pending_future(self) -> None:
|
||||
async def run() -> None:
|
||||
h = BridgeHandler()
|
||||
ws = _FakeWs()
|
||||
ws.send_raises = ConnectionResetError("broken pipe")
|
||||
h.phone_ws = ws
|
||||
|
||||
with self.assertRaises(BridgeError):
|
||||
await h.handle_command(method="GET", path="/ping")
|
||||
|
||||
# Failed-to-send future must be cleaned up so memory doesn't grow.
|
||||
self.assertEqual(h.pending, {})
|
||||
|
||||
_run(run())
|
||||
|
||||
# ── Module surface ───────────────────────────────────────────────────
|
||||
|
||||
def test_response_timeout_default_matches_legacy(self) -> None:
|
||||
# Legacy standalone relay used 30 s. Regression guard.
|
||||
self.assertEqual(RESPONSE_TIMEOUT, 30.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,168 @@
|
||||
"""
|
||||
hermes-relay plugin — Android notifications tool.
|
||||
|
||||
Registers ``android_notifications_recent`` into the hermes-agent tool
|
||||
registry. The tool reads the bounded in-memory deque maintained by the
|
||||
relay's :class:`NotificationsChannel`, populated when the user has
|
||||
explicitly granted Android's notification-listener permission to the
|
||||
companion phone app.
|
||||
|
||||
Permission model:
|
||||
* Off by default. The user must explicitly enable Hermes-Relay in
|
||||
Android Settings → Notification access. This is the same opt-in
|
||||
Wear OS, Android Auto, and Tasker have used for over a decade.
|
||||
* The agent only sees what the phone has chosen to share — there is
|
||||
no remote-trigger path that can re-enable the listener if the user
|
||||
revokes it.
|
||||
* The cache is in-memory only on the relay; restarting the relay
|
||||
drops everything.
|
||||
|
||||
Calls the relay over loopback (127.0.0.1) and falls through to a clean
|
||||
JSON error envelope on any failure so the LLM can render "I don't have
|
||||
notifications enabled" as natural language instead of crashing.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
|
||||
def _relay_url() -> str:
|
||||
"""Base URL of the local relay. Honours ``RELAY_PORT`` if set."""
|
||||
port = os.getenv("RELAY_PORT", "8767")
|
||||
return f"http://127.0.0.1:{port}"
|
||||
|
||||
|
||||
def _timeout() -> float:
|
||||
return float(os.getenv("RELAY_TOOL_TIMEOUT", "5"))
|
||||
|
||||
|
||||
def android_notifications_recent(limit: int = 20) -> str:
|
||||
"""List recent notifications the user has shared with this assistant.
|
||||
|
||||
Returns the most-recent ``limit`` notifications cached on the
|
||||
relay (capped at 100). Each entry has package_name, title, text,
|
||||
sub_text, posted_at (epoch ms), and key. Returns a structured
|
||||
error JSON envelope if the relay is unreachable or the user has
|
||||
not granted notification access on their phone.
|
||||
"""
|
||||
# Clamp the limit defensively — the LLM may try silly values.
|
||||
if not isinstance(limit, int) or limit < 1:
|
||||
limit = 20
|
||||
if limit > 100:
|
||||
limit = 100
|
||||
|
||||
url = f"{_relay_url()}/notifications/recent?limit={limit}"
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=_timeout()) as resp:
|
||||
if resp.status != 200:
|
||||
return json.dumps(
|
||||
{
|
||||
"status": "error",
|
||||
"message": f"Relay returned HTTP {resp.status}",
|
||||
}
|
||||
)
|
||||
raw = resp.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as exc:
|
||||
return json.dumps(
|
||||
{
|
||||
"status": "error",
|
||||
"message": f"Relay HTTP {exc.code}: {exc.reason}",
|
||||
}
|
||||
)
|
||||
except (urllib.error.URLError, OSError) as exc:
|
||||
return json.dumps(
|
||||
{
|
||||
"status": "error",
|
||||
"message": (
|
||||
"Cannot reach hermes-relay on loopback. Is the relay "
|
||||
f"running? ({exc})"
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
return json.dumps(
|
||||
{"status": "error", "message": "Relay returned non-JSON body"}
|
||||
)
|
||||
|
||||
notifications = data.get("notifications") or []
|
||||
return json.dumps(
|
||||
{
|
||||
"status": "ok",
|
||||
"count": len(notifications),
|
||||
"notifications": notifications,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ── Tool schema ─────────────────────────────────────────────────────────────
|
||||
|
||||
_SCHEMAS = {
|
||||
"android_notifications_recent": {
|
||||
"name": "android_notifications_recent",
|
||||
"description": (
|
||||
"List recent notifications the user has shared with this "
|
||||
"assistant. Returns up to `limit` of the most recent "
|
||||
"notifications the user's phone has forwarded after they "
|
||||
"opted in via Android's notification-access permission. "
|
||||
"Use this when the user asks 'what came in while I was "
|
||||
"away?' or 'summarize my unread messages'. Returns "
|
||||
"package_name, title, text, sub_text, posted_at (epoch "
|
||||
"ms), and key for each entry. Empty list means either no "
|
||||
"recent notifications or the user has not granted "
|
||||
"notification access yet."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"Maximum number of notifications to return "
|
||||
"(1-100, default 20, newest first)"
|
||||
),
|
||||
"default": 20,
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ── Tool handlers map ───────────────────────────────────────────────────────
|
||||
|
||||
_HANDLERS = {
|
||||
"android_notifications_recent": (
|
||||
lambda args, **kw: android_notifications_recent(**args)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ── Registry registration ──────────────────────────────────────────────────
|
||||
|
||||
try:
|
||||
from tools.registry import registry # type: ignore[import-not-found]
|
||||
|
||||
for tool_name, schema in _SCHEMAS.items():
|
||||
registry.register(
|
||||
name=tool_name,
|
||||
toolset="android",
|
||||
schema=schema,
|
||||
handler=_HANDLERS[tool_name],
|
||||
# No bridge connectivity required — this tool reads the
|
||||
# relay's in-memory cache directly via loopback HTTP.
|
||||
check_fn=lambda: True,
|
||||
requires_env=[],
|
||||
)
|
||||
except ImportError:
|
||||
# Running outside hermes-agent context (e.g. tests).
|
||||
pass
|
||||
@@ -1,427 +0,0 @@
|
||||
"""
|
||||
android_relay — WebSocket relay that bridges HTTP tool calls to a phone over WS.
|
||||
|
||||
The relay runs an aiohttp server exposing:
|
||||
- /ws WebSocket endpoint the phone connects to (?token=CODE for auth)
|
||||
- /ping, /screen, /tap, /tap_text, /type, /swipe, /open_app, /press_key,
|
||||
/screenshot, /scroll, /wait, /apps, /current_app HTTP endpoints matching
|
||||
the bridge API consumed by android_tool.py
|
||||
|
||||
Flow:
|
||||
1. Phone connects via WebSocket with ?token=<pairing_code>
|
||||
2. Python tool makes an HTTP request to e.g. /screen
|
||||
3. Relay wraps request as JSON command, sends over WS to phone
|
||||
4. Phone executes command, sends JSON response back over WS
|
||||
5. Relay returns the phone's response to the HTTP caller
|
||||
|
||||
Command JSON format:
|
||||
Relay -> Phone: {"request_id": "uuid", "method": "GET|POST", "path": "/screen",
|
||||
"params": {...}, "body": {...}}
|
||||
Phone -> Relay: {"request_id": "uuid", "result": {...}, "status": 200}
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from typing import Optional
|
||||
|
||||
import aiohttp
|
||||
from aiohttp import web
|
||||
|
||||
logger = logging.getLogger("android_relay")
|
||||
|
||||
# ── Module-level state ────────────────────────────────────────────────────────
|
||||
|
||||
_relay_lock = threading.Lock()
|
||||
_relay_instance: Optional["_RelayState"] = None
|
||||
|
||||
|
||||
class _RelayState:
|
||||
"""Holds all mutable state for one running relay instance."""
|
||||
|
||||
def __init__(self, pairing_code: str, port: int):
|
||||
self.pairing_code: str = pairing_code
|
||||
self.port: int = port
|
||||
|
||||
# asyncio loop running in the background thread
|
||||
self.loop: Optional[asyncio.AbstractEventLoop] = None
|
||||
self.thread: Optional[threading.Thread] = None
|
||||
|
||||
# aiohttp plumbing
|
||||
self.app: Optional[web.Application] = None
|
||||
self.runner: Optional[web.AppRunner] = None
|
||||
self.site: Optional[web.TCPSite] = None
|
||||
|
||||
# The single connected phone WebSocket (or None)
|
||||
self.phone_ws: Optional[web.WebSocketResponse] = None
|
||||
self.phone_ws_lock = asyncio.Lock() # created lazily in the event loop
|
||||
|
||||
# Pending requests: request_id -> asyncio.Future
|
||||
self.pending: dict[str, asyncio.Future] = {}
|
||||
self.pending_lock: Optional[asyncio.Lock] = None # created lazily
|
||||
|
||||
# Shutdown event
|
||||
self.shutdown_event: Optional[asyncio.Event] = None
|
||||
|
||||
|
||||
# ── Public API (called from sync code) ────────────────────────────────────────
|
||||
|
||||
def start_relay(pairing_code: str, port: int = 0) -> None:
|
||||
"""Start the relay in a background thread. No-op if already running."""
|
||||
global _relay_instance
|
||||
if port == 0:
|
||||
port = int(os.getenv("ANDROID_RELAY_PORT", "8766"))
|
||||
|
||||
with _relay_lock:
|
||||
if _relay_instance is not None and _relay_instance.thread is not None and _relay_instance.thread.is_alive():
|
||||
logger.info("Relay already running on port %d", _relay_instance.port)
|
||||
return
|
||||
|
||||
state = _RelayState(pairing_code, port)
|
||||
_relay_instance = state
|
||||
|
||||
ready = threading.Event()
|
||||
t = threading.Thread(target=_run_loop, args=(state, ready), daemon=True, name="android-relay")
|
||||
state.thread = t
|
||||
t.start()
|
||||
# Wait until the server is actually listening (up to 10 s)
|
||||
if not ready.wait(timeout=10):
|
||||
logger.error("Relay failed to start within 10 seconds")
|
||||
raise RuntimeError("Relay failed to start")
|
||||
logger.info("Relay started on port %d", port)
|
||||
|
||||
|
||||
def stop_relay() -> None:
|
||||
"""Gracefully stop the relay."""
|
||||
global _relay_instance
|
||||
with _relay_lock:
|
||||
state = _relay_instance
|
||||
if state is None:
|
||||
return
|
||||
_relay_instance = None
|
||||
|
||||
# Signal the event loop to shut down
|
||||
if state.loop is not None and state.shutdown_event is not None:
|
||||
state.loop.call_soon_threadsafe(state.shutdown_event.set)
|
||||
if state.thread is not None:
|
||||
state.thread.join(timeout=5)
|
||||
logger.info("Relay stopped")
|
||||
|
||||
|
||||
def is_relay_running() -> bool:
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
return s is not None and s.thread is not None and s.thread.is_alive()
|
||||
|
||||
|
||||
def is_phone_connected() -> bool:
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
if s is None:
|
||||
return False
|
||||
ws = s.phone_ws
|
||||
return ws is not None and not ws.closed
|
||||
|
||||
|
||||
def get_relay_url() -> str:
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
port = s.port if s else int(os.getenv("ANDROID_RELAY_PORT", "8766"))
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
|
||||
def set_pairing_code(code: str) -> None:
|
||||
"""Update the pairing code (e.g. when user reconnects with new code)."""
|
||||
with _relay_lock:
|
||||
s = _relay_instance
|
||||
if s is not None:
|
||||
s.pairing_code = code
|
||||
logger.info("Pairing code updated")
|
||||
|
||||
|
||||
# ── Background event-loop entry point ─────────────────────────────────────────
|
||||
|
||||
def _run_loop(state: _RelayState, ready: threading.Event) -> None:
|
||||
"""Runs in the background thread — creates an event loop and serves."""
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
state.loop = loop
|
||||
state.phone_ws_lock = asyncio.Lock()
|
||||
state.pending_lock = asyncio.Lock()
|
||||
state.shutdown_event = asyncio.Event()
|
||||
|
||||
try:
|
||||
loop.run_until_complete(_serve(state, ready))
|
||||
except Exception:
|
||||
logger.exception("Relay event loop crashed")
|
||||
finally:
|
||||
loop.run_until_complete(loop.shutdown_asyncgens())
|
||||
loop.close()
|
||||
|
||||
|
||||
async def _serve(state: _RelayState, ready: threading.Event) -> None:
|
||||
"""Build the aiohttp app, start the site, and block until shutdown."""
|
||||
app = web.Application()
|
||||
state.app = app
|
||||
|
||||
# WebSocket endpoint
|
||||
app.router.add_get("/ws", lambda req: _handle_ws(req, state))
|
||||
|
||||
# HTTP bridge endpoints (GET)
|
||||
for path in ("/ping", "/screen", "/screenshot", "/apps", "/current_app"):
|
||||
app.router.add_get(path, lambda req, p=path: _handle_http(req, state, p))
|
||||
|
||||
# HTTP bridge endpoints (POST)
|
||||
for path in ("/tap", "/tap_text", "/type", "/swipe", "/open_app", "/press_key", "/scroll", "/wait"):
|
||||
app.router.add_post(path, lambda req, p=path: _handle_http(req, state, p))
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
state.runner = runner
|
||||
await runner.setup()
|
||||
|
||||
site = web.TCPSite(runner, "0.0.0.0", state.port)
|
||||
state.site = site
|
||||
await site.start()
|
||||
|
||||
logger.info("Relay listening on 0.0.0.0:%d", state.port)
|
||||
ready.set()
|
||||
|
||||
# Block until shutdown is signalled
|
||||
await state.shutdown_event.wait()
|
||||
|
||||
# Cleanup
|
||||
await _cleanup_phone(state, reason="relay shutdown")
|
||||
await runner.cleanup()
|
||||
logger.info("Relay server cleaned up")
|
||||
|
||||
|
||||
# ── Rate limiting for WebSocket auth ─────────────────────────────────────────
|
||||
|
||||
_AUTH_MAX_ATTEMPTS = 5 # max failed attempts before blocking
|
||||
_AUTH_WINDOW_SECONDS = 60 # sliding window for counting failures
|
||||
_AUTH_BLOCK_SECONDS = 300 # how long to block an IP (5 minutes)
|
||||
_AUTH_CLEANUP_INTERVAL = 120 # seconds between cleanup sweeps
|
||||
|
||||
# {ip: [timestamp, timestamp, ...]} — tracks failed auth attempt times per IP
|
||||
_auth_failures: dict[str, list[float]] = {}
|
||||
# {ip: unblock_timestamp} — IPs currently blocked
|
||||
_auth_blocked: dict[str, float] = {}
|
||||
_auth_lock = threading.Lock()
|
||||
_auth_last_cleanup: float = 0.0
|
||||
|
||||
|
||||
def _auth_cleanup() -> None:
|
||||
"""Remove expired entries from the failure and block dicts."""
|
||||
global _auth_last_cleanup
|
||||
now = time.monotonic()
|
||||
if now - _auth_last_cleanup < _AUTH_CLEANUP_INTERVAL:
|
||||
return
|
||||
_auth_last_cleanup = now
|
||||
|
||||
# Remove expired blocks
|
||||
expired_blocks = [ip for ip, until in _auth_blocked.items() if now >= until]
|
||||
for ip in expired_blocks:
|
||||
del _auth_blocked[ip]
|
||||
|
||||
# Remove stale failure windows
|
||||
cutoff = now - _AUTH_WINDOW_SECONDS
|
||||
stale = []
|
||||
for ip, timestamps in _auth_failures.items():
|
||||
_auth_failures[ip] = [t for t in timestamps if t > cutoff]
|
||||
if not _auth_failures[ip]:
|
||||
stale.append(ip)
|
||||
for ip in stale:
|
||||
del _auth_failures[ip]
|
||||
|
||||
|
||||
def _auth_is_blocked(ip: str) -> bool:
|
||||
"""Check whether *ip* is currently blocked. Also triggers periodic cleanup."""
|
||||
now = time.monotonic()
|
||||
with _auth_lock:
|
||||
_auth_cleanup()
|
||||
until = _auth_blocked.get(ip)
|
||||
if until is not None:
|
||||
if now < until:
|
||||
return True
|
||||
# Block expired — remove it
|
||||
del _auth_blocked[ip]
|
||||
return False
|
||||
|
||||
|
||||
def _auth_record_failure(ip: str) -> None:
|
||||
"""Record a failed auth attempt for *ip*; block if threshold reached."""
|
||||
now = time.monotonic()
|
||||
with _auth_lock:
|
||||
timestamps = _auth_failures.setdefault(ip, [])
|
||||
timestamps.append(now)
|
||||
# Prune timestamps outside the window
|
||||
cutoff = now - _AUTH_WINDOW_SECONDS
|
||||
_auth_failures[ip] = [t for t in timestamps if t > cutoff]
|
||||
|
||||
if len(_auth_failures[ip]) >= _AUTH_MAX_ATTEMPTS:
|
||||
_auth_blocked[ip] = now + _AUTH_BLOCK_SECONDS
|
||||
_auth_failures.pop(ip, None)
|
||||
logger.warning(
|
||||
"IP %s blocked for %ds after %d failed auth attempts",
|
||||
ip, _AUTH_BLOCK_SECONDS, _AUTH_MAX_ATTEMPTS,
|
||||
)
|
||||
|
||||
|
||||
# ── WebSocket handler (phone side) ───────────────────────────────────────────
|
||||
|
||||
async def _handle_ws(request: web.Request, state: _RelayState) -> web.WebSocketResponse:
|
||||
# Rate limiting — check before token validation
|
||||
remote_ip = request.remote or "unknown"
|
||||
if _auth_is_blocked(remote_ip):
|
||||
logger.warning("Auth attempt from blocked IP %s — returning 429", remote_ip)
|
||||
raise web.HTTPTooManyRequests(text="Too many failed authentication attempts. Try again later.")
|
||||
|
||||
token = request.query.get("token", "")
|
||||
if token.upper() != state.pairing_code.upper():
|
||||
_auth_record_failure(remote_ip)
|
||||
logger.warning("Phone WS rejected — bad token (got %s) from %s", token, remote_ip)
|
||||
raise web.HTTPForbidden(text="Invalid pairing code")
|
||||
|
||||
ws = web.WebSocketResponse(heartbeat=15.0)
|
||||
await ws.prepare(request)
|
||||
|
||||
# Only one phone at a time — kick previous if any
|
||||
async with state.phone_ws_lock:
|
||||
if state.phone_ws is not None and not state.phone_ws.closed:
|
||||
logger.info("Replacing previous phone connection")
|
||||
await state.phone_ws.close(code=aiohttp.WSCloseCode.GOING_AWAY, message=b"replaced")
|
||||
state.phone_ws = ws
|
||||
|
||||
logger.info("Phone connected from %s", request.remote)
|
||||
|
||||
try:
|
||||
async for msg in ws:
|
||||
if msg.type == aiohttp.WSMsgType.TEXT:
|
||||
await _on_phone_message(state, msg.data)
|
||||
elif msg.type == aiohttp.WSMsgType.ERROR:
|
||||
logger.error("Phone WS error: %s", ws.exception())
|
||||
break
|
||||
finally:
|
||||
await _cleanup_phone(state, reason="phone disconnected")
|
||||
|
||||
return ws
|
||||
|
||||
|
||||
async def _on_phone_message(state: _RelayState, raw: str) -> None:
|
||||
"""Route an incoming message from the phone to the matching pending future."""
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("Non-JSON message from phone: %s", raw[:200])
|
||||
return
|
||||
|
||||
request_id = data.get("request_id")
|
||||
if not request_id:
|
||||
logger.warning("Phone message missing request_id: %s", raw[:200])
|
||||
return
|
||||
|
||||
async with state.pending_lock:
|
||||
future = state.pending.pop(request_id, None)
|
||||
|
||||
if future is None:
|
||||
logger.debug("No pending future for request_id=%s (possibly timed out)", request_id)
|
||||
return
|
||||
|
||||
if not future.done():
|
||||
future.set_result(data)
|
||||
|
||||
|
||||
async def _cleanup_phone(state: _RelayState, reason: str = "") -> None:
|
||||
"""Clean up phone connection and cancel all pending requests."""
|
||||
async with state.phone_ws_lock:
|
||||
ws = state.phone_ws
|
||||
state.phone_ws = None
|
||||
|
||||
if ws is not None and not ws.closed:
|
||||
await ws.close()
|
||||
|
||||
# Fail all pending futures
|
||||
async with state.pending_lock:
|
||||
pending = dict(state.pending)
|
||||
state.pending.clear()
|
||||
|
||||
for rid, fut in pending.items():
|
||||
if not fut.done():
|
||||
fut.set_exception(ConnectionError(f"Phone disconnected ({reason})"))
|
||||
|
||||
if pending:
|
||||
logger.info("Cancelled %d pending requests (%s)", len(pending), reason)
|
||||
|
||||
|
||||
# ── HTTP handler (tool side) ─────────────────────────────────────────────────
|
||||
|
||||
_RESPONSE_TIMEOUT = 30 # seconds
|
||||
|
||||
async def _handle_http(request: web.Request, state: _RelayState, path: str) -> web.Response:
|
||||
"""Forward an HTTP request from a tool to the phone over WebSocket."""
|
||||
ws = state.phone_ws
|
||||
if ws is None or ws.closed:
|
||||
return web.json_response(
|
||||
{"error": "No phone connected. Open the Hermes app on your phone and connect."},
|
||||
status=503,
|
||||
)
|
||||
|
||||
# Build the command envelope
|
||||
request_id = str(uuid.uuid4())
|
||||
method = request.method # GET or POST
|
||||
params = dict(request.query)
|
||||
|
||||
body = {}
|
||||
if method == "POST":
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
body = {}
|
||||
|
||||
command = {
|
||||
"request_id": request_id,
|
||||
"method": method,
|
||||
"path": path,
|
||||
"params": params,
|
||||
"body": body,
|
||||
}
|
||||
logger.info(">>> %s %s body=%s", method, path, json.dumps(body) if body else "{}")
|
||||
|
||||
# Register a future *before* sending so we never miss the reply
|
||||
future = state.loop.create_future()
|
||||
async with state.pending_lock:
|
||||
state.pending[request_id] = future
|
||||
|
||||
try:
|
||||
await ws.send_json(command)
|
||||
except Exception as exc:
|
||||
async with state.pending_lock:
|
||||
state.pending.pop(request_id, None)
|
||||
logger.error("Failed to send command to phone: %s", exc)
|
||||
return web.json_response(
|
||||
{"error": f"Failed to send command to phone: {exc}"},
|
||||
status=502,
|
||||
)
|
||||
|
||||
# Wait for the phone's response
|
||||
try:
|
||||
response_data = await asyncio.wait_for(future, timeout=_RESPONSE_TIMEOUT)
|
||||
except asyncio.TimeoutError:
|
||||
async with state.pending_lock:
|
||||
state.pending.pop(request_id, None)
|
||||
logger.warning("Phone did not respond within %ds for %s %s", _RESPONSE_TIMEOUT, method, path)
|
||||
return web.json_response(
|
||||
{"error": f"Phone did not respond within {_RESPONSE_TIMEOUT}s"},
|
||||
status=504,
|
||||
)
|
||||
except ConnectionError as exc:
|
||||
return web.json_response({"error": str(exc)}, status=502)
|
||||
|
||||
# Return the phone's result
|
||||
status = response_data.get("status", 200)
|
||||
result = response_data.get("result", {})
|
||||
return web.json_response(result, status=status)
|
||||
@@ -27,22 +27,25 @@ from typing import Optional
|
||||
# ── Config ────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Architecture: Phone connects OUT to Hermes server via WebSocket (NAT-friendly).
|
||||
# A relay server runs on localhost and bridges HTTP tool calls to the phone.
|
||||
# The unified Hermes-Relay server runs on localhost:8767 and multiplexes the
|
||||
# bridge channel alongside chat, terminal, media, and voice. The legacy
|
||||
# standalone bridge relay on port 8766 was retired in Phase 3 Wave 1.
|
||||
#
|
||||
# Tools ──HTTP──> Relay (localhost:8766) ──WebSocket──> Phone
|
||||
# Tools ──HTTP──> Unified Relay (localhost:8767) ──WSS bridge channel──> Phone
|
||||
#
|
||||
# For local/USB dev, tools can also talk directly to the phone's HTTP server
|
||||
# by setting ANDROID_BRIDGE_URL to the phone's IP.
|
||||
|
||||
def _bridge_url() -> str:
|
||||
"""URL of the relay (default) or direct phone connection."""
|
||||
return os.getenv("ANDROID_BRIDGE_URL", "http://localhost:8766")
|
||||
return os.getenv("ANDROID_BRIDGE_URL", "http://localhost:8767")
|
||||
|
||||
def _bridge_token() -> Optional[str]:
|
||||
return os.getenv("ANDROID_BRIDGE_TOKEN")
|
||||
|
||||
def _relay_port() -> int:
|
||||
return int(os.getenv("ANDROID_RELAY_PORT", "8766"))
|
||||
# Unified relay default. Override via ANDROID_RELAY_PORT or RELAY_PORT.
|
||||
return int(os.getenv("ANDROID_RELAY_PORT", os.getenv("RELAY_PORT", "8767")))
|
||||
|
||||
def _timeout() -> float:
|
||||
return float(os.getenv("ANDROID_BRIDGE_TIMEOUT", "30"))
|
||||
@@ -324,15 +327,21 @@ def _get_public_ip() -> str:
|
||||
|
||||
def android_setup(pairing_code: str) -> str:
|
||||
"""
|
||||
Start the Android bridge relay and configure the pairing code.
|
||||
The relay runs on this server and waits for the phone to connect via WebSocket.
|
||||
Configure the Android bridge to point at the unified Hermes-Relay.
|
||||
|
||||
The user needs to:
|
||||
1. Open the Hermes Bridge app on their phone
|
||||
2. Enter this server's public IP and the pairing code
|
||||
3. The phone connects to the relay automatically
|
||||
The unified relay (``plugin/relay/server.py``, port 8767) is the single
|
||||
WSS endpoint for chat, terminal, bridge, media, and voice. It's meant
|
||||
to run as a persistent service (``systemctl --user start hermes-relay``)
|
||||
rather than being spawned on demand per tool call, so this function no
|
||||
longer starts a standalone relay — it just updates the environment
|
||||
variables the ``android_*`` tools read and verifies the relay is up.
|
||||
|
||||
For the actual pairing dance (generating + pre-registering a fresh
|
||||
code, producing a QR for the phone to scan), use ``hermes-pair`` or
|
||||
``/hermes-relay-pair``. This function is the fallback for cases where
|
||||
the operator already has a code and just wants to tell the tool about
|
||||
it.
|
||||
|
||||
Call this when the user provides their pairing code from the Hermes Bridge app.
|
||||
Example: android_setup("K7V3NP")
|
||||
"""
|
||||
try:
|
||||
@@ -358,43 +367,64 @@ def android_setup(pairing_code: str) -> str:
|
||||
os.environ["ANDROID_BRIDGE_URL"] = relay_url
|
||||
os.environ["ANDROID_BRIDGE_TOKEN"] = pairing_code
|
||||
|
||||
# Start the relay server
|
||||
# Verify the unified relay is reachable and probe phone status.
|
||||
server_address = f"{public_ip}:{port}"
|
||||
relay_running = False
|
||||
phone_connected = False
|
||||
try:
|
||||
from tools.android_relay import start_relay, is_relay_running, is_phone_connected
|
||||
start_relay(pairing_code=pairing_code, port=port)
|
||||
health = requests.get(f"http://localhost:{port}/health", timeout=2)
|
||||
if health.status_code == 200:
|
||||
relay_running = True
|
||||
except Exception:
|
||||
relay_running = False
|
||||
|
||||
# Check if phone is already connected
|
||||
time.sleep(1)
|
||||
phone_connected = is_phone_connected()
|
||||
if relay_running:
|
||||
try:
|
||||
ping = requests.get(
|
||||
f"http://localhost:{port}/ping",
|
||||
headers=_auth_headers(),
|
||||
timeout=2,
|
||||
)
|
||||
if ping.status_code == 200:
|
||||
phone_connected = True
|
||||
except Exception:
|
||||
phone_connected = False
|
||||
|
||||
server_address = f"{public_ip}:{port}"
|
||||
|
||||
if phone_connected:
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": "Phone is connected and ready!",
|
||||
"phone_connected": True,
|
||||
"server_address": server_address,
|
||||
})
|
||||
else:
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": "Relay is running. Now tell the user to connect their phone.",
|
||||
"phone_connected": False,
|
||||
"server_address": server_address,
|
||||
"user_instructions": (
|
||||
f"Open the Hermes Bridge app on your phone and enter:\n"
|
||||
f" Server: {server_address}\n"
|
||||
f" Pairing code: {pairing_code}\n"
|
||||
f"Then tap Connect."
|
||||
),
|
||||
})
|
||||
except ImportError:
|
||||
if not relay_running:
|
||||
return json.dumps({
|
||||
"status": "error",
|
||||
"message": "android_relay module not found. Make sure hermes-relay plugin is installed.",
|
||||
"message": (
|
||||
"Unified Hermes-Relay is not running on "
|
||||
f"localhost:{port}. Start it with "
|
||||
"`systemctl --user start hermes-relay` and retry."
|
||||
),
|
||||
"server_address": server_address,
|
||||
})
|
||||
|
||||
if phone_connected:
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": "Phone is connected and ready!",
|
||||
"phone_connected": True,
|
||||
"server_address": server_address,
|
||||
})
|
||||
|
||||
return json.dumps({
|
||||
"status": "ok",
|
||||
"message": (
|
||||
"Relay is running. Pair the phone via `hermes-pair` "
|
||||
"or /hermes-relay-pair, then retry."
|
||||
),
|
||||
"phone_connected": False,
|
||||
"server_address": server_address,
|
||||
"user_instructions": (
|
||||
f"Open the Hermes app on your phone and scan the pairing QR.\n"
|
||||
f" Server: {server_address}\n"
|
||||
f" Pairing code: {pairing_code}\n"
|
||||
f"Then tap Connect."
|
||||
),
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
return json.dumps({"status": "error", "message": str(e)})
|
||||
|
||||
|
||||
@@ -8,7 +8,21 @@ Hermes-Relay is a native Android app for [Hermes Agent](https://hermes-agent.nou
|
||||
curl -fsSL https://raw.githubusercontent.com/Codename-11/hermes-relay/main/install.sh | bash
|
||||
```
|
||||
|
||||
This installs the server-side plugin. Grab the Android app from [GitHub Releases](https://github.com/Codename-11/hermes-relay/releases), then either type `/hermes-relay-pair` in any Hermes chat surface or run `hermes-pair` from a shell to generate a pairing QR. See [Installation & Setup](/guide/getting-started) for the full walkthrough.
|
||||
This installs the server-side plugin. One command, full features — sessions browser, conversation history, personality picker, command palette, memory management, and the WSS relay for terminal/voice all work out of the box on any standard `hermes-agent` install. Grab the Android app from [GitHub Releases](https://github.com/Codename-11/hermes-relay/releases), then either type `/hermes-relay-pair` in any Hermes chat surface or run `hermes-pair` from a shell to generate a pairing QR. See [Installation & Setup](/guide/getting-started) for the full walkthrough.
|
||||
|
||||
To uninstall later:
|
||||
|
||||
```bash
|
||||
bash ~/.hermes/hermes-relay/uninstall.sh
|
||||
```
|
||||
|
||||
Or via curl if the clone is already gone:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/Codename-11/hermes-relay/main/uninstall.sh | bash
|
||||
```
|
||||
|
||||
The uninstaller is idempotent and never touches state shared with other Hermes tools. Flags: `--dry-run`, `--keep-clone`, `--remove-secret`.
|
||||
|
||||
## Connection Model
|
||||
|
||||
|
||||
@@ -28,6 +28,14 @@ Want the service to survive logging out of SSH?
|
||||
loginctl enable-linger $USER
|
||||
```
|
||||
|
||||
To uninstall the relay (and the rest of the plugin) cleanly:
|
||||
|
||||
```bash
|
||||
bash ~/.hermes/hermes-relay/uninstall.sh
|
||||
```
|
||||
|
||||
The uninstaller stops the systemd unit, removes it, daemon-reloads, and reverses every other install step (pip package, plugin symlink, skills config entry, hermes-pair shim, git clone). It's idempotent and never touches `~/.hermes/.env`, the gateway's `state.db`, or the `hermes-agent` venv core. Use `--dry-run` to preview, `--keep-clone` to keep the git tree, `--remove-secret` to also wipe the QR signing identity.
|
||||
|
||||
**Manual run** (dev boxes, hosts without systemd-user):
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user