58 Commits
Author SHA1 Message Date
Bailey Dixon 3882b857e3 fix(relay): retire owned media and redact activity logs 2026-09-23 17:48:33 -04:00
Bailey Dixon 81c186c6ba fix(plugin): attach desktop screenshots as host images 2026-09-23 17:48:26 -04:00
Bailey Dixon 759ac490c9 fix(plugin): resolve Android screenshot media tokens for host tools 2026-09-23 16:37:50 -04:00
Bailey DixonandJack Hunzicker 179080d6d9 fix(plugin): share relay availability checks
Co-authored-by: Jack Hunzicker <JackHunzicker@users.noreply.github.com>
2026-09-09 20:57:22 -04:00
Bailey Dixon 7c5894213d fix(plugin): harden bridge credential recovery 2026-09-03 20:53:19 -04:00
nicolasestrem ed599995d6 fix(plugin): resolve bridge token from disk and make android_setup callable
Every android_* tool returned `401 Client Error: Unauthorized` on a phone
that was paired, connected and healthy (`/bridge/status` reported
phone_connected: true, accessibility_granted: true).

Cause: `_bridge_token()` read ANDROID_BRIDGE_TOKEN from the process
environment only. The env is snapshotted when the host starts, so a token
written afterwards — by `android_setup`, by `hermes-pair`, or by hand —
stayed invisible and `_require_bearer_session` rejected every bridge
dispatch until a full restart.

Resolution order is now env -> ~/.hermes/.env -> most-recently-seen paired
session in ~/.hermes/hermes-relay-sessions.json, with the disk fallbacks
cached for 30s so we don't stat two files on every bridge call. The process
environment still wins when set, and HERMES_HOME is honoured.

The documented recovery path was itself broken: android_setup's schema
required `pairing_code` while the signature had been renamed to
`bridge_session_token`, and the dispatcher calls func(**call_args) — so the
schema-conformant call raised TypeError and the canonical one was rejected
by the validator. The tool could not be invoked at all. Both spellings are
now accepted (canonical wins), neither is schema-required, a missing token
returns a structured error, and the description no longer mislabels the
value as a "6-character pairing code" or claims the tool performs pairing.

Also pin TestSetup to a temporary home. It exercises the real
android_setup, which persists ANDROID_BRIDGE_* to ~/.hermes/.env — running
the suite on a real host overwrote the machine's live paired session token
with a fixture value.

Verified: 43 passed (test_android_tool.py + test_android_tool_device_selector.py),
and against the physical device with ANDROID_BRIDGE_TOKEN unset from the
environment — /ping and /current_app both 200 via the new disk fallback.
2026-09-03 22:40:30 +02:00
Bailey Dixon 6a39e8dc1a feat(android): add granular bridge capability grants 2026-08-19 19:43:04 -04:00
Bailey Dixon 75bcd9180f feat(desktop): add optional CUA control engine 2026-08-13 19:33:11 -04:00
Bailey Dixon ca7ded3939 test(server): prove concurrent desktop routing 2026-08-12 18:24:44 -04:00
Bailey Dixon f97bbdd395 feat(desktop): add host-wide raw USB control 2026-08-12 11:38:09 -04:00
Bailey Dixon bbfb57b462 feat(desktop): harden targeted remote management 2026-08-12 10:37:12 -04:00
Bailey Dixon f5c2a2b888 feat(plugins): add live Android plugin surfaces 2026-08-02 18:14:42 -04:00
Bailey Dixon c05ee40db4 Merge PR #171 (bblicke1:feat/android-multi-device-bridge) into pr-batch — multi-device bridge targeting
# Conflicts:
#	DEVLOG.md
2026-07-07 12:44:17 -04:00
Bailey Dixon 71f3331f54 feat: add multi-device Android bridge targeting 2026-07-06 20:11:45 -04:00
Bailey DixonandClaude Fable 5 865c39bd86 fix(plugin): package-relative imports so the native plugin loader works (#165)
hermes-agent's native installer imports the plugin directory as
hermes_plugins.hermes_relay — no top-level 'plugin' package exists there,
so every absolute 'from plugin.X' import crashed 'hermes relay start'
with ModuleNotFoundError: No module named 'plugin'.

- Convert all runtime absolute plugin.* imports to package-relative form
  (relay voice/realtime chain, tailscale CLI, pair, enhancements, tools).
- android_tool's direct-script fallback now imports the sibling module
  bare instead of via 'plugin.tools.'.
- dashboard/plugin_api.py is exec'd standalone by the dashboard web
  server (spec_from_file_location, no parent package), so relative
  imports can't work there: add a _plugin_module() bootstrap that
  imports through the real parent package when one exists, and
  otherwise synthesizes it (bare ModuleType with __path__ at the plugin
  dir under a stable sys.modules alias) without exec'ing
  plugin/__init__.py side effects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 18:49:07 -04:00
Bailey DixonandClaude Opus 4.8 aa1b239e64 feat(attachments): in-app previews, sensitive-media blur, and richer capture
Inbound: new AttachmentViewer renders image/video/audio/pdf/text in-app
(Media3 + PdfRenderer) with a shared Share/Save/Open-externally toolbar; tapping
an attachment now previews in-app instead of firing ACTION_VIEW. Off-thread card
thumbnails, inline-image save menus, and configurable sensitive-media blur
(OFF/FLAGGED/ALL_IMAGES) applied in card, inline image, and viewer.

Sensitivity is model-emitted metadata only (no classifier): the relay carries a
`sensitive` bit via register_media -> X-Media-Sensitive header ->
FetchedMedia.sensitive -> Attachment.sensitive; the standard path uses a markdown
spoiler/sentinel convention. Adds D6 content re-sniff via _IMAGE_MAGIC.

Outbound: permissionless Photo Picker + camera capture + clipboard paste behind a
Photos/Files/Camera/Paste menu, unified through ingestAttachmentFromUri.

Design spec: docs/plans/2026-06-18-attachment-experience.md
Deferred: download progress/cancel (B3), multi-image gallery (A6), agent-side
sensitivity config gate (C5).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 21:50:55 -04:00
Bailey Dixon 5278e5b0bb fix(android): ship Play bridge core without device control 2026-05-19 20:16:27 -04:00
Bailey Dixon a5f3d0a2c3 fix(android): add bridge media sharing and mms handoff 2026-05-05 20:13:31 -04:00
Bailey Dixon ab1924d440 feat(desktop): approve computer control by grant 2026-04-26 21:06:00 -04:00
Bailey Dixon 66d958195d feat(desktop): enable computer use with desktop tools 2026-04-26 20:37:18 -04:00
Bailey Dixon 488a9d757f feat(desktop): enable approved computer actions 2026-04-26 18:14:49 -04:00
Bailey Dixon 9ebe12a0ad feat(desktop): add observe-first computer-use tools 2026-04-26 16:58:27 -04:00
Bailey DixonandClaude Opus 4.7 21b4cfd480 feat(desktop): remote-PC ergonomics — powershell, process, jobs, transfer, health tools
Adds 14 desktop_* tools to close the gaps surfaced from a real remote-PC session:
desktop_terminal timing out on long launches, no process management primitives,
no bulk file sync, PowerShell echoing instead of executing, no daemon-health
introspection.

Routed entirely through the existing `desktop` channel — no new channels,
no hermes-agent core changes:

- desktop_powershell             script via stdin → pwsh/powershell -Command -;
                                  bypasses cmd.exe quote-mangling.
- desktop_spawn_detached, _list_processes, _kill_process, _find_pid_by_port
                                  unref'd detached spawn for long jobs;
                                  cross-platform process listing via
                                  ps/tasklist /FO CSV (no /V — window-title
                                  enumeration was a hidden 30s+ latency
                                  landmine, same class that made desktop_terminal
                                  502); kill by pid or name; netstat/lsof/ss
                                  port lookup.
- desktop_job_{start,status,logs,cancel,list}
                                  long-running jobs with persistent
                                  stdout/stderr logs at
                                  ~/.hermes/desktop-jobs/<id>/. On-disk
                                  meta.json is source of truth across daemon
                                  restarts. taskkill /T on Windows so build
                                  trees (npm→node, gradle→java) die fully.
- desktop_copy_directory, _zip, _unzip, _checksum
                                  fs.cp recursive copy; zip/unzip via tar > zip
                                  > PowerShell probe; streamed sha256/sha1/md5.
- desktop_health                  connected client identity, uptime, advertised
                                  tools, last error, recent commands. Answered
                                  by the relay (new GET /desktop/health route)
                                  — does NOT round-trip through the client, so
                                  it remains callable when other tools are
                                  wedged. Heartbeat enriched with
                                  host/platform/arch/version/pid/uptime_ms +
                                  sticky last_error stamped from
                                  DesktopToolRouter.dispatch's catch arm.

Drift-prevention: chat.ts / shell.ts / daemon.ts each maintained their own
copy of the handler map. Replaced with single import from tools/handlerSet.ts
(DESKTOP_HANDLERS + DESKTOP_ADVERTISED_TOOLS). Adding the next tool is now a
one-file change.

Tests + smoke:
- plugin/tests/test_desktop_health.py (3 tests, green) — covers no-client
  200/connected:false, full surface after a desktop.status envelope, 403 on
  non-loopback.
- desktop/scripts/smoke-tools.mjs exercises PowerShell (literal "quotes" and
  $dollar to verify cmd-quote-bypass), process listing, sha256, full job
  lifecycle. PS confirmed pwsh selected, exit 0, output untouched.
- npm run type-check + npm run build clean. Full Python suite still 692
  passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 23:59:47 -04:00
Bailey DixonandClaude Opus 4.7 7117e61afd chore: recapture alpha.1 server-side state into git + merge feature/desktop-tui-mvp
Closes the drift between hermes-host's running hermes-relay and what
main had committed. Three sources converge here:

(1) feature/desktop-tui-mvp (3 commits never merged to main):
    - plugin/relay/channels/tui.py (508 LOC) — THE tui channel
      handler that spawns tui_gateway. Main has been running on the
      server but absent from git for a week.
    - docs/relay-protocol.md (450 LOC) — formal WSS envelope spec.
    - plugin/tests/test_tui_channel.py (520 LOC).
    - scripts/tui-smoke{,-teardown}.sh.
    - plugin/relay/auth.py +4 lines.
    - 4-line addition to plugin/relay/server.py for tui dispatch.

(2) alpha.1 hot-fix drift (live on the server, untracked in git):
    - plugin/relay/channels/desktop.py (424 LOC) — Phase B tool
      command channel: desktop.command/response/status, UUID-future
      correlation, single-client MVP. MERGED with the alpha.6
      DesktopChannel (161 LOC, workspace-awareness) into one
      DesktopHandler class. Backwards-compat alias
      `DesktopChannel = DesktopHandler` preserves alpha.6 import
      sites in server.py.
    - plugin/tools/desktop_tool.py (349 LOC) — registers 5 desktop_*
      tools (read_file/write_file/terminal/search_files/patch) via
      tools.registry. Adopted verbatim from server's working tree.
    - plugin/__init__.py — extended to register desktop tools via the
      plugin context API + matching plugins.enabled documentation.
      Adopted verbatim from server.

(3) Conflict resolution in server.py:
    - bridge.close() → desktop.close() → tui.close() lifecycle
      (both alpha.1's desktop hook and feature branch's tui hook
      called during shutdown).
    - _on_disconnect: server.desktop.detach_ws(ws) +
      server.tui.detach_ws(ws, reason=...) both run on disconnect.
    - alpha.9 /clipboard/inbox endpoint preserved in route table.

After this lands on main, hermes-host can `git checkout -- .`
(its working tree drift now matches main verbatim) and `git pull
origin main --ff-only` cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 21:17:09 -04:00
Bailey DixonandClaude Opus 4.7 fd77da043b feat(bridge): v0.5.0 agent-aware phone status, auto-return, activity log
Three cross-layer additions that close longstanding visibility gaps
between the phone, the relay, and the host-side agent:

  Agent awareness — unattended/screen/credential-lock state
  -----------------------------------------------------------
  PhoneSnapshot gains unattendedEnabled, credentialLockDetected, and
  screenOn fields. PhoneStatusPromptBuilder.buildBridgeLine() now
  appends explicit guidance — e.g. "Unattended access: off — commands
  only land when the screen is already on" or "Unattended access: on,
  but the device has a credential lock — commands will return
  keyguard_blocked and fail."

  BridgeStatusReporter.emitTick() emits a parallel `unattended` group
  in the bridge.status WSS envelope so the host-side `/bridge/status`
  cache (and the `android_phone_status` tool that reads it) sees the
  same state for non-phone frontends like Discord. Push triggers fire
  on toggle flip via ConnectionViewModel so the host cache updates in
  ~1s instead of waiting up to 30s for the periodic tick.

  android_phone_status tool description updated so the LLM proactively
  checks unattended.* fields and warns the user when commands will
  hit keyguard_blocked.

  Auto-return to Hermes-Relay
  ---------------------------
  android_return_to_hermes is an LLM-called tool that the agent
  routinely forgets, leaving the user stranded on Starbucks/Chrome/etc
  after the run. Two safety nets:

    1. Tightened tool descriptions (REQUIRED FINAL STEP, MANDATORY
       CLEANUP framing in android_open_app + android_return_to_hermes).
    2. New BridgeRunTracker singleton — coordinates two completion
       signals: Chat-tab SSE run.completed (fast, phone-only) and a
       12s bridge-idle timer (universal, works for Discord/CLI/web).
       Whichever fires first dispatches a local /return_to_hermes via
       handleLocalCommand. markReturnedToHermes() prevents double-fires
       when the LLM does call return explicitly.

  BridgeCommandHandler tracks foreground-shifting paths (/open_app,
  /send_intent) at respond() and arms/resets the idle timer accordingly.
  Reset hooks at both dispatch start and respond finish so slow-
  executing commands (screenshots, big tree reads) don't eat the idle
  budget.

  Bridge activity log wiring
  --------------------------
  The Activity Log card on the Bridge tab was scaffolded in Phase 3 but
  never wired — recordActivity() existed, the UI rendered the flow, but
  no code ever called it. BridgeCommandHandler now emits a
  BridgeActivityEntry per dispatched command (Success/Failed/Blocked)
  via a new onActivity callback. ConnectionViewModel pipes it through
  to BridgePreferencesRepository.appendEntry. High-frequency polls
  (/ping, /events, /current_app, /screen_hash) suppressed so the log
  shows user-meaningful activity, not noise. Per-route summarizers
  produce natural-looking entries: "tap (540, 1200)", "open_app
  com.starbucks.mobilecard", etc. resultText surfaces error strings
  on Failed/Blocked so users can see WHY without digging through logs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 21:42:23 -04:00
Bailey DixonandClaude Opus 4.7 197b54ddc3 feat(bridge): JIT permission-denied surfacing (v0.4.1)
Make permission-denied bridge tool failures legible to both the LLM and
the human, instead of bubbling up as opaque error strings that the agent
has to pattern-match its way through.

  - plugin/tools/resolve_result.py — new typed-union dataclass hierarchy
    with Found(value) / NotFound(detail) / PermissionDenied(permission,
    reason) variants and a from_bridge_response classifier. Reads both
    the v0.4.1 canonical wire keys (`code` / `permission`) and the
    legacy aliases (`error_code` / `required_permission`) so the rollout
    is forwards/backwards compatible across mixed-version installs.

  - plugin/tools/android_tool.py — Tier C agent-tool wrappers
    (android_search_contacts, android_send_sms, android_call,
    android_location) now run their bridge response through
    _maybe_jit_permission_response. On `code: permission_denied` the
    wrapper upgrades the response to a structured envelope with
    deterministic LLM-readable copy that names the exact Settings
    deep-link path: "User has not granted Contacts permission
    (android.permission.READ_CONTACTS). They can enable it in Settings
    > Apps > Hermes Relay > Permissions. Tool: android_search_contacts."

  - BridgeCommandHandler.respondFromResult now emits the canonical `code`
    + `permission` aliases ALONGSIDE the existing `error_code` +
    `required_permission` fields so both phone APK generations produce
    parseable envelopes. LocalDispatchResult also accepts either spelling.

  - VoiceModeOverlay — new PermissionDeniedChip composable surfaces
    above the mic button when a voice intent fails with
    permission_denied. Tap deep-links to ACTION_APPLICATION_DETAILS_SETTINGS
    for BuildConfig.APPLICATION_ID (so each flavor lands on its own
    package's permission page). VoiceUiState gains permissionDeniedCallout;
    VoiceViewModel.buildPermissionDeniedCallout reads the structured
    `permission` field off result.resultJson and builds copy like "I need
    Contacts to Send SMS here. Tap to open Settings." Callout cleared on
    chip tap and on the next mic-tap (fresh turn). Voice TTS already says
    "Permission needed. {hint}" from the prior session — chip is additive.

  - 17 new Python unit tests in plugin/tests/test_resolve_result.py covering
    the classifier, both wire-key spellings, success passthrough, non-
    permission error passthrough, and JIT upgrades for all four Tier C
    wrappers. Existing 39 Tier-C tests still pass with no regressions.

Docs: ROADMAP.md "Tiered permission checklist with JIT permission errors"
moved to a "shipped on feature/tiered-permissions" pointer; DEVLOG.md
entry for 2026-04-16; CHANGELOG.md [Unreleased] section gains v0.4.1
Bridge fast-follows entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-16 20:52:18 -04:00
Bailey DixonandClaude Opus 4.6 8fe34dc41b fix(v0.4): code-review gap fixes for bridge feature expansion
HIGH
- H1/M5 (relay GET dispatch): merge query params into bridge envelope
  body so phone-side handlers see them. Fixes android_events limit/since
  and android_read_screen include_bounds silently defaulting. Also
  renamed the android_read_screen query key from `bounds` to
  `include_bounds` so the Python tool key matches what the Kotlin
  handler reads.
- H2 (searchNodes walker): hoist nextIndex counter outside the per-window
  loop AND switch the increment predicate to match walk/findNodeById's
  canonical "interesting" filter so IDs from find_nodes match the global
  scheme readAllWindows + findNodeById use. Previously nodeIds from
  find_nodes silently resolved to the wrong node on multi-window screens
  (and could drift even on single-window screens because the increment
  fired on every visit, not only on emitted nodes).
- H3 (scroll leak): recycle rootInActiveWindow node in a try/finally.
- H4 (SMS multipart): cache divideMessage result, don't call twice.
- H5 (voice intents): convert OpenApp/Tap/Scroll/Back/Home builders to
  bridge.command envelope shape matching SendSms. Previously these
  dispatched tool.call envelopes that BridgeCommandHandler silently
  dropped — the intents appeared to succeed but never reached the phone.

MEDIUM
- M1 (/call /send_sms): return 503 when safetyManager is null instead
  of silently bypassing the confirmation modal.
- M2 (EventStore TOCTOU): re-check isStreaming inside the lock in
  append so setStreaming(false) → clear is atomic.
- M4 (longPress nodeId): use reader.findNodeById instead of
  viewIdResourceName match, matching /tap and /scroll semantics and
  the Python schema's advertised contract. Removed the now-unused
  findNodeByResourceId helper.

LOW
- L3 (WakeLockManager.initialize): synchronized check-and-set to
  close the theoretical concurrent-init race.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:46:39 -04:00
Bailey DixonandClaude Opus 4.6 407efc6d63 Merge feature/C1-C4-sideload-perms: location/contacts/call/sms tools
Resolved conflicts:
- ActionExecutor.kt imports merged (added annotation/PendingIntent/
  BroadcastReceiver/IntentFilter/PackageManager/Location/LocationManager/
  Build/ContextCompat/withTimeoutOrNull); Tier C constants joined the
  long_press/parent_walk constants in the companion. Functions
  location/searchContacts/makeCall/sendSms not wrapped in wakeForAction.
- BridgeCommandHandler.kt imports added BuildFlavor + EventStore from
  both branches; /location/search_contacts/call/send_sms cases added
  after /clipboard /media. /call and /send_sms unconditionally call
  safetyManager.awaitConfirmation.
- server.py routes + handlers (additive).
- android_tool.py docstring + function defs + schemas + handlers.

Other C1-C4 files (FeatureFlags BuildFlavor.isSideload helper, sideload
manifest permissions, VoiceBridgeIntentHandlerImpl /send_sms wiring)
auto-merged cleanly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:28:29 -04:00
Bailey DixonandClaude Opus 4.6 937580d072 Merge feature/B1-event-stream: EventStore + events/event_stream tools
Resolved conflicts: server.py handlers + routes (additive), and
android_tool.py (docstring, function defs, schema block, handlers
map). ConflResolution kept all pre-existing tools and appended
android_events + android_event_stream to the end of each list.

EventStore.kt is a new file (no conflict). HermesAccessibilityService
auto-merged (onAccessibilityEvent hook + EventStore.append call).
BridgeCommandHandler auto-merged (/events + /events/stream cases).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:20:48 -04:00
Bailey DixonandClaude Opus 4.6 2da98cd0ab Merge feature/B4-send-intent: raw Intent + broadcast escape hatches
Resolved conflicts:
- ActionExecutor.kt imports merged (ActivityNotFoundException,
  ComponentName, Uri joined with A6's ClipData/Manager/Context and
  A9's Rect, keeping WakeLockManager import). sendIntent/sendBroadcast
  are NOT wrapped in wakeForAction — they're Intent dispatches, not
  gesture strokes.
- server.py handlers (both function defs + route registrations).
- android_tool.py docstring, function defs, schema block (macro +
  clipboard + screen_hash/diff_screen + send_intent + broadcast all
  preserved additively), and handlers map.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:16:46 -04:00
Bailey DixonandClaude Opus 4.6 25970c3e24 Merge feature/A4-describe-node: describe_node + nodeId tap/scroll
Resolved conflicts:
- ActionExecutor.scroll gained optional centerX/centerY params (A4)
  while preserving the A8 WakeLockManager.wakeForAction wrapper.
- BridgeCommandHandler docstring + /tap + /scroll + /describe_node
  cases auto-merged (additive).
- ScreenReader.findNodeById walker ALIGNED to P1 interesting-only
  emission semantics. A4's original pre-order-total counter would
  have broken nodeId round-trip against P1's `w<win>:${out.size}`
  scheme. Rewritten walkForId to replicate P1's `interesting`
  predicate exactly (text/contentDesc/clickable/longClickable/
  scrollable/editable AND non-empty bounds) and share the counter
  GLOBALLY across windows, matching readAllWindows' shared
  `collected` list. Earlier/later windows still contribute to the
  global counter but can only claim the match when currentWindow
  == wantedWindow.
- android_tool.py docstring, function defs, schema block, handler
  map — kept all additive entries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:11:37 -04:00
Bailey DixonandClaude Opus 4.6 f8b0195a91 Merge feature/A5-screen-hash: SHA-256 screen fingerprint + diff tool
Resolved conflicts: android_tool.py (4 spots — docstring, function
defs, schemas block where clipboard_write boundary collided with
screen_hash/diff_screen, and handlers map), server.py (handler +
route), BridgeCommandHandler.kt docstring. ScreenHasher.kt is a new
file so no conflict. Kept all additive entries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:07:20 -04:00
Bailey DixonandClaude Opus 4.6 501d0cca8c Merge feature/A3-find-nodes: filtered searchNodes tool
Resolved conflicts: ScreenReader.kt — kept A3's searchNodes +
searchWalk functions above P1's findNodeBoundsByText (not below).
server.py and android_tool.py conflicts were additive (wave-local
handler + route + tool). Test file count assertion kept as >= 14.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:04:49 -04:00
Bailey DixonandClaude Opus 4.6 51dead05a3 Merge feature/A7-media: media playback broadcast
Resolved conflicts: import lists, BridgeCommandHandler case, server.py
handler+route pair, test count assertion relaxed to >= 14, tool list
docstring. ActionExecutor.mediaControl auto-merged cleanly (not wrapped
in wakeForAction since it's a broadcast, not a gesture).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 22:00:33 -04:00
Bailey DixonandClaude Opus 4.6 c2d9eb3e21 Merge feature/A1-long-press: long-press gesture
Resolved conflicts: ActionExecutor.kt — kept drag() + longPress() both
after swipe, kept Dispatchers import from A6/clipboard side. Upgraded
A1's single-root snapshotRoot() nodeId lookup to P1 multi-window
snapshotAllWindows() pattern (matches tapText). longPress takes a
viewIdResourceName nodeId (distinct from P1 walk IDs) so its own
helper findNodeByResourceId is kept as-is.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:58:55 -04:00
Bailey DixonandClaude Opus 4.6 cccafa912e Merge feature/A2-drag: drag gesture
Resolved conflicts: ActionExecutor.kt — kept Dispatchers/withContext imports (needed by clipboard).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:56:52 -04:00
Bailey DixonandClaude Opus 4.6 c5ff028000 Merge feature/A6-clipboard: clipboard read/write bridge
Resolved conflicts:
- ActionExecutor.kt imports: kept WakeLockManager + Dispatchers
- android_tool.py _SCHEMAS: both android_macro and clipboard entries
- android_tool.py _HANDLERS: added both dispatchers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:53:39 -04:00
Bailey DixonandClaude Opus 4.6 35808e7905 feat(C1-C4): sideload-only tier - location, contacts, call, send_sms
C1 android_location: last-known GPS via LocationManager across
providers, staleness warning, no fresh-fix requests from background.

C2 android_search_contacts: ContactsContract filter + phone number
resolution, privacy-respecting logging.

C3 android_call: auto-dial via ACTION_CALL on sideload, fallback to
ACTION_DIAL on googlePlay / permission-denied. Destructive-verb
confirmation modal gates every call.

C4 android_send_sms: direct SmsManager.sendTextMessage +
sendMultipartTextMessage with PendingIntent result callback (actual
wait for send completion, not fire-and-forget). API-version-aware
SmsManager retrieval. Voice-to-bridge SendSms intent handler now
emits a real /send_sms bridge.command envelope instead of a
malformed tool.call payload; contact->number resolution marked
TODO(C4) with a sketch since fire-and-forget dispatch lacks
response correlation. Destructive-verb confirmation modal gates
every send.

All four permissions added to app/src/sideload/AndroidManifest.xml
only - NOT the main manifest. Flavor gate via
FeatureFlags.BuildFlavor.isSideload in BridgeCommandHandler. Tools
return 'sideload-only' errors on googlePlay devices. Every call/send
logs the full payload to the safety-rails activity log via the
confirmation modal's method + text fields.

Tests: 39 stdlib-unittest cases across
plugin/tests/test_android_{location,search_contacts,call,send_sms}.py
- happy / denied / timeout / schema coverage for each tool. Existing
test_android_tool.py tool-count assertion bumped 14 -> 18.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:45:43 -04:00
Bailey DixonandClaude Opus 4.6 9bab4356aa feat(B1): add android_events + android_event_stream
Real-time AccessibilityEvent ring buffer (500 entries, thread-safe,
throttled to 1 event per type+package per 100ms). Hooks
HermesAccessibilityService.onAccessibilityEvent when streaming is
enabled (off by default — explicit opt-in via android_event_stream).
Two tools: android_events(limit, since) polls recent entries,
android_event_stream(enabled) toggles capture and clears buffer on
disable. Signal-rich event types only: click, text changed, window
content/state changed, scroll.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:39:50 -04:00
Bailey DixonandClaude Opus 4.6 1ec7a2e8ee feat(B4): add android_send_intent and android_broadcast
Power-user escape hatch for launching arbitrary Activities and
sending broadcasts. Both take action/data/package/extras; send_intent
also accepts component + category. FLAG_ACTIVITY_NEW_TASK added for
Activity launches. Gated through BridgeSafetyManager package
blocklist — a blocklisted target package refuses. ActivityNotFound
and SecurityException are soft-failed into ActionResult errors,
not crashes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:20:58 -04:00
Bailey DixonandClaude Opus 4.6 1e5ad146f4 feat(A4): android_describe_node + wire nodeId for /tap and /scroll
New describe_node tool returns the full property bag (bounds, classes,
text, state flags, hintText, viewIdResourceName) for a nodeId from
the P1 stable-ID scheme. Also resolves that P1 emitted nodeIds but
/tap and /scroll ignored them — BridgeCommandHandler now parses
nodeId, resolves via ScreenReader.findNodeById, dispatches against
the node's bounds center. Closes the end-to-end nodeId contract.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:20:36 -04:00
Bailey DixonandClaude Opus 4.6 3543e6679b feat(A5): add android_screen_hash + android_diff_screen
Cheap change detection for navigation loops. SHA-256 over per-node
fingerprints (className + text + contentDescription + bounds +
viewIdResourceName) across the full multi-window accessibility
tree. diff_screen reports changed + new hash + node_count in one
call so the agent can update its reference without an extra
round-trip. ~100x cheaper than re-reading the full tree for
'did anything change?' polling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:19:07 -04:00
Bailey DixonandClaude Opus 4.6 9c4acbcf0f feat(A6): add android_clipboard_read and android_clipboard_write
Clipboard bridge via ClipboardManager. Label ClipData as "hermes" so
other apps can see the source. Handle empty clipboard as empty
string (not error). On API 31+ the system shows a privacy toast on
write, documented in the tool description.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:18:30 -04:00
Bailey DixonandClaude Opus 4.6 e2ba96b192 feat(A10): add android_macro batched workflow tool
Pure-Python orchestrator that dispatches to other android_* tools
in order, stopping on first failure. Returns a structured trace
with completed-count, per-step results, and error details.
Complements android_navigate (vision-driven) for known workflows
where the steps are deterministic and batching cuts round-trips.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:18:19 -04:00
Bailey DixonandClaude Opus 4.6 dd61db78d9 feat(A7): add android_media system-wide playback control
Control whatever media app is currently playing (Spotify, YouTube
Music, Pocket Casts, etc.) via ACTION_MEDIA_BUTTON broadcast with
KEYCODE_MEDIA_* keycodes. DOWN+UP ordered broadcast pair. Actions:
play, pause, toggle, next, previous. No special permissions — media
button is a system-wide interface every compliant player handles.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:17:47 -04:00
Bailey DixonandClaude Opus 4.6 164cdfb5c1 feat(A1): add android_long_press gesture tool
Long-press at coordinates or on an accessibility node via
ACTION_LONG_CLICK / GestureDescription. Wrapped in WakeLockManager
wake scope and BridgeSafetyManager package gate. Duration clamped
to 100-3000ms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:17:26 -04:00
Bailey DixonandClaude Opus 4.6 83bfa23f6b feat(A3): add android_find_nodes filtered search
Targeted node search by text/class/clickable criteria across all
accessibility windows. Avoids dumping the full tree for simple
existence queries. Reuses the P1 multi-window walker and emits
nodes with stable w<N>:<M> IDs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:17:16 -04:00
Bailey DixonandClaude Opus 4.6 52843410d3 feat(A2): add android_drag gesture tool
Drag gesture from (startX, startY) to (endX, endY) via a single-stroke
GestureDescription. Wrapped in WakeLockManager wake scope and
BridgeSafetyManager package gate. Duration clamped to 100-3000ms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 21:16:07 -04:00
Bailey DixonandClaude Opus 4.6 7f7d92955a docs: rename misleading 'Bridge pairing code' → 'Manual pairing code (fallback)'
Three coordinated text-only edits across plugin tool docstring, in-app
Settings card, and user-docs prose. No logic changes, no behavior changes.

The in-app card and matching configuration.md doc described the locally-
generated 6-char code as the "Phase 3 bridge feature" approval mechanism.
That was speculative when written and turned out to be wrong: Phase 3's
bridge gate is the master toggle in BridgeViewModel + the foreground
service notification, not an in-app code-approval flow. The locally-
generated code is actually the auth fallback path in AuthManager.
authenticate() — used when no QR-issued code is present, requiring the
host to pre-register the matching code with the relay. Renamed accordingly.

android_setup tool docstring + parameter rename:
- First line now says "FALLBACK helper" so LLMs reading the tool registry
  get the right signal. Was previously "Configure the Android bridge to
  point at the unified Hermes-Relay" which sounded canonical
- Parameter renamed `pairing_code` → `bridge_session_token`. The function
  stores the value in ANDROID_BRIDGE_TOKEN which is sent as the bearer
  token on every bridge HTTP call — it expects a long-lived session token,
  not a one-shot pairing code. The old name was misleading
- Clarified user_instructions: stop telling users to "scan the QR with this
  pairing code" (mixing flows). Just say "run hermes-pair, scan the QR"

Phase 3 status table cleanup:
- user-docs/guide/index.md and user-docs/reference/relay-server.md status
  tables previously said "Phase 3" for Bridge. Phase 3 is now in production
  on the sideload track (with safety rails, Tier 5 master toggle, accessibility
  service, MediaProjection consent flow). Updated to "Beta (sideload track)".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 21:49:42 -04:00
Bailey DixonandClaude Opus 4.6 990870cee3 feat(phase3-status): dynamic phone-status prompt + android_phone_status tool + bridge UI hardening
Three-layer feature so the agent finally knows what the phone can do, plus
follow-up fixes discovered while smoke-testing the merged Phase 3 build.

Layer 1 — phone-side dynamic system prompt (replaces the static one-liner):
- New PhoneStatusPromptBuilder.kt builds a transparent block from real
  bridge/permission state, capped under 100 words, returns null when
  everything is off (no empty system messages)
- 4 new sub-toggles in ConnectionViewModel: bridge state, current app,
  battery, safety status. Privacy-sensitive ones (current_app, battery)
  default OFF
- ChatSettingsScreen gains a live preview Card showing exactly what
  will be sent on the next message
- ChatViewModel deletes APP_CONTEXT_PROMPT, calls the builder via a
  guarded-reflection capturePhoneSnapshot() helper

Layer 2 — relay backend (GET /bridge/status, loopback only):
- BridgeStatusReporter expanded to push the full nested device/bridge/safety
  contract every 30s; new pushNow() method called on master toggle flips
- BridgeHandler caches latest_status + last_seen_at
- New handle_bridge_status route mirrors /pairing/register loopback gate
- 7 new stdlib unittest tests in test_bridge_status.py — all green

Layer 2 — symmetric trio (mirrors the pair feature):
- New plugin/tools/android_phone_status.py — Hermes tool, stdlib only
- New plugin/status.py + hermes-status shim — operator CLI with --json/--port,
  three exit codes (0/1/2 for connected/relay-down/no-phone)
- New skills/devops/hermes-relay-status/SKILL.md — slash command
- install.sh + uninstall.sh updated for the second shim
- 19 new stdlib unittest tests — all green

Bridge UI hardening (master gate, MediaProjection, labels):
- HermesAccessibilityService now feeds cachedMasterEnabled itself via a
  service-scoped DataStore observer. The previous push-from-outside
  pattern was never wired and the cache stayed false forever, 403'ing
  every command except /ping and /current_app
- MainActivity registers an ActivityResultLauncher for MediaProjection;
  new ScreenCaptureRequester process-singleton bridges non-Activity
  callers (BridgeViewModel.requestScreenCapture()). Bridge tab's Screen
  Capture row is now tappable instead of inert
- BridgeViewModel.testNotificationListener() + an onTestNotificationListener
  lambda through BridgePermissionChecklist for parity with the other rows
- Sideload flavor strings: app_name → "Hermes Dev", a11y_service_label →
  "Hermes-Bridge Dev", notification_companion_label → "Hermes Dev …".
  googlePlay's a11y_service_label flipped to "Hermes-Bridge" (with hyphen)
  for consistency. Disambiguates side-by-side installs in launcher /
  recents / Settings → Apps
- BridgeForegroundService: Intent.flags = … → addFlags(…) (the property
  setter form fails because Intent.setFlags returns Intent, not void)
- BridgeViewModel: removed deprecated StateFlow.distinctUntilChanged()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-12 20:43:58 -04:00