Compare commits

..
Author SHA1 Message Date
Bailey Dixon 5100c524f6 Merge pull request #351 from Codename-11/dev
release: Desktop beta.2 and Server 1.8.0
2026-08-14 15:04:03 -04:00
Bailey Dixon c609ae867f Merge pull request #352 from Codename-11/chore/backmerge-desktop-beta1
chore: back-merge Desktop beta.1 release history
2026-08-14 14:49:23 -04:00
Bailey Dixon 107f8c7720 chore: merge desktop beta.1 release history into dev 2026-08-14 14:49:08 -04:00
Bailey Dixon 8963e4fafd Merge pull request #350 from Codename-11/release/server-1.8.0
release(server): server-v1.8.0
2026-08-14 14:46:57 -04:00
Bailey Dixon 5d9624e2ef release(server): server-v1.8.0 2026-08-14 14:35:01 -04:00
Bailey Dixon 4b063d3fd7 Merge pull request #349 from Codename-11/release/desktop-0.4.0-beta.2
release(desktop): desktop-v0.4.0-beta.2
2026-08-14 12:56:04 -04:00
Bailey Dixon 9b9d7b654c release(desktop): desktop-v0.4.0-beta.2 2026-08-14 12:45:56 -04:00
Bailey Dixon a26e17e72c Merge pull request #348 from Codename-11/fix/cua-windows-health-compat
feat(desktop): enhance activity and control diagnostics
2026-08-14 12:38:12 -04:00
Bailey Dixon a3a6a9bb13 fix(desktop): satisfy tray release lint 2026-08-14 12:36:46 -04:00
Bailey Dixon 169bd09559 merge: sync desktop activity work with dev
# Conflicts:
#	CHANGELOG.md
2026-08-14 11:32:56 -04:00
Bailey Dixon 45d631e7ac feat(desktop): enhance activity and control diagnostics 2026-08-14 11:30:36 -04:00
Bailey Dixon eb6a6c95d2 merge: integrate official desktop relay plugin 2026-08-14 07:58:10 -04:00
Bailey Dixon 3b102663c2 feat(plugin): add official desktop relay surface 2026-08-14 07:56:26 -04:00
Bailey Dixon 0e5fc4c606 Merge branch 'fix/android-proactive-thread-entry' into dev 2026-08-14 07:50:07 -04:00
Bailey Dixon c3189f2cbb fix(android): open proactive messages as threads 2026-08-14 07:49:39 -04:00
Bailey Dixon 0d6c3bd6b0 Merge pull request #346 from Codename-11/dev
release(desktop): desktop-v0.4.0-beta.1
2026-08-13 21:18:41 -04:00
Bailey Dixon 06d88ad40a Merge pull request #345 from Codename-11/release/desktop-0.4.0-beta.1
release(desktop): desktop-v0.4.0-beta.1
2026-08-13 21:11:57 -04:00
Bailey Dixon 8ae5b3fbc2 release(desktop): desktop-v0.4.0-beta.1 2026-08-13 21:04:07 -04:00
Bailey Dixon 39b7a8f108 Merge pull request #344 from Codename-11/fix/desktop-updater-cua-hardening
feat(desktop): adopt CUA as primary control backend
2026-08-13 20:57:50 -04:00
Bailey Dixon af6e167692 fix(desktop): normalize Windows installer paths 2026-08-13 20:51:17 -04:00
Bailey Dixon 274bd6ae98 fix(desktop): honor CUA health schema 2026-08-13 20:45:58 -04:00
Bailey Dixon 9fc55b379a fix(desktop): clarify CUA readiness fallback 2026-08-13 20:34:45 -04:00
Bailey Dixon 559a0ffdc8 feat(desktop): make CUA the primary control backend 2026-08-13 20:16:03 -04:00
Bailey Dixon 75bcd9180f feat(desktop): add optional CUA control engine 2026-08-13 19:33:11 -04:00
Bailey Dixon 9c995a443d fix(desktop): harden bundle updates 2026-08-13 19:06:25 -04:00
80 changed files with 7221 additions and 237 deletions
+28
View File
@@ -20,6 +20,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this
- **Android keeps profile management and retained automation truthful.** Custom Endpoint list and mutation routes now follow the selected Hermes profile, while completed one-shot cron jobs show their retained outcome and expose only valid Runs/Delete actions.
- **Android and Relay recover more generated media reliably.** Android accepts upstream-valid wrapped, punctuated, adjacent, spaced, and Windows `MEDIA:` markers without consuming fenced examples, and Relay translates Docker-visible workspace, home, cache, and configured-mount paths before applying its existing credential, sandbox, and size checks.
## [1.8.0] - 2026-08-14
### Added
- **Official Hermes Desktop can surface Relay through its supported runtime Plugin SDK.** The unified plugin package now includes an opt-in, profile-scoped Desktop pane for Relay status, paired devices, bridge activity, media, pairing, revocation, and remote-access management. Loading, startup, reconnects, profile changes, and updates never open it; only labeled sidebar, status-bar, or command-palette actions register and reveal the movable native pane.
## [0.4.0-beta.2] - 2026-08-14
### Added
- **Desktop Activity now keeps inspectable local evidence.** Commands, files, devices, connection lifecycle, and computer control share a truthful event stepper with dedicated failure details; screenshot events can retain bounded local PNG evidence and open it in a larger borderless viewer. Settings controls retention as Off, 1 day, 7 days, or 30 days and shows local file usage.
### Fixed
- **Tunnel state stays responsive through interruption and retry.** The CLI UI distinguishes connected, reconnecting, and stopped states, exposes retry attempt/timing and a Retry now action, records connection failures and recovery in Activity, and shows compact connection cards only while the main UI is hidden.
- **Windows CUA readiness no longer depends on the flaky whole-desktop health scan.** Hermes-Relay verifies the canonical runtime, manifest, required tools, daemon, and safe permission mode before starting structured sessions, while accessibility health remains an explicit CLI/UI diagnostic that can be rechecked without forcing the compatibility backend. This temporary workaround is scoped to the upstream fixed-timeout issue and keeps individual actions fail-closed.
## [0.4.0-beta.1] - 2026-08-14
### Added
- **CUA Driver is the preferred Windows structured-control engine.** New local settings prefer a verified CUA runtime for window-targeted background actions, fresh snapshot tokens, and optional per-session animated agent cursors without moving the physical pointer; Windows Input is the explicit compatibility backend and backend choice is fixed for each control session. Full-display observation remains on the read-only system capture path. CLI and UI can explicitly install, check, or update the canonical CUA package after verifying the upstream release manifest and installer checksum; nothing is bundled or updated automatically, driver telemetry stays off for Hermes sessions, and activity records contain only bounded, redacted control metadata.
### Fixed
- **Windows bundle updates fail closed when installed processes retain a binary lock.** Setup waits for the invoking CLI, quiesces the tray and its short-lived CLI children, checks every payload extraction before writing release metadata, preserves custom install directories, and returns a failure instead of reporting a mixed-version installation.
- **CUA readiness follows the published driver contract.** Hermes accepts the documented `ok` health state, distinguishes an installed-but-degraded runtime from a missing installation, and constructs trusted Windows installer paths consistently across verification environments.
## [1.7.0] - 2026-08-13
### Added
+9 -12
View File
@@ -1,30 +1,27 @@
# Hermes-Relay CLI v__VERSION__
**Release Date:** 2026-08-13
**Release Date:** 2026-08-14
This alpha makes the compact **Hermes-Relay CLI UI** responsive during connection changes, expands host and capability management, and presents live route security and diagnostics without turning the tray into a full desktop client.
This beta makes connection recovery and Activity evidence inspectable in the compact management UI, and keeps the preferred CUA control engine usable when its upstream whole-desktop accessibility probe times out.
**Experimental phase.** Assets remain unsigned, so Windows SmartScreen and macOS Gatekeeper may warn on first launch. Standalone CLI binaries ship for Windows x64, Linux x64, and macOS x64/arm64; the management tray is Windows-only.
**Beta phase.** Assets remain unsigned, so Windows SmartScreen and macOS Gatekeeper may warn on first launch. Standalone CLI binaries ship for Windows x64, Linux x64, and macOS x64/arm64; the management UI is Windows-only.
## What's changed
### Added
- **Host management hub.** Each paired Hermes host has local identity, pairing facts, access and capability controls, authorized-client revocation, re-pairing, and guarded removal.
- **Evidence-first activity.** Overview shows the latest three events and detailed views retain bounded command, output, exit, duration, and truncation evidence.
- **Live route details.** The Agent-to-PC path shows route type, encryption state, endpoint, packet motion, and a connection test with reachability and latency.
- **Inspectable Activity evidence.** Commands, files, device work, connection lifecycle, and computer control share a consistent event stepper with dedicated failure details.
- **Optional screenshot retention.** Screenshot events can keep bounded local PNG evidence for Off, 1 day, 7 days, or 30 days and open it in a larger borderless viewer. Evidence stays outside the JSON activity log.
### Changed
- **Connection lifecycle stays responsive.** Connect, disconnect, and snapshot work run outside the UI thread with immediate transition feedback and single-flight live polling.
- **Access presets are explicit.** Restricted, Ask Every Time, Standard, Full Access, and Custom map visibly onto individual command, file, screen/input, and hardware capabilities.
- **Tailscale is the recommended remote route.** Direct TLS and Hermes Secure Link remain supported; Hermes Reach is marked experimental and stays below supported routes.
- **Connection state is live and actionable.** The UI distinguishes connected, reconnecting, and stopped states, shows retry timing, and offers Retry now without freezing the popup.
- **Connection notices stay out of the way.** Compact connect, disconnect, and reconnect cards appear only while the main management UI is hidden.
### Fixed
- **Legacy LAN and Tailscale routes no longer appear as Custom VPN.** Route testing infers generic saved roles from the endpoint and reports the correct network path.
- **PowerShell output remains complete.** Scalar, pipeline, JSON, native output, errors, exit status, and truncation metadata return reliably through desktop RPC.
- **Tray placement follows the real notification area.** Responsive geometry uses the tray monitor and DPI and remains anchored above the icon.
- **CUA readiness no longer depends on the flaky global accessibility scan.** Hermes verifies the canonical runtime, required tools, daemon, and safe permission mode before structured control; explicit accessibility health remains available for diagnosis and individual actions still fail closed.
- **Connection errors retain useful context.** Activity records bounded retry and recovery evidence without flooding one event per backoff attempt.
## Install
+6 -12
View File
@@ -1,8 +1,8 @@
# Hermes-Relay-Server v__VERSION__
**Release Date:** August 13, 2026
**Release Date:** August 14, 2026
This release adds an operator-owned secure ingress path, promotes Tailscale as the easiest supported remote route, and introduces Hermes Reach as a disabled-by-default experimental broker for outbound-only environments.
This release adds an official, opt-in Relay pane for Hermes Desktop through the supported runtime Plugin SDK. It keeps Relay management profile-scoped and user-invoked without opening a pane during startup, reconnects, profile changes, or plugin updates.
Standard chat, session history, and Vanilla Hermes voice remain upstream-owned and do not require this plugin.
@@ -10,19 +10,13 @@ Standard chat, session history, and Vanilla Hermes voice remain upstream-owned a
### Added
- **Hermes Secure Link.** A pinned-TLS ingress can expose Relay, API, and Dashboard namespaces through one self-hosted endpoint while retaining each service's native authentication.
- **Experimental Hermes Reach.** An optional self-hosted rendezvous broker forwards opaque inner Secure Link TLS records, with hashed credentials, replay protection, bounded streams, persistence, and revocation.
- **Remote-access status.** Dashboard and pairing metadata distinguish reachability from transport protection and show supported services without exposing certificate pins.
- **Official Hermes Desktop pane.** The unified plugin package registers a movable native pane for Relay status, paired devices, bridge activity, media, pairing, revocation, and remote-access management.
- **Explicit entry points.** Labeled sidebar, status-bar, and command-palette actions register and reveal the pane lazily; repeated opens reuse the same surface.
- **Profile-scoped state.** Cached Relay state follows the active Hermes profile and is disposed cleanly when the plugin unloads.
### Changed
- **Tailscale is recommended for remote access.** Tailscale Serve remains the simplest supported path; direct TLS and Secure Link are self-hosted alternatives, while Reach stays advanced and experimental.
- **Pairing carries reviewed transport trust.** QR payloads include the Secure Link endpoint and pin before the first network request; rotation requires explicit re-pairing.
### Fixed
- **Route credentials remain scoped and revocable.** Reach credentials are issued only through trusted Secure Link ingress, replaced atomically per Relay session, bounded by session expiry, and removed on revocation.
- **Proxy namespaces preserve credential isolation.** API and Dashboard headers, cookies, redirects, methods, sizes, timeouts, and loopback authority are constrained independently.
- **Plugin loading stays passive.** Loading, startup, reconnects, profile changes, and updates never reveal the pane or perform pane-owned network work.
## Install / update
+8
View File
@@ -219,6 +219,14 @@ It pairs against the **same relay and credential store** as the Android app —
On Windows, the default installer adds the optional compact **Hermes-Relay CLI UI** tray popup for host selection and pairing, connection and daemon state, per-host Ask/Trusted/Full Access, local grant dialogs, authorized-client revocation, activity, settings, and emergency stop. It is a management surface only—chat, TUI, plugins, voice, and agent sessions remain CLI/upstream concerns.
Structured Windows computer control prefers a compatible local CUA Driver
runtime for window-targeted background actions and virtual per-session agent
cursors. It remains behind Hermes host policy, grants, targeting, audit, and
emergency stop; Windows input is an explicit compatibility backend. CUA is not
bundled or updated automatically, but the local CLI/UI can explicitly install,
check, or update its verified canonical package. It is never exposed as a raw
remote tool surface. See the [desktop tools guide](https://hermes-relay.dev/docs/desktop/tools.html#computer-use-engines).
- **Docs:** [CLI guide](https://hermes-relay.dev/docs/desktop/) · [`desktop/README.md`](desktop/README.md)
- **AI-agent setup recipe:** `/hermes-relay-desktop-setup`
+42 -2
View File
@@ -6,6 +6,27 @@ For shipped work, see `DEVLOG.md`. For architectural decisions, see `docs/decisi
---
## Certify the official Desktop Relay plugin
The unified `plugin/desktop/plugin.js` implementation is covered by source-level
SDK contract, packaging, explicit-open, no-auto-open, close, unload, and profile
cache-isolation tests. A physical official Hermes Desktop session is still
required before calling the UX live-certified:
- Test default and named local profiles, ordinary authenticated remote mode,
and SSH mode with differently named local/remote profile mapping.
- In two full app windows, prove enabling, registration, explicit open,
requests, close/reopen, hot reload, and disable/unload remain window-local.
- Prove startup, reconnect, profile change, layout restore/reset, update, and
background events never open or focus Relay.
- Drag and dock the pane across native zones, close it, reopen it from all three
labeled actions, and verify no private-hook fallback is needed.
- Exercise Relay running/unreachable, zero/one/multiple devices, pairing,
revocation, bridge activity, media, remote access, and renderer error logging
without exposing credentials, pairing payloads, filesystem paths, or tokens.
---
## Structured desktop hardware capabilities
Structured access and per-host USB policy now ship with typed, serial-bound ADB
@@ -935,7 +956,7 @@ Phase 1 (end-to-end spine) shipped on `Codename-11/phone-platform` — `send_mes
- **Outbound buffering — ✅ relay-side DONE (2026-06-29).** `ProactiveChannel.push()` now queues agent→phone messages in a bounded deque (drop-oldest, 24 h TTL) when no phone is subscribed and returns `{queued: true}` (not 503); `_flush_outbound` delivers FIFO on the next subscribe (stale pruned). Inspect/cancel via `peek_outbound`/`cancel_outbound` + loopback `GET`/`DELETE /phone/outbound`. **UI surfacing of the queued state** (host-side, since the queue exists while the phone is OFFLINE): (a) ✅ **desktop CLI `relay queue` / `relay queue --clear` / `--cancel <id>` DONE (2026-06-29)** over the new endpoints (loopback-only — run on the relay host); a dashboard Relay-tab view is the optional GUI equivalent; (b) **remaining** — in the threaded agent surface, mark messages that arrived-while-away, and show the user's OWN pending replies (the Phase 3 reply queue) with a sending/Cancel affordance — that's where phone-side "queued + cancel" belongs.
- **Threads surface (unified-session model — see ADR 12 + the Refinement above).** Build order, each shippable: **(1)** source tags in the session drawer (`source=phone` → clean **Threads** chip + thread-spool icon, NOT a phone glyph) — also delivers the "source attribution in Chat" goal; **(2)** open a Thread in Chat from its session-store history (reuse the existing message-history path); **(3)** route the live `proactive` push into the session view + notification + unread, demoting `ProactiveInboxStore` to cache/outbox; **(4)** reply from the Chat composer via `proactive.reply` + persist the user turn + local `Sending/Queued/Failed` status — **MVP**; **(5)** a **Threads capability row** in the best-path UI + a pinned **Threads** entry atop the drawer (thread-spool icon, shown only when relay-paired + opted-in) + retire `HermesInboxScreen`, re-point the notification deep-link + Settings "View messages"; **(6)** outbox/retry on reconnect; **(7)** relay `proactive.reply.ack` (honest Delivered) + `proactive.cancel`; **(8)** multi-thread `chat_id` (named/project Threads). **Verify gate before (1):** confirm the app's session-list/history path surfaces a `source=phone` session cleanly (upstream `session.list` returns all sources flat, so it should — but check whether the drawer currently filters it out). Honesty call: do NOT show "Delivered" until (7) lands (can't confirm it client-side before the ack).
- **Status (2026-06-29, implemented UNBUILT — verify in Studio):** **CODE-COMPLETE on `dev`:** slice **1** (drawer source tags + `ThreadSpoolGlyph` + Threads filter), **2** (open a Thread from history — free via the existing `loadSessionHistory` path), **3-parse** (carry `reply_to` on `ProactiveMessage`), **4** (composer reply in a `source=phone` session routes over `proactive.reply`; `MessageDeliveryStatus` SENDING→DELIVERED/FAILED on the bubble), **5** (Threads capability row in `SessionPathCard` + `threadsCapabilityActive` drawer wiring), **7** (relay `proactive.reply.ack` + `proactive.cancel` — 25/25 `unittest` green — and client ack handling). **DONE since (2026-06-29, built + on phone):** live **in-thread reply rendering** (an agent reply lands in the open Thread as an ASSISTANT bubble, suppressing the notification/inbox — `injectIntoThread`); **user-created named Threads** ("+ New Thread"); **retire `HermesInboxScreen`** (deleted; route + nav removed; notification tap + Settings "View messages" re-pointed to Chat; surface renamed "Hermes messages" → **"Threads"**); relay slice-7 ack/cancel **DEPLOYED** to the host so **"Delivered" is live**. **DEFERRED (reasons):** per-session **unread badge**; **outbox/retry** (needs multiplexer connection-state); **exact-Thread deep-link** from the notification (opens Chat today, not the specific thread — needs select-session-on-entry); **remove the now-orphaned `ProactiveInboxStore`** (viewer-less write-only log); **agent-initiated** named Threads (upstream `send_message` thread param). On-device verifies for the create-flow: fresh-`chat_id` auto-create, the `…:dm:<chat_id>` id form, `renameSession` on a phone session.
- **Status (2026-06-29, implemented UNBUILT — verify in Studio):** **CODE-COMPLETE on `dev`:** slice **1** (drawer source tags + `ThreadSpoolGlyph` + Threads filter), **2** (open a Thread from history — free via the existing `loadSessionHistory` path), **3-parse** (carry `reply_to` on `ProactiveMessage`), **4** (composer reply in a `source=phone` session routes over `proactive.reply`; `MessageDeliveryStatus` SENDING→DELIVERED/FAILED on the bubble), **5** (Threads capability row in `SessionPathCard` + `threadsCapabilityActive` drawer wiring), **7** (relay `proactive.reply.ack` + `proactive.cancel` — 25/25 `unittest` green — and client ack handling). **DONE since (2026-06-29, built + on phone):** live **in-thread reply rendering**; **user-created named Threads** ("+ New Thread"); **retire `HermesInboxScreen`**; relay slice-7 ack/cancel **DEPLOYED** to the host so **"Delivered" is live**. **DONE (2026-08-14):** notification taps survive cold start and open the exact `chat_id`; agent-initiated outbound messages appear as connection-scoped provisional Threads backed by the bounded proactive store, then promote to the real `source=phone` session after the first reply. **DEFERRED:** per-session **unread badge**; **outbox/retry** (needs multiplexer connection-state); **agent-initiated** named Threads (upstream `send_message` thread param). On-device verifies for the create-flow: fresh-`chat_id` auto-create, the `…:dm:<chat_id>` id form, `renameSession` on a phone session.
- **User-created Threads (slice 8, Discord-style) — CODE-COMPLETE on `dev` (built + installed 2026-06-29; on-device behavior pending).** "+ New Thread" in the drawer's Threads view → name dialog → `ChatViewModel.startNewThread` mints a fresh `chat_id`; the first composer message opens it over `proactive.reply` (gateway auto-creates the `source=phone` session) → `switchToCreatedThread` polls + switches to the real session + applies the name. Existing-thread replies route by the `chat_id` parsed from the session id (`…:dm:<chat_id>`; opaque id → home fallback). **On-device verifies:** (1) a fresh-`chat_id` no-`reply_to` inbound creates a new `source=phone` session; (2) the phone session id carries the `…:dm:<chat_id>` form the client parses; (3) `renameSession` titles a phone session. **Remaining slice-8:** AGENT-initiated named Threads (the upstream `send_message` thread/chat_id param so the agent can open its own named Threads).
- **`chat_id` not exposed by `/api/sessions` (root cause of the 2026-06-29 on-device create-flow bugs — fixed client-side).** Confirmed on the host: a phone session's `id` is a timestamp (e.g. `20260629_204755_94f391d6`); the real `chat_id` lives in the `session_key` (`agent:main:phone:dm:<chat_id>`) and a `chat_id` column — but `/api/sessions` returns **neither `chat_id` nor `session_key`**, only `source` + the timestamp `id`. So the client could not map a session ↔ its `chat_id`, which broke create-thread switch/rename + reply routing + in-thread injection. **Client workaround shipped:** find a created thread by session-list **diff** (the new `source=phone` session), keep an in-memory `sessionId → chat_id` map (learned at creation + from incoming `phone.message`s) for reply routing, and inject by source (+ learned chat_id) rather than a parsed id. **Limitation:** for a thread the app didn't create *this* session (agent-created, another device, or after an app restart) `chat_id` is unknown until a message arrives while viewing it → its replies fall back to the home channel until then. **RESOLVED via the plugin (2026-06-29, per upstream-or-plugin policy):** the relay now exposes `GET /phone/threads` (`plugin/relay/session_store.py` reads the gateway store read-only → `[{session_id, chat_id, title}]`; `server.py` `handle_phone_threads`, bearer for the app / loopback for diag; 5 unit tests). The app (`RelayHttpClient.fetchPhoneThreads` → `ConnectionViewModel.phoneThreadChatIds` on every `auth.ok` → `ChatViewModel.seedThreadChatIds`, authoritative over the learned map) now routes replies correctly for **any** Thread — incl. ones it didn't create + after restart. Deployed + verified live. **Still-nice-to-have (lower priority): the upstream PR** to add `chat_id`/`session_key` to `/api/sessions` (the standard-path proper fix; the relay route then becomes redundant + the client prefers upstream when present).
- **Threads as named/project conversations (Discord-parity — folds into multi-thread #8).** A stable *named* `chat_id` per project = a persistent, agent-reachable project Thread (Discord named-thread parity for "persist a session for a project"). Enables: the agent **opening** a new named Thread for a background job/topic (a relay/gateway "open thread" affordance + a `send_message`-adjacent tool); cron/job updates landing in their own Thread; and replying to a Thread from any surface (desktop CLI / dashboard) since it is just a gateway session. Also evaluate per-Thread profile binding (a project Thread uses the "work" profile — ties to profile=contact).
@@ -964,7 +985,7 @@ The gateway-platform model is the *correct + sufficient architecture* (the phone
session store; the relay buffer is only the live/offline-delivery layer, not a
parallel history database.
- **Profile = contact mapping (new idea, fold in).** Multiple Hermes **profiles** (distinct agent personas/configs) could each be a distinct thread *source*/"contact" — DMing different agents. Maps cleanly onto the per-thread `chat_id` + source-attribution work; lets the app feel like a contact list of agents.
- **Per-thread notification controls + deep-link (Discord-parity affordances).** Per-thread notification channels, mute/DND/quiet-hours (Phase 3 partially), and a notification that **deep-links into the exact thread** (tap → land in that conversation) so dipping in/out while multitasking is frictionless.
- **Per-thread notification controls (Discord-parity affordances).** Exact-thread notification deep-linking is shipped. Remaining: per-thread notification channels and mute/DND/quiet-hours controls (Phase 3 partially).
- **Agent-initiated rich content.** Agent → phone thread with **images/cards** (relay media infra + `InboundAttachmentCard`/`HermesCardBubble` already exist on the chat side — reuse). Inbound (phone → agent) reply media stays deferred (text-first), but outbound rich content is low-cost parity.
- **In-thread "agent is working" indicator.** A typing/working state in the thread while the agent thinks/runs tools (Discord typing-dots parity) — the chat surface already has thinking indicators to reuse.
@@ -1230,6 +1251,25 @@ When the answer becomes clearer, this section becomes either an ADR in `docs/dec
## Smaller deferred items
- **Certify the preferred CUA Driver backend (ADR 56).** The canonical-runtime
probe, bounded adapter, server-owned control-session envelope, per-session
grant state, local engine/status controls, telemetry-off process environment,
and Hermes snapshot-token primitives now exist. Before graduating the engine,
finish end-to-end enforcement of app/display/folder scopes and sensitive
pixel/accessibility denial or redaction, harden the grant-bridge ACL and nonce
lifecycle, and complete live Windows certification proving the physical cursor and
foreground app stay unchanged, stale or cross-window tokens fail, two remote
control sessions receive isolated animated cursors, and foreground escalation
never happens implicitly. Exercise revoke on grant expiry, disconnect,
re-pair, policy downgrade, emergency stop, Windows-session change, and daemon
shutdown. The explicit local CUA install/update surface now verifies upstream
manifest identity and installer SHA-256; add Windows publisher verification
when upstream signs the installer. Keep raw CUA tools, configuration,
recording, replay, and JavaScript outside the remote agent surface.
Remove the temporary Windows readiness/health split once
[trycua/cua#3103](https://github.com/trycua/cua/issues/3103) ships in the
supported CUA range; restore a mandatory health gate only if the upstream
probe is bounded and cannot leave UI Automation falsely busy.
- **MediaProjection consent flow** — wired in MainActivity (2026-04-12), needs end-to-end test on a real device
- **WorkManager upgrade for auto-disable timer** — currently a coroutine `Job + delay()` in `AutoDisableWorker.kt`; documented at top of file. Upgrade when androidx.work joins the classpath
- **Wave 3 voice-bridge multi-turn confirmation** — currently a 5s TTS countdown with cancel; conversational confirmation is the follow-up
@@ -34,6 +34,8 @@ data class ProactiveInboxEntry(
* field).
*/
val chatId: String? = null,
/** Owning saved connection. Null only for entries written by older builds. */
val connectionId: String? = null,
)
private val Context.proactiveInboxStore: DataStore<Preferences> by
@@ -49,10 +51,10 @@ private const val MAX_ENTRIES = 100
* newest-first, deduped by id (so a re-delivered message doesn't double up), and
* capped at [MAX_ENTRIES]. Survives app restart.
*
* Demoted (2026-06-29): the agent conversation now lives as a Thread in Chat (the
* gateway session is the durable history), so the in-app inbox view is retired.
* This store is only fed for messages NOT shown in an open Thread; it currently
* has no viewer and is fully retireable — see TODO.
* Demoted (2026-06-29): once a phone gateway session exists, it is the durable
* history. Outbound agent messages arrive before that session exists, so this
* bounded store also backs the provisional Thread until the user's first reply
* promotes it to a real `source=phone` session.
*/
class ProactiveInboxRepository(private val context: Context) {
@@ -61,8 +61,8 @@ class ProactiveMessageHandler(
/**
* Show an inbound message inline in the Chat **Thread** it belongs to, when
* that Thread is currently open. Returns true if it was shown there — in
* which case the message is NOT also notified or added to the inbox (you're
* already looking at the conversation). The unified-Threads counterpart of
* which case the message is persisted but not also notified (you're already
* looking at the conversation). The unified-Threads counterpart of
* [toSession]; wired after construction.
*/
var injectIntoThread: ((ProactiveMessage) -> Boolean)? = null,
@@ -94,13 +94,15 @@ class ProactiveMessageHandler(
/** Route a parsed message: into the open Thread if it belongs there, else
* the durable inbox log + the surface its hint selects. */
private fun dispatch(msg: ProactiveMessage) {
// Unified Threads: if this message belongs to the Thread currently open
// in Chat, render it inline there and STOP — no notification, no inbox
// entry (you're already looking at the conversation).
if (injectIntoThread?.invoke(msg) == true) return
// Otherwise the inbox is the durable log of agent-initiated messages and
// the surfacing hint selects the additional surface.
// Persist first even when the currently open Thread consumes the live
// message. Agent-initiated outbound sends do not create a gateway
// session until the phone replies, so this cache is the provisional
// Thread transcript during that gap.
toInbox?.invoke(msg)
// Unified Threads: if this message belongs to the Thread currently open
// in Chat, render it inline there and stop before raising a notification.
if (injectIntoThread?.invoke(msg) == true) return
// The surfacing hint selects the additional surface.
when (msg.surfacing?.lowercase()) {
"inbox" -> { /* inbox only — already recorded above */ }
"session" -> {
@@ -9,6 +9,7 @@ import android.content.Context
import android.content.Intent
import android.content.pm.PackageManager
import android.os.Build
import android.net.Uri
import android.util.Log
import androidx.core.app.NotificationCompat
import androidx.core.app.NotificationManagerCompat
@@ -47,10 +48,13 @@ object ProactiveMessageNotifier {
/**
* Tap route — opens Chat, where the message lives as a Thread. Must match
* `Screen.Chat.route()` in RelayApp. Routed via the EXTRA_NAV_ROUTE deep-link
* path (MainActivity → NavRouteRequest → RelayApp collector). Opening the
* exact Thread by chat_id is a follow-up (see TODO).
* path (MainActivity → NavRouteRequest → RelayApp collector), carrying the
* `chat_id` so RelayApp opens the exact real or provisional Thread.
*/
private const val TAP_ROUTE = "chat"
private fun tapRoute(chatId: String?): String =
chatId?.takeIf { it.isNotBlank() }
?.let { "chat?proactiveChatId=${Uri.encode(it)}" }
?: "chat"
/**
* Post (or replace) a proactive-message notification.
@@ -80,7 +84,7 @@ object ProactiveMessageNotifier {
val tapIntent = Intent(context, MainActivity::class.java).apply {
flags = Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP
putExtra(MainActivity.EXTRA_NAV_ROUTE, TAP_ROUTE)
putExtra(MainActivity.EXTRA_NAV_ROUTE, tapRoute(chatId))
}
val pendingFlags = PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE
// Distinct requestCode per slot so each notification gets its own
@@ -336,17 +336,20 @@ sealed class Screen(
// NavHost, and the NavigationBarItem click must navigate via [route]()
// so no unresolved `{openAgentSheet}` leaks into the destination.
data object Chat : Screen(
"chat?openAgentSheet={openAgentSheet}&sessionId={sessionId}&profile={profile}",
"chat?openAgentSheet={openAgentSheet}&sessionId={sessionId}&profile={profile}" +
"&proactiveChatId={proactiveChatId}",
"Chat",
Icons.AutoMirrored.Filled.Chat,
) {
const val ARG_OPEN_AGENT_SHEET: String = "openAgentSheet"
const val ARG_SESSION_ID: String = "sessionId"
const val ARG_PROFILE: String = "profile"
const val ARG_PROACTIVE_CHAT_ID: String = "proactiveChatId"
fun route(
openAgentSheet: Boolean = false,
sessionId: String? = null,
profile: String? = null,
proactiveChatId: String? = null,
): String {
val params = buildList {
if (openAgentSheet) add("$ARG_OPEN_AGENT_SHEET=true")
@@ -356,6 +359,9 @@ sealed class Screen(
profile?.takeIf { it.isNotBlank() }?.let {
add("$ARG_PROFILE=${android.net.Uri.encode(it)}")
}
proactiveChatId?.takeIf { it.isNotBlank() }?.let {
add("$ARG_PROACTIVE_CHAT_ID=${android.net.Uri.encode(it)}")
}
}
return if (params.isEmpty()) "chat" else "chat?${params.joinToString("&")}"
}
@@ -1685,6 +1691,11 @@ fun RelayApp() {
nullable = true
defaultValue = null
},
navArgument(Screen.Chat.ARG_PROACTIVE_CHAT_ID) {
type = NavType.StringType
nullable = true
defaultValue = null
},
),
) { backStackEntry ->
// Responsive bubble width based on screen width. The "Blend"
@@ -1715,6 +1726,35 @@ fun RelayApp() {
val requestedProfileRoute = backStackEntry.arguments
?.getString(Screen.Chat.ARG_PROFILE)
?.takeIf { it.isNotBlank() }
val requestedProactiveChatId = backStackEntry.arguments
?.getString(Screen.Chat.ARG_PROACTIVE_CHAT_ID)
?.takeIf { it.isNotBlank() }
val proactiveInboxEntries by connectionViewModel.inboxMessages.collectAsState()
val phoneThreadChatIds by connectionViewModel.phoneThreadChatIds.collectAsState()
LaunchedEffect(
requestedProactiveChatId,
proactiveInboxEntries,
phoneThreadChatIds,
) {
val chatId = requestedProactiveChatId ?: return@LaunchedEffect
val realSessionId = phoneThreadChatIds.entries
.firstOrNull { it.value == chatId }
?.key
if (realSessionId != null) {
chatViewModel.switchSession(realSessionId)
} else {
val entries = proactiveInboxEntries.filter {
(it.connectionId == null || it.connectionId == activeConnectionId) &&
(it.chatId ?: "phone") == chatId
}
if (entries.isEmpty()) return@LaunchedEffect
chatViewModel.openProactiveThread(chatId, entries)
}
backStackEntry.arguments?.putString(
Screen.Chat.ARG_PROACTIVE_CHAT_ID,
null,
)
}
LaunchedEffect(
requestedSessionId,
requestedProfileRoute,
@@ -1739,7 +1739,10 @@ fun ActiveCardRoutesSection(
var routeEditorOriginal by remember(connection.id) {
mutableStateOf<EndpointCandidate?>(null)
}
val hasTailscaleRoute = endpoints.any { it.role.equals("tailscale", ignoreCase = true) }
val hasTailscaleRoute = hasConfiguredTailscaleRoute(
endpoints = endpoints,
primaryEndpointUrl = connection.primaryEndpointUrl,
)
val tailscalePreferred = preferredRole?.equals("tailscale", ignoreCase = true) == true
val routeNeedsAttention = activeEndpoint == null && liveState != RelayUiState.Connected
val showTailscaleUnavailableHint =
@@ -2171,6 +2174,12 @@ fun ActiveCardRoutesSection(
}
}
internal fun hasConfiguredTailscaleRoute(
endpoints: List<EndpointCandidate>,
primaryEndpointUrl: String,
): Boolean = endpoints.any { it.role.equals("tailscale", ignoreCase = true) } ||
Connection.inferRouteRole(primaryEndpointUrl) == "tailscale"
/**
* Numbered step row for the Manual pairing code fallback. Tightly
* coupled to its Card 3 layout — step badge sizing + content shape —
@@ -104,6 +104,15 @@ data class ProfileSessionRow(
val session: ChatSession,
)
data class ProvisionalThreadRow(
val chatId: String,
val title: String,
val messageCount: Int,
val lastActivityAt: Long,
)
private const val PROVISIONAL_THREAD_PREFIX = "proactive:"
internal fun sessionWorkLabels(session: ChatSession): List<String> = buildList {
val repo = (session.gitRepoRoot ?: session.workingDirectory)
?.trimEnd('/', '\\')
@@ -169,6 +178,8 @@ fun SessionDrawerContent(
* filter view. The first message the user types opens the conversation.
*/
onNewThread: ((String) -> Unit)? = null,
provisionalThreads: List<ProvisionalThreadRow> = emptyList(),
onSelectProvisionalThread: ((String) -> Unit)? = null,
/** Gateway sources currently hidden from the drawer (default: cron+webhook). */
hiddenSources: Set<String> = emptySet(),
/** Toggle a source's visibility (persisted). Null hides the source filter. */
@@ -191,7 +202,20 @@ fun SessionDrawerContent(
// Threads affordance shows when the capability is active OR there's already at least one
// agent Thread (source=phone) in the list. If the filter is on Threads but they've
// vanished, fall back to All so the drawer never gets stuck on an empty hidden filter.
val showThreads = threadsCapabilityActive || sessions.any { isThreadSource(it.source) }
val provisionalSessions = provisionalThreads.map { thread ->
ChatSession(
sessionId = "$PROVISIONAL_THREAD_PREFIX${thread.chatId}",
title = thread.title,
model = null,
messageCount = thread.messageCount,
updatedAt = thread.lastActivityAt,
startedAt = thread.lastActivityAt,
lastActivityAt = thread.lastActivityAt,
source = "phone",
)
}
val allSessions = sessions + provisionalSessions
val showThreads = threadsCapabilityActive || allSessions.any { isThreadSource(it.source) }
val activeFilter = resolveSessionDrawerFilter(filter, showThreads, archiveSupported)
// External gateway sources present (discord/telegram/cron/…) for the source
// filter dropdown. Own chats (tui/api_server) + phone Threads aren't listed.
@@ -200,7 +224,7 @@ fun SessionDrawerContent(
.distinct()
.filter { sourceBadge(it) != null }
.sorted()
val visibleSessions = sessions
val visibleSessions = allSessions
.asSequence()
.filter { session ->
when (activeFilter) {
@@ -529,7 +553,7 @@ fun SessionDrawerContent(
horizontalAlignment = Alignment.CenterHorizontally
) {
Text(
text = if (sessions.isEmpty()) stringResource(R.string.drawer_no_sessions) else stringResource(R.string.drawer_no_matching_sessions),
text = if (allSessions.isEmpty()) stringResource(R.string.drawer_no_sessions) else stringResource(R.string.drawer_no_matching_sessions),
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
@@ -553,7 +577,16 @@ fun SessionDrawerContent(
pinned = session.pinned,
archived = session.archived,
archiveSupported = archiveSupported,
onClick = { onSelectSession(session.sessionId) },
onClick = {
if (session.sessionId.startsWith(PROVISIONAL_THREAD_PREFIX)) {
onSelectProvisionalThread?.invoke(
session.sessionId.removePrefix(PROVISIONAL_THREAD_PREFIX),
)
} else {
onSelectSession(session.sessionId)
}
},
actionsEnabled = !session.sessionId.startsWith(PROVISIONAL_THREAD_PREFIX),
onTogglePinned = {
onSetSessionPinned(session.sessionId, !session.pinned)
},
@@ -748,6 +781,7 @@ private fun SessionItem(
archived: Boolean,
archiveSupported: Boolean,
onClick: () -> Unit,
actionsEnabled: Boolean = true,
onTogglePinned: () -> Unit,
onToggleArchived: () -> Unit,
onRename: () -> Unit,
@@ -888,7 +922,7 @@ private fun SessionItem(
}
}
Box {
if (actionsEnabled) Box {
IconButton(
onClick = { menuOpen = true },
modifier = Modifier.size(36.dp),
@@ -173,6 +173,7 @@ import com.hermesandroid.relay.data.HermesCardAction
import com.hermesandroid.relay.data.MessageRole
import com.hermesandroid.relay.data.PhysicalKeyboardEnterBehavior
import com.hermesandroid.relay.data.ProfilePresentationPolicy
import com.hermesandroid.relay.data.ProactiveInboxEntry
import com.hermesandroid.relay.data.SessionActivityState
import com.hermesandroid.relay.data.VoicePresentationMode
import com.hermesandroid.relay.data.hermesProcessNotificationOrNull
@@ -232,6 +233,7 @@ import com.hermesandroid.relay.ui.components.ThinkingMatrixColor
import com.hermesandroid.relay.ui.components.ThinkingMatrixPattern
import com.hermesandroid.relay.ui.components.SessionDrawerContent
import com.hermesandroid.relay.ui.components.ProfileSessionRow
import com.hermesandroid.relay.ui.components.ProvisionalThreadRow
import com.hermesandroid.relay.ui.components.ProfileDisplayManagerDialog
import com.hermesandroid.relay.ui.components.ProfileShelf
import com.hermesandroid.relay.ui.components.ProfileSwitcherSheet
@@ -2066,6 +2068,22 @@ fun ChatScreen(
val threadsCapabilityActive = threadsProactiveEnabled &&
threadsAuthState is com.hermesandroid.relay.auth.AuthState.Paired
val hiddenSources by connectionViewModel.hiddenSources.collectAsState()
val proactiveInboxEntries by connectionViewModel.inboxMessages.collectAsState()
val phoneThreadChatIds by connectionViewModel.phoneThreadChatIds.collectAsState()
val provisionalThreadEntries = buildProvisionalThreadRows(
entries = proactiveInboxEntries,
activeConnectionId = activeConnection?.id,
realThreadChatIds = phoneThreadChatIds.values,
)
val provisionalThreads = provisionalThreadEntries.map { (chatId, entries) ->
val latest = entries.maxBy { it.receivedAt }
ProvisionalThreadRow(
chatId = chatId,
title = latest.title.ifBlank { "Hermes" },
messageCount = entries.size,
lastActivityAt = latest.receivedAt,
)
}
SessionDrawerContent(
sessions = sessions,
@@ -2112,6 +2130,14 @@ fun ChatScreen(
chatViewModel.startNewThread(name)
scope.launch { drawerState.close() }
},
provisionalThreads = provisionalThreads,
onSelectProvisionalThread = { chatId ->
chatViewModel.openProactiveThread(
chatId,
provisionalThreadEntries[chatId].orEmpty(),
)
scope.launch { drawerState.close() }
},
hiddenSources = hiddenSources,
onToggleSourceHidden = { source, hidden ->
connectionViewModel.setSourceHidden(source, hidden)
@@ -4230,6 +4256,15 @@ fun ChatScreen(
}
}
internal fun buildProvisionalThreadRows(
entries: List<ProactiveInboxEntry>,
activeConnectionId: String?,
realThreadChatIds: Collection<String>,
): Map<String, List<ProactiveInboxEntry>> = entries
.filter { it.connectionId == null || it.connectionId == activeConnectionId }
.groupBy { it.chatId ?: "phone" }
.filterKeys { it !in realThreadChatIds }
// --- Helper functions ---
@Composable
@@ -1,8 +1,8 @@
package com.hermesandroid.relay.util
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.SharedFlow
import kotlinx.coroutines.flow.asSharedFlow
import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.receiveAsFlow
/**
* Cross-layer one-shot navigation requests.
@@ -18,23 +18,21 @@ import kotlinx.coroutines.flow.asSharedFlow
* `BridgeSafetySettingsScreen` instead of dropping the user on `MainActivity`'s
* home screen.
*
* Buffer: `extraBufferCapacity = 4` so back-to-back tryEmit calls during
* `onCreate → setContent` don't drop on the floor before `RelayApp`'s
* collector subscribes. Replay 0 — late subscribers shouldn't replay stale
* navigation intents from prior process lifetimes.
* A buffered [Channel] is intentional here: notification taps are consumed in
* `MainActivity.onCreate` before Compose installs RelayApp's collector. A
* replay-0 SharedFlow drops those cold-start requests when no subscriber exists.
* The channel retains up to four one-shot routes and hands each to the single
* app-root collector exactly once.
*/
object NavRouteRequest {
private val _requests = MutableSharedFlow<String>(
replay = 0,
extraBufferCapacity = 4,
)
private val channel = Channel<String>(capacity = 4)
val requests: SharedFlow<String> = _requests.asSharedFlow()
val requests: Flow<String> = channel.receiveAsFlow()
/** Fire-and-forget emit. Safe to call from any thread, including the main thread. */
fun tryRequest(route: String): Boolean = _requests.tryEmit(route)
fun tryRequest(route: String): Boolean = channel.trySend(route).isSuccess
suspend fun request(route: String) {
_requests.emit(route)
channel.send(route)
}
}
@@ -33,6 +33,7 @@ import com.hermesandroid.relay.data.MessageDeliveryStatus
import com.hermesandroid.relay.data.MessageRole
import com.hermesandroid.relay.data.parseChatQuotedPrompt
import com.hermesandroid.relay.data.Profile
import com.hermesandroid.relay.data.ProactiveInboxEntry
import com.hermesandroid.relay.data.RealtimeConversationContextMessage
import com.hermesandroid.relay.data.RealtimeTurnTrace
import com.hermesandroid.relay.data.SessionActivityState
@@ -2932,6 +2933,12 @@ class ChatViewModel : ViewModel() {
fun injectThreadMessage(msg: ProactiveMessage): Boolean {
val handler = chatHandler ?: return false
val msgChatId = msg.chatId?.takeIf { it.isNotBlank() }
pendingThread?.let { pending ->
if (msgChatId == null || msgChatId == pending.chatId) {
handler.addAgentThreadMessage(msg.text, msg.messageId, msg.title)
return true
}
}
// A freshly-created thread whose real session we're still switching to:
// show the agent's first reply in the draft view now (the switch
// reconciles it from history). Covers the gap before currentSessionId is
@@ -3550,6 +3557,8 @@ class ChatViewModel : ViewModel() {
fun createNewChat() {
val handler = chatHandler ?: return
pendingThread = null
creatingThread = null
// Gateway turns continue as detached siblings; SSE remains exclusive.
releaseTurnForNavigation(handler)
@@ -3666,6 +3675,43 @@ class ChatViewModel : ViewModel() {
onSessionChanged?.invoke(null)
}
/**
* Open an agent-initiated Thread before the gateway has a `source=phone`
* session for it. Outbound platform sends do not create gateway sessions;
* the first phone reply does. Until then the durable proactive inbox is the
* provisional transcript. The existing pending-thread send path promotes
* this draft to the real gateway session after the user's first reply.
*/
fun openProactiveThread(chatId: String, entries: List<ProactiveInboxEntry>) {
val handler = chatHandler ?: return
val normalizedChatId = chatId.ifBlank { "phone" }
val ordered = entries
.filter { (it.chatId ?: "phone") == normalizedChatId }
.sortedBy { it.receivedAt }
if (ordered.isEmpty()) return
releaseTurnForNavigation(handler)
cancelAnswerRecovery(settleUi = false)
historyLoadGeneration.incrementAndGet()
pendingThread = PendingThread(
chatId = normalizedChatId,
name = ordered.last().title.ifBlank { "Hermes" },
)
creatingThread = null
gatewayClient?.clearSession()
handler.setSessionId(null)
selectBackgroundProcessSession(null)
handler.clearMessages()
ordered.forEach { entry ->
handler.addAgentThreadMessage(entry.text, entry.id, entry.title)
}
_contextUsage.value = null
_contextWindow.value = null
dismissPendingAskNotification()
_pendingAsk.value = null
onSessionChanged?.invoke(null)
}
/**
* After a "+ New Thread" first send, poll the session list until the gateway
* has created the new `source=phone` session, then switch to it (loading its
@@ -3711,6 +3757,8 @@ class ChatViewModel : ViewModel() {
fun switchSession(sessionId: String) {
val handler = chatHandler ?: return
if (streamingEndpoint != "gateway" && apiClient == null) return
pendingThread = null
creatingThread = null
// Keep a Gateway sibling alive and detach its callbacks. SSE remains a
// single exclusive stream and is interrupted on navigation.
@@ -2362,6 +2362,7 @@ class ConnectionViewModel(application: Application) : AndroidViewModel(applicati
text = msg.text,
receivedAt = msg.sentAt ?: System.currentTimeMillis(),
chatId = msg.chatId,
connectionId = connectionStore.activeConnectionId.value,
),
)
}
@@ -0,0 +1,44 @@
package com.hermesandroid.relay.ui.components
import com.hermesandroid.relay.data.ApiEndpoint
import com.hermesandroid.relay.data.EndpointCandidate
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class TailscaleRoutePresenceTest {
@Test
fun primaryTailnetIpCountsAsConfiguredRoute() {
assertTrue(
hasConfiguredTailscaleRoute(
endpoints = emptyList(),
primaryEndpointUrl = "http://100.75.1.2:9119",
),
)
}
@Test
fun explicitTailscaleCandidateStillCounts() {
assertTrue(
hasConfiguredTailscaleRoute(
endpoints = listOf(
EndpointCandidate(
role = "tailscale",
api = ApiEndpoint("server.ts.net", 8642),
),
),
primaryEndpointUrl = "http://192.168.1.2:9119",
),
)
}
@Test
fun ordinaryLanDoesNotCount() {
assertFalse(
hasConfiguredTailscaleRoute(
endpoints = emptyList(),
primaryEndpointUrl = "http://192.168.1.2:9119",
),
)
}
}
@@ -0,0 +1,48 @@
package com.hermesandroid.relay.ui.screens
import com.hermesandroid.relay.data.ProactiveInboxEntry
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class ProvisionalThreadRowsTest {
@Test
fun rowsAreConnectionScopedAndDisappearWhenRealThreadExists() {
val entries = listOf(
entry("one", connectionId = "connection-a", chatId = "phone"),
entry("two", connectionId = "connection-b", chatId = "phone"),
entry("three", connectionId = "connection-a", chatId = "project"),
)
val rows = buildProvisionalThreadRows(
entries = entries,
activeConnectionId = "connection-a",
realThreadChatIds = listOf("project"),
)
assertEquals(listOf("one"), rows.getValue("phone").map { it.id })
assertFalse("project" in rows)
}
@Test
fun legacyUnscopedEntriesRemainVisibleOnTheActiveConnection() {
val rows = buildProvisionalThreadRows(
entries = listOf(entry("legacy", connectionId = null, chatId = null)),
activeConnectionId = "connection-a",
realThreadChatIds = emptyList(),
)
assertTrue("phone" in rows)
}
private fun entry(id: String, connectionId: String?, chatId: String?) =
ProactiveInboxEntry(
id = id,
title = "Hermes",
text = id,
receivedAt = 1L,
chatId = chatId,
connectionId = connectionId,
)
}
@@ -0,0 +1,23 @@
package com.hermesandroid.relay.util
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import kotlinx.coroutines.withTimeoutOrNull
class NavRouteRequestTest {
@Test
fun requestBeforeCollector_isRetainedAndConsumedOnce() = runBlocking {
assertTrue(NavRouteRequest.tryRequest("chat?proactiveChatId=phone"))
assertEquals(
"chat?proactiveChatId=phone",
withTimeout(1_000) { NavRouteRequest.requests.first() },
)
assertNull(withTimeoutOrNull(50) { NavRouteRequest.requests.first() })
}
}
+83 -4
View File
@@ -80,11 +80,15 @@ aborts, and non-zero process exits; and keeps request context collapsed until
explicitly expanded. Events record handler duration and request ID where
available. The compact Overview still shows only the three newest events.
Settings also keeps activity compact: it previews the three newest events and
opens a dedicated Activity page. Selecting an event opens bounded request,
stdout, stderr, result, exit, timing, and truncation evidence; sensitive request
inputs are excluded. Handler failures and aborts are **Issues**; non-zero process exits are
opens a dedicated Activity page. Selecting an event opens a truthful lifecycle
stepper plus bounded request, stdout, stderr, result, exit, timing, and
truncation evidence; sensitive request inputs are excluded. Screenshot events
can retain an opaque local PNG outside the JSON log and open it in a larger
borderless viewer. **Settings → Activity → Screenshot evidence** controls this
as Off, 1 day, 7 days (default), or 30 days, shows local file count/usage, and
caps storage at 20 files and 10 MB per image. Handler failures and aborts are **Issues**; non-zero process exits are
shown separately because probing commands may legitimately use them. **Clear**
removes both current and rotated local audit history after confirmation.
removes current/rotated audit history and retained screenshot evidence after confirmation.
Clicking a card under **Hosts** opens that host's detail page; it does not change
the active connection. The detail page is the per-host hub for its local display
@@ -441,6 +445,81 @@ assist/control grant is active because approved input inherits that privilege.
Default computer-use policy blocks password managers, credential prompts, banking/payment/crypto surfaces, OS security/admin settings, and private-key/token material. `~/.hermes/desktop-control.json` lets operators tighten or extend that baseline.
#### Preferred CUA Driver engine
On Windows, Hermes-Relay prefers a compatible local
[CUA Driver](https://github.com/trycua/cua) runtime for structured
computer-control engine. It stays behind the same `desktop_computer_*` tools:
the agent does not receive CUA's raw tool surface, configuration, updater,
recording, replay, JavaScript, application-launch, or process-termination
operations.
Windows input is the explicit compatibility backend. A backend is selected once
when each authenticated control session starts and cannot change mid-session;
changing the setting affects only new sessions. If preferred CUA is unavailable
before a session starts, that session can use compatibility mode.
Inspect the detected runtime and selected/effective engine with:
```powershell
hermes-relay computer-use status --json
hermes-relay computer-use cua status
hermes-relay computer-use cua health # explicit accessibility recheck
hermes-relay computer-use cua check-update
hermes-relay computer-use cua install --yes
hermes-relay computer-use cua update --yes
hermes-relay computer-use engine cua # preferred; requires a ready runtime
hermes-relay computer-use engine legacy # explicit compatibility backend
hermes-relay computer-use cursor on
```
The management UI exposes the same controls under **Settings → Computer
control**. CUA is selected only when its canonical Windows package resolves from
`%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe`, its supported
version and manifest agree, its required tools are present, and its permission mode
is not unrestricted. The live health report is an explicit diagnostic while the
temporary Windows workaround for trycua/cua#3103 is active. Hermes ignores an unrelated
or stale `cua-driver.exe` found earlier on `PATH`.
Background dispatch is mandatory. If an application cannot accept a
background action, the action fails instead of silently stealing focus.
**Allow foreground escalation** is reserved for a later explicitly approved
path; this release always reports it off and dispatches CUA actions in the
background. **Animated agent cursor** shows a virtual, session-scoped pointer
for agent activity; it does not move the operator's physical Windows cursor and
is not another hardware pointer. Hermes binds driver sessions and snapshot
tokens to its own control authority, target PID/window, grant, and fresh
snapshot; an element token cannot be reused across windows or after it is
consumed or expires.
CUA Driver is not bundled with the Hermes-Relay installer. The explicit
`computer-use cua install|update --yes` commands use the canonical upstream
GitHub release manifest and installer. Hermes verifies the manifest's
repository/product/version and installer SHA-256 before execution under a
sanitized child-process environment, then checks
the canonical binary's path, version, own manifest, tool surface, and permission
mode. Accessibility health can be rechecked separately. This is release-metadata/checksum validation—not a Windows
publisher signature. A native update newer than the supported `>=0.19.3,
<0.20.0` range is displayed but refused. There is no silent install/update,
and every child invocation forces CUA telemetry off. `hermes-relay update`
continues to manage only the CLI and management UI.
Window-scoped snapshots and semantic actions use the selected control backend.
The existing full-display screenshot remains a separate read-only
`system_capture` path, so observation does not cause a mid-session backend
switch. The local audit and UI Activity timeline record bounded high-level
evidence such as backend, background dispatch, control session, target
application/window identifiers, action, phase, and verification state.
Accessibility text, screenshot bytes, entered values, and raw CUA responses are
excluded from that drilldown.
CUA improves structured screen/input isolation, but it is not a sandbox for
general commands. If Commands, PowerShell, or terminal execution is allowed,
that trusted path can still use ordinary operating-system automation. Disable
raw command access when CUA's scoped UI-control boundary is part of the security
model. Full Access can remove ordinary task prompts, but it does not bypass
authenticated targeting, sensitive-surface checks, snapshot freshness, UAC or
Windows-session boundaries, audit, driver health, or emergency stop.
### Devices — server-side session management
```sh
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@hermes-relay/cli",
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@hermes-relay/cli",
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"license": "MIT",
"bin": {
"hermes-relay": "bin/hermes-relay.js"
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@hermes-relay/cli",
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"description": "Thin-client CLI for Hermes-Relay — talk to a remote Hermes agent over WSS with pairing auth, stream-renders tool calls and responses to plain stdout.",
"type": "module",
"bin": {
+118
View File
@@ -0,0 +1,118 @@
[CmdletBinding()]
param(
[switch]$Run,
[string]$CuaDriver = "$env:USERPROFILE\.cua-driver\packages\current\cua-driver.exe"
)
$ErrorActionPreference = 'Stop'
if (-not $Run) {
Write-Host 'Hermes CUA live acceptance is opt-in because it opens and controls two Calculator windows.'
Write-Host 'Re-run with: powershell -ExecutionPolicy Bypass -File scripts/test-cua-windows.ps1 -Run'
exit 0
}
if (-not (Test-Path -LiteralPath $CuaDriver -PathType Leaf)) {
throw "Canonical CUA Driver was not found at $CuaDriver"
}
Add-Type -TypeDefinition @'
using System;
using System.Runtime.InteropServices;
public static class HermesCuaAcceptanceNative {
[StructLayout(LayoutKind.Sequential)] public struct POINT { public int X; public int Y; }
[DllImport("user32.dll")] public static extern bool GetCursorPos(out POINT point);
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
}
'@
function Invoke-Cua([string]$Tool, [hashtable]$Payload) {
$json = $Payload | ConvertTo-Json -Depth 12 -Compress
$raw = $json | & $CuaDriver call $Tool
if ($LASTEXITCODE -ne 0) { throw "CUA $Tool failed with exit code $LASTEXITCODE" }
$result = $raw | ConvertFrom-Json
if ($result.isError -eq $true) { throw "CUA $Tool rejected the request: $raw" }
return $result
}
function Get-DesktopSentinel {
$point = New-Object HermesCuaAcceptanceNative+POINT
[void][HermesCuaAcceptanceNative]::GetCursorPos([ref]$point)
[pscustomobject]@{
CursorX = $point.X
CursorY = $point.Y
Foreground = [HermesCuaAcceptanceNative]::GetForegroundWindow().ToInt64()
}
}
function Assert-Unchanged([object]$Before, [string]$Step) {
$after = Get-DesktopSentinel
if ($after.CursorX -ne $Before.CursorX -or $after.CursorY -ne $Before.CursorY) {
throw "$Step moved the physical cursor from ($($Before.CursorX),$($Before.CursorY)) to ($($after.CursorX),$($after.CursorY))"
}
if ($after.Foreground -ne $Before.Foreground) {
throw "$Step changed the foreground HWND from $($Before.Foreground) to $($after.Foreground)"
}
}
function Get-Window([object]$Launch) {
$window = @($Launch.windows | Where-Object { $_.is_on_screen -ne $false }) | Select-Object -First 1
if (-not $window) { throw "CUA launch did not return a usable window: $($Launch | ConvertTo-Json -Depth 8 -Compress)" }
return $window
}
$sessionA = "hermes-live-a-$([guid]::NewGuid().ToString('N'))"
$sessionB = "hermes-live-b-$([guid]::NewGuid().ToString('N'))"
$opened = @()
try {
$health = Invoke-Cua health_report @{}
if ($health.overall -ne 'healthy') { throw "CUA health must be healthy, got $($health.overall)" }
[void](Invoke-Cua start_session @{ session = $sessionA; capture_scope = 'window' })
[void](Invoke-Cua start_session @{ session = $sessionB; capture_scope = 'window' })
$sentinel = Get-DesktopSentinel
$launchA = Invoke-Cua launch_app @{ aumid = 'Microsoft.WindowsCalculator_8wekyb3d8bbwe!App'; creates_new_application_instance = $true; session = $sessionA }
$launchB = Invoke-Cua launch_app @{ aumid = 'Microsoft.WindowsCalculator_8wekyb3d8bbwe!App'; creates_new_application_instance = $true; session = $sessionB }
$opened = @($launchA.pid, $launchB.pid)
Start-Sleep -Milliseconds 800
Assert-Unchanged $sentinel 'background launch'
$windowA = Get-Window $launchA
$windowB = Get-Window $launchB
if ($windowA.window_id -eq $windowB.window_id) { throw 'isolated sessions resolved to the same Calculator window' }
$beforeA = Invoke-Cua get_window_state @{ pid = $launchA.pid; window_id = $windowA.window_id; session = $sessionA; query = 'Seven'; include_screenshot = $false }
$seven = @($beforeA.elements | Where-Object { $_.label -eq 'Seven' -or $_.name -eq 'Seven' }) | Select-Object -First 1
if (-not $seven.element_token) { throw 'Calculator Seven element did not expose an opaque token' }
[void](Invoke-Cua click @{ pid = $launchA.pid; window_id = $windowA.window_id; element_token = $seven.element_token; session = $sessionA; scope = 'window'; delivery_mode = 'background' })
Assert-Unchanged $sentinel 'background element action'
$afterA = Invoke-Cua get_window_state @{ pid = $launchA.pid; window_id = $windowA.window_id; session = $sessionA; query = 'Display'; include_screenshot = $false }
if ($afterA.snapshot_id -eq $beforeA.snapshot_id) { throw 'post-action snapshot did not advance its generation' }
$staleRaw = (@{ pid = $launchA.pid; window_id = $windowA.window_id; element_token = $seven.element_token; session = $sessionA; scope = 'window'; delivery_mode = 'background' } | ConvertTo-Json -Compress) | & $CuaDriver call click
$stale = $staleRaw | ConvertFrom-Json
if ($stale.isError -ne $true) { throw 'stale element token was accepted after a newer snapshot' }
$cursorA = Invoke-Cua get_agent_cursor_state @{ session = $sessionA }
$cursorB = Invoke-Cua get_agent_cursor_state @{ session = $sessionB }
if (($cursorA | ConvertTo-Json -Depth 8 -Compress) -eq ($cursorB | ConvertTo-Json -Depth 8 -Compress)) {
throw 'two control sessions did not expose isolated cursor state'
}
[void](Invoke-Cua end_session @{ session = $sessionA })
$endedRaw = (@{ session = $sessionA } | ConvertTo-Json -Compress) | & $CuaDriver call get_session_state
$ended = $endedRaw | ConvertFrom-Json
if ($ended.code -ne 'session_not_started') { throw 'ended session remained active' }
Write-Host 'PASS: physical cursor and foreground stayed unchanged.'
Write-Host 'PASS: background Calculator action was bracketed by snapshots.'
Write-Host 'PASS: stale token rejection and two isolated cursor sessions were verified.'
Write-Host 'PASS: ending a session immediately revoked its CUA state.'
} finally {
foreach ($session in @($sessionA, $sessionB)) {
try { [void](Invoke-Cua end_session @{ session = $session }) } catch { Write-Warning $_ }
}
Write-Host "Calculator processes created by this acceptance run: $($opened -join ', '). Close them manually after inspection."
}
+34 -3
View File
@@ -5,7 +5,8 @@
// the audit flagged as the biggest desktop-tools transparency gap.
import type { ParsedArgs } from '../cli.js'
import { auditLogPath, readRecentAudit } from '../lib/auditLog.js'
import { auditLogPath, auditScreenshotEvidenceStatus, clearAuditScreenshotEvidence, pruneAuditScreenshotEvidence, readRecentAudit } from '../lib/auditLog.js'
import { readDesktopUseSettings, setActivityScreenshotRetention } from '../lib/desktopUseSettings.js'
import { renderTable } from '../lib/table.js'
import { SYMBOLS, theme as makeTheme } from '../lib/theme.js'
import { printUsage, type UsageSpec } from '../lib/usage.js'
@@ -13,10 +14,12 @@ import { printUsage, type UsageSpec } from '../lib/usage.js'
const AUDIT_USAGE: UsageSpec = {
name: 'audit',
summary: 'show recent desktop-tool activity the agent ran on this machine',
usage: ['audit [--limit <n>] [--json]'],
usage: ['audit [--limit <n>] [--json]', 'audit screenshots [on|off] [--days <1|7|30>] [--yes] [--json]'],
flags: [
{ flag: '--limit <n>', desc: 'How many recent entries to show (default 50)' },
{ flag: '--json', desc: 'Emit raw audit entries as JSON' }
{ flag: '--json', desc: 'Emit raw audit entries as JSON' },
{ flag: '--days <1|7|30>', desc: 'Local screenshot retention period' },
{ flag: '--yes', desc: 'Confirm a retention change' }
],
examples: ['hermes-relay audit', 'hermes-relay audit --limit 20']
}
@@ -36,6 +39,34 @@ export async function auditCommand(args: ParsedArgs): Promise<number> {
return 0
}
if (args.positional[0] === 'screenshots') {
const settings = await readDesktopUseSettings()
const mode = args.positional[1]
if (mode === 'on' || mode === 'off') {
if (args.flags.yes !== true) {
process.stderr.write('audit screenshots: retention changes require --yes\n')
return 1
}
const rawDays = typeof args.flags.days === 'string' ? Number(args.flags.days) : settings.activity_screenshot_retention_days
if (rawDays !== 1 && rawDays !== 7 && rawDays !== 30) {
process.stderr.write('audit screenshots: --days must be 1, 7, or 30\n')
return 1
}
await setActivityScreenshotRetention(mode === 'on', rawDays)
if (mode === 'off') await clearAuditScreenshotEvidence()
else await pruneAuditScreenshotEvidence(rawDays)
} else if (mode) {
process.stderr.write('audit screenshots: expected on or off\n')
return 1
}
const current = await readDesktopUseSettings()
const evidence = await auditScreenshotEvidenceStatus()
const result = { enabled: current.activity_screenshot_retention_enabled, days: current.activity_screenshot_retention_days, ...evidence }
if (args.flags.json) process.stdout.write(JSON.stringify(result, null, 2) + '\n')
else process.stdout.write(`Screenshot evidence: ${result.enabled ? `${result.days} days` : 'off'} · ${result.count} file${result.count === 1 ? '' : 's'}\n`)
return 0
}
const rawLimit = typeof args.flags.limit === 'string' ? parseInt(args.flags.limit, 10) : 50
const limit = Number.isFinite(rawLimit) && rawLimit > 0 ? rawLimit : 50
+163 -4
View File
@@ -5,11 +5,19 @@ import { readDaemonStatus, isDaemonProcessAlive } from '../lib/daemonStatus.js'
import {
readDesktopUseSettings,
requestComputerGrantCancellation,
setComputerControlSettings,
setDesktopUseEnabled
} from '../lib/desktopUseSettings.js'
import { listPendingGrantRequests } from '../lib/grantBridge.js'
import { theme as makeTheme } from '../lib/theme.js'
import { printUsage, type UsageSpec, unknownSubcommand } from '../lib/usage.js'
import { CuaDriverAdapter, type CuaRuntimeStatus } from '../tools/cuaDriver.js'
import {
checkCuaUpdate,
getCuaManagementStatus,
installCuaDriver,
updateCuaDriver
} from '../tools/cuaManagement.js'
const COMPUTER_USE_USAGE: UsageSpec = {
name: 'computer-use',
@@ -18,21 +26,31 @@ const COMPUTER_USE_USAGE: UsageSpec = {
'computer-use status [--json]',
'computer-use enable [--yes]',
'computer-use disable',
'computer-use cancel'
'computer-use cancel',
'computer-use engine <legacy|cua>',
'computer-use cursor <on|off>',
'computer-use cua <status|health|install|check-update|update> [--json] [--yes]'
],
subcommands: [
{ verb: 'status', desc: 'Show preference, daemon state, active grant, and pending requests' },
{ verb: 'enable', desc: 'Persist desktop-use enablement after explicit confirmation' },
{ verb: 'disable', desc: 'Disable desktop use and request cancellation of any active grant' },
{ verb: 'cancel', desc: 'Cancel the active task-scoped desktop grant' }
{ verb: 'cancel', desc: 'Cancel the active task-scoped desktop grant' },
{ verb: 'engine', desc: 'Choose legacy Windows input or a ready CUA Driver backend' },
{ verb: 'cursor', desc: 'Show or hide the CUA virtual agent cursor' },
{ verb: 'cua', desc: 'Manage the canonical CUA Driver package and recheck accessibility health' }
],
flags: [
{ flag: '--json', desc: 'Emit machine-readable status' },
{ flag: '--yes', desc: 'Confirm enablement non-interactively' }
{ flag: '--yes', desc: 'Confirm enablement, CUA installation, or CUA update explicitly' }
],
examples: [
'hermes-relay computer-use status',
'hermes-relay computer-use enable',
'hermes-relay computer-use cua status',
'hermes-relay computer-use cua health',
'hermes-relay computer-use cua check-update',
'hermes-relay computer-use cua install --yes',
'hermes-relay computer-use cancel',
'hermes-relay computer-use disable'
]
@@ -62,6 +80,24 @@ async function statusPayload(): Promise<Record<string, unknown>> {
const activeGrant = daemonAlive && daemon?.computer_grant?.active === true
? daemon.computer_grant
: null
let cua: CuaRuntimeStatus | null = null
try {
cua = await CuaDriverAdapter.status()
} catch {
// The optional backend must not make ordinary desktop-use status fail.
}
const cuaReason = cua?.reason?.toLowerCase() ?? ''
const cuaState = !cua?.available
? 'not_installed'
: cua.ready
? 'ready'
: /(?:incompatible|unsupported|version|manifest|permission mode|missing required tools)/.test(cuaReason)
? 'incompatible'
: /(?:degraded|health)/.test(cuaReason)
? 'degraded'
: 'error'
const cuaReady = cuaState === 'ready'
const lifecycle = daemonAlive ? daemon?.computer_control : undefined
return {
enabled: settings.computer_use_enabled,
daemon_alive: daemonAlive,
@@ -69,7 +105,32 @@ async function statusPayload(): Promise<Record<string, unknown>> {
daemon_computer_use_enabled: daemonAlive ? (daemon?.computer_use_enabled ?? false) : false,
active_grant: activeGrant,
pending_grants: pending.length,
restart_required: daemonAlive && daemon?.computer_use_enabled !== settings.computer_use_enabled
restart_required: daemonAlive && daemon?.computer_use_enabled !== settings.computer_use_enabled,
computer_control_engine: {
selected: settings.computer_control_engine,
effective: settings.computer_control_engine === 'cua'
? lifecycle?.active_backend === 'cua'
? 'cua'
: lifecycle?.active_backend === 'legacy_compat'
? 'legacy'
: cuaReady ? 'cua' : 'legacy'
: 'legacy',
available: cua?.available === true,
state: cuaState,
version: cua?.binaryVersion ?? null,
health: cua?.health ?? null,
path: cua?.binaryPath ?? null,
cursor_enabled: settings.cua_cursor_enabled,
active_sessions: lifecycle?.active_sessions ?? 0,
active_backend: lifecycle?.active_backend ?? 'idle',
last_action: lifecycle?.last_action ?? null,
foreground_escalation_enabled: false,
message: settings.computer_control_engine === 'cua' && !cuaReady
? cuaState === 'degraded'
? `CUA Driver is installed, but UI Automation is degraded; new sessions use Windows Input compatibility mode. ${cua?.reason ?? ''}`.trim()
: `CUA Driver is unavailable before control starts; new sessions use Windows Input compatibility mode. ${cua?.reason ?? ''}`.trim()
: cua?.reason ?? null
}
}
}
@@ -81,6 +142,68 @@ export async function computerUseCommand(args: ParsedArgs): Promise<number> {
}
const subcommand = args.positional[0] ?? 'status'
if (subcommand === 'cua') {
const action = args.positional[1] ?? 'status'
const json = args.flags.json === true
try {
if (action === 'health') {
const health = await CuaDriverAdapter.healthStatus()
if (json) process.stdout.write(JSON.stringify(health, null, 2) + '\n')
else {
process.stdout.write(t.bold('CUA Driver accessibility health') + `\n state: ${health.state}\n`)
if (health.reason) process.stdout.write(t.warnLine(` ${health.reason}`) + '\n')
process.stdout.write(t.muted(' This diagnostic does not disable the runtime while the temporary Windows compatibility policy is active.') + '\n')
}
// The probe result is data, not command failure. Callers (including the
// tray) inspect state while still receiving the JSON for degradation.
return 0
}
const payload = action === 'status'
? await getCuaManagementStatus()
: action === 'check-update'
? await checkCuaUpdate()
: action === 'install'
? args.flags.yes === true
? await installCuaDriver()
: null
: action === 'update'
? args.flags.yes === true
? await updateCuaDriver()
: null
: undefined
if (payload === undefined) {
process.stderr.write(t.err('cua action must be status, health, install, check-update, or update') + '\n')
return 1
}
if (payload === null) {
process.stderr.write(t.err(`CUA ${action} requires explicit confirmation with --yes`) + '\n')
return 1
}
if (json) {
process.stdout.write(JSON.stringify(payload, null, 2) + '\n')
return 0
}
const version = payload.current_version ?? 'not installed'
process.stdout.write(t.bold('CUA Driver') + `\n version: ${version}\n`)
process.stdout.write(` package: ${payload.canonical_path ?? 'not installed'}\n`)
process.stdout.write(` compatibility: ${payload.compatible ? 'supported' : payload.compatibility_reason ?? 'not ready'}\n`)
if (payload.stale_path_shim) {
process.stdout.write(t.warnLine(` PATH resolves a competing copy: ${payload.discovered_path}`) + '\n')
process.stdout.write(t.muted(' Hermes uses the canonical package/current install instead.') + '\n')
}
if (payload.update) {
if (payload.update.error) process.stdout.write(t.warnLine(` update check: ${payload.update.error}`) + '\n')
else if (payload.update.update_available) {
process.stdout.write(` update: ${payload.update.latest_version}${payload.update.compatible ? ' available' : ' available but unsupported'}\n`)
} else process.stdout.write(' update: up to date\n')
}
if (payload.operation) process.stdout.write(t.okLine(`CUA ${payload.operation.kind} completed`) + '\n')
return 0
} catch (error) {
process.stderr.write(t.err(error instanceof Error ? error.message : String(error)) + '\n')
return 1
}
}
if (subcommand === 'status') {
const payload = await statusPayload()
if (args.flags.json) {
@@ -129,6 +252,42 @@ export async function computerUseCommand(args: ParsedArgs): Promise<number> {
return 0
}
if (subcommand === 'engine') {
const engine = args.positional[1]
if (engine !== 'legacy' && engine !== 'cua') {
process.stderr.write(t.err('engine must be legacy or cua') + '\n')
return 1
}
if (engine === 'cua') {
const payload = await statusPayload()
const status = payload.computer_control_engine as { state?: string }
if (status.state !== 'ready') {
process.stderr.write(t.err('CUA Driver is not ready; engine selection was not changed') + '\n')
return 1
}
}
await setComputerControlSettings({ computer_control_engine: engine })
process.stdout.write(t.okLine(`computer control engine set to ${engine}`) + '\n')
return 0
}
if (subcommand === 'cursor') {
const value = args.positional[1]
if (value !== 'on' && value !== 'off') {
process.stderr.write(t.err(`${subcommand} must be on or off`) + '\n')
return 1
}
const payload = await statusPayload()
const status = payload.computer_control_engine as { selected?: string; state?: string }
if (status.selected !== 'cua' || status.state !== 'ready') {
process.stderr.write(t.err(`CUA Driver must be selected and ready before changing ${subcommand}`) + '\n')
return 1
}
await setComputerControlSettings({ cua_cursor_enabled: value === 'on' })
process.stdout.write(t.okLine(`CUA ${subcommand} ${value}`) + '\n')
return 0
}
return unknownSubcommand(COMPUTER_USE_USAGE, subcommand, t)
}
+28 -8
View File
@@ -50,6 +50,7 @@ import {
type DaemonStatus
} from '../lib/daemonStatus.js'
import { rpcErrorMessage, asRpcResult } from '../lib/rpc.js'
import { appendAudit } from '../lib/auditLog.js'
import { effectiveHostAccessMode, effectiveHostCapabilityPolicies, getHostAccessMode, getHostCapabilityPolicies } from '../lib/hostAccessPolicy.js'
import { theme as makeTheme } from '../lib/theme.js'
import { printUsage, type UsageSpec } from '../lib/usage.js'
@@ -62,12 +63,13 @@ import {
shouldAdvertiseComputerUse
} from '../tools/handlerSet.js'
import {
cancelComputerGrant,
cancelAllComputerGrants,
configureComputerUseRuntime,
getActiveComputerGrant,
expireComputerControlSessions,
setComputerGrantChangeListener,
type ComputerGrant
} from '../tools/computerGrants.js'
import { closeCuaControlSession, setComputerControlLifecycleListener } from '../tools/cuaDriver.js'
import { DesktopToolRouter } from '../tools/router.js'
import { configureCapabilityPolicies } from '../tools/capabilityRuntime.js'
import { adbBackendAvailable } from '../tools/handlers/adb.js'
@@ -853,17 +855,26 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
? (info as { attempt?: number; delayMs?: number })
: {}
log.warn({ event: 'reconnecting', attempt: attempt ?? null, delay_ms: delayMs ?? null })
updateStatus({ state: 'reconnecting', last_event: 'reconnecting' })
const retryAt = nowSec() + Math.ceil((delayMs ?? 0) / 1000)
updateStatus({ state: 'reconnecting', last_event: 'reconnecting', reconnect_attempt: attempt ?? null, retry_at: retryAt, last_error: 'Relay connection interrupted' })
if ((attempt ?? 1) === 1) {
void appendAudit({
ts: Date.now(), kind: 'connection.state', tool: 'daemon.reconnecting', category: 'system', ok: false,
host_url: configuredUrl, summary: 'Automatic reconnect started', error: 'Relay connection interrupted'
})
}
})
relay.on('reconnected', () => {
log.info({ event: 'reconnected' })
updateStatus({ state: 'connected', last_event: 'reconnected' })
updateStatus({ state: 'connected', last_event: 'reconnected', reconnect_attempt: null, retry_at: null, last_error: null })
void appendAudit({ ts: Date.now(), kind: 'connection.state', tool: 'daemon.reconnected', category: 'system', ok: true, host_url: configuredUrl, summary: 'Relay tunnel restored' })
})
relay.on('exit', (code: unknown) => {
// Transport gave up (auth.fail, reconnect gate returned false, or
// reconnect attempts exhausted). Daemon exits non-zero so the
// service manager decides whether to restart.
log.error({ event: 'transport_exited', code: typeof code === 'number' ? code : null })
void appendAudit({ ts: Date.now(), kind: 'connection.state', tool: 'daemon.disconnected', category: 'system', ok: false, host_url: configuredUrl, summary: 'Relay transport stopped', error: 'Automatic reconnect stopped' })
// Defer exit so the log line flushes before the process dies.
setImmediate(() => process.exit(1))
})
@@ -873,6 +884,8 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
const outcome = await relay.whenAuthResolved()
if (!outcome.ok) {
log.error({ event: 'auth_failed', reason: outcome.reason })
updateStatus({ state: 'stopped', last_event: 'auth_failed', last_error: outcome.reason, reconnect_attempt: null, retry_at: null })
void appendAudit({ ts: Date.now(), kind: 'connection.state', tool: 'daemon.auth_failed', category: 'system', ok: false, host_url: configuredUrl, summary: 'Relay authentication failed', error: outcome.reason })
try {
relay.kill()
} catch {
@@ -886,7 +899,7 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
server_version: relay.serverVersion ?? null,
transport: relay.authMeta?.transportHint ?? null
})
updateStatus({ state: 'connected', server_version: relay.serverVersion ?? null, last_event: 'authed' })
updateStatus({ state: 'connected', server_version: relay.serverVersion ?? null, last_event: 'authed', reconnect_attempt: null, retry_at: null, last_error: null })
// Signal downstream handlers that we're running headless. The router
// also checks this env var in its detectInteractive() fallback, so any
@@ -925,8 +938,14 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
expires_at: grant?.expires_at ?? null,
reason: grant?.reason
})
const restoreGrantListener = setComputerGrantChangeListener(grant => {
const restoreGrantListener = setComputerGrantChangeListener((grant, controlSessionId) => {
updateStatus({ computer_grant: toDaemonGrantStatus(grant), last_event: 'grant_changed' })
if (!grant && controlSessionId) {
void closeCuaControlSession(controlSessionId, 'computer grant ended')
}
})
const restoreControlLifecycleListener = setComputerControlLifecycleListener(computerControl => {
updateStatus({ computer_control: computerControl })
})
let cancellationCheckRunning = false
@@ -936,10 +955,10 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
try {
const request = await consumeComputerGrantCancellation()
if (request) {
const result = cancelComputerGrant(request.reason)
const result = { cancelled: cancelAllComputerGrants(request.reason) }
log.info({ event: 'computer_grant_cancelled_locally', reason: request.reason, result })
} else {
getActiveComputerGrant()
expireComputerControlSessions()
}
} catch (error) {
log.warn({ event: 'computer_grant_control_failed', message: rpcErrorMessage(error) })
@@ -1027,6 +1046,7 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
clearInterval(statusHeartbeat)
clearInterval(grantControlInterval)
restoreGrantListener()
restoreControlLifecycleListener()
try {
await clearDaemonStatus()
} catch {
+145 -3
View File
@@ -9,7 +9,8 @@
//
// Best-effort by design: a logging failure must never break a tool dispatch.
import { appendFile, mkdir, readFile, rename, stat } from 'node:fs/promises'
import { appendFile, mkdir, readFile, readdir, rename, stat, unlink, writeFile } from 'node:fs/promises'
import { createHash } from 'node:crypto'
import { homedir } from 'node:os'
import { dirname, join } from 'node:path'
@@ -17,7 +18,7 @@ export interface AuditEntry {
/** Epoch milliseconds when the command completed. */
ts: number
/** Stable event type for consumers that do not want to infer it from fields. */
kind?: 'tool.completed'
kind?: 'tool.completed' | 'connection.state'
tool: string
category?: AuditCategory
ok: boolean
@@ -25,6 +26,20 @@ export interface AuditEntry {
request_id?: string
/** Relay identity that issued the action, when known. */
host_url?: string
relay_session_id?: string
requester_device_id?: string
run_id?: string
target_device_id?: string
backend?: 'cua' | 'legacy_compat' | 'system_capture'
dispatch?: 'background' | 'foreground_compatibility'
control_session_id?: string
target_app?: string
target_title?: string
target_pid?: number
target_window_id?: number
action?: string
verification?: 'snapshot_captured' | 'not_requested' | 'failed'
phase?: 'structured_primary' | 'explicit_compatibility' | 'pre_session_safe_fallback'
/** Handler wall time, excluding relay transport latency. */
duration_ms?: number
/** Process exit code when the handler returns one. Non-zero is attention-worthy. */
@@ -45,6 +60,11 @@ export interface AuditEntry {
/** Short success summary (path / exit code / first stdout line). */
summary?: string
error?: string
/** Opaque local evidence identifier. Screenshot pixels never enter JSONL. */
screenshot_evidence_id?: string
screenshot_mime_type?: 'image/png'
screenshot_width?: number
screenshot_height?: number
}
export type AuditCategory = 'command' | 'files' | 'screen' | 'input' | 'devices' | 'system' | 'other'
@@ -52,11 +72,109 @@ export type AuditCategory = 'command' | 'files' | 'screen' | 'input' | 'devices'
/** Rotate the log once it crosses ~1 MB, keeping a single `.1` backup. */
const MAX_BYTES = 1_000_000
const MAX_DETAIL_BYTES = 32_768
const MAX_SCREENSHOT_BYTES = 10_000_000
const MAX_SCREENSHOT_FILES = 20
export function auditLogPath(): string {
return join(homedir(), '.hermes', 'desktop-audit.jsonl')
}
export function auditEvidenceDirectory(): string {
return process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR ?? join(homedir(), '.hermes', 'activity-evidence')
}
type ScreenshotEvidence = Pick<AuditEntry, 'screenshot_evidence_id' | 'screenshot_mime_type' | 'screenshot_width' | 'screenshot_height'>
function screenshotPayload(result: unknown): { base64: string; width?: number; height?: number } | null {
if (!result || typeof result !== 'object' || Array.isArray(result)) return null
const record = result as Record<string, unknown>
const nested = record.after_screenshot
if (nested && typeof nested === 'object' && !Array.isArray(nested)) {
const found = screenshotPayload(nested)
if (found) return found
}
const base64 = typeof record.screenshot_base64 === 'string'
? record.screenshot_base64
: typeof record.bytes_base64 === 'string'
? record.bytes_base64
: null
if (!base64) return null
const display = record.display && typeof record.display === 'object' && !Array.isArray(record.display)
? record.display as Record<string, unknown>
: null
const width = typeof record.screenshot_width === 'number' ? record.screenshot_width
: typeof display?.width === 'number' ? display.width : undefined
const height = typeof record.screenshot_height === 'number' ? record.screenshot_height
: typeof display?.height === 'number' ? display.height : undefined
return { base64, width, height }
}
async function pruneScreenshotEvidence(directory: string, retentionDays: number): Promise<void> {
const now = Date.now()
const maxAgeMs = retentionDays * 24 * 60 * 60 * 1000
const records = await Promise.all((await readdir(directory, { withFileTypes: true }))
.filter(entry => entry.isFile() && /^[a-f0-9]{32}\.png$/.test(entry.name))
.map(async entry => ({ name: entry.name, stats: await stat(join(directory, entry.name)) })))
records.sort((left, right) => right.stats.mtimeMs - left.stats.mtimeMs)
await Promise.all(records
.filter((record, index) => index >= MAX_SCREENSHOT_FILES || now - record.stats.mtimeMs > maxAgeMs)
.map(record => unlink(join(directory, record.name)).catch(() => {})))
}
export async function pruneAuditScreenshotEvidence(retentionDays: 1 | 7 | 30): Promise<void> {
try {
await pruneScreenshotEvidence(auditEvidenceDirectory(), retentionDays)
} catch {
/* Missing or unreadable evidence is equivalent to an empty store. */
}
}
/** Retain a bounded, local screenshot for the activity evidence viewer. The
* audit log stores only an opaque identifier; clearing activity removes the
* evidence directory too. */
export async function persistAuditScreenshot(result: unknown, requestId: string, retentionDays = 7): Promise<ScreenshotEvidence> {
const payload = screenshotPayload(result)
if (!payload) return {}
let bytes: Buffer
try {
bytes = Buffer.from(payload.base64, 'base64')
} catch {
return {}
}
if (!bytes.length || bytes.length > MAX_SCREENSHOT_BYTES || bytes.subarray(0, 8).toString('hex') !== '89504e470d0a1a0a') return {}
const id = createHash('sha256').update(`${requestId}:${Date.now()}`).digest('hex').slice(0, 32)
const directory = auditEvidenceDirectory()
try {
await mkdir(directory, { recursive: true })
await writeFile(join(directory, `${id}.png`), bytes, { mode: 0o600, flag: 'wx' })
await pruneScreenshotEvidence(directory, retentionDays)
return {
screenshot_evidence_id: id,
screenshot_mime_type: 'image/png',
...(payload.width ? { screenshot_width: payload.width } : {}),
...(payload.height ? { screenshot_height: payload.height } : {})
}
} catch {
return {}
}
}
export async function clearAuditScreenshotEvidence(): Promise<void> {
const { rm } = await import('node:fs/promises')
await rm(auditEvidenceDirectory(), { recursive: true, force: true })
}
export async function auditScreenshotEvidenceStatus(): Promise<{ count: number; bytes: number }> {
try {
const records = await Promise.all((await readdir(auditEvidenceDirectory(), { withFileTypes: true }))
.filter(entry => entry.isFile() && /^[a-f0-9]{32}\.png$/.test(entry.name))
.map(entry => stat(join(auditEvidenceDirectory(), entry.name))))
return { count: records.length, bytes: records.reduce((total, value) => total + value.size, 0) }
} catch {
return { count: 0, bytes: 0 }
}
}
export async function appendAudit(entry: AuditEntry): Promise<void> {
const path = auditLogPath()
try {
@@ -127,7 +245,8 @@ function boundedText(value: string): { text: string; truncated: boolean } {
* bodies, environment values, or unbounded process output. */
export function auditDetails(
args: Record<string, unknown>,
result?: unknown
result?: unknown,
options: { redactComputerContent?: boolean } = {}
): Pick<AuditEntry, 'request_detail' | 'stdout' | 'stderr' | 'result_detail' | 'request_truncated' | 'stdout_truncated' | 'stderr_truncated' | 'result_truncated'> {
const safeRequest: Record<string, unknown> = {}
for (const key of [
@@ -141,6 +260,29 @@ export function auditDetails(
request_detail: request.text,
request_truncated: request.truncated || undefined
}
if (options.redactComputerContent) {
if (result && typeof result === 'object' && !Array.isArray(result)) {
const record = result as Record<string, unknown>
const target = record.target && typeof record.target === 'object' && !Array.isArray(record.target)
? record.target as Record<string, unknown>
: {}
const safeResult = {
backend: record.backend,
action: record.action,
status: record.status,
code: record.code,
target: {
pid: target.pid,
windowId: target.windowId
},
redacted: true
}
const value = boundedText(JSON.stringify(safeResult, null, 2))
details.result_detail = value.text
details.result_truncated = value.truncated || undefined
}
return details
}
if (!result || typeof result !== 'object') {
if (result !== undefined) {
const value = boundedText(String(result))
+5
View File
@@ -14,6 +14,7 @@ import { execFileSync } from 'node:child_process'
import type { ProcessPrivilege } from './processPrivilege.js'
import type { HostAccessMode } from './hostAccessPolicy.js'
import type { ComputerControlLifecycleStatus } from '../tools/cuaDriver.js'
export type DaemonState = 'starting' | 'connected' | 'reconnecting' | 'stopped'
@@ -44,12 +45,16 @@ export interface DaemonStatus {
advertised_tools?: number
voice_url?: string | null
last_event?: string
reconnect_attempt?: number | null
retry_at?: number | null
last_error?: string | null
username?: string
privilege?: ProcessPrivilege
computer_use_enabled?: boolean
access_mode?: HostAccessMode
tools_enabled?: boolean
computer_grant?: DaemonComputerGrantStatus
computer_control?: ComputerControlLifecycleStatus
}
export function daemonStatusPath(): string {
+51 -3
View File
@@ -7,6 +7,10 @@ import { grantBridgeDir } from './grantBridge.js'
export interface DesktopUseSettings {
computer_use_enabled: boolean
computer_control_engine: 'legacy' | 'cua'
cua_cursor_enabled: boolean
activity_screenshot_retention_enabled: boolean
activity_screenshot_retention_days: 1 | 7 | 30
updated_at?: string
}
@@ -27,22 +31,36 @@ export function computerGrantCancellationPath(): string {
function normalizeSettings(value: unknown): DesktopUseSettings {
if (!value || typeof value !== 'object' || Array.isArray(value)) {
return { computer_use_enabled: false }
return defaultSettings()
}
const raw = value as Partial<DesktopUseSettings>
return {
computer_use_enabled: raw.computer_use_enabled === true,
computer_control_engine: raw.computer_control_engine === 'legacy' ? 'legacy' : 'cua',
cua_cursor_enabled: raw.cua_cursor_enabled === true,
activity_screenshot_retention_enabled: raw.activity_screenshot_retention_enabled !== false,
activity_screenshot_retention_days: raw.activity_screenshot_retention_days === 1 || raw.activity_screenshot_retention_days === 30 ? raw.activity_screenshot_retention_days : 7,
updated_at: typeof raw.updated_at === 'string' ? raw.updated_at : undefined
}
}
function defaultSettings(): DesktopUseSettings {
return {
computer_use_enabled: false,
computer_control_engine: 'cua',
cua_cursor_enabled: false,
activity_screenshot_retention_enabled: true,
activity_screenshot_retention_days: 7
}
}
export function readDesktopUseSettingsSync(
filePath = desktopUseSettingsPath()
): DesktopUseSettings {
try {
return normalizeSettings(JSON.parse(readFileSync(filePath, 'utf8')))
} catch {
return { computer_use_enabled: false }
return defaultSettings()
}
}
@@ -52,7 +70,7 @@ export async function readDesktopUseSettings(
try {
return normalizeSettings(JSON.parse(await readFile(filePath, 'utf8')))
} catch {
return { computer_use_enabled: false }
return defaultSettings()
}
}
@@ -68,6 +86,7 @@ export async function setDesktopUseEnabled(
filePath = desktopUseSettingsPath()
): Promise<DesktopUseSettings> {
const settings: DesktopUseSettings = {
...await readDesktopUseSettings(filePath),
computer_use_enabled: enabled,
updated_at: new Date().toISOString()
}
@@ -75,6 +94,35 @@ export async function setDesktopUseEnabled(
return settings
}
export async function setComputerControlSettings(
update: Partial<Pick<DesktopUseSettings,
'computer_control_engine' | 'cua_cursor_enabled'>>,
filePath = desktopUseSettingsPath()
): Promise<DesktopUseSettings> {
const settings: DesktopUseSettings = {
...await readDesktopUseSettings(filePath),
...update,
updated_at: new Date().toISOString()
}
await writeJsonAtomic(filePath, settings)
return settings
}
export async function setActivityScreenshotRetention(
enabled: boolean,
days: 1 | 7 | 30,
filePath = desktopUseSettingsPath()
): Promise<DesktopUseSettings> {
const settings: DesktopUseSettings = {
...await readDesktopUseSettings(filePath),
activity_screenshot_retention_enabled: enabled,
activity_screenshot_retention_days: days,
updated_at: new Date().toISOString()
}
await writeJsonAtomic(filePath, settings)
return settings
}
export async function requestComputerGrantCancellation(
reason = 'cancelled from local desktop controls',
filePath = computerGrantCancellationPath()
@@ -0,0 +1,152 @@
import { randomUUID } from 'node:crypto'
export const LEGACY_CONTROL_SESSION_ID = 'legacy-local-control-session'
export interface ComputerControlAuthority {
/** Server-attested identity for one attached desktop router lifecycle. */
controlSessionId: string
/** Canonical relay URL when known. Never supplied by remote tool arguments. */
hostUrl?: string
/** Request currently being dispatched. Bound by the router, not the caller. */
requestId?: string
/** Reserved for authenticated relay identity once the wire protocol supplies it. */
relaySessionId?: string
requesterDeviceId?: string
chatSessionId?: string
runId?: string
targetDeviceId?: string
}
export interface ComputerTarget {
pid: number
windowId: number
app?: string
title?: string
executable?: string
display?: string
folder?: string
}
export interface SnapshotBindingInput {
authority: ComputerControlAuthority
grantId: string | null
target: ComputerTarget
snapshotGeneration: string
driverElementToken?: string
ttlMs?: number
}
interface SnapshotBinding extends SnapshotBindingInput {
expiresAt: number
}
export type SensitiveTargetDecision =
| { allowed: true; reason: 'not_sensitive' }
| { allowed: false; reason: 'missing_target_identity' | 'sensitive_target'; matched?: string }
const DEFAULT_SENSITIVE_PATTERNS = Object.freeze([
/\b(password|passkey|credential|authenticator|security key)\b/i,
/\b(1password|bitwarden|keepass|lastpass|dashlane|proton pass)\b/i,
/\b(bank|banking|payment|checkout|wallet|crypto|cryptocurrency)\b/i,
/\b(certificate|private[ _-]?key|api[ _-]?key|access[ _-]?token|secret)\b/i,
/\b(user account control|windows security|credential manager|security settings)\b/i
])
/**
* Hermes-owned state for a single router lifecycle. It intentionally does not
* execute input. It binds remote requests, local grants, snapshots, and opaque
* element handles before an execution backend (legacy or CUA) is invoked.
*/
export class ComputerControlSecurityState {
private readonly seenRequestIds = new Map<string, { at: number; controlSessionId: string }>()
private readonly snapshotBindings = new Map<string, SnapshotBinding>()
constructor(
readonly authority: ComputerControlAuthority,
private readonly maxRememberedRequests = 2_048
) {}
bindRequest(
requestId: string,
authority: ComputerControlAuthority = this.authority
): ComputerControlAuthority | null {
const normalized = requestId.trim()
if (!normalized || normalized.length > 256 || this.seenRequestIds.has(normalized)) return null
this.seenRequestIds.set(normalized, { at: Date.now(), controlSessionId: authority.controlSessionId })
while (this.seenRequestIds.size > this.maxRememberedRequests) {
const oldest = this.seenRequestIds.keys().next().value as string | undefined
if (!oldest) break
this.seenRequestIds.delete(oldest)
}
return { ...authority, requestId: normalized }
}
issueSnapshotToken(input: SnapshotBindingInput): string {
if (!input.authority.controlSessionId.trim()) throw new Error('snapshot authority requires a control session id')
const token = `hermes-snapshot-${randomUUID()}`
const ttlMs = Math.max(1_000, Math.min(input.ttlMs ?? 60_000, 300_000))
this.snapshotBindings.set(token, { ...input, expiresAt: Date.now() + ttlMs })
return token
}
consumeSnapshotToken(
token: string,
expected: Omit<SnapshotBindingInput, 'ttlMs' | 'driverElementToken' | 'snapshotGeneration'> & {
snapshotGeneration?: string
}
): SnapshotBinding | null {
const binding = this.snapshotBindings.get(token)
this.snapshotBindings.delete(token)
if (!binding || binding.expiresAt <= Date.now()) return null
if (binding.authority.controlSessionId !== expected.authority.controlSessionId) return null
if (binding.grantId !== expected.grantId) return null
if (binding.target.pid !== expected.target.pid || binding.target.windowId !== expected.target.windowId) return null
if (expected.snapshotGeneration && binding.snapshotGeneration !== expected.snapshotGeneration) return null
return binding
}
revoke(): void {
this.seenRequestIds.clear()
this.snapshotBindings.clear()
}
/** Revoke only artifacts owned by one concurrent relay control session. */
revokeAuthority(controlSessionId: string): void {
for (const [requestId, binding] of this.seenRequestIds) {
if (binding.controlSessionId === controlSessionId) this.seenRequestIds.delete(requestId)
}
for (const [token, binding] of this.snapshotBindings) {
if (binding.authority.controlSessionId === controlSessionId) this.snapshotBindings.delete(token)
}
}
}
export function hasAuthenticatedControlIdentity(
authority: ComputerControlAuthority | undefined
): authority is ComputerControlAuthority & Required<Pick<ComputerControlAuthority,
'relaySessionId' | 'requesterDeviceId' | 'runId' | 'targetDeviceId'>> {
return !!authority && [
authority.controlSessionId,
authority.relaySessionId,
authority.requesterDeviceId,
authority.runId,
authority.targetDeviceId
].every(value => typeof value === 'string' && value.trim().length > 0)
}
/** Fail closed when a brokered action does not identify its app/window. */
export function evaluateSensitiveTarget(
target: Pick<ComputerTarget, 'app' | 'title' | 'executable'>,
extraPatterns: readonly RegExp[] = []
): SensitiveTargetDecision {
const identity = [target.app, target.title, target.executable]
.filter((value): value is string => typeof value === 'string' && value.trim().length > 0)
.join(' | ')
if (!identity) return { allowed: false, reason: 'missing_target_identity' }
for (const pattern of [...DEFAULT_SENSITIVE_PATTERNS, ...extraPatterns]) {
pattern.lastIndex = 0
const match = pattern.exec(identity)
if (match) return { allowed: false, reason: 'sensitive_target', matched: match[0] }
}
return { allowed: true, reason: 'not_sensitive' }
}
+128 -40
View File
@@ -1,9 +1,16 @@
import { randomUUID } from 'node:crypto'
import {
DEFAULT_CAPABILITY_POLICIES,
presetCapabilityPolicies,
type CapabilityPolicies,
type HostAccessMode
} from '../lib/hostAccessPolicy.js'
import {
LEGACY_CONTROL_SESSION_ID,
type ComputerControlAuthority,
type ComputerTarget
} from './computerControlSecurity.js'
export type ComputerGrantMode = 'observe' | 'assist' | 'control'
@@ -30,14 +37,34 @@ export interface ComputerUseRuntime {
capabilities: CapabilityPolicies
}
let activeGrant: ComputerGrant | null = null
let grantChangeListener: ((grant: ComputerGrant | null) => void) | null = null
let runtime: ComputerUseRuntime = {
interface ControlSessionState {
activeGrant: ComputerGrant | null
runtime: ComputerUseRuntime
}
const defaultRuntime = (): ComputerUseRuntime => ({
url: null,
computerUseConsented: false,
consentSource: 'none',
accessMode: 'ask',
capabilities: { ...DEFAULT_CAPABILITY_POLICIES }
})
const controlSessions = new Map<string, ControlSessionState>()
let grantChangeListener: ((grant: ComputerGrant | null, controlSessionId?: string) => void) | null = null
function authorityKey(authority?: ComputerControlAuthority): string {
return authority?.controlSessionId || LEGACY_CONTROL_SESSION_ID
}
function sessionState(authority?: ComputerControlAuthority): ControlSessionState {
const key = authorityKey(authority)
let state = controlSessions.get(key)
if (!state) {
state = { activeGrant: null, runtime: defaultRuntime() }
controlSessions.set(key, state)
}
return state
}
function nowMs(): number {
@@ -45,7 +72,7 @@ function nowMs(): number {
}
function newGrantId(): string {
return `computer-grant-${nowMs().toString(36)}-${Math.random().toString(36).slice(2, 8)}`
return `computer-grant-${randomUUID()}`
}
function parseScope(value: unknown): ComputerGrantScope {
@@ -82,18 +109,19 @@ export function normalizeComputerGrantDurationSeconds(value: unknown): number {
: 900
}
function expireIfNeeded(): void {
if (!activeGrant) {
function expireIfNeeded(authority?: ComputerControlAuthority): void {
const state = sessionState(authority)
if (!state.activeGrant) {
return
}
if (Date.parse(activeGrant.expires_at) <= nowMs()) {
activeGrant = null
grantChangeListener?.(null)
if (Date.parse(state.activeGrant.expires_at) <= nowMs()) {
state.activeGrant = null
grantChangeListener?.(null, authorityKey(authority))
}
}
export function setComputerGrantChangeListener(
listener: ((grant: ComputerGrant | null) => void) | null
listener: ((grant: ComputerGrant | null, controlSessionId?: string) => void) | null
): () => void {
const previous = grantChangeListener
grantChangeListener = listener
@@ -102,24 +130,25 @@ export function setComputerGrantChangeListener(
}
}
export function getActiveComputerGrant(): ComputerGrant | null {
expireIfNeeded()
return activeGrant
export function getActiveComputerGrant(authority?: ComputerControlAuthority): ComputerGrant | null {
expireIfNeeded(authority)
return sessionState(authority).activeGrant
}
export function getComputerGrantSummary(): Record<string, unknown> {
if (runtime.capabilities.screen_input === 'allow') {
export function getComputerGrantSummary(authority?: ComputerControlAuthority): Record<string, unknown> {
const state = sessionState(authority)
if (state.runtime.capabilities.screen_input === 'allow') {
return {
active: true,
mode: runtime.accessMode === 'full_access' ? 'full_access' : 'capability_allow',
mode: state.runtime.accessMode === 'full_access' ? 'full_access' : 'capability_allow',
expires_at: null,
scope: null,
reason: runtime.accessMode === 'full_access'
reason: state.runtime.accessMode === 'full_access'
? 'This host has Full Access.'
: 'Screen and input are allowed by this host policy.'
}
}
const grant = getActiveComputerGrant()
const grant = getActiveComputerGrant(authority)
if (!grant) {
return {
active: false,
@@ -138,16 +167,18 @@ export function getComputerGrantSummary(): Record<string, unknown> {
}
}
export function configureComputerUseRuntime(next: Partial<ComputerUseRuntime>): void {
const capabilities = next.capabilities ?? (next.accessMode ? presetCapabilityPolicies(next.accessMode) : runtime.capabilities)
runtime = {
...runtime,
export function configureComputerUseRuntime(next: Partial<ComputerUseRuntime>, authority?: ComputerControlAuthority): void {
const state = sessionState(authority)
const capabilities = next.capabilities ?? (next.accessMode ? presetCapabilityPolicies(next.accessMode) : state.runtime.capabilities)
state.runtime = {
...state.runtime,
...next,
capabilities
}
}
export function getComputerUseRuntimeSummary(): Record<string, unknown> {
export function getComputerUseRuntimeSummary(authority?: ComputerControlAuthority): Record<string, unknown> {
const runtime = sessionState(authority).runtime
return {
url: runtime.url,
consented: runtime.computerUseConsented,
@@ -165,12 +196,14 @@ export interface RequestComputerGrantInput {
reason?: unknown
}
export function requestComputerGrant(input: RequestComputerGrantInput): Record<string, unknown> {
export function requestComputerGrant(input: RequestComputerGrantInput, authority?: ComputerControlAuthority): Record<string, unknown> {
const state = sessionState(authority)
const runtime = state.runtime
if (runtime.capabilities.screen_input === 'allow') {
return {
ok: true,
full_access: runtime.accessMode === 'full_access',
grant: getComputerGrantSummary(),
grant: getComputerGrantSummary(authority),
message: 'This host already has Full Access; no task grant is required.'
}
}
@@ -184,7 +217,7 @@ export function requestComputerGrant(input: RequestComputerGrantInput): Record<s
code: 'computer_use_consent_required',
message:
'Assist/control grants require local desktop-tool consent for this relay URL before task-scoped input grants can be created.',
grant: getComputerGrantSummary()
grant: getComputerGrantSummary(authority)
}
}
@@ -197,12 +230,12 @@ export function requestComputerGrant(input: RequestComputerGrantInput): Record<s
created_at: createdAt.toISOString(),
expires_at: new Date(createdAt.getTime() + durationSeconds * 1000).toISOString()
}
activeGrant = grant
grantChangeListener?.(grant)
state.activeGrant = grant
grantChangeListener?.(grant, authorityKey(authority))
return {
ok: true,
grant: getComputerGrantSummary(),
grant: getComputerGrantSummary(authority),
message:
mode === 'observe'
? 'Observe grant active. Screenshot/status tools may run.'
@@ -210,29 +243,84 @@ export function requestComputerGrant(input: RequestComputerGrantInput): Record<s
}
}
export function cancelComputerGrant(reason = 'cancelled'): Record<string, unknown> {
const previous = getActiveComputerGrant()
activeGrant = null
if (previous) grantChangeListener?.(null)
export function cancelComputerGrant(reason = 'cancelled', authority?: ComputerControlAuthority): Record<string, unknown> {
const state = sessionState(authority)
const previous = getActiveComputerGrant(authority)
state.activeGrant = null
if (previous) grantChangeListener?.(null, authorityKey(authority))
return {
ok: true,
cancelled: previous !== null,
previous_grant: previous,
reason,
grant: getComputerGrantSummary()
grant: getComputerGrantSummary(authority)
}
}
export function hasComputerInputGrant(): boolean {
export function hasComputerInputGrant(authority?: ComputerControlAuthority): boolean {
const runtime = sessionState(authority).runtime
if (runtime.capabilities.screen_input === 'allow') return true
const grant = getActiveComputerGrant()
const grant = getActiveComputerGrant(authority)
return grant?.mode === 'assist' || grant?.mode === 'control'
}
export function hasComputerObserveGrant(): boolean {
return runtime.capabilities.screen_input === 'allow' || getActiveComputerGrant() !== null
export function hasComputerObserveGrant(authority?: ComputerControlAuthority): boolean {
const runtime = sessionState(authority).runtime
return runtime.capabilities.screen_input === 'allow' || getActiveComputerGrant(authority) !== null
}
export function hasFullHostAccess(): boolean {
return runtime.accessMode === 'full_access'
/** Seed a new router lifecycle from the legacy process-level configuration. */
export function initializeComputerControlSession(authority: ComputerControlAuthority): void {
if (controlSessions.has(authorityKey(authority))) return
const legacy = sessionState().runtime
controlSessions.set(authorityKey(authority), {
activeGrant: null,
runtime: { ...legacy, capabilities: { ...legacy.capabilities } }
})
}
export function hasFullHostAccess(authority?: ComputerControlAuthority): boolean {
return sessionState(authority).runtime.accessMode === 'full_access'
}
export function revokeComputerControlSession(authority: ComputerControlAuthority, reason = 'control session ended'): void {
cancelComputerGrant(reason, authority)
controlSessions.delete(authorityKey(authority))
}
export function cancelAllComputerGrants(_reason = 'cancelled locally'): number {
let cancelled = 0
for (const [controlSessionId, state] of controlSessions) {
if (!state.activeGrant) continue
state.activeGrant = null
cancelled += 1
grantChangeListener?.(null, controlSessionId)
}
return cancelled
}
export function expireComputerControlSessions(): void {
for (const controlSessionId of controlSessions.keys()) {
expireIfNeeded({ controlSessionId })
}
}
/** Enforce the scope shown in the approval prompt before a broker acts. */
export function computerGrantAllowsTarget(authority: ComputerControlAuthority, target: ComputerTarget): boolean {
const runtime = sessionState(authority).runtime
if (runtime.capabilities.screen_input === 'allow') return true
const grant = getActiveComputerGrant(authority)
if (!grant) return false
if (grant.scope.display && grant.scope.display !== target.display) return false
if (grant.scope.app) {
const actual = [target.app, target.executable].filter(Boolean).join(' ').toLowerCase()
if (!actual || !actual.includes(grant.scope.app.toLowerCase())) return false
}
if (grant.scope.folder) {
if (!target.folder) return false
const expected = grant.scope.folder.replaceAll('\\', '/').replace(/\/+$/, '').toLowerCase()
const actual = target.folder.replaceAll('\\', '/').toLowerCase()
if (actual !== expected && !actual.startsWith(`${expected}/`)) return false
}
return true
}
+866
View File
@@ -0,0 +1,866 @@
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
import { createHash } from 'node:crypto'
import { access, realpath } from 'node:fs/promises'
import { homedir } from 'node:os'
import { basename, dirname, join, relative, resolve, sep } from 'node:path'
import { readDesktopUseSettingsSync } from '../lib/desktopUseSettings.js'
const SUPPORTED_MIN_VERSION = [0, 19, 3] as const
const SUPPORTED_MAX_VERSION = [0, 20, 0] as const
const DEFAULT_TIMEOUT_MS = 8_000
const MAX_OUTPUT_BYTES = 4 * 1024 * 1024
const REQUIRED_TOOLS = Object.freeze([
'health_report',
'start_session',
'end_session',
'list_windows',
'get_window_state',
'click',
'set_value',
'press_key',
'scroll'
])
const ALLOWED_TOOLS = Object.freeze([...REQUIRED_TOOLS, 'set_agent_cursor_enabled'])
export interface CuaProcessResult {
stdout: string
stderr: string
exitCode: number
}
export interface CuaProcessRunner {
run(
executable: string,
args: readonly string[],
options?: { stdin?: string; timeoutMs?: number; signal?: AbortSignal; env?: NodeJS.ProcessEnv }
): Promise<CuaProcessResult>
}
export interface CuaRuntimeStatus {
available: boolean
ready: boolean
binaryPath?: string
binaryVersion?: string
permissionMode?: 'standard' | 'bounded'
health?: 'ok' | 'not_checked'
reason?: string
}
export interface CuaHealthStatus {
state: 'healthy' | 'degraded' | 'error'
checkedAt: string
overall?: string
reason?: string
temporaryWindowsCompatibility: true
}
export interface CuaControlSessionIdentity {
controlSessionId: string
targetDeviceId: string
hostUrl?: string
requestId?: string
relaySessionId?: string
requesterDeviceId?: string
chatSessionId?: string
runId?: string
}
export interface CuaWindowTarget {
pid: number
windowId: number
}
export interface CuaSnapshotOptions extends CuaWindowTarget {
query?: string
includeScreenshot?: boolean
maxElements?: number
maxDepth?: number
}
export interface CuaElementTarget extends CuaWindowTarget {
elementToken: string
}
export type CuaToolResult = Record<string, unknown>
export type ComputerControlBackend = 'cua' | 'legacy_compat'
export interface ComputerControlBackendSelection {
backend: ComputerControlBackend
reason: 'cua_ready' | 'explicit_compatibility' | 'cua_unavailable_before_session'
selectedAt: string
cursorEnabled: boolean
}
export interface CuaActionEvent {
action: string
target_app?: string
target_pid?: number
target_window_id?: number
verification: 'snapshot_captured' | 'not_requested' | 'failed'
occurred_at: string
}
export interface ComputerControlLifecycleStatus {
active_sessions: number
cursor_enabled: boolean
active_backend: 'cua' | 'legacy_compat' | 'mixed' | 'idle'
last_action: CuaActionEvent | null
}
interface CuaManifest {
schema_version?: unknown
binary_version?: unknown
binary_path?: unknown
}
interface CuaHealthReport {
schema_version?: unknown
driver_version?: unknown
overall?: unknown
}
interface JsonRpcResponse {
id?: number
result?: unknown
error?: { message?: unknown }
}
/**
* CUA is a separately maintained child process. Do not copy the relay daemon's
* environment into it: that environment can contain pairing tokens, provider
* credentials, and other secrets unrelated to desktop control.
*/
export function cuaDriverEnvironment(extra: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv {
const allowed = [
'SystemRoot', 'WINDIR', 'COMSPEC', 'PATHEXT', 'PATH', 'TEMP', 'TMP',
'LOCALAPPDATA', 'APPDATA', 'USERPROFILE', 'HOMEDRIVE', 'HOMEPATH',
'PROCESSOR_ARCHITECTURE', 'PROCESSOR_ARCHITEW6432', 'NUMBER_OF_PROCESSORS'
]
const env: NodeJS.ProcessEnv = {}
for (const key of allowed) {
const value = process.env[key]
if (value !== undefined) env[key] = value
}
return { ...env, ...extra, CUA_DRIVER_RS_TELEMETRY_ENABLED: '0' }
}
class CuaMcpClient {
private readonly child: ChildProcessWithoutNullStreams
private readonly pending = new Map<number, { resolve(value: unknown): void; reject(error: Error): void; timer: NodeJS.Timeout }>()
private nextId = 1
private stdout = ''
private closed = false
private constructor(binaryPath: string) {
this.child = spawn(binaryPath, ['mcp', '--socket', '\\\\.\\pipe\\cua-driver'], {
stdio: ['pipe', 'pipe', 'pipe'],
windowsHide: true,
env: cuaDriverEnvironment()
})
this.child.stdout.on('data', (chunk: Buffer) => this.receive(chunk))
this.child.on('error', error => this.failAll(new CuaRuntimeError(`CUA MCP transport failed: ${error.message}`, 'transport')))
this.child.on('close', code => this.failAll(new CuaRuntimeError(`CUA MCP transport closed (${code ?? 'unknown'})`, 'transport')))
}
static async connect(binaryPath: string, expectedVersion: string): Promise<CuaMcpClient> {
const client = new CuaMcpClient(binaryPath)
const initialized = await client.request('initialize', {
protocolVersion: '2025-06-18',
capabilities: {},
clientInfo: { name: 'hermes-relay-desktop', version: '1' }
}) as { protocolVersion?: unknown; serverInfo?: { version?: unknown } }
if (initialized.protocolVersion !== '2025-06-18') {
client.close()
throw new CuaRuntimeError('CUA MCP protocol version is incompatible', 'incompatible')
}
if (initialized.serverInfo?.version !== expectedVersion) {
client.close()
throw new CuaRuntimeError(
`CUA MCP daemon version ${String(initialized.serverInfo?.version ?? 'unknown')} does not match the verified binary ${expectedVersion}`,
'incompatible'
)
}
client.notify('notifications/initialized', {})
return client
}
async call(tool: string, args: Record<string, unknown>): Promise<CuaToolResult> {
const result = await this.request('tools/call', { name: tool, arguments: args }) as {
isError?: unknown
structuredContent?: unknown
content?: Array<{ type?: unknown; text?: unknown; data?: unknown; mimeType?: unknown }>
}
if (result.isError === true) {
// Driver errors can quote UI labels or entered values. Treat the broker
// result as sensitive and expose only the stable operation name.
throw new CuaRuntimeError(`CUA Driver ${tool} rejected the action`, 'transport')
}
if (result.structuredContent && typeof result.structuredContent === 'object' && !Array.isArray(result.structuredContent)) {
const structured = { ...(result.structuredContent as CuaToolResult) }
const image = result.content?.find(item => item.type === 'image' && typeof item.data === 'string')
if (image) {
structured.screenshot_base64 = image.data
structured.screenshot_mime_type = image.mimeType
}
return structured
}
const text = result.content?.find(item => item.type === 'text' && typeof item.text === 'string')?.text
return text ? parseJsonObject(String(text), `CUA Driver ${tool}`) : {}
}
close(): void {
if (this.closed) return
this.closed = true
this.child.stdin.end()
const timer = setTimeout(() => this.child.kill('SIGKILL'), 1_000)
timer.unref?.()
}
private request(method: string, params: Record<string, unknown>): Promise<unknown> {
if (this.closed) return Promise.reject(new CuaRuntimeError('CUA MCP transport is closed', 'closed'))
const id = this.nextId++
return new Promise((resolvePromise, reject) => {
const timer = setTimeout(() => {
this.pending.delete(id)
reject(new CuaRuntimeError(`CUA MCP ${method} timed out`, 'transport'))
}, DEFAULT_TIMEOUT_MS)
timer.unref?.()
this.pending.set(id, { resolve: resolvePromise, reject, timer })
this.child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', id, method, params })}\n`)
})
}
private notify(method: string, params: Record<string, unknown>): void {
this.child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', method, params })}\n`)
}
private receive(chunk: Buffer): void {
this.stdout += chunk.toString('utf8')
if (Buffer.byteLength(this.stdout, 'utf8') > MAX_OUTPUT_BYTES) {
this.failAll(new CuaRuntimeError('CUA MCP response exceeded the output limit', 'transport'))
this.child.kill('SIGKILL')
return
}
let newline = this.stdout.indexOf('\n')
while (newline >= 0) {
const line = this.stdout.slice(0, newline).trim()
this.stdout = this.stdout.slice(newline + 1)
if (line) this.resolveLine(line)
newline = this.stdout.indexOf('\n')
}
}
private resolveLine(line: string): void {
let message: JsonRpcResponse
try { message = JSON.parse(line) as JsonRpcResponse } catch { return }
if (typeof message.id !== 'number') return
const pending = this.pending.get(message.id)
if (!pending) return
this.pending.delete(message.id)
clearTimeout(pending.timer)
if (message.error) pending.reject(new CuaRuntimeError(`CUA MCP error: ${String(message.error.message ?? 'unknown')}`, 'transport'))
else pending.resolve(message.result)
}
private failAll(error: Error): void {
if (this.closed && this.pending.size === 0) return
this.closed = true
for (const pending of this.pending.values()) {
clearTimeout(pending.timer)
pending.reject(error)
}
this.pending.clear()
}
}
export interface CuaRuntimeOptions {
platform?: NodeJS.Platform
homeDir?: string
runner?: CuaProcessRunner
}
export class CuaRuntimeError extends Error {
constructor(
message: string,
readonly code: 'unavailable' | 'incompatible' | 'degraded' | 'transport' | 'closed'
) {
super(message)
this.name = 'CuaRuntimeError'
}
}
export class SpawnCuaProcessRunner implements CuaProcessRunner {
run(
executable: string,
args: readonly string[],
options: { stdin?: string; timeoutMs?: number; signal?: AbortSignal; env?: NodeJS.ProcessEnv } = {}
): Promise<CuaProcessResult> {
return new Promise((resolvePromise, reject) => {
const child = spawn(executable, [...args], {
stdio: ['pipe', 'pipe', 'pipe'],
windowsHide: true,
env: options.env ?? cuaDriverEnvironment()
})
let stdout = ''
let stderr = ''
let settled = false
let outputBytes = 0
const finish = (error?: Error, exitCode = -1) => {
if (settled) return
settled = true
clearTimeout(timer)
options.signal?.removeEventListener('abort', abort)
if (error) reject(error)
else resolvePromise({ stdout, stderr, exitCode })
}
const abort = () => {
child.kill('SIGKILL')
finish(new CuaRuntimeError('CUA Driver process was cancelled', 'transport'))
}
const append = (current: string, chunk: Buffer): string => {
outputBytes += chunk.length
if (outputBytes > MAX_OUTPUT_BYTES) {
child.kill('SIGKILL')
finish(new CuaRuntimeError('CUA Driver response exceeded the output limit', 'transport'))
return current
}
return current + chunk.toString('utf8')
}
const timer = setTimeout(() => {
child.kill('SIGKILL')
finish(new CuaRuntimeError('CUA Driver process timed out', 'transport'))
}, options.timeoutMs ?? DEFAULT_TIMEOUT_MS)
timer.unref?.()
options.signal?.addEventListener('abort', abort, { once: true })
child.stdout.on('data', (chunk: Buffer) => { stdout = append(stdout, chunk) })
child.stderr.on('data', (chunk: Buffer) => { stderr = append(stderr, chunk) })
child.on('error', error => finish(new CuaRuntimeError(`CUA Driver process failed: ${error.message}`, 'transport')))
child.on('close', code => finish(undefined, code ?? -1))
child.stdin.end(options.stdin ?? '')
})
}
}
function compareVersion(left: readonly number[], right: readonly number[]): number {
for (let i = 0; i < 3; i += 1) {
const delta = (left[i] ?? 0) - (right[i] ?? 0)
if (delta !== 0) return delta
}
return 0
}
function parseVersion(value: string): [number, number, number] | null {
const match = /(?:^|\s)(\d+)\.(\d+)\.(\d+)(?:\s|$)/.exec(value.trim())
return match ? [Number(match[1]), Number(match[2]), Number(match[3])] : null
}
function parseJsonObject(text: string, label: string): Record<string, unknown> {
let parsed: unknown
try {
parsed = JSON.parse(text)
} catch {
throw new CuaRuntimeError(`${label} returned invalid JSON`, 'transport')
}
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new CuaRuntimeError(`${label} returned a non-object response`, 'transport')
}
return parsed as Record<string, unknown>
}
function validatePositiveInteger(value: number, name: string): void {
if (!Number.isSafeInteger(value) || value <= 0) {
throw new CuaRuntimeError(`${name} must be a positive integer`, 'transport')
}
}
function validateElementToken(value: string): void {
if (!/^e[0-9a-f]+$/i.test(value)) {
throw new CuaRuntimeError('elementToken is not a valid opaque CUA element token', 'transport')
}
}
function derivedSessionId(identity: CuaControlSessionIdentity): string {
const fields = [
identity.controlSessionId,
identity.targetDeviceId,
identity.hostUrl ?? '',
identity.relaySessionId ?? '',
identity.requesterDeviceId ?? '',
identity.chatSessionId ?? '',
identity.runId ?? ''
]
if (!identity.controlSessionId.trim() || !identity.targetDeviceId.trim()) {
throw new CuaRuntimeError('CUA control session identity is incomplete', 'transport')
}
return `hermes-${createHash('sha256').update(fields.join('\u0000')).digest('hex').slice(0, 32)}`
}
function controlIdentityFingerprint(identity: CuaControlSessionIdentity): string {
return createHash('sha256').update([
identity.controlSessionId,
identity.targetDeviceId,
identity.hostUrl ?? '',
identity.relaySessionId ?? '',
identity.requesterDeviceId ?? '',
identity.chatSessionId ?? '',
identity.runId ?? ''
].join('\u0000')).digest('hex')
}
async function canonicalBinary(options: CuaRuntimeOptions): Promise<string> {
const platform = options.platform ?? process.platform
if (platform !== 'win32') {
throw new CuaRuntimeError(`CUA Driver adapter is not enabled on ${platform}`, 'unavailable')
}
const home = resolve(options.homeDir ?? homedir())
const packages = join(home, '.cua-driver', 'packages')
const releases = join(packages, 'releases')
const candidate = join(packages, 'current', 'cua-driver.exe')
let resolvedCandidate: string
let resolvedReleases: string
try {
;[resolvedCandidate, resolvedReleases] = await Promise.all([realpath(candidate), realpath(releases)])
await access(resolvedCandidate)
} catch {
throw new CuaRuntimeError('Canonical CUA Driver package is not installed', 'unavailable')
}
const inside = relative(resolvedReleases, resolvedCandidate)
if (!inside || inside.startsWith(`..${sep}`) || inside === '..' || resolve(resolvedCandidate) === resolve(resolvedReleases)) {
throw new CuaRuntimeError('Canonical CUA Driver package resolved outside its release store', 'incompatible')
}
if (basename(resolvedCandidate).toLowerCase() !== 'cua-driver.exe' || basename(dirname(resolvedCandidate)).length === 0) {
throw new CuaRuntimeError('Canonical CUA Driver executable path is invalid', 'incompatible')
}
return resolvedCandidate
}
export class CuaDriverAdapter {
readonly binaryPath: string
readonly binaryVersion: string
readonly permissionMode: 'standard' | 'bounded'
private readonly sessions = new Map<string, CuaControlSession>()
private constructor(
binaryPath: string,
binaryVersion: string,
permissionMode: 'standard' | 'bounded',
private readonly runner: CuaProcessRunner,
private readonly usePersistentMcp: boolean,
private readonly supportsCursorToggle: boolean
) {
this.binaryPath = binaryPath
this.binaryVersion = binaryVersion
this.permissionMode = permissionMode
}
static async connect(options: CuaRuntimeOptions = {}): Promise<CuaDriverAdapter> {
const runner = options.runner ?? new SpawnCuaProcessRunner()
const binaryPath = await canonicalBinary(options)
const run = async (args: readonly string[], stdin?: string): Promise<string> => {
const result = await runner.run(binaryPath, args, {
stdin,
env: cuaDriverEnvironment()
})
if (result.exitCode !== 0) {
throw new CuaRuntimeError(`CUA Driver probe ${args[0] ?? 'command'} failed`, 'transport')
}
return result.stdout.trim()
}
const versionText = await run(['--version'])
const versionTuple = parseVersion(versionText)
if (!versionTuple || compareVersion(versionTuple, SUPPORTED_MIN_VERSION) < 0 || compareVersion(versionTuple, SUPPORTED_MAX_VERSION) >= 0) {
throw new CuaRuntimeError(`Unsupported CUA Driver version: ${versionText || 'unknown'}`, 'incompatible')
}
const manifest = parseJsonObject(await run(['manifest', '--pretty']), 'CUA Driver manifest') as CuaManifest
if (manifest.schema_version !== '1' || manifest.binary_version !== versionTuple.join('.')) {
throw new CuaRuntimeError('CUA Driver manifest schema or version is incompatible', 'incompatible')
}
const manifestPath = typeof manifest.binary_path === 'string' ? await realpath(manifest.binary_path).catch(() => '') : ''
if (resolve(manifestPath).toLowerCase() !== resolve(binaryPath).toLowerCase()) {
throw new CuaRuntimeError('CUA Driver manifest identifies a different executable', 'incompatible')
}
const toolNames = new Set((await run(['list-tools'])).split(/\r?\n/).map(line => line.split(':', 1)[0]?.trim()).filter(Boolean))
const missingTools = REQUIRED_TOOLS.filter(tool => !toolNames.has(tool))
if (missingTools.length > 0) {
throw new CuaRuntimeError(`CUA Driver is missing required tools: ${missingTools.join(', ')}`, 'incompatible')
}
const statusText = await run(['status'])
const permissionMatch = /permission mode:\s*(standard|bounded|unrestricted)\b/i.exec(statusText)
if (!permissionMatch || permissionMatch[1]?.toLowerCase() === 'unrestricted') {
throw new CuaRuntimeError('CUA Driver permission mode is unavailable or unrestricted', 'incompatible')
}
const permissionMode = permissionMatch[1]!.toLowerCase() as 'standard' | 'bounded'
// Temporary Windows compatibility policy for trycua/cua#3103. The global
// health_report performs a whole-desktop UIA walk with a fixed timeout and
// can poison the driver's busy flag after a false timeout. Runtime
// readiness is therefore based on the canonical binary, manifest, tool
// contract, daemon status, and safe permission mode. Individual structured
// actions still fail closed. Keep health_report as an explicit diagnostic
// via healthStatus(), and remove this split when upstream fixes #3103.
return new CuaDriverAdapter(
binaryPath,
versionTuple.join('.'),
permissionMode,
runner,
options.runner === undefined,
toolNames.has('set_agent_cursor_enabled')
)
}
static async status(options: CuaRuntimeOptions = {}): Promise<CuaRuntimeStatus> {
try {
const adapter = await CuaDriverAdapter.connect(options)
return {
available: true,
ready: true,
binaryPath: adapter.binaryPath,
binaryVersion: adapter.binaryVersion,
permissionMode: adapter.permissionMode,
health: 'not_checked',
reason: 'Runtime checks passed; Windows accessibility health is checked separately.'
}
} catch (error) {
const reason = error instanceof Error ? error.message : String(error)
return { available: error instanceof CuaRuntimeError && error.code !== 'unavailable', ready: false, reason }
}
}
static async healthStatus(options: CuaRuntimeOptions = {}): Promise<CuaHealthStatus> {
const checkedAt = new Date().toISOString()
try {
const adapter = await CuaDriverAdapter.connect(options)
const runner = options.runner ?? new SpawnCuaProcessRunner()
const result = await runner.run(adapter.binaryPath, ['call', 'health_report'], {
stdin: '{}',
timeoutMs: DEFAULT_TIMEOUT_MS,
env: cuaDriverEnvironment()
})
if (result.exitCode !== 0) {
return {
state: 'error', checkedAt, temporaryWindowsCompatibility: true,
reason: `CUA Driver health probe exited ${result.exitCode}`
}
}
const health = parseJsonObject(result.stdout.trim(), 'CUA Driver health report') as CuaHealthReport
if (health.schema_version !== '1' || health.driver_version !== adapter.binaryVersion) {
return {
state: 'error', checkedAt, temporaryWindowsCompatibility: true,
reason: 'CUA Driver health report schema or version is incompatible'
}
}
const overall = String(health.overall ?? 'unknown')
return overall === 'ok'
? { state: 'healthy', checkedAt, overall, temporaryWindowsCompatibility: true }
: {
state: 'degraded', checkedAt, overall, temporaryWindowsCompatibility: true,
reason: `CUA Driver reported ${overall}; runtime remains available and actions still fail closed.`
}
} catch (error) {
return {
state: 'error', checkedAt, temporaryWindowsCompatibility: true,
reason: error instanceof Error ? error.message : String(error)
}
}
}
async openSession(identity: CuaControlSessionIdentity, signal?: AbortSignal, cursorEnabled = false): Promise<CuaControlSession> {
const id = derivedSessionId(identity)
if (this.sessions.has(id)) {
throw new CuaRuntimeError('CUA control session is already active', 'transport')
}
if (signal?.aborted) throw new CuaRuntimeError('CUA control session was cancelled', 'transport')
const mcp = this.usePersistentMcp ? await CuaMcpClient.connect(this.binaryPath, this.binaryVersion) : null
const invoke = async (tool: string, args: Record<string, unknown>, invokeSignal?: AbortSignal): Promise<CuaToolResult> => {
if (invokeSignal?.aborted) throw new CuaRuntimeError('CUA action was cancelled', 'transport')
if (!ALLOWED_TOOLS.includes(tool)) throw new CuaRuntimeError('Attempted to invoke a non-allowlisted CUA Driver tool', 'transport')
if (mcp) return mcp.call(tool, args)
const result = await this.runner.run(this.binaryPath, ['call', tool], {
stdin: JSON.stringify(args),
timeoutMs: DEFAULT_TIMEOUT_MS,
signal: invokeSignal,
env: cuaDriverEnvironment()
})
if (result.exitCode !== 0) {
throw new CuaRuntimeError(`CUA Driver ${tool} failed`, 'transport')
}
const payload = parseJsonObject(result.stdout.trim(), `CUA Driver ${tool}`)
if (payload.isError === true) throw new CuaRuntimeError(`CUA Driver ${tool} rejected the action`, 'transport')
return payload
}
await invoke('start_session', { session: id, capture_scope: 'window' }, signal).catch(error => {
mcp?.close()
throw error
})
if (this.supportsCursorToggle) {
await invoke('set_agent_cursor_enabled', { session: id, enabled: cursorEnabled }, signal).catch(error => {
mcp?.close()
throw error
})
}
const session = new CuaControlSession(
id,
invoke,
() => mcp?.close(),
() => this.sessions.delete(id)
)
this.sessions.set(id, session)
return session
}
async closeAll(reason = 'Hermes control authority ended'): Promise<void> {
const sessions = [...this.sessions.values()]
await Promise.allSettled(sessions.map(session => session.close(reason)))
}
}
export class CuaControlSession {
private closed = false
constructor(
private readonly id: string,
private readonly invoke: (tool: string, args: Record<string, unknown>, signal?: AbortSignal) => Promise<CuaToolResult>,
private readonly closeTransport: () => void,
private readonly onClose: () => void
) {}
private ensureOpen(): void {
if (this.closed) throw new CuaRuntimeError('CUA control session is closed', 'closed')
}
async listWindows(pid?: number, signal?: AbortSignal): Promise<CuaToolResult> {
this.ensureOpen()
if (pid !== undefined) validatePositiveInteger(pid, 'pid')
return this.invoke('list_windows', pid === undefined ? {} : { pid }, signal)
}
async snapshot(options: CuaSnapshotOptions, signal?: AbortSignal): Promise<CuaToolResult> {
this.ensureOpen()
validatePositiveInteger(options.pid, 'pid')
validatePositiveInteger(options.windowId, 'windowId')
return this.invoke('get_window_state', {
pid: options.pid,
window_id: options.windowId,
session: this.id,
query: options.query,
include_screenshot: options.includeScreenshot,
max_elements: options.maxElements ?? 1_000,
max_depth: options.maxDepth ?? 20
}, signal)
}
async clickElement(target: CuaElementTarget, signal?: AbortSignal): Promise<CuaToolResult> {
this.ensureOpen()
validatePositiveInteger(target.pid, 'pid')
validatePositiveInteger(target.windowId, 'windowId')
validateElementToken(target.elementToken)
return this.invoke('click', {
pid: target.pid,
window_id: target.windowId,
element_token: target.elementToken,
session: this.id,
scope: 'window',
delivery_mode: 'background'
}, signal)
}
async setElementValue(target: CuaElementTarget, value: string, signal?: AbortSignal): Promise<CuaToolResult> {
this.ensureOpen()
validatePositiveInteger(target.pid, 'pid')
validatePositiveInteger(target.windowId, 'windowId')
validateElementToken(target.elementToken)
return this.invoke('set_value', {
pid: target.pid,
window_id: target.windowId,
element_token: target.elementToken,
value,
session: this.id
}, signal)
}
async pressKey(target: CuaWindowTarget, key: string, signal?: AbortSignal): Promise<CuaToolResult> {
this.ensureOpen()
validatePositiveInteger(target.pid, 'pid')
validatePositiveInteger(target.windowId, 'windowId')
if (!/^(?:return|tab|escape|up|down|left|right|space|delete|home|end|pageup|pagedown|f(?:[1-9]|1[0-2])|[a-z0-9])$/i.test(key)) {
throw new CuaRuntimeError('key is not in the allowlisted CUA key vocabulary', 'transport')
}
return this.invoke('press_key', {
pid: target.pid,
window_id: target.windowId,
key: key.toLowerCase(),
session: this.id,
scope: 'window',
delivery_mode: 'background'
}, signal)
}
async scroll(target: CuaElementTarget, direction: 'up' | 'down' | 'left' | 'right', amount: number, signal?: AbortSignal): Promise<CuaToolResult> {
this.ensureOpen()
validatePositiveInteger(target.pid, 'pid')
validatePositiveInteger(target.windowId, 'windowId')
validateElementToken(target.elementToken)
if (!Number.isSafeInteger(amount) || amount < 1 || amount > 20) {
throw new CuaRuntimeError('scroll amount must be an integer from 1 to 20', 'transport')
}
return this.invoke('scroll', {
pid: target.pid,
window_id: target.windowId,
element_token: target.elementToken,
direction,
amount,
session: this.id,
delivery_mode: 'background'
}, signal)
}
async close(_reason = 'Hermes control session ended'): Promise<void> {
if (this.closed) return
this.closed = true
this.onClose()
try {
await this.invoke('end_session', { session: this.id })
} finally {
this.closeTransport()
}
}
}
let sharedAdapter: Promise<CuaDriverAdapter> | null = null
const sharedSessions = new Map<string, Promise<CuaControlSession>>()
const sharedSessionIdentities = new Map<string, string>()
const backendSelections = new Map<string, ComputerControlBackendSelection>()
let lastActionEvent: CuaActionEvent | null = null
let lifecycleListener: ((status: ComputerControlLifecycleStatus) => void) | null = null
export function setComputerControlLifecycleListener(
listener: ((status: ComputerControlLifecycleStatus) => void) | null
): () => void {
lifecycleListener = listener
listener?.(computerControlLifecycleStatus())
return () => { if (lifecycleListener === listener) lifecycleListener = null }
}
function publishLifecycle(): void {
lifecycleListener?.(computerControlLifecycleStatus())
}
let testSessionFactory: ((identity: CuaControlSessionIdentity) => Promise<CuaControlSession>) | null = null
/** Test-only dependency seam; production callers must never configure this. */
export function setCuaControlSessionFactoryForTests(
factory: ((identity: CuaControlSessionIdentity) => Promise<CuaControlSession>) | null
): void {
testSessionFactory = factory
sharedSessions.clear()
sharedSessionIdentities.clear()
backendSelections.clear()
sharedAdapter = null
}
/** Select once per outer Hermes control session; an active session never changes backend. */
export async function selectComputerControlBackend(
controlSessionId: string,
selected: 'legacy' | 'cua',
cursorEnabled: boolean
): Promise<ComputerControlBackendSelection> {
const existing = backendSelections.get(controlSessionId)
if (existing) return existing
let selection: ComputerControlBackendSelection
if (selected === 'legacy') {
selection = {
backend: 'legacy_compat',
reason: 'explicit_compatibility',
selectedAt: new Date().toISOString(),
cursorEnabled: false
}
} else if (testSessionFactory) {
selection = { backend: 'cua', reason: 'cua_ready', selectedAt: new Date().toISOString(), cursorEnabled }
} else {
try {
sharedAdapter ??= CuaDriverAdapter.connect()
await sharedAdapter
selection = { backend: 'cua', reason: 'cua_ready', selectedAt: new Date().toISOString(), cursorEnabled }
} catch {
sharedAdapter = null
selection = {
backend: 'legacy_compat',
reason: 'cua_unavailable_before_session',
selectedAt: new Date().toISOString(),
cursorEnabled: false
}
}
}
backendSelections.set(controlSessionId, selection)
publishLifecycle()
return selection
}
export function recordCuaActionEvent(event: Omit<CuaActionEvent, 'occurred_at'>): void {
lastActionEvent = { ...event, occurred_at: new Date().toISOString() }
publishLifecycle()
}
export function computerControlLifecycleStatus(): ComputerControlLifecycleStatus {
const active = [...backendSelections.values()]
const backends = new Set(active.map(item => item.backend))
return {
active_sessions: active.length,
cursor_enabled: active.some(item => item.backend === 'cua' && item.cursorEnabled),
active_backend: active.length === 0 ? 'idle' : backends.size > 1 ? 'mixed' : active[0]!.backend,
last_action: lastActionEvent
}
}
/** Return the one bounded CUA session owned by an authenticated Hermes control session. */
export async function getCuaControlSession(identity: CuaControlSessionIdentity): Promise<CuaControlSession> {
const key = identity.controlSessionId
const fingerprint = controlIdentityFingerprint(identity)
const existingFingerprint = sharedSessionIdentities.get(key)
if (existingFingerprint && existingFingerprint !== fingerprint) {
throw new CuaRuntimeError('CUA control session identity changed after session creation', 'incompatible')
}
let session = sharedSessions.get(key)
if (!session) {
if (testSessionFactory) {
session = testSessionFactory(identity)
} else {
sharedAdapter ??= CuaDriverAdapter.connect()
const settings = readDesktopUseSettingsSync()
const selection = backendSelections.get(key)
session = sharedAdapter.then(adapter => adapter.openSession(
identity,
undefined,
selection?.backend === 'cua' ? selection.cursorEnabled : settings.cua_cursor_enabled
))
}
sharedSessions.set(key, session)
sharedSessionIdentities.set(key, fingerprint)
void session.catch(() => {
sharedSessions.delete(key)
sharedSessionIdentities.delete(key)
})
}
return session
}
/** Revoke a CUA cursor/session when its outer Hermes authority ends. */
export async function closeCuaControlSession(controlSessionId: string, reason?: string): Promise<void> {
const session = sharedSessions.get(controlSessionId)
sharedSessions.delete(controlSessionId)
sharedSessionIdentities.delete(controlSessionId)
backendSelections.delete(controlSessionId)
publishLifecycle()
if (session) await session.then(value => value.close(reason)).catch(() => undefined)
}
export async function closeAllCuaControlSessions(reason?: string): Promise<void> {
const sessions = [...sharedSessions.values()]
sharedSessions.clear()
sharedSessionIdentities.clear()
backendSelections.clear()
publishLifecycle()
await Promise.allSettled(sessions.map(session => session.then(value => value.close(reason))))
sharedAdapter = null
}
+390
View File
@@ -0,0 +1,390 @@
import { createHash } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { access, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises'
import { homedir, tmpdir } from 'node:os'
import { delimiter, join, relative, resolve, sep, win32 } from 'node:path'
import {
CuaDriverAdapter,
SpawnCuaProcessRunner,
cuaDriverEnvironment,
type CuaProcessRunner
} from './cuaDriver.js'
export const CUA_SUPPORTED_MIN_VERSION = '0.19.3'
export const CUA_SUPPORTED_MAX_EXCLUSIVE = '0.20.0'
const TRUSTED_REPOSITORY = 'trycua/cua'
const TRUSTED_PRODUCT = 'cua-driver-rs'
const RELEASE_BASE = 'https://github.com/trycua/cua/releases/download'
const MAX_INSTALLER_BYTES = 2 * 1024 * 1024
interface FetchResponse {
ok: boolean
status: number
arrayBuffer(): Promise<ArrayBuffer>
json(): Promise<unknown>
}
export type CuaManagementFetch = (url: string) => Promise<FetchResponse>
export interface CuaUpdateStatus {
checked_at?: string
current_version?: string
latest_version?: string
update_available: boolean
compatible: boolean
error?: string
release_notes_url?: string
}
export interface CuaManagementStatus {
installed: boolean
canonical_path: string | null
discovered_path: string | null
stale_path_shim: boolean
current_version: string | null
compatible: boolean
compatibility_reason: string | null
binary_sha256: string | null
release_source: typeof TRUSTED_REPOSITORY
telemetry_enabled: false
bundled: false
supported_range: {
minimum: typeof CUA_SUPPORTED_MIN_VERSION
maximum_exclusive: typeof CUA_SUPPORTED_MAX_EXCLUSIVE
}
update?: CuaUpdateStatus
operation?: {
kind: 'install' | 'update'
state: 'completed'
version: string
release_manifest_verified: true
installer_checksum_verified: true
runtime_verified: true
}
}
export interface CuaManagementOptions {
platform?: NodeJS.Platform
arch?: string
homeDir?: string
path?: string
runner?: CuaProcessRunner
fetch?: CuaManagementFetch
systemRoot?: string
}
interface TrustedRelease {
version: string
installerUrl: string
installerSha256: string
}
function versionTuple(value: string): [number, number, number] | null {
const match = /(?:^|\s|v)(\d+)\.(\d+)\.(\d+)(?:\s|$)/.exec(value.trim())
return match ? [Number(match[1]), Number(match[2]), Number(match[3])] : null
}
function compareVersion(left: string, right: string): number | null {
const a = versionTuple(left)
const b = versionTuple(right)
if (!a || !b) return null
for (let index = 0; index < 3; index += 1) {
const delta = a[index] - b[index]
if (delta !== 0) return delta
}
return 0
}
export function isSupportedCuaVersion(value: string): boolean {
const minimum = compareVersion(value, CUA_SUPPORTED_MIN_VERSION)
const maximum = compareVersion(value, CUA_SUPPORTED_MAX_EXCLUSIVE)
return minimum !== null && maximum !== null && minimum >= 0 && maximum < 0
}
function canonicalPaths(homeDir: string): { executable: string; releases: string } {
const packages = join(resolve(homeDir), '.cua-driver', 'packages')
return {
executable: join(packages, 'current', 'cua-driver.exe'),
releases: join(packages, 'releases')
}
}
async function resolveCanonical(options: CuaManagementOptions): Promise<string | null> {
if ((options.platform ?? process.platform) !== 'win32') return null
const paths = canonicalPaths(options.homeDir ?? homedir())
try {
const [binary, releases] = await Promise.all([
realpath(paths.executable),
realpath(paths.releases)
])
await access(binary)
const inside = relative(releases, binary)
if (!inside || inside === '..' || inside.startsWith(`..${sep}`)) return null
return binary
} catch {
return null
}
}
async function firstPathCandidate(options: CuaManagementOptions): Promise<string | null> {
if ((options.platform ?? process.platform) !== 'win32') return null
for (const entry of (options.path ?? process.env.PATH ?? '').split(delimiter)) {
const trimmed = entry.trim().replace(/^"|"$/g, '')
if (!trimmed) continue
const candidate = join(trimmed, 'cua-driver.exe')
try {
await access(candidate)
return await realpath(candidate)
} catch {
// Continue to the next PATH entry.
}
}
return null
}
async function sha256File(path: string): Promise<string> {
return new Promise((resolvePromise, reject) => {
const hash = createHash('sha256')
const stream = createReadStream(path)
stream.on('data', chunk => hash.update(chunk))
stream.on('error', reject)
stream.on('end', () => resolvePromise(hash.digest('hex')))
})
}
function safeError(error: unknown): string {
return (error instanceof Error ? error.message : String(error)).slice(0, 500)
}
async function binaryVersion(binary: string, runner: CuaProcessRunner): Promise<string | null> {
const result = await runner.run(binary, ['--version'], {
timeoutMs: 8_000,
env: cuaDriverEnvironment()
})
if (result.exitCode !== 0) return null
const tuple = versionTuple(result.stdout)
return tuple?.join('.') ?? null
}
export async function getCuaManagementStatus(
options: CuaManagementOptions = {}
): Promise<CuaManagementStatus> {
const runner = options.runner ?? new SpawnCuaProcessRunner()
const [canonical, discovered] = await Promise.all([
resolveCanonical(options),
firstPathCandidate(options)
])
let currentVersion: string | null = null
let binarySha256: string | null = null
let compatibilityReason: string | null = null
if (canonical) {
try {
;[currentVersion, binarySha256] = await Promise.all([
binaryVersion(canonical, runner),
sha256File(canonical)
])
} catch (error) {
compatibilityReason = safeError(error)
}
}
const compatible = !!currentVersion && isSupportedCuaVersion(currentVersion)
if (canonical && !compatibilityReason && !compatible) {
compatibilityReason = currentVersion
? `CUA Driver ${currentVersion} is outside the supported range`
: 'CUA Driver version could not be verified'
}
return {
installed: canonical !== null,
canonical_path: canonical,
discovered_path: discovered,
stale_path_shim: !!canonical && !!discovered && resolve(canonical).toLowerCase() !== resolve(discovered).toLowerCase(),
current_version: currentVersion,
compatible,
compatibility_reason: compatibilityReason,
binary_sha256: binarySha256,
release_source: TRUSTED_REPOSITORY,
telemetry_enabled: false,
bundled: false,
supported_range: {
minimum: CUA_SUPPORTED_MIN_VERSION,
maximum_exclusive: CUA_SUPPORTED_MAX_EXCLUSIVE
}
}
}
function normalizeUpdatePayload(payload: Record<string, unknown>): CuaUpdateStatus {
const latest = typeof payload.latest_version === 'string' ? payload.latest_version : undefined
const error = typeof payload.error === 'string' && payload.error.trim() ? payload.error.slice(0, 500) : undefined
return {
checked_at: typeof payload.checked_at === 'string' ? payload.checked_at : undefined,
current_version: typeof payload.current_version === 'string' ? payload.current_version : undefined,
latest_version: latest,
update_available: payload.update_available === true,
compatible: !!latest && isSupportedCuaVersion(latest),
error,
release_notes_url: typeof payload.release_notes_url === 'string' ? payload.release_notes_url : undefined
}
}
export async function checkCuaUpdate(
options: CuaManagementOptions = {}
): Promise<CuaManagementStatus> {
const runner = options.runner ?? new SpawnCuaProcessRunner()
let status = await getCuaManagementStatus({ ...options, runner })
for (let attempt = 0; !status.installed && attempt < 2; attempt += 1) {
await new Promise(resolvePromise => setTimeout(resolvePromise, 50))
status = await getCuaManagementStatus({ ...options, runner })
}
if (!status.canonical_path) {
return { ...status, update: { update_available: false, compatible: false, error: 'CUA Driver is not installed' } }
}
const result = await runner.run(status.canonical_path, ['check-update', '--json', '--no-cache'], {
timeoutMs: 30_000,
env: cuaDriverEnvironment()
})
if (result.exitCode !== 0) {
return {
...status,
update: {
update_available: false,
compatible: false,
error: result.stderr.trim().slice(0, 500) || `check-update exited ${result.exitCode}`
}
}
}
try {
const payload = JSON.parse(result.stdout) as Record<string, unknown>
return { ...status, update: normalizeUpdatePayload(payload) }
} catch {
return { ...status, update: { update_available: false, compatible: false, error: 'CUA Driver returned invalid update JSON' } }
}
}
async function trustedRelease(version: string, fetchImpl: CuaManagementFetch): Promise<TrustedRelease> {
if (!isSupportedCuaVersion(version)) {
throw new Error(`CUA Driver ${version} is outside the Hermes-supported range`)
}
const tag = `cua-driver-rs-v${version}`
const base = `${RELEASE_BASE}/${tag}`
const manifestResponse = await fetchImpl(`${base}/release-manifest.json`)
if (!manifestResponse.ok) throw new Error(`CUA release manifest request failed (${manifestResponse.status})`)
const manifest = await manifestResponse.json() as Record<string, unknown>
if (
manifest.schemaVersion !== 1 || manifest.repository !== TRUSTED_REPOSITORY ||
manifest.product !== TRUSTED_PRODUCT || manifest.version !== version || manifest.tag !== tag
) {
throw new Error('CUA release publisher or version manifest is invalid')
}
const assets = Array.isArray(manifest.assets) ? manifest.assets : []
const installer = assets.find(asset =>
!!asset && typeof asset === 'object' && (asset as Record<string, unknown>).name === 'install.ps1'
) as Record<string, unknown> | undefined
const checksum = typeof installer?.sha256 === 'string' ? installer.sha256.toLowerCase() : ''
if (!/^[0-9a-f]{64}$/.test(checksum)) throw new Error('CUA release installer checksum is missing')
return { version, installerUrl: `${base}/install.ps1`, installerSha256: checksum }
}
async function applyTrustedRelease(
release: TrustedRelease,
runner: CuaProcessRunner,
fetchImpl: CuaManagementFetch,
systemRootOverride?: string
): Promise<void> {
const response = await fetchImpl(release.installerUrl)
if (!response.ok) throw new Error(`CUA installer request failed (${response.status})`)
const script = Buffer.from(await response.arrayBuffer())
if (script.byteLength === 0 || script.byteLength > MAX_INSTALLER_BYTES) {
throw new Error('CUA installer size is invalid')
}
const actual = createHash('sha256').update(script).digest('hex')
if (actual !== release.installerSha256) throw new Error('CUA installer checksum verification failed')
const directory = await mkdtemp(join(tmpdir(), 'hermes-cua-install-'))
const path = join(directory, 'install.ps1')
try {
await writeFile(path, script, { mode: 0o600 })
const configuredSystemRoot = systemRootOverride ?? process.env.SystemRoot ?? process.env.WINDIR
if (!configuredSystemRoot || !configuredSystemRoot.match(/^[A-Za-z]:[\\/]/)) {
throw new Error('Windows system PowerShell path is unavailable')
}
const powershell = win32.join(win32.resolve(configuredSystemRoot), 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe')
const result = await runner.run(powershell, [
'-NoProfile',
'-NonInteractive',
'-ExecutionPolicy',
'Bypass',
'-File',
path
], {
timeoutMs: 15 * 60_000,
env: cuaDriverEnvironment({
CUA_DRIVER_RS_VERSION: release.version,
CUA_DRIVER_INSTALL_CHANNEL: 'install_script'
})
})
if (result.exitCode !== 0) {
throw new Error(result.stderr.trim().slice(0, 500) || `CUA installer exited ${result.exitCode}`)
}
} finally {
await rm(directory, { recursive: true, force: true })
}
}
async function mutateCua(
kind: 'install' | 'update',
version: string,
options: CuaManagementOptions
): Promise<CuaManagementStatus> {
if ((options.platform ?? process.platform) !== 'win32') throw new Error('CUA management is currently available on Windows only')
const runner = options.runner ?? new SpawnCuaProcessRunner()
const fetchImpl = options.fetch ?? (globalThis.fetch as unknown as CuaManagementFetch)
const release = await trustedRelease(version, fetchImpl)
await applyTrustedRelease(release, runner, fetchImpl, options.systemRoot)
const status = await getCuaManagementStatus({ ...options, runner })
if (!status.installed || status.current_version !== version || !status.compatible) {
throw new Error(
`CUA canonical package did not pass post-install version verification ` +
`(installed=${status.installed}, version=${status.current_version ?? 'unknown'}, compatible=${status.compatible})`
)
}
const runtime = await CuaDriverAdapter.status({
platform: options.platform,
homeDir: options.homeDir,
runner
})
if (
!runtime.ready || runtime.binaryVersion !== version ||
!runtime.binaryPath || !status.canonical_path ||
resolve(runtime.binaryPath).toLowerCase() !== resolve(status.canonical_path).toLowerCase()
) {
throw new Error(runtime.reason ?? 'CUA canonical manifest, path, or health verification failed')
}
return {
...status,
operation: {
kind,
state: 'completed',
version,
release_manifest_verified: true,
installer_checksum_verified: true,
runtime_verified: true
}
}
}
export async function installCuaDriver(options: CuaManagementOptions = {}): Promise<CuaManagementStatus> {
const existing = await getCuaManagementStatus(options)
if (existing.installed) return existing
return mutateCua('install', CUA_SUPPORTED_MIN_VERSION, options)
}
export async function updateCuaDriver(options: CuaManagementOptions = {}): Promise<CuaManagementStatus> {
const checked = await checkCuaUpdate(options)
const update = checked.update
if (!update || update.error) throw new Error(update?.error ?? 'CUA update check failed')
if (!update.update_available) return checked
if (!update.latest_version || !update.compatible) {
throw new Error(`CUA Driver ${update.latest_version ?? 'unknown'} is available but unsupported by this Hermes Relay build`)
}
return mutateCua('update', update.latest_version, options)
}
+243 -19
View File
@@ -2,6 +2,7 @@ import { spawn } from 'node:child_process'
import {
cancelComputerGrant,
computerGrantAllowsTarget,
getActiveComputerGrant,
getComputerGrantSummary,
getComputerUseRuntimeSummary,
@@ -13,12 +14,22 @@ import {
requestComputerGrant,
type ComputerGrantMode
} from '../computerGrants.js'
import { evaluateSensitiveTarget, hasAuthenticatedControlIdentity, type ComputerTarget } from '../computerControlSecurity.js'
import {
CuaDriverAdapter,
closeCuaControlSession,
computerControlLifecycleStatus,
getCuaControlSession,
recordCuaActionEvent,
selectComputerControlBackend
} from '../cuaDriver.js'
import { approveComputerGrant } from '../computerActionApproval.js'
import {
runComputerInputAction,
validateComputerAction
} from '../computerInput.js'
import type { ToolHandler } from '../router.js'
import type { ToolContext, ToolHandler } from '../router.js'
import { readDesktopUseSettingsSync } from '../../lib/desktopUseSettings.js'
import { screenshotHandler } from './screenshot.js'
const STATUS_TIMEOUT_MS = 5_000
@@ -212,13 +223,18 @@ function pngDimensions(base64: string): { width: number; height: number } | null
}
}
function failure(code: string, message: string, extra: Record<string, unknown> = {}): Record<string, unknown> {
function failure(
code: string,
message: string,
extra: Record<string, unknown> = {},
authority?: ToolContext['controlSession']
): Record<string, unknown> {
return {
ok: false,
code,
message,
...EXPERIMENTAL_META,
grant: getComputerGrantSummary(),
grant: getComputerGrantSummary(authority),
...extra
}
}
@@ -230,17 +246,178 @@ function parseGrantMode(value: unknown): ComputerGrantMode | null {
return null
}
function positiveInteger(value: unknown): number | null {
return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : null
}
function cuaTarget(args: Record<string, unknown>): Pick<ComputerTarget, 'pid' | 'windowId'> | null {
const pid = positiveInteger(args.pid)
const windowId = positiveInteger(args.window_id)
if (pid === null || windowId === null) return null
return { pid, windowId }
}
async function cuaPreflight(args: Record<string, unknown>, ctx: ToolContext): Promise<
| { ok: true; target: ComputerTarget; grantId: string | null; session: Awaited<ReturnType<typeof getCuaControlSession>> }
| { ok: false; result: Record<string, unknown> }
> {
if (!hasAuthenticatedControlIdentity(ctx.controlSession) || !ctx.controlSecurity) {
return { ok: false, result: failure('authenticated_control_session_required', 'Structured CUA control requires server-attested relay, requester, run, and target identity.', {}, ctx.controlSession) }
}
const requested = cuaTarget(args)
if (!requested) return { ok: false, result: failure('invalid_target', 'Structured CUA control requires positive pid and window_id.', {}, ctx.controlSession) }
const session = await getCuaControlSession({ ...ctx.controlSession, targetDeviceId: ctx.controlSession.targetDeviceId })
const listed = await session.listWindows(requested.pid, ctx.abortSignal)
const windows = Array.isArray(listed.windows) ? listed.windows : []
const exact = windows.find(item => isObject(item) && item.window_id === requested.windowId)
if (!isObject(exact)) {
return { ok: false, result: failure('target_not_found', 'CUA Driver did not report the exact requested PID and window.', {}, ctx.controlSession) }
}
const target: ComputerTarget = {
...requested,
app: typeof exact.app_name === 'string' ? exact.app_name : undefined,
title: typeof exact.title === 'string' ? exact.title : undefined,
executable: typeof exact.executable === 'string' ? exact.executable : undefined
}
const sensitive = evaluateSensitiveTarget(target)
if (!sensitive.allowed) {
return { ok: false, result: failure('sensitive_target_blocked', 'Structured control is blocked for missing or sensitive application identity.', { reason: sensitive.reason }, ctx.controlSession) }
}
if (!computerGrantAllowsTarget(ctx.controlSession, target)) {
return { ok: false, result: failure('grant_target_mismatch', 'The active computer grant does not cover this application target.', {}, ctx.controlSession) }
}
return { ok: true, target, grantId: getActiveComputerGrant(ctx.controlSession)?.id ?? null, session }
}
async function cuaSnapshot(args: Record<string, unknown>, ctx: ToolContext): Promise<Record<string, unknown>> {
const check = await cuaPreflight(args, ctx)
if (!check.ok) return check.result
const authority = ctx.controlSession!
const raw = await check.session.snapshot({
pid: check.target.pid,
windowId: check.target.windowId,
query: argString(args.query).trim() || undefined,
includeScreenshot: args.include_screenshot !== false,
maxElements: typeof args.max_elements === 'number' ? args.max_elements : undefined,
maxDepth: typeof args.max_depth === 'number' ? args.max_depth : undefined
}, ctx.abortSignal)
const generation = typeof raw.snapshot_id === 'string' ? raw.snapshot_id : ''
if (!generation) return failure('invalid_backend_response', 'CUA Driver snapshot did not include a generation.', {}, authority)
const elements = Array.isArray(raw.elements) ? raw.elements : []
const safeElements = elements.slice(0, 1_000).map(item => {
if (!isObject(item)) return item
const driverToken = typeof item.element_token === 'string' ? item.element_token : undefined
const output = { ...item }
delete output.element_token
if (driverToken) {
output.snapshot_token = ctx.controlSecurity!.issueSnapshotToken({
authority,
grantId: check.grantId,
target: check.target,
snapshotGeneration: generation,
driverElementToken: driverToken
})
}
return output
})
return {
ok: true,
...EXPERIMENTAL_META,
backend: 'cua_driver',
snapshot_generation: generation,
target: check.target,
elements: safeElements,
tree_markdown: raw.tree_markdown,
screenshot_base64: raw.screenshot_base64,
screenshot_width: raw.screenshot_width,
screenshot_height: raw.screenshot_height,
truncated: elements.length > safeElements.length
}
}
async function cuaAction(args: Record<string, unknown>, ctx: ToolContext): Promise<Record<string, unknown>> {
const check = await cuaPreflight(args, ctx)
if (!check.ok) return check.result
const authority = ctx.controlSession!
const token = argString(args.snapshot_token).trim()
const binding = token ? ctx.controlSecurity!.consumeSnapshotToken(token, {
authority,
grantId: check.grantId,
target: check.target,
...(typeof args.snapshot_generation === 'string' ? { snapshotGeneration: args.snapshot_generation } : {})
}) : null
if (!binding?.driverElementToken) {
return failure('invalid_or_stale_snapshot', 'A fresh one-use snapshot_token for this exact session, grant, PID, and window is required.', {}, authority)
}
const element = { pid: check.target.pid, windowId: check.target.windowId, elementToken: binding.driverElementToken }
const action = argString(args.action).trim()
let result: Record<string, unknown>
if (action === 'click_element') result = await check.session.clickElement(element, ctx.abortSignal)
else if (action === 'set_value') result = await check.session.setElementValue(element, argString(args.value), ctx.abortSignal)
else if (action === 'press_key') result = await check.session.pressKey(check.target, argString(args.key), ctx.abortSignal)
else if (action === 'scroll_element') {
const direction = argString(args.direction) as 'up' | 'down' | 'left' | 'right'
if (!['up', 'down', 'left', 'right'].includes(direction)) return failure('invalid_request', 'direction must be up, down, left, or right.', {}, authority)
result = await check.session.scroll(element, direction, positiveInteger(args.amount) ?? 1, ctx.abortSignal)
} else return failure('invalid_request', 'Unsupported structured CUA action.', {}, authority)
const after = await check.session.snapshot({ pid: check.target.pid, windowId: check.target.windowId, includeScreenshot: false }, ctx.abortSignal)
recordCuaActionEvent({
action,
target_app: check.target.app,
target_pid: check.target.pid,
target_window_id: check.target.windowId,
verification: 'snapshot_captured'
})
return {
ok: true,
...EXPERIMENTAL_META,
backend: 'cua',
dispatch: 'background',
control_session_id: authority.controlSessionId,
target_app: check.target.app,
target_title: check.target.title,
target_pid: check.target.pid,
target_window_id: check.target.windowId,
action,
verification: 'snapshot_captured',
phase: 'structured_primary',
result,
verification_snapshot: after
}
}
export const computerStatusHandler: ToolHandler = async (_args, ctx) => {
const grant = getActiveComputerGrant()
const runtime = getComputerUseRuntimeSummary()
const grant = getActiveComputerGrant(ctx.controlSession)
const runtime = getComputerUseRuntimeSummary(ctx.controlSession)
const inputBackendReady = runtime.consented === true && process.platform === 'win32'
const fullAccess = runtime.full_access === true
const settings = readDesktopUseSettingsSync()
const cua = settings.computer_control_engine === 'cua' ? await CuaDriverAdapter.status() : null
const lifecycle = computerControlLifecycleStatus()
return {
ok: true,
...EXPERIMENTAL_META,
platform: process.platform,
displays: await getDisplays(),
runtime,
computer_control_engine: {
...lifecycle,
selected: settings.computer_control_engine,
effective: lifecycle.active_backend === 'cua'
? 'cua'
: lifecycle.active_backend === 'legacy_compat'
? 'legacy'
: settings.computer_control_engine === 'cua' && cua?.ready ? 'cua' : 'legacy',
cursor_enabled: settings.cua_cursor_enabled,
foreground_escalation_enabled: false,
cua,
message: settings.computer_control_engine === 'cua' && !cua?.ready
? cua?.available
? `CUA Driver is installed, but not ready; new sessions use Windows Input compatibility mode. ${cua.reason ?? ''}`.trim()
: `CUA Driver is unavailable before control starts; new sessions use Windows Input compatibility mode. ${cua?.reason ?? ''}`.trim()
: null
},
permissions: {
screenshot: fullAccess ? 'full_access' : grant ? 'granted' : 'grant_required',
input: fullAccess || grant?.mode === 'assist' || grant?.mode === 'control'
@@ -248,9 +425,9 @@ export const computerStatusHandler: ToolHandler = async (_args, ctx) => {
? fullAccess ? 'full_access' : 'granted_until_expiry'
: 'grant_active_but_input_backend_unavailable'
: 'not_granted',
accessibility: 'not_implemented'
accessibility: cua?.ready ? 'available' : 'unavailable'
},
grant: getComputerGrantSummary(),
grant: getComputerGrantSummary(ctx.controlSession),
overlay: {
visible: ctx.interactive,
state: ctx.interactive ? 'cli_grant_prompt_available' : 'not_available',
@@ -267,13 +444,25 @@ export const computerStatusHandler: ToolHandler = async (_args, ctx) => {
}
export const computerScreenshotHandler: ToolHandler = async (args, ctx) => {
if (!hasComputerObserveGrant()) {
if (!hasComputerObserveGrant(ctx.controlSession)) {
return failure(
'grant_required',
'Screenshot observe mode requires an active observe/assist/control grant. Call desktop_computer_grant_request first.'
'Screenshot observe mode requires an active observe/assist/control grant. Call desktop_computer_grant_request first.',
{},
ctx.controlSession
)
}
const settings = readDesktopUseSettingsSync()
const selection = ctx.controlSession
? await selectComputerControlBackend(ctx.controlSession.controlSessionId, settings.computer_control_engine, settings.cua_cursor_enabled)
: null
if (selection?.backend === 'cua') {
// Window-scoped observation belongs to CUA. A caller may still request the
// existing read-only display capture; it is not an input-backend fallback.
if (args.pid !== undefined || args.window_id !== undefined) return cuaSnapshot(args, ctx)
}
if (args.region !== undefined && args.region !== null) {
return failure(
'not_implemented',
@@ -299,6 +488,7 @@ export const computerScreenshotHandler: ToolHandler = async (args, ctx) => {
ok: true,
...EXPERIMENTAL_META,
mode: 'observe',
backend: 'system_capture',
format: result.format,
bytes_base64: result.bytes_base64,
saved_path: result.saved_path,
@@ -318,7 +508,7 @@ export const computerScreenshotHandler: ToolHandler = async (args, ctx) => {
applied: false,
reason: 'Sensitive-window redaction is planned but not implemented yet.'
},
grant: getComputerGrantSummary()
grant: getComputerGrantSummary(ctx.controlSession)
}
}
@@ -327,17 +517,39 @@ export const computerActionHandler: ToolHandler = async (args, ctx) => {
if (!action) {
return failure('invalid_request', 'desktop_computer_action requires an action name.')
}
const settings = readDesktopUseSettingsSync()
const selection = ctx.controlSession
? await selectComputerControlBackend(ctx.controlSession.controlSessionId, settings.computer_control_engine, settings.cua_cursor_enabled)
: null
if (selection?.backend === 'cua') {
if (!['click_element', 'set_value', 'press_key', 'scroll_element'].includes(action)) {
return failure(
'cua_structured_action_required',
'CUA is selected; legacy coordinate and foreground-routed input is disabled. Take a structured snapshot and use an element action.',
{ action },
ctx.controlSession
)
}
if (!hasComputerInputGrant(ctx.controlSession)) {
return failure('grant_required', 'Structured host input requires an active assist/control grant.', { action }, ctx.controlSession)
}
return cuaAction(args, ctx)
}
if (['click_element', 'set_value', 'press_key', 'scroll_element'].includes(action)) {
return failure('cua_unavailable', 'Structured CUA actions are unavailable in the compatibility backend.', { action }, ctx.controlSession)
}
const displays = await getDisplays()
const validation = validateComputerAction(args, displays)
if (!validation.ok) {
return failure(validation.code, validation.message, { action })
}
if (!hasComputerInputGrant()) {
if (!hasComputerInputGrant(ctx.controlSession)) {
return failure(
'grant_required',
'Host input is disabled. Request and locally approve an assist/control grant first.',
{ action }
{ action },
ctx.controlSession
)
}
@@ -352,12 +564,21 @@ export const computerActionHandler: ToolHandler = async (args, ctx) => {
performed_at: new Date().toISOString(),
duration_ms: Date.now() - started,
input_backend: inputResult.backend,
backend: 'legacy_compat',
dispatch: 'foreground_compatibility',
phase: selection?.reason === 'cua_unavailable_before_session' ? 'pre_session_safe_fallback' : 'explicit_compatibility',
control_session_id: ctx.controlSession?.controlSessionId,
verification: normalized.returnScreenshot ? 'snapshot_captured' : 'not_requested',
platform: inputResult.platform,
grant: getComputerGrantSummary()
grant: getComputerGrantSummary(ctx.controlSession)
}
if (normalized.returnScreenshot) {
response.after_screenshot = await computerScreenshotHandler({ display: args.display ?? 'primary' }, ctx)
}
recordCuaActionEvent({
action: normalized.action,
verification: normalized.returnScreenshot ? 'snapshot_captured' : 'not_requested'
})
return response
}
@@ -366,7 +587,7 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
if (!mode) {
return failure('invalid_request', 'mode must be one of observe, assist, or control.')
}
const runtime = getComputerUseRuntimeSummary()
const runtime = getComputerUseRuntimeSummary(ctx.controlSession)
if (runtime.full_access === true) {
return {
...requestComputerGrant({
@@ -374,7 +595,7 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
scope: args.scope,
duration_seconds: args.duration_seconds,
reason: args.reason
}),
}, ctx.controlSession),
...EXPERIMENTAL_META
}
}
@@ -382,7 +603,9 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
if (runtime.consented !== true) {
return failure(
'computer_use_consent_required',
'Assist/control grants require local desktop-tool consent for this relay URL before task-scoped input grants can be created.'
'Assist/control grants require local desktop-tool consent for this relay URL before task-scoped input grants can be created.',
{},
ctx.controlSession
)
}
const approval = await approveComputerGrant({
@@ -406,17 +629,18 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
scope: args.scope,
duration_seconds: args.duration_seconds,
reason: args.reason
}),
}, ctx.controlSession),
...EXPERIMENTAL_META
}
}
export const computerCancelHandler: ToolHandler = async (args) => {
export const computerCancelHandler: ToolHandler = async (args, ctx) => {
const reason = typeof args.reason === 'string' && args.reason.trim()
? args.reason.trim()
: 'cancelled by desktop_computer_cancel'
if (ctx.controlSession) await closeCuaControlSession(ctx.controlSession.controlSessionId, reason)
return {
...cancelComputerGrant(reason),
...cancelComputerGrant(reason, ctx.controlSession),
...EXPERIMENTAL_META
}
}
+186 -9
View File
@@ -30,18 +30,31 @@ import {
auditDetails,
categorizeTool,
previewArgs,
persistAuditScreenshot,
resultExitCode,
summarizeResult
} from '../lib/auditLog.js'
import { VERSION } from '../version.js'
import { desktopDeviceId } from '../deviceIdentity.js'
import { getComputerGrantSummary, getComputerUseRuntimeSummary } from './computerGrants.js'
import { readDesktopUseSettingsSync } from '../lib/desktopUseSettings.js'
import {
getComputerGrantSummary,
getComputerUseRuntimeSummary,
initializeComputerControlSession,
revokeComputerControlSession
} from './computerGrants.js'
import {
ComputerControlSecurityState,
type ComputerControlAuthority
} from './computerControlSecurity.js'
import { closeAllCuaControlSessions, closeCuaControlSession } from './cuaDriver.js'
/** The payload shape server → client for a single tool invocation. */
export interface ToolCallPayload {
request_id: string
tool: string
args: Record<string, unknown>
control_session?: unknown
}
/** Either a success with a free-form result, or a failure with an error
@@ -67,6 +80,10 @@ export interface ToolContext {
cwd: string
abortSignal: AbortSignal
interactive: boolean
/** Server-attested control identity. Optional only for legacy handlers. */
controlSession?: ComputerControlAuthority
/** Hermes-owned snapshot/token binding state for this router lifecycle. */
controlSecurity?: ComputerControlSecurityState
}
/** A tool handler. Throws → router responds with `{ok:false, error}`. */
@@ -122,6 +139,98 @@ function isToolCallPayload(x: unknown): x is ToolCallPayload {
)
}
interface ControlSessionEndPayload {
version: 1
id: string
target_device_id: string
reason?: string
}
function parseControlSessionEnd(value: unknown): ControlSessionEndPayload | null {
if (!value || typeof value !== 'object' || Array.isArray(value)) return null
const raw = value as Record<string, unknown>
const field = (key: string): string | null => {
const value = raw[key]
if (typeof value !== 'string') return null
const normalized = value.trim()
return normalized && normalized.length <= 256 && !/[\u0000-\u001f\u007f]/u.test(normalized) ? normalized : null
}
if (raw.version !== 1) return null
const id = field('id')
const targetDeviceId = field('target_device_id')
if (!id || !targetDeviceId) return null
const reason = raw.reason === undefined ? undefined : field('reason') ?? undefined
return { version: 1, id, target_device_id: targetDeviceId, ...(reason ? { reason } : {}) }
}
function parseControlAuthority(
value: unknown,
requestId: string,
hostUrl: string | undefined
): ComputerControlAuthority | null {
if (!value || typeof value !== 'object' || Array.isArray(value)) return null
const raw = value as Record<string, unknown>
const required = (field: string): string | null => {
const current = raw[field]
if (typeof current !== 'string') return null
const normalized = current.trim()
if (!normalized || normalized.length > 256 || /[\u0000-\u001f\u007f]/u.test(normalized)) return null
return normalized
}
if (raw.version !== 1) return null
const id = required('id')
const boundRequestId = required('request_id')
const requesterDeviceId = required('requester_device_id')
const targetDeviceId = required('target_device_id')
const runId = required('run_id')
if (
!id || !boundRequestId || boundRequestId !== requestId || !requesterDeviceId ||
!targetDeviceId || targetDeviceId !== desktopDeviceId() || !runId
) return null
const optional = (field: string): string | undefined => {
const current = raw[field]
if (typeof current !== 'string') return undefined
const normalized = current.trim()
return normalized && normalized.length <= 256 && !/[\u0000-\u001f\u007f]/u.test(normalized)
? normalized
: undefined
}
return {
controlSessionId: id,
// The relay-owned control session is the authenticated execution-session
// identity currently available on the wire. Keep the adapter contract
// explicit while avoiding any identity supplied through tool args.
relaySessionId: id,
requesterDeviceId,
targetDeviceId,
requestId,
chatSessionId: optional('chat_session_id'),
runId,
...(hostUrl ? { hostUrl } : {})
}
}
function computerAuditMetadata(result: unknown): Record<string, unknown> {
if (!result || typeof result !== 'object' || Array.isArray(result)) return {}
const source = result as Record<string, unknown>
const metadata: Record<string, unknown> = {}
// Window titles frequently contain document names, account names, or page
// content. Keep the app and numeric target for drilldown, but never persist
// the title in the local activity log.
for (const key of ['backend', 'dispatch', 'control_session_id', 'target_app', 'action', 'verification', 'phase']) {
if (typeof source[key] === 'string') metadata[key] = String(source[key]).slice(0, 256)
}
for (const key of ['target_pid', 'target_window_id']) {
if (typeof source[key] === 'number' && Number.isSafeInteger(source[key])) metadata[key] = source[key]
}
return metadata
}
export function toolResultSucceeded(tool: string, result: unknown): boolean {
if (!tool.startsWith('desktop_computer_')) return true
return !result || typeof result !== 'object' || Array.isArray(result) || (result as Record<string, unknown>).ok !== false
}
/** Default interactive detection — true iff stdin is a TTY AND we're not
* flagged as the daemon subcommand. The daemon command sets
* HERMES_RELAY_DAEMON=1 in its own process.env before constructing the
@@ -151,6 +260,9 @@ export class DesktopToolRouter {
* timeout, abort). Surfaced in the heartbeat so an agent or the dashboard
* can ask "is this client healthy?" without parsing transcript history. */
private lastError: { message: string; tool: string; ts: number } | null = null
private readonly controlAuthority: ComputerControlAuthority
private readonly controlSecurity: ComputerControlSecurityState
private readonly activeControlAuthorities = new Map<string, ComputerControlAuthority>()
constructor(opts: DesktopToolRouterOpts) {
this.handlers = opts.handlers
@@ -161,6 +273,13 @@ export class DesktopToolRouter {
// capture it once at construct time so a short shell session that
// happens to get its stdin redirected mid-flight doesn't flip mode.
this.interactive = opts.interactive ?? detectInteractive()
this.controlAuthority = {
controlSessionId: `router-${desktopDeviceId()}-${this.startedAtMs}`,
...(this.hostUrl ? { hostUrl: this.hostUrl } : {})
}
this.controlSecurity = new ComputerControlSecurityState(this.controlAuthority)
initializeComputerControlSession(this.controlAuthority)
this.activeControlAuthorities.set(this.controlAuthority.controlSessionId, this.controlAuthority)
}
/** Install the `onChannel('desktop')` listener and start heartbeats.
@@ -188,6 +307,17 @@ export class DesktopToolRouter {
void this.dispatch(payload)
return
}
if (type === 'desktop.control_session_end') {
const ended = parseControlSessionEnd(payload)
if (!ended || ended.target_device_id !== desktopDeviceId()) return
const authority = this.activeControlAuthorities.get(ended.id)
if (!authority || authority.targetDeviceId !== ended.target_device_id) return
this.activeControlAuthorities.delete(ended.id)
this.controlSecurity.revokeAuthority(ended.id)
revokeComputerControlSession(authority, ended.reason ?? 'relay control session ended')
void closeCuaControlSession(ended.id, ended.reason ?? 'relay control session ended')
return
}
// Unknown desktop.* types — ignore; server may extend later.
})
@@ -202,6 +332,7 @@ export class DesktopToolRouter {
/** Remove the channel listener and stop heartbeats. Idempotent. */
detach(): void {
if (!this.attached) {
this.revokeControlAuthority()
return
}
this.attached = false
@@ -215,6 +346,16 @@ export class DesktopToolRouter {
/* ignore */
}
this.relay = null
this.revokeControlAuthority()
}
private revokeControlAuthority(): void {
this.controlSecurity.revoke()
for (const authority of this.activeControlAuthorities.values()) {
revokeComputerControlSession(authority, 'desktop router detached')
}
this.activeControlAuthorities.clear()
void closeAllCuaControlSessions('desktop router detached')
}
/** Broadcast the advertised-tools heartbeat. Safe to call when detached
@@ -241,8 +382,8 @@ export class DesktopToolRouter {
device_name: os.hostname()
}
if (this.advertisedTools.some(name => name.startsWith('desktop_computer_'))) {
const runtime = getComputerUseRuntimeSummary()
const grant = getComputerGrantSummary()
const runtime = getComputerUseRuntimeSummary(this.controlAuthority)
const grant = getComputerGrantSummary(this.controlAuthority)
const inputGrantActive =
grant.active === true && (grant.mode === 'assist' || grant.mode === 'control')
payload.computer_use = {
@@ -289,6 +430,22 @@ export class DesktopToolRouter {
return
}
const relayAuthority = parseControlAuthority(cmd.control_session, request_id, this.hostUrl)
if (tool.startsWith('desktop_computer_') && cmd.control_session !== undefined && !relayAuthority) {
this.sendResponse({ request_id, ok: false, error: 'invalid server control_session binding' })
return
}
const requestAuthority = relayAuthority ?? this.controlAuthority
if (relayAuthority) {
initializeComputerControlSession(relayAuthority)
this.activeControlAuthorities.set(relayAuthority.controlSessionId, relayAuthority)
}
const controlSession = this.controlSecurity.bindRequest(request_id, requestAuthority)
if (!controlSession) {
this.sendResponse({ request_id, ok: false, error: 'duplicate or invalid desktop request_id' })
return
}
const controller = new AbortController()
const timeoutMs = tool.startsWith('desktop_computer_') || tool.startsWith('desktop_adb_') || tool.startsWith('desktop_usb_')
? COMPUTER_USE_HANDLER_TIMEOUT_MS
@@ -303,7 +460,9 @@ export class DesktopToolRouter {
const ctx: ToolContext = {
cwd: process.cwd(),
abortSignal: controller.signal,
interactive: this.interactive
interactive: this.interactive,
controlSession,
controlSecurity: this.controlSecurity
}
try {
@@ -314,20 +473,34 @@ export class DesktopToolRouter {
// work was completable.
this.sendResponse({ request_id, ok: true, result })
// Local audit trail — fire-and-forget so logging never delays the reply.
void appendAudit({
const retention = readDesktopUseSettingsSync()
const canRetainScreenshot = tool.includes('screenshot') || tool === 'desktop_computer_action'
const screenshotEvidence = retention.activity_screenshot_retention_enabled && canRetainScreenshot
? persistAuditScreenshot(result, request_id, retention.activity_screenshot_retention_days)
: Promise.resolve({})
void screenshotEvidence.then(screenshotEvidence => appendAudit({
ts: Date.now(),
kind: 'tool.completed',
tool,
category: categorizeTool(tool),
ok: true,
ok: toolResultSucceeded(tool, result),
request_id,
host_url: this.hostUrl,
duration_ms: Date.now() - startedAt,
exit_code: resultExitCode(result),
args_preview: previewArgs(args),
summary: summarizeResult(result),
...auditDetails(args, result)
})
...(controlSession.relaySessionId ? { relay_session_id: controlSession.relaySessionId } : {}),
...(controlSession.requesterDeviceId ? { requester_device_id: controlSession.requesterDeviceId } : {}),
...(controlSession.runId ? { run_id: controlSession.runId } : {}),
...(controlSession.targetDeviceId ? { target_device_id: controlSession.targetDeviceId } : {}),
...computerAuditMetadata(result),
...screenshotEvidence,
...(!toolResultSucceeded(tool, result) && result && typeof result === 'object' && !Array.isArray(result) && typeof (result as Record<string, unknown>).code === 'string'
? { error: String((result as Record<string, unknown>).code).slice(0, 128) }
: {}),
...auditDetails(args, result, { redactComputerContent: tool.startsWith('desktop_computer_') })
}))
} catch (e) {
clearTimeout(timeoutTimer)
// Distinguish aborts (timeout or transport teardown) from genuine
@@ -355,7 +528,11 @@ export class DesktopToolRouter {
host_url: this.hostUrl,
duration_ms: Date.now() - startedAt,
args_preview: previewArgs(args),
...auditDetails(args),
...(controlSession.relaySessionId ? { relay_session_id: controlSession.relaySessionId } : {}),
...(controlSession.requesterDeviceId ? { requester_device_id: controlSession.requesterDeviceId } : {}),
...(controlSession.runId ? { run_id: controlSession.runId } : {}),
...(controlSession.targetDeviceId ? { target_device_id: controlSession.targetDeviceId } : {}),
...auditDetails(args, undefined, { redactComputerContent: tool.startsWith('desktop_computer_') }),
error: message
})
}
+1 -1
View File
@@ -1,2 +1,2 @@
// Regenerated from package.json by gen:version script. Do not edit by hand.
export const VERSION = "0.4.0-alpha.8" as const
export const VERSION = "0.4.0-beta.2" as const
+6 -1
View File
@@ -13,6 +13,7 @@ export interface WindowsInstallerLaunchOptions {
cliPath?: string
trayPath?: string
delayMs?: number
callerPid?: number
}
/**
@@ -30,9 +31,12 @@ export function windowsInstallerLaunchPlan(
installDir = '',
cliPath = '',
trayPath = '',
delayMs = 1200
delayMs = 1200,
callerPid = process.pid
} = options
const script = [
'$callerPid = [int]$env:HERMES_RELAY_SETUP_CALLER_PID',
'if ($callerPid -gt 0) { Wait-Process -Id $callerPid -ErrorAction SilentlyContinue }',
`Start-Sleep -Milliseconds ${Math.max(0, Math.floor(delayMs))}`,
'$installer = $env:HERMES_RELAY_SETUP_PATH',
"$installerArgs = if ($env:HERMES_RELAY_SETUP_SILENT -eq '1') { @('/S') } else { @() }",
@@ -64,6 +68,7 @@ export function windowsInstallerLaunchPlan(
env: {
...process.env,
HERMES_RELAY_SETUP_PATH: installerPath,
HERMES_RELAY_SETUP_CALLER_PID: String(callerPid),
HERMES_RELAY_SETUP_SILENT: silent ? '1' : '0',
HERMES_RELAY_SETUP_RESTART_DAEMON: restartDaemon ? '1' : '0',
HERMES_RELAY_SETUP_INSTALL_DIR: installDir,
+47 -1
View File
@@ -1,7 +1,11 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { mkdtemp, readFile, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { auditDetails, categorizeTool, resultExitCode, summarizeResult } from '../src/lib/auditLog.js'
import { auditDetails, categorizeTool, persistAuditScreenshot, resultExitCode, summarizeResult } from '../src/lib/auditLog.js'
import { toolResultSucceeded } from '../src/tools/router.js'
test('audit events classify the activity surfaces used by the tray', () => {
assert.equal(categorizeTool('desktop_powershell'), 'command')
@@ -31,3 +35,45 @@ test('audit events preserve process exit outcome separately from dispatch succes
assert.equal(summarizeResult(result), 'exit 17')
assert.equal(resultExitCode({ path: 'C:\\temp\\file.txt' }), undefined)
})
test('computer-use audit redacts screenshots, UI trees, labels, and entered values', () => {
const details = auditDetails(
{ pid: 42, window_id: 7, value: 'top-secret-password' },
{
backend: 'cua_driver',
target: { pid: 42, windowId: 7, title: 'Password Manager' },
tree_markdown: 'Password: hunter2',
screenshot_base64: 'sensitive-pixels',
elements: [{ label: 'API token sk-live-secret' }]
},
{ redactComputerContent: true }
)
const serialized = JSON.stringify(details)
assert.doesNotMatch(serialized, /hunter2|sensitive-pixels|sk-live-secret|top-secret-password|Password Manager/)
assert.match(details.result_detail ?? '', /"redacted": true/)
assert.match(details.result_detail ?? '', /"pid": 42/)
})
test('semantic computer-control rejection is audited as failed without changing transport semantics', () => {
assert.equal(toolResultSucceeded('desktop_computer_action', { ok: false, code: 'sensitive_target_blocked' }), false)
assert.equal(toolResultSucceeded('desktop_computer_action', { ok: true }), true)
assert.equal(toolResultSucceeded('desktop_read_file', { ok: false }), true)
})
test('screenshot evidence is retained as a private opaque PNG instead of JSON content', async () => {
const directory = await mkdtemp(join(tmpdir(), 'hermes-activity-evidence-'))
const previous = process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR
process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR = directory
try {
const png = Buffer.from('89504e470d0a1a0a00000000', 'hex').toString('base64')
const evidence = await persistAuditScreenshot({ screenshot_base64: png, screenshot_width: 640, screenshot_height: 480 }, 'request-1', 7)
assert.match(evidence.screenshot_evidence_id ?? '', /^[a-f0-9]{32}$/)
assert.equal(evidence.screenshot_width, 640)
assert.equal((await readFile(join(directory, `${evidence.screenshot_evidence_id}.png`))).subarray(0, 8).toString('hex'), '89504e470d0a1a0a')
assert.equal(JSON.stringify(evidence).includes(png), false)
} finally {
if (previous === undefined) delete process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR
else process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR = previous
await rm(directory, { recursive: true, force: true })
}
})
@@ -0,0 +1,190 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import {
computerGrantAllowsTarget,
configureComputerUseRuntime,
getComputerGrantSummary,
initializeComputerControlSession,
cancelAllComputerGrants,
requestComputerGrant,
revokeComputerControlSession
} from '../src/tools/computerGrants.js'
import {
ComputerControlSecurityState,
evaluateSensitiveTarget,
hasAuthenticatedControlIdentity,
type ComputerControlAuthority
} from '../src/tools/computerControlSecurity.js'
const authority = (id: string): ComputerControlAuthority => ({
controlSessionId: id,
hostUrl: 'wss://relay.example'
})
test('control request ids are accepted once per router lifecycle', () => {
const state = new ComputerControlSecurityState(authority('router-a'))
assert.equal(state.bindRequest('request-1')?.requestId, 'request-1')
assert.equal(state.bindRequest('request-1'), null)
assert.equal(state.bindRequest(' '), null)
})
test('snapshot tokens are opaque, one-use, and bound to grant and target', () => {
const auth = authority('router-snapshot')
const state = new ComputerControlSecurityState(auth)
const token = state.issueSnapshotToken({
authority: auth,
grantId: 'grant-a',
target: { pid: 42, windowId: 7, app: 'Notepad' },
snapshotGeneration: 'generation-1',
driverElementToken: 'driver-private-token'
})
assert.match(token, /^hermes-snapshot-/)
assert.equal(state.consumeSnapshotToken(token, {
authority: auth,
grantId: 'grant-b',
target: { pid: 42, windowId: 7 }
}), null)
assert.equal(state.consumeSnapshotToken(token, {
authority: auth,
grantId: 'grant-a',
target: { pid: 42, windowId: 7 }
}), null)
})
test('snapshot tokens return their backend binding only to the exact target', () => {
const auth = authority('router-target')
const state = new ComputerControlSecurityState(auth)
const token = state.issueSnapshotToken({
authority: auth,
grantId: null,
target: { pid: 9, windowId: 11 },
snapshotGeneration: 'g1',
driverElementToken: 'opaque-driver-token'
})
const binding = state.consumeSnapshotToken(token, {
authority: auth,
grantId: null,
target: { pid: 9, windowId: 11 },
snapshotGeneration: 'g1'
})
assert.equal(binding?.driverElementToken, 'opaque-driver-token')
assert.equal(state.consumeSnapshotToken(token, {
authority: auth,
grantId: null,
target: { pid: 9, windowId: 11 }
}), null)
})
test('authority revocation removes only that concurrent session artifacts', () => {
const first = authority('router-first')
const second = authority('router-second')
const state = new ComputerControlSecurityState(first)
assert.ok(state.bindRequest('request-first', first))
assert.ok(state.bindRequest('request-second', second))
const firstToken = state.issueSnapshotToken({
authority: first, grantId: null, target: { pid: 1, windowId: 1 }, snapshotGeneration: 'g1'
})
const secondToken = state.issueSnapshotToken({
authority: second, grantId: null, target: { pid: 2, windowId: 2 }, snapshotGeneration: 'g2'
})
state.revokeAuthority(first.controlSessionId)
assert.ok(state.bindRequest('request-first', first))
assert.equal(state.bindRequest('request-second', second), null)
assert.equal(state.consumeSnapshotToken(firstToken, { authority: first, grantId: null, target: { pid: 1, windowId: 1 } }), null)
assert.ok(state.consumeSnapshotToken(secondToken, { authority: second, grantId: null, target: { pid: 2, windowId: 2 } }))
})
test('sensitive target policy fails closed without identity and blocks baseline surfaces', () => {
assert.deepEqual(evaluateSensitiveTarget({}), { allowed: false, reason: 'missing_target_identity' })
assert.equal(evaluateSensitiveTarget({ app: 'Bitwarden' }).allowed, false)
assert.equal(evaluateSensitiveTarget({ title: 'Windows Security settings' }).allowed, false)
assert.deepEqual(evaluateSensitiveTarget({ app: 'Notepad', title: 'notes.txt' }), {
allowed: true,
reason: 'not_sensitive'
})
})
test('authenticated control identity requires relay-attested requester/run/target fields', () => {
assert.equal(hasAuthenticatedControlIdentity(authority('local-only')), false)
assert.equal(hasAuthenticatedControlIdentity({
controlSessionId: 'control-1',
relaySessionId: 'relay-1',
requesterDeviceId: 'requester-1',
runId: 'run-1',
targetDeviceId: 'desktop-1'
}), true)
})
test('router control sessions inherit host policy but keep grants isolated', () => {
configureComputerUseRuntime({
url: 'wss://relay.example',
computerUseConsented: true,
consentSource: 'stored',
accessMode: 'ask',
capabilities: {
commands: 'ask', files: 'ask', screen_input: 'ask', usb: 'ask', microphone: 'disabled', camera: 'disabled'
}
})
const first = authority('router-first')
const second = authority('router-second')
initializeComputerControlSession(first)
initializeComputerControlSession(second)
requestComputerGrant({
mode: 'control',
scope: { app: 'Notepad', display: 'primary', folder: 'C:\\work' },
duration_seconds: 60,
reason: 'edit notes'
}, first)
assert.equal(getComputerGrantSummary(first).active, true)
assert.equal(getComputerGrantSummary(second).active, false)
assert.equal(computerGrantAllowsTarget(first, {
pid: 1,
windowId: 2,
app: 'Windows Notepad',
display: 'primary',
folder: 'C:\\work\\notes'
}), true)
assert.equal(computerGrantAllowsTarget(first, {
pid: 1,
windowId: 2,
app: 'Browser',
display: 'primary',
folder: 'C:\\work\\notes'
}), false)
revokeComputerControlSession(first)
revokeComputerControlSession(second)
})
test('Full Access configured before router creation is preserved without sharing grants', () => {
configureComputerUseRuntime({
computerUseConsented: true,
accessMode: 'full_access'
})
const full = authority('router-full')
initializeComputerControlSession(full)
assert.equal(getComputerGrantSummary(full).mode, 'full_access')
revokeComputerControlSession(full)
configureComputerUseRuntime({ accessMode: 'ask' })
})
test('local emergency cancellation revokes every active control session', () => {
configureComputerUseRuntime({
computerUseConsented: true,
accessMode: 'ask',
capabilities: {
commands: 'ask', files: 'ask', screen_input: 'ask', usb: 'ask', microphone: 'disabled', camera: 'disabled'
}
})
const first = authority('emergency-first')
const second = authority('emergency-second')
initializeComputerControlSession(first)
initializeComputerControlSession(second)
requestComputerGrant({ mode: 'control', reason: 'first' }, first)
requestComputerGrant({ mode: 'observe', reason: 'second' }, second)
assert.equal(cancelAllComputerGrants('emergency stop'), 2)
assert.equal(getComputerGrantSummary(first).active, false)
assert.equal(getComputerGrantSummary(second).active, false)
revokeComputerControlSession(first)
revokeComputerControlSession(second)
})
+260
View File
@@ -0,0 +1,260 @@
import assert from 'node:assert/strict'
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import { ComputerControlSecurityState, type ComputerControlAuthority } from '../src/tools/computerControlSecurity.js'
import {
cancelComputerGrant,
configureComputerUseRuntime,
requestComputerGrant
} from '../src/tools/computerGrants.js'
import {
setCuaControlSessionFactoryForTests,
type CuaControlSession,
type CuaControlSessionIdentity
} from '../src/tools/cuaDriver.js'
import {
computerActionHandler,
computerCancelHandler,
computerScreenshotHandler
} from '../src/tools/handlers/computer.js'
import type { ToolContext } from '../src/tools/router.js'
interface FakeCall {
name: string
args?: unknown
}
class FakeCuaSession {
readonly calls: FakeCall[] = []
closed = false
snapshotNumber = 0
appName = 'Calculator'
async listWindows(pid?: number): Promise<Record<string, unknown>> {
this.calls.push({ name: 'listWindows', args: pid })
return {
windows: [{
window_id: 200,
app_name: this.appName,
title: this.appName,
executable: 'C:\\WindowsApps\\CalculatorApp.exe'
}]
}
}
async snapshot(args: unknown): Promise<Record<string, unknown>> {
this.snapshotNumber += 1
this.calls.push({ name: 'snapshot', args })
return {
snapshot_id: `s0000000${this.snapshotNumber}`,
elements: [{ element_index: 7, element_token: 'e1234abcd', role: 'button', label: 'Seven' }],
tree_markdown: '[7] button Seven'
}
}
async clickElement(args: unknown): Promise<Record<string, unknown>> {
this.calls.push({ name: 'clickElement', args })
return { clicked: true }
}
async setElementValue(args: unknown, value: string): Promise<Record<string, unknown>> {
this.calls.push({ name: 'setElementValue', args: { args, value } })
return { changed: true }
}
async pressKey(args: unknown, key: string): Promise<Record<string, unknown>> {
this.calls.push({ name: 'pressKey', args: { args, key } })
return { pressed: true }
}
async scroll(args: unknown, direction: string, amount: number): Promise<Record<string, unknown>> {
this.calls.push({ name: 'scroll', args: { args, direction, amount } })
return { scrolled: true }
}
async close(reason?: string): Promise<void> {
this.closed = true
this.calls.push({ name: 'close', args: reason })
}
}
async function fixture(): Promise<{
authority: ComputerControlAuthority
ctx: ToolContext
session: FakeCuaSession
cleanup(): Promise<void>
}> {
const dir = await mkdtemp(join(tmpdir(), 'hermes-cua-handlers-'))
const settingsPath = join(dir, 'desktop-settings.json')
await writeFile(settingsPath, JSON.stringify({
computer_use_enabled: true,
computer_control_engine: 'cua',
cua_cursor_enabled: true,
cua_foreground_escalation_enabled: false
}))
process.env.HERMES_RELAY_DESKTOP_SETTINGS_PATH = settingsPath
const authority: ComputerControlAuthority = {
controlSessionId: 'control-handler-test',
requestId: 'request-handler-test',
relaySessionId: 'relay-session',
requesterDeviceId: 'requester-device',
runId: 'run-1',
targetDeviceId: 'desktop-target'
}
const session = new FakeCuaSession()
setCuaControlSessionFactoryForTests(async (_identity: CuaControlSessionIdentity) => session as unknown as CuaControlSession)
configureComputerUseRuntime({
url: 'wss://relay.example.test',
computerUseConsented: true,
consentSource: 'stored',
accessMode: 'ask'
}, authority)
requestComputerGrant({ mode: 'control', scope: { app: 'Calculator' }, duration_seconds: 60 }, authority)
const ctx: ToolContext = {
cwd: dir,
abortSignal: new AbortController().signal,
interactive: false,
controlSession: authority,
controlSecurity: new ComputerControlSecurityState(authority)
}
return {
authority,
ctx,
session,
cleanup: async () => {
cancelComputerGrant('test cleanup', authority)
setCuaControlSessionFactoryForTests(null)
delete process.env.HERMES_RELAY_DESKTOP_SETTINGS_PATH
await rm(dir, { recursive: true, force: true })
}
}
}
test('CUA handlers issue a Hermes token, execute once, and verify with a fresh snapshot', async () => {
const item = await fixture()
try {
const observed = await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx) as {
ok: boolean
backend: string
elements: Array<{ snapshot_token: string; element_token?: string }>
}
assert.equal(observed.ok, true)
assert.equal(observed.backend, 'cua_driver')
assert.equal(observed.elements[0]!.element_token, undefined)
assert.match(observed.elements[0]!.snapshot_token, /^hermes-snapshot-/)
const acted = await computerActionHandler({
action: 'click_element',
pid: 100,
window_id: 200,
snapshot_token: observed.elements[0]!.snapshot_token,
snapshot_generation: 's00000001'
}, item.ctx) as { ok: boolean; backend: string; verification_snapshot: { snapshot_id: string } }
assert.equal(acted.ok, true)
assert.equal(acted.backend, 'cua')
assert.equal(acted.verification_snapshot.snapshot_id, 's00000002')
assert.deepEqual(item.session.calls.map(call => call.name), [
'listWindows', 'snapshot', 'listWindows', 'clickElement', 'snapshot'
])
const replayed = await computerActionHandler({
action: 'click_element',
pid: 100,
window_id: 200,
snapshot_token: observed.elements[0]!.snapshot_token
}, item.ctx) as { ok: boolean; code: string }
assert.equal(replayed.ok, false)
assert.equal(replayed.code, 'invalid_or_stale_snapshot')
assert.equal(item.session.calls.filter(call => call.name === 'clickElement').length, 1)
} finally {
await item.cleanup()
}
})
test('CUA handlers reject unauthenticated and mismatched targets before input', async () => {
const item = await fixture()
try {
const unauthenticatedAuthority = { controlSessionId: 'legacy-only' }
requestComputerGrant({ mode: 'observe', duration_seconds: 60 }, unauthenticatedAuthority)
const unauthenticated = await computerScreenshotHandler(
{ pid: 100, window_id: 200 },
{ ...item.ctx, controlSession: unauthenticatedAuthority }
) as { ok: boolean; code: string }
assert.equal(unauthenticated.ok, false)
assert.equal(unauthenticated.code, 'authenticated_control_session_required')
cancelComputerGrant('test cleanup', unauthenticatedAuthority)
const missing = await computerScreenshotHandler({ pid: 100, window_id: 201 }, item.ctx) as {
ok: boolean
code: string
}
assert.equal(missing.ok, false)
assert.equal(missing.code, 'target_not_found')
requestComputerGrant({ mode: 'control', scope: { app: 'Notepad' }, duration_seconds: 60 }, item.authority)
const wrongGrant = await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx) as {
ok: boolean
code: string
}
assert.equal(wrongGrant.ok, false)
assert.equal(wrongGrant.code, 'grant_target_mismatch')
item.session.appName = 'Windows Security'
configureComputerUseRuntime({
url: 'wss://relay.example.test',
computerUseConsented: true,
consentSource: 'stored',
accessMode: 'full_access'
}, item.authority)
const sensitive = await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx) as {
ok: boolean
code: string
}
assert.equal(sensitive.ok, false)
assert.equal(sensitive.code, 'sensitive_target_blocked')
assert.equal(item.session.calls.some(call => call.name === 'clickElement'), false)
} finally {
await item.cleanup()
}
})
test('desktop computer cancel closes the matching CUA session', async () => {
const item = await fixture()
try {
await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx)
const result = await computerCancelHandler({ reason: 'operator stop' }, item.ctx) as { ok: boolean }
assert.equal(result.ok, true)
assert.equal(item.session.closed, true)
assert.deepEqual(item.session.calls.at(-1), { name: 'close', args: 'operator stop' })
} finally {
await item.cleanup()
}
})
test('selected CUA retains system display observation but never falls back to legacy input', async () => {
const item = await fixture()
try {
if (process.platform === 'win32') {
const screenshot = await computerScreenshotHandler({ display: 'primary' }, item.ctx) as {
ok: boolean
backend: string
}
assert.equal(screenshot.ok, true)
assert.equal(screenshot.backend, 'system_capture')
}
const coordinate = await computerActionHandler({
action: 'left_click',
x: 100,
y: 200
}, item.ctx) as { ok: boolean; code: string }
assert.equal(coordinate.ok, false)
assert.equal(coordinate.code, 'cua_structured_action_required')
assert.equal(item.session.calls.some(call => call.name === 'clickElement'), false)
} finally {
await item.cleanup()
}
})
+167
View File
@@ -0,0 +1,167 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { desktopDeviceId } from '../src/deviceIdentity.js'
import type { RelayTransport } from '../src/transport/RelayTransport.js'
import { configureComputerUseRuntime, getComputerGrantSummary, requestComputerGrant } from '../src/tools/computerGrants.js'
import {
DesktopToolRouter,
type ToolContext
} from '../src/tools/router.js'
type ChannelListener = (type: string, payload: Record<string, unknown>) => void
function harness(handler: (ctx: ToolContext) => void) {
let listener: ChannelListener | null = null
const sent: Array<{ channel: string; type: string; payload: Record<string, unknown> }> = []
const relay = {
onChannel(channel: string, next: ChannelListener | null) {
assert.equal(channel, 'desktop')
listener = next
},
sendChannel(channel: string, type: string, payload: Record<string, unknown>) {
sent.push({ channel, type, payload })
}
} as unknown as RelayTransport
const router = new DesktopToolRouter({
consentGranted: true,
interactive: false,
hostUrl: 'wss://relay.example',
handlers: {
desktop_computer_snapshot: async (_args, ctx) => {
handler(ctx)
return { ok: true }
}
}
})
router.attach(relay)
return {
router,
sent,
emit(type: string, payload: Record<string, unknown>) {
assert.ok(listener)
listener(type, payload)
}
}
}
function nextTurn(): Promise<void> {
return new Promise(resolve => setImmediate(resolve))
}
test('router passes a strictly validated server control session to handlers', async () => {
let received: ToolContext | undefined
const testHarness = harness(ctx => { received = ctx })
const requestId = 'request-1'
testHarness.emit('desktop.command', {
request_id: requestId,
tool: 'desktop_computer_snapshot',
args: {},
control_session: {
version: 1,
id: 'control-1',
request_id: requestId,
requester_device_id: 'paired-agent-1',
target_device_id: desktopDeviceId(),
chat_session_id: 'chat-1',
run_id: 'run-1'
}
})
await nextTurn()
assert.equal(received?.controlSession?.controlSessionId, 'control-1')
assert.equal(received?.controlSession?.relaySessionId, 'control-1')
assert.equal(received?.controlSession?.requestId, requestId)
assert.equal(received?.controlSession?.requesterDeviceId, 'paired-agent-1')
assert.equal(received?.controlSession?.targetDeviceId, desktopDeviceId())
assert.equal(received?.controlSession?.runId, 'run-1')
assert.equal(received?.controlSession?.chatSessionId, 'chat-1')
assert.ok(testHarness.sent.some(item =>
item.type === 'desktop.response' && item.payload.request_id === requestId && item.payload.ok === true
))
testHarness.router.detach()
})
test('router rejects a supplied control session whose request binding is invalid', async () => {
let invoked = false
const testHarness = harness(() => { invoked = true })
testHarness.emit('desktop.command', {
request_id: 'request-outer',
tool: 'desktop_computer_snapshot',
args: {},
control_session: {
version: 1,
id: 'control-1',
request_id: 'request-other',
requester_device_id: 'paired-agent-1',
target_device_id: desktopDeviceId(),
run_id: 'run-1'
}
})
await nextTurn()
assert.equal(invoked, false)
const response = testHarness.sent.find(item =>
item.type === 'desktop.response' && item.payload.request_id === 'request-outer'
)
assert.equal(response?.payload.ok, false)
assert.equal(response?.payload.error, 'invalid server control_session binding')
testHarness.router.detach()
})
test('router preserves legacy computer commands when the server omits identity', async () => {
let received: ToolContext | undefined
const testHarness = harness(ctx => { received = ctx })
testHarness.emit('desktop.command', {
request_id: 'legacy-request',
tool: 'desktop_computer_snapshot',
args: {}
})
await nextTurn()
assert.match(received?.controlSession?.controlSessionId ?? '', /^router-/)
assert.equal(received?.controlSession?.relaySessionId, undefined)
testHarness.router.detach()
})
test('relay control-session end revokes only the exact local target authority', async () => {
let received: ToolContext | undefined
const testHarness = harness(ctx => {
received = ctx
configureComputerUseRuntime({ computerUseConsented: true, accessMode: 'ask' }, ctx.controlSession)
requestComputerGrant({ mode: 'control', duration_seconds: 60 }, ctx.controlSession)
})
const requestId = 'request-ending'
testHarness.emit('desktop.command', {
request_id: requestId,
tool: 'desktop_computer_snapshot',
args: {},
control_session: {
version: 1,
id: 'control-ending',
request_id: requestId,
requester_device_id: 'paired-agent-1',
target_device_id: desktopDeviceId(),
run_id: 'run-ending'
}
})
await nextTurn()
assert.equal(getComputerGrantSummary(received?.controlSession).active, true)
testHarness.emit('desktop.control_session_end', {
version: 1,
id: 'control-ending',
target_device_id: 'another-desktop',
reason: 'wrong target'
})
assert.equal(getComputerGrantSummary(received?.controlSession).active, true)
testHarness.emit('desktop.control_session_end', {
version: 1,
id: 'control-ending',
target_device_id: desktopDeviceId(),
reason: 'run ended'
})
assert.equal(getComputerGrantSummary(received?.controlSession).active, false)
testHarness.router.detach()
})
+201
View File
@@ -0,0 +1,201 @@
import assert from 'node:assert/strict'
import { mkdtemp, mkdir, rm, symlink, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import {
CuaDriverAdapter,
CuaRuntimeError,
closeAllCuaControlSessions,
getCuaControlSession,
selectComputerControlBackend,
setCuaControlSessionFactoryForTests,
type CuaControlSession,
type CuaProcessResult,
type CuaProcessRunner
} from '../src/tools/cuaDriver.js'
const REQUIRED_TOOL_LINES = [
'health_report', 'start_session', 'end_session', 'list_windows', 'get_window_state',
'click', 'set_value', 'press_key', 'scroll'
].map(name => `${name}: test tool`).join('\n')
class FakeRunner implements CuaProcessRunner {
readonly calls: Array<{ executable: string; args: readonly string[]; stdin?: string }> = []
constructor(
private readonly binaryPath: string,
private readonly health = 'ok',
private readonly version = '0.19.3',
private readonly permissionMode = 'standard'
) {}
async run(executable: string, args: readonly string[], options = {}): Promise<CuaProcessResult> {
this.calls.push({ executable, args, stdin: options.stdin })
const command = args.join(' ')
if (command === '--version') return ok(`cua-driver ${this.version}`)
if (command === 'manifest --pretty') {
return ok(JSON.stringify({ schema_version: '1', binary_version: this.version, binary_path: this.binaryPath }))
}
if (command === 'list-tools') return ok(REQUIRED_TOOL_LINES)
if (command === 'status') return ok(`Cua Driver daemon is running\n permission mode: ${this.permissionMode} (test)`)
if (command === 'call health_report') {
return ok(JSON.stringify({ schema_version: '1', driver_version: this.version, overall: this.health }))
}
return ok(JSON.stringify({ ok: true, tool: args[1] }))
}
}
function ok(stdout: string): CuaProcessResult {
return { stdout, stderr: '', exitCode: 0 }
}
async function fakeInstall(): Promise<{ home: string; binary: string; cleanup(): Promise<void> }> {
const home = await mkdtemp(join(tmpdir(), 'hermes-cua-'))
const releases = join(home, '.cua-driver', 'packages', 'releases')
const release = join(releases, '0.19.3-x86_64-pc-windows-msvc')
const current = join(home, '.cua-driver', 'packages', 'current')
await mkdir(release, { recursive: true })
const binary = join(release, 'cua-driver.exe')
await writeFile(binary, '')
await symlink(release, current, 'junction')
return { home, binary, cleanup: () => rm(home, { recursive: true, force: true }) }
}
test('discovers only the canonical package/current executable and negotiates readiness', async () => {
const install = await fakeInstall()
try {
const runner = new FakeRunner(install.binary)
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner })
assert.equal(adapter.binaryPath, install.binary)
assert.equal(adapter.binaryVersion, '0.19.3')
assert.equal(adapter.permissionMode, 'standard')
assert.equal(runner.calls[0]?.executable, install.binary)
assert.deepEqual(runner.calls.map(call => call.args.join(' ')).slice(0, 4), [
'--version', 'manifest --pretty', 'list-tools', 'status'
])
assert.equal(runner.calls.some(call => call.args.join(' ') === 'call health_report'), false)
} finally {
await install.cleanup()
}
})
test('keeps runtime ready when global health is degraded but still rejects unrestricted permission mode', async () => {
const install = await fakeInstall()
try {
const adapter = await CuaDriverAdapter.connect({
platform: 'win32', homeDir: install.home, runner: new FakeRunner(install.binary, 'degraded')
})
assert.equal(adapter.binaryVersion, '0.19.3')
await assert.rejects(
CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner: new FakeRunner(install.binary, 'ok', '0.19.3', 'unrestricted') }),
(error: unknown) => error instanceof CuaRuntimeError && error.code === 'incompatible'
)
} finally {
await install.cleanup()
}
})
test('explicit health recheck reports degradation without changing runtime readiness', async () => {
const install = await fakeInstall()
try {
const runner = new FakeRunner(install.binary, 'degraded')
const health = await CuaDriverAdapter.healthStatus({ platform: 'win32', homeDir: install.home, runner })
assert.equal(health.state, 'degraded')
assert.equal(health.overall, 'degraded')
assert.equal(health.temporaryWindowsCompatibility, true)
assert.equal(runner.calls.filter(call => call.args.join(' ') === 'call health_report').length, 1)
const runtime = await CuaDriverAdapter.status({ platform: 'win32', homeDir: install.home, runner })
assert.equal(runtime.ready, true)
assert.equal(runtime.health, 'not_checked')
} finally {
await install.cleanup()
}
})
test('uses a locally derived session and exposes only typed background actions', async () => {
const install = await fakeInstall()
try {
const runner = new FakeRunner(install.binary)
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner })
const session = await adapter.openSession({ controlSessionId: 'control-1', targetDeviceId: 'desktop-1', runId: 'run-1' })
await session.snapshot({ pid: 123, windowId: 456, includeScreenshot: false })
await session.clickElement({ pid: 123, windowId: 456, elementToken: 'e123abc' })
await session.pressKey({ pid: 123, windowId: 456 }, 'escape')
await session.close()
const invocations = runner.calls.filter(call => call.args[0] === 'call' && call.args[1] !== 'health_report')
assert.deepEqual(invocations.map(call => call.args[1]), [
'start_session', 'get_window_state', 'click', 'press_key', 'end_session'
])
const start = JSON.parse(invocations[0]!.stdin!) as { session: string; capture_scope: string }
assert.match(start.session, /^hermes-[0-9a-f]{32}$/)
assert.equal(start.capture_scope, 'window')
const click = JSON.parse(invocations[2]!.stdin!) as { session: string; delivery_mode: string; scope: string }
assert.equal(click.session, start.session)
assert.equal(click.delivery_mode, 'background')
assert.equal(click.scope, 'window')
await assert.rejects(session.listWindows(), /closed/)
} finally {
await install.cleanup()
}
})
test('rejects invalid tokens and keys before invoking the driver', async () => {
const install = await fakeInstall()
try {
const runner = new FakeRunner(install.binary)
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner })
const session = await adapter.openSession({ controlSessionId: 'control-2', targetDeviceId: 'desktop-1' })
const before = runner.calls.length
await assert.rejects(session.clickElement({ pid: 1, windowId: 2, elementToken: '../bad' }), /elementToken/)
await assert.rejects(session.pressKey({ pid: 1, windowId: 2 }, 'win+r'), /allowlisted/)
assert.equal(runner.calls.length, before)
await session.close()
} finally {
await install.cleanup()
}
})
test('shared control sessions reject identity changes under the same authority id', async () => {
const fake = { close: async () => undefined } as unknown as CuaControlSession
setCuaControlSessionFactoryForTests(async () => fake)
try {
await getCuaControlSession({
controlSessionId: 'control-bound',
targetDeviceId: 'desktop-1',
relaySessionId: 'relay-1',
requesterDeviceId: 'agent-1',
runId: 'run-1'
})
await assert.rejects(getCuaControlSession({
controlSessionId: 'control-bound',
targetDeviceId: 'desktop-1',
relaySessionId: 'relay-1',
requesterDeviceId: 'agent-2',
runId: 'run-1'
}), /identity changed/)
} finally {
await closeAllCuaControlSessions('test cleanup')
setCuaControlSessionFactoryForTests(null)
}
})
test('control backend selection is immutable for the active session', async () => {
const fake = { close: async () => undefined } as unknown as CuaControlSession
setCuaControlSessionFactoryForTests(async () => fake)
try {
const primary = await selectComputerControlBackend('backend-cua', 'cua', true)
const attemptedDowngrade = await selectComputerControlBackend('backend-cua', 'legacy', false)
assert.equal(primary.backend, 'cua')
assert.deepEqual(attemptedDowngrade, primary)
const compatibility = await selectComputerControlBackend('backend-legacy', 'legacy', false)
const attemptedUpgrade = await selectComputerControlBackend('backend-legacy', 'cua', true)
assert.equal(compatibility.backend, 'legacy_compat')
assert.deepEqual(attemptedUpgrade, compatibility)
} finally {
await closeAllCuaControlSessions('test cleanup')
setCuaControlSessionFactoryForTests(null)
}
})
+149
View File
@@ -0,0 +1,149 @@
import assert from 'node:assert/strict'
import { mkdir, mkdtemp, rm, symlink, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import {
CuaDriverAdapter,
CuaRuntimeError,
type CuaProcessResult,
type CuaProcessRunner
} from '../src/tools/cuaDriver.js'
const tools = [
'health_report', 'start_session', 'end_session', 'list_windows', 'get_window_state',
'click', 'set_value', 'press_key', 'scroll'
]
class StatefulFakeCua implements CuaProcessRunner {
readonly calls: Array<{ args: readonly string[]; payload: Record<string, unknown> | null; signal?: AbortSignal }> = []
rejectClicksAsStale = false
constructor(private readonly binary: string) {}
async run(
_executable: string,
args: readonly string[],
options: { stdin?: string; signal?: AbortSignal } = {}
): Promise<CuaProcessResult> {
const payload = options.stdin ? JSON.parse(options.stdin) as Record<string, unknown> : null
this.calls.push({ args: [...args], payload, signal: options.signal })
if (options.signal?.aborted) throw new CuaRuntimeError('fake action cancelled', 'transport')
const command = args.join(' ')
if (command === '--version') return ok('cua-driver 0.19.3')
if (command === 'manifest --pretty') {
return ok(JSON.stringify({ schema_version: '1', binary_version: '0.19.3', binary_path: this.binary }))
}
if (command === 'list-tools') return ok(tools.map(tool => `${tool}: fake`).join('\n'))
if (command === 'status') return ok('permission mode: bounded')
if (command === 'call health_report') {
return ok(JSON.stringify({ schema_version: '1', driver_version: '0.19.3', overall: 'ok' }))
}
if (command === 'call get_window_state') {
return ok(JSON.stringify({
snapshot_id: 's1234abcd',
elements: [{ element_index: 7, element_token: 'e1234abcd', role: 'button', label: 'Seven' }]
}))
}
if (command === 'call click' && this.rejectClicksAsStale) {
return ok(JSON.stringify({ isError: true, code: 'stale_element_token' }))
}
return ok(JSON.stringify({ ok: true }))
}
}
function ok(stdout: string): CuaProcessResult {
return { stdout, stderr: '', exitCode: 0 }
}
async function harness(): Promise<{
adapter: CuaDriverAdapter
runner: StatefulFakeCua
cleanup(): Promise<void>
}> {
const home = await mkdtemp(join(tmpdir(), 'hermes-cua-integration-'))
const release = join(home, '.cua-driver', 'packages', 'releases', '0.19.3-test')
await mkdir(release, { recursive: true })
const binary = join(release, 'cua-driver.exe')
await writeFile(binary, '')
await symlink(release, join(home, '.cua-driver', 'packages', 'current'), 'junction')
const runner = new StatefulFakeCua(binary)
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: home, runner })
return { adapter, runner, cleanup: () => rm(home, { recursive: true, force: true }) }
}
function identity(id: string) {
return { controlSessionId: id, targetDeviceId: 'desktop-target', runId: 'run-1' }
}
test('semantic CUA action can be bracketed by fresh snapshots without foreground dispatch', async () => {
const { adapter, runner, cleanup } = await harness()
try {
const session = await adapter.openSession(identity('control-snapshot'))
const before = await session.snapshot({ pid: 100, windowId: 200, includeScreenshot: true })
const token = (before.elements as Array<{ element_token: string }>)[0]!.element_token
await session.clickElement({ pid: 100, windowId: 200, elementToken: token })
await session.snapshot({ pid: 100, windowId: 200, includeScreenshot: false })
await session.close()
const calls = runner.calls.filter(call => call.args[0] === 'call' && call.args[1] !== 'health_report')
assert.deepEqual(calls.map(call => call.args[1]), [
'start_session', 'get_window_state', 'click', 'get_window_state', 'end_session'
])
assert.deepEqual(calls[2]!.payload, {
pid: 100,
window_id: 200,
element_token: 'e1234abcd',
session: calls[0]!.payload!.session,
scope: 'window',
delivery_mode: 'background'
})
} finally {
await cleanup()
}
})
test('stale driver token errors and cancellation both fail closed', async () => {
const { adapter, runner, cleanup } = await harness()
try {
const session = await adapter.openSession(identity('control-failure'))
runner.rejectClicksAsStale = true
await assert.rejects(
session.clickElement({ pid: 100, windowId: 200, elementToken: 'edeadbeef' }),
/rejected the action/
)
const controller = new AbortController()
controller.abort()
await assert.rejects(
session.pressKey({ pid: 100, windowId: 200 }, 'escape', controller.signal),
/cancelled/
)
const key = runner.calls.find(call => call.args[1] === 'press_key')
assert.equal(key, undefined)
await session.close()
} finally {
await cleanup()
}
})
test('two control authorities get isolated virtual cursor sessions and independent teardown', async () => {
const { adapter, runner, cleanup } = await harness()
try {
const first = await adapter.openSession(identity('control-one'))
const second = await adapter.openSession(identity('control-two'))
const starts = runner.calls.filter(call => call.args[1] === 'start_session')
assert.equal(starts.length, 2)
assert.notEqual(starts[0]!.payload!.session, starts[1]!.payload!.session)
assert.match(String(starts[0]!.payload!.session), /^hermes-[0-9a-f]{32}$/)
assert.equal(starts[0]!.payload!.capture_scope, 'window')
assert.equal(starts[1]!.payload!.capture_scope, 'window')
await first.close()
await assert.rejects(first.snapshot({ pid: 100, windowId: 200 }), /closed/)
await second.snapshot({ pid: 300, windowId: 400 })
await second.close()
const ends = runner.calls.filter(call => call.args[1] === 'end_session').map(call => call.payload!.session)
assert.deepEqual(new Set(ends), new Set(starts.map(call => call.payload!.session)))
} finally {
await cleanup()
}
})
+326
View File
@@ -0,0 +1,326 @@
import assert from 'node:assert/strict'
import { createHash } from 'node:crypto'
import { access, mkdir, mkdtemp, rm, symlink, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import test from 'node:test'
import {
checkCuaUpdate,
getCuaManagementStatus,
installCuaDriver,
isSupportedCuaVersion,
updateCuaDriver,
type CuaManagementFetch
} from '../src/tools/cuaManagement.js'
import { CuaDriverAdapter, type CuaProcessResult, type CuaProcessRunner } from '../src/tools/cuaDriver.js'
import { computerUseCommand } from '../src/commands/computerUse.js'
const ok = (stdout = ''): CuaProcessResult => ({ stdout, stderr: '', exitCode: 0 })
async function packageHome(version = '0.19.3'): Promise<{ root: string; binary: string }> {
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-management-'))
const release = join(root, '.cua-driver', 'packages', 'releases', `${version}-x86_64-pc-windows-msvc`)
const current = join(root, '.cua-driver', 'packages', 'current')
await mkdir(release, { recursive: true })
const binary = join(release, 'cua-driver.exe')
await writeFile(binary, `fake-cua-${version}`)
await symlink(release, current, 'junction')
return { root, binary }
}
class ProbeRunner implements CuaProcessRunner {
constructor(
readonly version = '0.19.3',
readonly latest = version,
readonly onRun?: (executable: string, args: readonly string[], env?: NodeJS.ProcessEnv) => Promise<void> | void,
readonly health: string = 'ok'
) {}
async run(executable: string, args: readonly string[], options: { env?: NodeJS.ProcessEnv } = {}): Promise<CuaProcessResult> {
await this.onRun?.(executable, args, options.env)
if (args[0] === '--version') return ok(`cua-driver ${this.version}`)
if (args[0] === 'check-update') return ok(JSON.stringify({
checked_at: '2026-08-13T00:00:00Z',
current_version: this.version,
latest_version: this.latest,
update_available: this.latest !== this.version,
error: null
}))
if (args[0] === 'manifest') return ok(JSON.stringify({
schema_version: '1',
binary_version: this.version,
binary_path: executable
}))
if (args[0] === 'list-tools') return ok([
'health_report:', 'start_session:', 'end_session:', 'list_windows:',
'get_window_state:', 'click:', 'set_value:', 'press_key:', 'scroll:'
].join('\n'))
if (args[0] === 'status') return ok('permission mode: standard')
if (args[0] === 'call' && args[1] === 'health_report') return ok(JSON.stringify({
schema_version: '1', driver_version: this.version, overall: this.health
}))
return ok()
}
}
test('supported CUA range is bounded to the adapter contract', () => {
assert.equal(isSupportedCuaVersion('0.19.2'), false)
assert.equal(isSupportedCuaVersion('0.19.3'), true)
assert.equal(isSupportedCuaVersion('0.19.99'), true)
assert.equal(isSupportedCuaVersion('0.20.0'), false)
})
test('status prefers canonical package/current and reports a competing PATH shim', async () => {
const home = await packageHome()
const stale = await mkdtemp(join(tmpdir(), 'hermes-cua-stale-'))
await writeFile(join(stale, 'cua-driver.exe'), 'stale')
try {
const status = await getCuaManagementStatus({
platform: 'win32', homeDir: home.root, path: stale, runner: new ProbeRunner()
})
assert.equal(status.installed, true)
assert.equal(status.current_version, '0.19.3')
assert.equal(status.compatible, true)
assert.equal(status.stale_path_shim, true)
assert.equal(status.canonical_path, home.binary)
assert.equal(status.release_source, 'trycua/cua')
assert.equal(status.telemetry_enabled, false)
} finally {
await rm(home.root, { recursive: true, force: true })
await rm(stale, { recursive: true, force: true })
}
})
test('check-update exposes a newer incompatible release without applying it', async () => {
const home = await packageHome()
try {
const status = await checkCuaUpdate({
platform: 'win32', homeDir: home.root, path: '', runner: new ProbeRunner('0.19.3', '0.20.0')
})
assert.equal(status.update?.update_available, true)
assert.equal(status.update?.latest_version, '0.20.0')
assert.equal(status.update?.compatible, false)
} finally {
await rm(home.root, { recursive: true, force: true })
}
})
test('update refuses an unsupported latest release before any download or apply', async () => {
const home = await packageHome()
let fetches = 0
let powershellRuns = 0
try {
await assert.rejects(updateCuaDriver({
platform: 'win32',
homeDir: home.root,
path: '',
runner: new ProbeRunner('0.19.3', '0.20.0', executable => {
if (executable.toLowerCase() === 'powershell.exe') powershellRuns += 1
}),
fetch: async () => {
fetches += 1
throw new Error('must not fetch')
}
}), /available but unsupported/)
assert.equal(fetches, 0)
assert.equal(powershellRuns, 0)
} finally {
await rm(home.root, { recursive: true, force: true })
}
})
test('install verifies trusted release metadata/checksum and sanitizes installer environment', async () => {
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-install-test-'))
const script = Buffer.from('Write-Output canonical-installer')
const scriptSha = createHash('sha256').update(script).digest('hex')
const fetchImpl: CuaManagementFetch = async url => ({
ok: true,
status: 200,
async arrayBuffer() { return script.buffer.slice(script.byteOffset, script.byteOffset + script.byteLength) },
async json() {
assert.match(url, /release-manifest\.json$/)
return {
schemaVersion: 1,
repository: 'trycua/cua',
product: 'cua-driver-rs',
version: '0.19.3',
tag: 'cua-driver-rs-v0.19.3',
assets: [{ name: 'install.ps1', sha256: scriptSha }]
}
}
})
let installerEnvironment: NodeJS.ProcessEnv | undefined
let installerPath: string | undefined
const runner = new ProbeRunner('0.19.3', '0.19.3', async (executable, args, env) => {
if (!executable.toLowerCase().endsWith('\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe')) return
installerEnvironment = env
installerPath = args[args.indexOf('-File') + 1]
const release = join(root, '.cua-driver', 'packages', 'releases', '0.19.3-x86_64-pc-windows-msvc')
await mkdir(release, { recursive: true })
await writeFile(join(release, 'cua-driver.exe'), 'installed')
await symlink(release, join(root, '.cua-driver', 'packages', 'current'), 'junction')
})
const priorSecret = process.env.OPENAI_API_KEY
process.env.OPENAI_API_KEY = 'must-not-leak'
try {
const status = await installCuaDriver({
platform: 'win32', homeDir: root, path: '', runner, fetch: fetchImpl, systemRoot: 'C:\\Windows'
})
assert.equal(status.operation?.release_manifest_verified, true)
assert.equal(status.operation?.installer_checksum_verified, true)
assert.equal(status.operation?.runtime_verified, true)
assert.equal(installerEnvironment?.OPENAI_API_KEY, undefined)
assert.equal(installerEnvironment?.CUA_DRIVER_RS_TELEMETRY_ENABLED, '0')
assert.equal(installerEnvironment?.CUA_DRIVER_RS_VERSION, '0.19.3')
assert.ok(installerPath)
await assert.rejects(access(installerPath!))
} finally {
if (priorSecret === undefined) delete process.env.OPENAI_API_KEY
else process.env.OPENAI_API_KEY = priorSecret
await rm(root, { recursive: true, force: true })
}
})
test('install rejects invalid release identity metadata before downloading the installer', async () => {
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-invalid-manifest-'))
let requests = 0
try {
await assert.rejects(installCuaDriver({
platform: 'win32', homeDir: root, path: '', runner: new ProbeRunner(),
fetch: async url => {
requests += 1
assert.match(url, /release-manifest\.json$/)
return {
ok: true,
status: 200,
async arrayBuffer() { return new ArrayBuffer(0) },
async json() {
return {
schemaVersion: 1,
repository: 'attacker/fork',
product: 'cua-driver-rs',
version: '0.19.3',
tag: 'cua-driver-rs-v0.19.3',
assets: [{ name: 'install.ps1', sha256: 'a'.repeat(64) }]
}
}
}
}
}), /publisher or version manifest is invalid/)
assert.equal(requests, 1)
} finally {
await rm(root, { recursive: true, force: true })
}
})
test('install rejects an installer whose bytes do not match release metadata', async () => {
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-bad-checksum-'))
let powershellRuns = 0
try {
await assert.rejects(installCuaDriver({
platform: 'win32', homeDir: root, path: '',
runner: new ProbeRunner('0.19.3', '0.19.3', executable => {
if (executable.toLowerCase() === 'powershell.exe') powershellRuns += 1
}),
fetch: async url => ({
ok: true,
status: 200,
async arrayBuffer() { return Buffer.from('tampered').buffer },
async json() {
assert.match(url, /release-manifest\.json$/)
return {
schemaVersion: 1, repository: 'trycua/cua', product: 'cua-driver-rs',
version: '0.19.3', tag: 'cua-driver-rs-v0.19.3',
assets: [{ name: 'install.ps1', sha256: 'a'.repeat(64) }]
}
}
})
}), /checksum verification failed/)
assert.equal(powershellRuns, 0)
} finally {
await rm(root, { recursive: true, force: true })
}
})
test('post-install runtime verification succeeds while explicit health remains degraded', async () => {
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-degraded-'))
const script = Buffer.from('installer')
const checksum = createHash('sha256').update(script).digest('hex')
const runner = new ProbeRunner('0.19.3', '0.19.3', async executable => {
if (!executable.toLowerCase().endsWith('\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe')) return
const release = join(root, '.cua-driver', 'packages', 'releases', '0.19.3-x86_64-pc-windows-msvc')
await mkdir(release, { recursive: true })
await writeFile(join(release, 'cua-driver.exe'), 'installed')
await symlink(release, join(root, '.cua-driver', 'packages', 'current'), 'junction')
}, 'degraded')
try {
const installed = await installCuaDriver({
platform: 'win32', homeDir: root, path: '', runner, systemRoot: 'C:\\Windows',
fetch: async url => ({
ok: true,
status: 200,
async arrayBuffer() { return script.buffer.slice(script.byteOffset, script.byteOffset + script.byteLength) },
async json() {
assert.match(url, /release-manifest\.json$/)
return {
schemaVersion: 1, repository: 'trycua/cua', product: 'cua-driver-rs',
version: '0.19.3', tag: 'cua-driver-rs-v0.19.3',
assets: [{ name: 'install.ps1', sha256: checksum }]
}
}
})
})
assert.equal(installed.operation?.runtime_verified, true)
const health = await CuaDriverAdapter.healthStatus({ platform: 'win32', homeDir: root, runner })
assert.equal(health.state, 'degraded')
} finally {
await rm(root, { recursive: true, force: true })
}
})
test('CLI install and update require explicit confirmation', async () => {
let stderr = ''
const original = process.stderr.write
process.stderr.write = ((chunk: string | Uint8Array) => {
stderr += chunk.toString()
return true
}) as typeof process.stderr.write
try {
assert.equal(await computerUseCommand({ command: 'computer-use', flags: {}, positional: ['cua', 'install'] }), 1)
assert.match(stderr, /requires explicit confirmation with --yes/)
} finally {
process.stderr.write = original
}
})
test('install rejects an unverified installer before process execution', async () => {
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-bad-installer-'))
let processStarted = false
const fetchImpl: CuaManagementFetch = async url => ({
ok: true,
status: 200,
async arrayBuffer() { return Buffer.from('tampered').buffer },
async json() {
assert.match(url, /release-manifest\.json$/)
return {
schemaVersion: 1,
repository: 'trycua/cua',
product: 'cua-driver-rs',
version: '0.19.3',
tag: 'cua-driver-rs-v0.19.3',
assets: [{ name: 'install.ps1', sha256: '0'.repeat(64) }]
}
}
})
const runner = new ProbeRunner('0.19.3', '0.19.3', () => { processStarted = true })
try {
await assert.rejects(
installCuaDriver({ platform: 'win32', homeDir: root, path: '', runner, fetch: fetchImpl }),
/checksum verification failed/
)
assert.equal(processStarted, false)
} finally {
await rm(root, { recursive: true, force: true })
}
})
+41
View File
@@ -9,6 +9,8 @@ import {
readDesktopUseSettings,
readDesktopUseSettingsSync,
requestComputerGrantCancellation,
setActivityScreenshotRetention,
setComputerControlSettings,
setDesktopUseEnabled
} from '../src/lib/desktopUseSettings.js'
import { shouldAdvertiseComputerUse } from '../src/tools/handlerSet.js'
@@ -34,6 +36,45 @@ test('desktop-use preference defaults off and persists explicit changes', async
}
})
test('computer control settings default fail-closed and survive desktop-use changes', async () => {
const dir = await mkdtemp(join(tmpdir(), 'hermes-computer-control-'))
const settingsPath = join(dir, 'desktop-settings.json')
try {
assert.deepEqual(await readDesktopUseSettings(settingsPath), {
computer_use_enabled: false,
computer_control_engine: 'cua',
cua_cursor_enabled: false,
activity_screenshot_retention_enabled: true,
activity_screenshot_retention_days: 7
})
await setComputerControlSettings({
computer_control_engine: 'cua',
cua_cursor_enabled: true
}, settingsPath)
await setDesktopUseEnabled(true, settingsPath)
const settings = await readDesktopUseSettings(settingsPath)
assert.equal(settings.computer_use_enabled, true)
assert.equal(settings.computer_control_engine, 'cua')
assert.equal(settings.cua_cursor_enabled, true)
} finally {
await rm(dir, { recursive: true, force: true })
}
})
test('screenshot evidence retention is explicit and survives other desktop setting changes', async () => {
const dir = await mkdtemp(join(tmpdir(), 'hermes-screenshot-retention-'))
const settingsPath = join(dir, 'desktop-settings.json')
try {
await setActivityScreenshotRetention(false, 30, settingsPath)
await setDesktopUseEnabled(true, settingsPath)
const settings = await readDesktopUseSettings(settingsPath)
assert.equal(settings.activity_screenshot_retention_enabled, false)
assert.equal(settings.activity_screenshot_retention_days, 30)
} finally {
await rm(dir, { recursive: true, force: true })
}
})
test('grant cancellation bridge is consumed exactly once', async () => {
const dir = await mkdtemp(join(tmpdir(), 'hermes-desktop-cancel-'))
const cancelPath = join(dir, 'cancel-active.json')
+30 -1
View File
@@ -31,6 +31,8 @@ test('installer launch is delayed until the running CLI can exit', () => {
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_PATH, 'C:\\Temp\\hermes setup.exe')
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_SILENT, '1')
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_INSTALL_DIR, 'C:\\Hermes custom')
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_CALLER_PID, String(process.pid))
assert.match(plan.args.join(' '), /Wait-Process -Id \$callerPid/)
assert.match(plan.args.join(' '), /Start-Sleep/)
assert.match(plan.args.join(' '), /\/D=/)
assert.match(plan.args.join(' '), /Remove-Item -LiteralPath \$installer/)
@@ -44,7 +46,7 @@ test('PowerShell launches an installer whose path contains spaces via environmen
const fakeInstaller = join(scratch, 'fake setup.exe')
await copyFile(join(process.env.WINDIR ?? 'C:\\Windows', 'System32', 'whoami.exe'), fakeInstaller)
try {
const plan = windowsInstallerLaunchPlan(fakeInstaller, { silent: false, delayMs: 0 })
const plan = windowsInstallerLaunchPlan(fakeInstaller, { silent: false, delayMs: 0, callerPid: 0 })
const result = spawnSync(plan.program, plan.args, { ...plan.options, detached: false, stdio: 'pipe' })
assert.equal(result.status, 0, result.stderr?.toString())
} finally {
@@ -76,6 +78,25 @@ test('NSIS bundle cleans cooperative-update and local-development backups', asyn
}
})
test('NSIS bundle quiesces tray children and retries failed payload extraction once', async () => {
const script = await readFile(new URL('../tray/installer/hermes-relay.nsi', import.meta.url), 'utf8')
const core = script.slice(script.indexOf('Section "Hermes-Relay CLI and management UI"'), script.indexOf('Section "Start tray when I sign in"'))
const trayStop = core.indexOf('/IM hermes-relay-tray.exe /T /F')
const cliDrain = core.indexOf('/IM hermes-relay.exe /T /F')
const firstInstall = core.indexOf('File /oname=hermes-relay.exe "${CLI_EXE}"')
const retry = core.indexOf('install_attempt_failed:')
assert.ok(trayStop >= 0, 'installer should terminate the tray process tree')
assert.ok(cliDrain < firstInstall, 'installed processes must be drained before replacement')
assert.ok(retry > firstInstall, 'installer should retry the first failed extraction attempt')
assert.match(core, /StrCpy \$InstallAttempt "1"[\s\S]*Goto install_attempt/)
assert.match(core, /ClearErrors[\s\S]*File \/oname=hermes-relay\.exe[\s\S]*File \/oname=hermes-relay-tray\.exe[\s\S]*IfErrors install_attempt_failed/)
assert.match(core, /SetErrorLevel 1\s+Quit/)
assert.match(core, /MessageBox MB_ICONSTOP .* \/SD IDOK/)
assert.match(core, /taskkill\.exe" \/IM hermes-relay-tray\.exe \/T \/F/)
assert.match(core, /taskkill\.exe" \/IM hermes-relay\.exe \/T \/F/)
})
test('local tray installation embeds production assets instead of loading devUrl', async () => {
const script = await readFile(new URL('../scripts/dev-install-tray.mjs', import.meta.url), 'utf8')
assert.match(script, /'--features', 'custom-protocol'/)
@@ -87,6 +108,14 @@ test('CLI opens the GUI process without forcing a hidden Windows startup state',
assert.doesNotMatch(source, /windowsHide:\s*true/)
})
test('tray update helper preserves the current install directory and cleans its installer', async () => {
const source = await readFile(new URL('../tray/src/main.rs', import.meta.url), 'utf8')
assert.match(source, /\.env\("HERMES_UPDATE_INSTALL_DIR", install_dir\)/)
assert.match(source, /\$installerArgs=@\('\/S',\('\/D=' \+ \$env:HERMES_UPDATE_INSTALL_DIR\)\)/)
assert.match(source, /Remove-Item -LiteralPath \$env:HERMES_UPDATE_INSTALLER -Force/)
assert.match(source, /exit \$exitCode/)
})
test('POSIX installer only advertises artifacts produced by the release workflow', async () => {
const script = await readFile(new URL('../scripts/install.sh', import.meta.url), 'utf8')
assert.doesNotMatch(script, /hermes-relay-linux-arm64/)
+2 -1
View File
@@ -1232,8 +1232,9 @@ checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "hermes-relay-tray"
version = "0.4.0-alpha.8"
version = "0.4.0-beta.2"
dependencies = [
"base64 0.22.1",
"serde",
"serde_json",
"tauri",
+2 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "hermes-relay-tray"
version = "0.4.0-alpha.8"
version = "0.4.0-beta.2"
description = "Compact Windows management UI for Hermes-Relay CLI"
authors = ["Axiom Labs"]
edition = "2021"
@@ -14,6 +14,7 @@ path = "src/main.rs"
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1"
base64 = "0.22"
tauri = { version = "2", features = ["tray-icon", "image-png"] }
[features]
+32 -4
View File
@@ -19,6 +19,7 @@ VIAddVersionKey "FileVersion" "${VERSION}"
VIAddVersionKey "LegalCopyright" "MIT License"
Var ExistingStartup
Var InstallAttempt
Function .onInit
ReadRegStr $ExistingStartup HKCU "Software\Microsoft\Windows\CurrentVersion\Run" "HermesRelayTray"
@@ -42,20 +43,47 @@ FunctionEnd
Section "Hermes-Relay CLI and management UI" SEC_CORE
SectionIn RO
IfFileExists "$INSTDIR\hermes-relay.exe" 0 tray_stop
; Stop the tray first so its snapshot polling cannot launch another CLI
; process while setup is trying to replace hermes-relay.exe. The following
; /T also terminates a poll already in flight before payload extraction.
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay-tray.exe /T /F'
IfFileExists "$INSTDIR\hermes-relay.exe" 0 processes_quiesced
nsExec::ExecToLog '"$INSTDIR\hermes-relay.exe" daemon stop'
tray_stop:
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay-tray.exe /F'
Sleep 250
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay.exe /T /F'
processes_quiesced:
; Drain the daemon and any short-lived tray/management CLI children that
; raced with shutdown. An explicit bundle update is the lifecycle boundary
; for installed Hermes-Relay processes.
Sleep 350
Delete "$INSTDIR\hermes-relay.new.exe"
Delete "$INSTDIR\hermes-relay.old.exe"
Delete "$INSTDIR\hermes-relay.exe.bak"
Delete "$INSTDIR\hermes-relay-tray.exe.bak"
StrCpy $InstallAttempt "0"
install_attempt:
SetOutPath "$INSTDIR"
ClearErrors
File /oname=hermes-relay.exe "${CLI_EXE}"
File /oname=hermes-relay-tray.exe "${TRAY_EXE}"
File /oname=hermes-relay-path.ps1 "${PATH_HELPER}"
File /oname=hermes-relay-ui.cmd "${UI_SHIM}"
IfErrors install_attempt_failed
; ClearErrors/IfErrors covers both CLI and tray writes, so setup cannot
; publish registry metadata after a locked or partial replacement.
Goto cli_verified
install_attempt_failed:
StrCmp $InstallAttempt "0" 0 cli_verification_failed
StrCpy $InstallAttempt "1"
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay-tray.exe /T /F'
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay.exe /T /F'
Sleep 350
Goto install_attempt
cli_verification_failed:
MessageBox MB_ICONSTOP "Hermes-Relay CLI UI ${VERSION} could not replace the running installation. Close Hermes-Relay processes and run setup again." /SD IDOK
SetErrorLevel 1
Quit
cli_verified:
WriteUninstaller "$INSTDIR\uninstall-hermes-relay.exe"
WriteRegStr HKCU "Software\HermesRelay" "InstallDir" "$INSTDIR"
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@hermes-relay/tray-ui",
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@hermes-relay/tray-ui",
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"dependencies": {
"@tauri-apps/api": "^2.8.0",
"lucide-react": "^0.468.0",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@hermes-relay/tray-ui",
"private": true,
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"type": "module",
"scripts": {
"dev": "vite --host 127.0.0.1",
+523 -20
View File
@@ -5,6 +5,7 @@ compile_error!("hermes-relay-tray is a Windows-only optional systray");
#[cfg(windows)]
mod app {
use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::{
@@ -15,7 +16,7 @@ mod app {
os::windows::process::CommandExt,
path::{Path, PathBuf},
process::{Command, Output, Stdio},
sync::{mpsc, Arc, Mutex},
sync::{mpsc, Arc, Mutex, OnceLock},
thread,
time::{Duration, SystemTime, UNIX_EPOCH},
};
@@ -225,6 +226,10 @@ mod app {
privilege: Option<String>,
username: Option<String>,
updated_at: Option<u64>,
last_event: Option<String>,
reconnect_attempt: Option<u64>,
retry_at: Option<u64>,
last_error: Option<String>,
}
fn stopped() -> String {
@@ -269,6 +274,26 @@ mod app {
#[serde(skip_serializing_if = "Option::is_none")]
host_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
backend: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
dispatch: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
control_session_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
target_app: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
target_title: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
target_pid: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
target_window_id: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
action: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
verification: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
phase: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
duration_ms: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
exit_code: Option<i64>,
@@ -294,6 +319,14 @@ mod app {
result_truncated: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
error: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
screenshot_evidence_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
screenshot_mime_type: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
screenshot_width: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
screenshot_height: Option<u64>,
}
#[derive(Debug, Clone, Deserialize, Serialize)]
@@ -312,15 +345,100 @@ mod app {
active_url: Option<String>,
daemon: DaemonStatus,
activity: Vec<Activity>,
activity_screenshot_retention: ActivityScreenshotRetention,
pending_grants: Vec<PendingGrantRequest>,
startup_enabled: bool,
daemon_autostart_enabled: bool,
hardware_availability: HardwareAvailability,
computer_control_engine: Option<ComputerControlEngine>,
ui_version: &'static str,
cli_version: Option<String>,
cli_path: Option<String>,
}
#[derive(Debug, Clone, Deserialize, Serialize)]
struct ActivityScreenshotRetention {
#[serde(default)]
enabled: bool,
#[serde(default = "default_retention_days")]
days: u64,
#[serde(default)]
count: u64,
#[serde(default)]
bytes: u64,
}
fn default_retention_days() -> u64 {
7
}
impl Default for ActivityScreenshotRetention {
fn default() -> Self {
Self {
enabled: true,
days: 7,
count: 0,
bytes: 0,
}
}
}
#[derive(Debug, Clone, Deserialize, Serialize)]
struct ComputerControlEngine {
selected: String,
effective: String,
available: bool,
state: String,
version: Option<String>,
health: Option<String>,
path: Option<String>,
#[serde(default)]
cursor_enabled: bool,
#[serde(default)]
foreground_escalation_enabled: bool,
#[serde(default)]
active_sessions: Option<u64>,
active_backend: Option<String>,
last_action: Option<Value>,
message: Option<String>,
}
type ComputerControlEngineCache = Option<(SystemTime, Option<ComputerControlEngine>)>;
static COMPUTER_CONTROL_ENGINE_CACHE: OnceLock<Mutex<ComputerControlEngineCache>> =
OnceLock::new();
fn probe_computer_control_engine() -> Option<ComputerControlEngine> {
run_json(&["computer-use", "status", "--json"])
.ok()
.and_then(|value| value.get("computer_control_engine").cloned())
.and_then(|value| serde_json::from_value::<ComputerControlEngine>(value).ok())
}
fn cached_computer_control_engine() -> Option<ComputerControlEngine> {
let cache = COMPUTER_CONTROL_ENGINE_CACHE.get_or_init(|| Mutex::new(None));
if let Ok(guard) = cache.lock() {
if let Some((checked_at, status)) = guard.as_ref() {
if checked_at.elapsed().unwrap_or_default() < Duration::from_secs(30) {
return status.clone();
}
}
}
let status = probe_computer_control_engine();
if let Ok(mut guard) = cache.lock() {
*guard = Some((SystemTime::now(), status.clone()));
}
status
}
fn clear_computer_control_engine_cache() {
if let Some(cache) = COMPUTER_CONTROL_ENGINE_CACHE.get() {
if let Ok(mut guard) = cache.lock() {
*guard = None;
}
}
}
#[derive(Debug, Serialize)]
struct HardwareAvailability {
usb: bool,
@@ -521,17 +639,36 @@ mod app {
let Ok(path) = home_dir().map(|h| h.join(".hermes").join("desktop-audit.jsonl")) else {
return Vec::new();
};
let Ok(text) = fs::read_to_string(path) else {
return Vec::new();
};
text.lines()
.rev()
.take(30)
.filter_map(|line| serde_json::from_str(line).ok())
.collect::<Vec<_>>()
let text = [path.with_extension("jsonl.1"), path]
.into_iter()
.rev()
.collect()
.filter_map(|file| fs::read_to_string(file).ok())
.collect::<Vec<_>>()
.join("\n");
let mut activity = text
.lines()
.filter_map(|line| serde_json::from_str(line).ok())
.collect::<Vec<Activity>>();
let evidence_directory = home_dir()
.ok()
.map(|home| home.join(".hermes").join("activity-evidence"));
for entry in &mut activity {
let available = entry
.screenshot_evidence_id
.as_ref()
.zip(evidence_directory.as_ref())
.is_some_and(|(id, directory)| directory.join(format!("{id}.png")).is_file());
if !available {
entry.screenshot_evidence_id = None;
entry.screenshot_mime_type = None;
entry.screenshot_width = None;
entry.screenshot_height = None;
}
}
activity.sort_by_key(|entry| entry.ts);
if activity.len() > 200 {
activity.drain(..activity.len() - 200);
}
activity
}
fn append_management_event(
@@ -579,6 +716,90 @@ mod app {
fs::remove_file(&backup)
.map_err(|error| format!("cannot remove rotated activity: {error}"))?;
}
let evidence = directory.join("activity-evidence");
if evidence.exists() {
fs::remove_dir_all(&evidence)
.map_err(|error| format!("cannot clear screenshot evidence: {error}"))?;
}
Ok(())
}
fn append_management_error(tool: &str, summary: &str, error: &str) {
let Ok(directory) = home_dir().map(|home| home.join(".hermes")) else {
return;
};
if fs::create_dir_all(&directory).is_err() {
return;
}
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_millis() as u64;
let event = serde_json::json!({
"ts": timestamp,
"kind": "management.completed",
"tool": tool,
"category": "system",
"ok": false,
"summary": summary,
"error": error.chars().take(512).collect::<String>(),
});
if let Ok(mut file) = OpenOptions::new()
.create(true)
.append(true)
.open(directory.join("desktop-audit.jsonl"))
{
let _ = writeln!(file, "{event}");
}
}
#[tauri::command]
fn get_activity_screenshot(evidence_id: String) -> Result<String, String> {
if evidence_id.len() != 32 || !evidence_id.bytes().all(|byte| byte.is_ascii_hexdigit()) {
return Err("invalid screenshot evidence identifier".to_string());
}
let path = home_dir()?
.join(".hermes")
.join("activity-evidence")
.join(format!("{}.png", evidence_id.to_ascii_lowercase()));
let bytes =
fs::read(path).map_err(|_| "screenshot evidence is no longer available".to_string())?;
if bytes.is_empty()
|| bytes.len() > 10_000_000
|| !bytes.starts_with(&[0x89, b'P', b'N', b'G', 0x0d, 0x0a, 0x1a, 0x0a])
{
return Err("screenshot evidence is invalid".to_string());
}
Ok(format!(
"data:image/png;base64,{}",
BASE64_STANDARD.encode(bytes)
))
}
#[tauri::command]
fn set_activity_screenshot_retention(enabled: bool, days: u64) -> Result<(), String> {
if !matches!(days, 1 | 7 | 30) {
return Err("screenshot retention must be 1, 7, or 30 days".to_string());
}
let days = days.to_string();
run_cli_checked(&[
"audit",
"screenshots",
if enabled { "on" } else { "off" },
"--days",
&days,
"--yes",
])?;
append_management_event(
"activity.retention",
if enabled {
"Screenshot evidence retention changed"
} else {
"Screenshot evidence retention disabled"
},
None,
None,
);
Ok(())
}
@@ -643,21 +864,122 @@ mod app {
.and_then(|value| serde_json::from_value::<Vec<PendingGrantRequest>>(value).ok())
.unwrap_or_default();
let (cli_version, cli_path) = cli_details();
let computer_control_engine = cached_computer_control_engine();
let activity_screenshot_retention = run_json(&["audit", "screenshots", "--json"])
.ok()
.and_then(|value| serde_json::from_value(value).ok())
.unwrap_or_default();
Ok(Snapshot {
hosts,
active_url: selected,
daemon,
activity: read_activity(),
activity_screenshot_retention,
pending_grants,
startup_enabled: startup_enabled(),
daemon_autostart_enabled: daemon_autostart_enabled(),
hardware_availability: hardware_availability(),
computer_control_engine,
ui_version: env!("CARGO_PKG_VERSION"),
cli_version,
cli_path,
})
}
fn computer_control_engine_status() -> Result<ComputerControlEngine, String> {
run_json(&["computer-use", "status", "--json"])?
.get("computer_control_engine")
.cloned()
.ok_or_else(|| {
"the installed CLI does not report a computer control engine".to_string()
})
.and_then(|value| {
serde_json::from_value(value).map_err(|_| {
"the installed CLI returned an invalid computer control status".to_string()
})
})
}
#[tauri::command]
async fn computer_cua_status() -> Result<Value, String> {
tauri::async_runtime::spawn_blocking(|| {
run_json(&["computer-use", "cua", "status", "--json"])
})
.await
.map_err(|error| format!("CUA status task failed: {error}"))?
}
#[tauri::command]
async fn computer_cua_health() -> Result<Value, String> {
tauri::async_runtime::spawn_blocking(|| {
run_json(&["computer-use", "cua", "health", "--json"])
})
.await
.map_err(|error| format!("CUA health task failed: {error}"))?
}
#[tauri::command]
async fn computer_cua_install() -> Result<Value, String> {
tauri::async_runtime::spawn_blocking(|| {
let result = run_json(&["computer-use", "cua", "install", "--yes", "--json"])?;
clear_computer_control_engine_cache();
Ok(result)
})
.await
.map_err(|error| format!("CUA install task failed: {error}"))?
}
#[tauri::command]
async fn computer_cua_check_update() -> Result<Value, String> {
tauri::async_runtime::spawn_blocking(|| {
run_json(&["computer-use", "cua", "check-update", "--json"])
})
.await
.map_err(|error| format!("CUA update check task failed: {error}"))?
}
#[tauri::command]
async fn computer_cua_update() -> Result<Value, String> {
tauri::async_runtime::spawn_blocking(|| {
let result = run_json(&["computer-use", "cua", "update", "--yes", "--json"])?;
clear_computer_control_engine_cache();
Ok(result)
})
.await
.map_err(|error| format!("CUA update task failed: {error}"))?
}
#[tauri::command]
fn set_computer_control_engine(engine: String) -> Result<(), String> {
if engine != "legacy" && engine != "cua" {
return Err("invalid computer control engine".to_string());
}
if engine == "cua" {
let status = computer_control_engine_status()?;
if !status.available || status.state != "ready" {
return Err("CUA Driver is not ready; engine selection was not changed".to_string());
}
}
let was_running = daemon_is_running();
run_cli_checked(&["computer-use", "engine", engine.as_str()])?;
clear_computer_control_engine_cache();
restart_daemon_if_running(was_running)
}
#[tauri::command]
fn set_cua_cursor_enabled(enabled: bool) -> Result<(), String> {
let status = computer_control_engine_status()?;
if status.selected != "cua" || status.state != "ready" {
return Err(
"CUA Driver must be selected and ready before changing its cursor".to_string(),
);
}
let was_running = daemon_is_running();
run_cli_checked(&["computer-use", "cursor", if enabled { "on" } else { "off" }])?;
clear_computer_control_engine_cache();
restart_daemon_if_running(was_running)
}
#[tauri::command]
async fn get_snapshot() -> Result<Snapshot, String> {
tauri::async_runtime::spawn_blocking(build_snapshot)
@@ -706,14 +1028,21 @@ mod app {
.parent()
.map(|directory| directory.join("hermes-relay.exe"))
.ok_or_else(|| "cannot resolve the installed CLI path".to_string())?;
let install_dir = tray_exe
.parent()
.ok_or_else(|| "cannot resolve the desktop install directory".to_string())?;
let helper_script = "$ErrorActionPreference='Stop'; "
.to_string()
+ "$targetPid=[int]$env:HERMES_UPDATE_PID; "
+ "Wait-Process -Id $targetPid -ErrorAction SilentlyContinue; "
+ "$result=Start-Process -FilePath $env:HERMES_UPDATE_INSTALLER -ArgumentList '/S' -Wait -PassThru; "
+ "if ($result.ExitCode -eq 0) { "
+ "$exitCode=1; try { "
+ "$installerArgs=@('/S',('/D=' + $env:HERMES_UPDATE_INSTALL_DIR)); "
+ "$result=Start-Process -FilePath $env:HERMES_UPDATE_INSTALLER -ArgumentList $installerArgs -Wait -PassThru; "
+ "$exitCode=$result.ExitCode; if ($exitCode -eq 0) { "
+ "if ($env:HERMES_UPDATE_RESTART_DAEMON -eq '1') { & $env:HERMES_UPDATE_CLI daemon start | Out-Null }; "
+ "Start-Process -FilePath $env:HERMES_UPDATE_TRAY }";
+ "Start-Process -FilePath $env:HERMES_UPDATE_TRAY } "
+ "} finally { Remove-Item -LiteralPath $env:HERMES_UPDATE_INSTALLER -Force -ErrorAction SilentlyContinue }; "
+ "exit $exitCode";
Command::new("powershell.exe")
.args([
"-NoProfile",
@@ -725,6 +1054,7 @@ mod app {
])
.env("HERMES_UPDATE_PID", std::process::id().to_string())
.env("HERMES_UPDATE_INSTALLER", installer)
.env("HERMES_UPDATE_INSTALL_DIR", install_dir)
.env("HERMES_UPDATE_TRAY", tray_exe)
.env("HERMES_UPDATE_CLI", cli_exe)
.env(
@@ -1126,7 +1456,10 @@ mod app {
#[tauri::command]
async fn connect_daemon() -> Result<(), String> {
tauri::async_runtime::spawn_blocking(|| {
run_cli_checked(&["daemon", "start"])?;
if let Err(error) = run_cli_checked(&["daemon", "start"]) {
append_management_error("daemon.start", "Relay daemon failed to connect", &error);
return Err(error);
}
append_management_event("daemon.start", "Relay daemon connected", None, None);
Ok(())
})
@@ -1136,7 +1469,10 @@ mod app {
#[tauri::command]
async fn disconnect_daemon() -> Result<(), String> {
tauri::async_runtime::spawn_blocking(|| {
run_cli_checked(&["daemon", "stop"])?;
if let Err(error) = run_cli_checked(&["daemon", "stop"]) {
append_management_error("daemon.stop", "Relay daemon failed to disconnect", &error);
return Err(error);
}
append_management_event("daemon.stop", "Relay daemon disconnected", None, None);
Ok(())
})
@@ -1144,10 +1480,17 @@ mod app {
.map_err(|error| format!("disconnect daemon task failed: {error}"))?
}
#[tauri::command]
fn restart_daemon() -> Result<(), String> {
run_cli_checked(&["daemon", "restart"])?;
append_management_event("daemon.restart", "Relay daemon restarted", None, None);
Ok(())
async fn restart_daemon() -> Result<(), String> {
tauri::async_runtime::spawn_blocking(|| {
if let Err(error) = run_cli_checked(&["daemon", "restart"]) {
append_management_error("daemon.restart", "Relay daemon failed to restart", &error);
return Err(error);
}
append_management_event("daemon.restart", "Relay daemon restarted", None, None);
Ok(())
})
.await
.map_err(|error| format!("restart daemon task failed: {error}"))?
}
#[tauri::command]
@@ -1484,6 +1827,154 @@ mod app {
present_grant_window_inner(&app, expanded, &tray_position)
}
#[derive(Serialize)]
struct ConnectionNotice<'a> {
tone: &'a str,
title: &'a str,
detail: String,
}
fn daemon_status_from_file() -> Option<DaemonStatus> {
let path = home_dir().ok()?.join(".hermes").join("daemon-status.json");
serde_json::from_slice(&fs::read(path).ok()?).ok()
}
fn present_connection_notice(app: &AppHandle, notice: ConnectionNotice<'_>) {
if app
.get_webview_window("main")
.is_some_and(|window| window.is_visible().unwrap_or(false))
{
if let Some(window) = app.get_webview_window("notice") {
let _ = window.hide();
}
return;
}
let Some(window) = app.get_webview_window("notice") else {
return;
};
let Some(monitor) = window
.current_monitor()
.ok()
.flatten()
.or_else(|| window.primary_monitor().ok().flatten())
else {
return;
};
let scale = monitor.scale_factor();
let size = PhysicalSize::new(
(360.0 * scale).round() as u32,
(126.0 * scale).round() as u32,
);
let _ = window.set_size(Size::Physical(size));
let _ = window.set_position(bottom_right_position(monitor.work_area(), size, scale));
let Ok(payload) = serde_json::to_string(&notice) else {
return;
};
let _ = window.eval(format!("window.dispatchEvent(new CustomEvent('hermes-connection-notice', {{ detail: {payload} }}))"));
let _ = window.show();
}
fn start_connection_watcher(app: AppHandle) {
thread::spawn(move || {
let mut previous = None::<String>;
loop {
let status = daemon_status_from_file();
let state = status
.as_ref()
.map(|value| value.state.clone())
.unwrap_or_else(|| "stopped".to_string());
if let Some(before) = previous.as_deref() {
if before != state {
let notice = match state.as_str() {
"connected" => Some(ConnectionNotice {
tone: "connected",
title: if before == "reconnecting" {
"Tunnel restored"
} else {
"Tunnel connected"
},
detail: status
.as_ref()
.and_then(|value| value.url.clone())
.unwrap_or_else(|| "Remote access is ready".to_string()),
}),
"reconnecting" => Some(ConnectionNotice {
tone: "warning",
title: "Connection interrupted",
detail: format!(
"Retrying automatically · attempt {}",
status
.as_ref()
.and_then(|value| value.reconnect_attempt)
.unwrap_or(1)
),
}),
"stopped"
if before == "connected"
|| before == "reconnecting"
|| before == "starting" =>
{
Some(ConnectionNotice {
tone: "offline",
title: "Tunnel disconnected",
detail: status
.as_ref()
.and_then(|value| value.last_error.clone())
.unwrap_or_else(|| "Remote access is offline".to_string()),
})
}
_ => None,
};
if let Some(notice) = notice {
let handle = app.clone();
let _ = app.run_on_main_thread(move || {
present_connection_notice(&handle, notice)
});
}
}
}
previous = Some(state);
thread::sleep(Duration::from_millis(500));
}
});
}
#[tauri::command]
fn open_management_from_notice(app: AppHandle, tray_position: State<'_, TrayPositionState>) {
if let Some(window) = app.get_webview_window("notice") {
let _ = window.hide();
}
let anchor = tray_position.0.lock().ok().and_then(|value| *value);
reveal_main_window(&app, anchor);
}
#[tauri::command]
fn present_activity_screenshot(app: AppHandle, evidence_id: String) -> Result<(), String> {
let _ = get_activity_screenshot(evidence_id.clone())?;
let window = app
.get_webview_window("evidence")
.ok_or_else(|| "screenshot viewer is unavailable".to_string())?;
let payload = serde_json::to_string(&serde_json::json!({ "evidenceId": evidence_id }))
.map_err(|error| error.to_string())?;
window.eval(format!("window.dispatchEvent(new CustomEvent('hermes-screenshot-evidence', {{ detail: {payload} }}))")).map_err(|error| error.to_string())?;
let monitor = app
.get_webview_window("main")
.and_then(|main| main.current_monitor().ok().flatten())
.or_else(|| window.current_monitor().ok().flatten())
.or_else(|| window.primary_monitor().ok().flatten());
if let (Some(monitor), Ok(size)) = (monitor, window.outer_size()) {
let work = monitor.work_area();
let x =
work.position.x + ((work.size.width as i64 - size.width as i64).max(0) / 2) as i32;
let y = work.position.y
+ ((work.size.height as i64 - size.height as i64).max(0) / 2) as i32;
let _ = window.set_position(PhysicalPosition::new(x, y));
}
window.show().map_err(|error| error.to_string())?;
let _ = window.set_focus();
Ok(())
}
fn start_grant_watcher(app: AppHandle, tray_position: TrayPositionState) {
thread::spawn(move || {
let mut active_id = None::<String>;
@@ -1549,6 +2040,7 @@ mod app {
pair_host,
test_host_route,
open_management_from_grant,
open_management_from_notice,
forget_host,
connect_daemon,
disconnect_daemon,
@@ -1562,7 +2054,17 @@ mod app {
open_external_url,
set_startup,
set_daemon_autostart,
set_computer_control_engine,
set_cua_cursor_enabled,
computer_cua_status,
computer_cua_health,
computer_cua_install,
computer_cua_check_update,
computer_cua_update,
clear_activity,
get_activity_screenshot,
set_activity_screenshot_retention,
present_activity_screenshot,
present_grant_window
])
.setup(|app| {
@@ -1571,6 +2073,7 @@ mod app {
let tray_anchor = anchor.clone();
let menu_anchor = anchor.clone();
start_grant_watcher(app.handle().clone(), anchor);
start_connection_watcher(app.handle().clone());
start_activation_watcher(app.handle().clone());
let tray_actions = start_tray_action_worker(app.handle().clone());
let click_actions = tray_actions.clone();
+45 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "Hermes-Relay CLI UI",
"version": "0.4.0-alpha.8",
"version": "0.4.0-beta.2",
"identifier": "com.axiomlabs.hermes-relay-tray",
"build": {
"beforeDevCommand": "npm run dev",
@@ -47,6 +47,50 @@
"alwaysOnTop": true,
"skipTaskbar": true,
"center": false
},
{
"label": "notice",
"title": "Hermes-Relay connection status",
"width": 360,
"height": 126,
"resizable": false,
"fullscreen": false,
"decorations": false,
"transparent": true,
"backgroundColor": [
0,
0,
0,
0
],
"shadow": false,
"visible": false,
"alwaysOnTop": true,
"skipTaskbar": true,
"center": false
},
{
"label": "evidence",
"title": "Hermes-Relay screenshot evidence",
"width": 900,
"height": 650,
"minWidth": 560,
"minHeight": 420,
"resizable": true,
"fullscreen": false,
"decorations": false,
"transparent": true,
"backgroundColor": [
0,
0,
0,
0
],
"shadow": true,
"visible": false,
"alwaysOnTop": true,
"skipTaskbar": true,
"center": true
}
],
"security": {
+51 -3
View File
@@ -102,6 +102,30 @@ fn management_window_owns_the_expected_narrow_surfaces() {
"Experimental",
"open_management_from_grant",
"Open in UI",
"computer_control_engine",
"Computer control",
"CUA Driver",
"Not installed",
"Incompatible",
"Degraded",
"Ready",
"Animated agent cursor",
"Background only",
"Preferred structured engine",
"Compatibility",
"Active session",
"Event timeline",
"Post-action snapshot captured",
"Authenticated control session",
"computer_cua_status",
"computer_cua_health",
"computer_cua_install",
"computer_cua_check_update",
"computer_cua_update",
"verified compatible driver",
"set_computer_control_engine",
"set_cua_cursor_enabled",
"visual overlays—not additional Windows hardware pointers",
] {
assert!(
source.contains(required) || security.contains(required),
@@ -128,7 +152,9 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
.and_then(|windows| windows.iter().find(|window| window["label"] == "grant"))
.expect("grant window configuration");
assert!(ui.contains("isGrantWindow ? <GrantWindow /> : <ManagementApp />"));
assert!(ui.contains("isGrantWindow ? <GrantWindow />"));
assert!(ui.contains("isNoticeWindow ? <ConnectionNoticeWindow />"));
assert!(ui.contains("isEvidenceWindow ? <EvidenceWindow />"));
assert!(ui.contains("present_grant_window"));
assert!(ui.contains("Remote access request"));
assert!(native.contains("start_grant_watcher"));
@@ -140,7 +166,8 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
serde_json::json!([0, 0, 0, 0])
);
assert_eq!(grant_window["shadow"], false);
assert!(ui.contains("(snapshot.daemon.configured_url ?? snapshot.daemon.url) === host.url"));
assert!(ui.contains("const daemonTargetsHost"));
assert!(ui.contains("(snapshot?.daemon.configured_url ?? snapshot?.daemon.url) === host.url"));
assert!(ui.contains("formatGrantScope(grant.scope)"));
assert!(ui.contains("grantAction(grant.scope)"));
assert!(ui.contains("Requested action"));
@@ -149,6 +176,10 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
assert!(native.contains("management.completed"));
assert!(native.contains("append_management_event"));
assert!(native.contains("fn clear_activity"));
assert!(native.contains("fn get_activity_screenshot"));
assert!(native.contains("fn set_activity_screenshot_retention"));
assert!(native.contains("fn start_connection_watcher"));
assert!(native.contains("fn present_connection_notice"));
assert!(native.contains("skip_serializing_if = \"Option::is_none\""));
assert!(native.contains("popup_position"));
assert!(native.contains("window.outer_size()"));
@@ -160,6 +191,19 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
assert!(native.contains("spawn_blocking(build_snapshot)"));
assert!(native.contains("async fn check_desktop_update"));
assert!(native.contains("async fn install_desktop_update"));
for command in [
"async fn computer_cua_status",
"async fn computer_cua_health",
"async fn computer_cua_install",
"async fn computer_cua_check_update",
"async fn computer_cua_update",
] {
assert!(
native.contains(command),
"blocking CUA management command: {command}"
);
}
assert!(native.matches("spawn_blocking(||").count() >= 6);
assert!(native.contains("fn set_host_capability"));
assert!(native.contains("fn hardware_availability"));
assert!(native.contains("HERMES_RELAY_CODE"));
@@ -214,7 +258,7 @@ fn management_window_keeps_the_reviewed_compact_geometry() {
assert!(config.contains("\"minWidth\": 340"));
assert!(config.contains("\"minHeight\": 460"));
assert!(config.contains("\"resizable\": false"));
assert_eq!(config.matches("\"alwaysOnTop\": true").count(), 2);
assert_eq!(config.matches("\"alwaysOnTop\": true").count(), 4);
assert!(!ui.contains("toggleMaximize"));
assert!(!ui.contains("data-tauri-drag-region"));
assert!(!capability.contains("allow-start-dragging"));
@@ -227,6 +271,10 @@ fn management_window_keeps_the_reviewed_compact_geometry() {
assert!(ui.contains("packet packet-outbound"));
assert!(ui.contains("packet packet-inbound"));
assert!(ui.contains("connectionTransition"));
assert!(ui.contains("daemon.reconnect_attempt"));
assert!(ui.contains("Retry now"));
assert!(ui.contains("Screenshot evidence"));
assert!(ui.contains("View larger"));
assert!(ui.contains("refreshInFlight"));
assert!(ui.contains("Starting daemon and opening relay tunnel"));
assert!(ui.contains("Stopping the local daemon"));
+207 -23
View File
@@ -6,10 +6,10 @@ import {
CircleHelp, Clock3, Download, ExternalLink, Eye, FileText, FolderOpen, Home, Info, Laptop, Link2,
Copy, LoaderCircle, LogOut, Monitor, MousePointer2, Power, Radio, RefreshCw, Server,
Settings, ShieldCheck, TerminalSquare, Trash2, Unplug, UserRoundX, X, Usb,
LockKeyhole, SlidersHorizontal, Mic, Video
LockKeyhole, SlidersHorizontal, Mic, Video, MousePointerClick, Maximize2, RotateCcw
} from 'lucide-react'
import logo from '../icons/icon-256.png'
import type { AccessMode, Activity, AuthorizedClient, Capability, CapabilityMode, Host, PendingGrantRequest, Snapshot, UpdateReport } from './types'
import type { AccessMode, Activity, AuthorizedClient, Capability, CapabilityMode, CuaHealthStatus, CuaManagementStatus, Host, PendingGrantRequest, Snapshot, UpdateReport } from './types'
import { describeTransportSecurity } from '../../src/transportSecurity'
import { displayLabel as displayRouteLabel, inferEndpointRole } from '../../src/endpoint'
@@ -18,7 +18,10 @@ type PendingAction = { type: 'access'; mode: AccessMode } | { type: 'capability'
type RouteTestResult = { label: string; url: string; reachable: boolean; elapsed_ms: number; encrypted: boolean; security: string; error?: string | null }
type RouteTestReport = { best?: RouteTestResult | null; routes?: RouteTestResult[] }
const isGrantWindow = '__TAURI_INTERNALS__' in window && getCurrentWindow().label === 'grant'
const windowLabel = '__TAURI_INTERNALS__' in window ? getCurrentWindow().label : 'main'
const isGrantWindow = windowLabel === 'grant'
const isNoticeWindow = windowLabel === 'notice'
const isEvidenceWindow = windowLabel === 'evidence'
const demo: Snapshot = {
hosts: [{ url: 'wss://home-hermes.local:8767', name: 'Docker-Server', server_version: '1.6.3', endpoint_role: 'tailscale', paired_at: 1786458000, is_active: true, access_mode: 'full-access', capabilities: { commands: 'allow', files: 'allow', screen_input: 'allow', usb: 'allow', microphone: 'allow', camera: 'allow' } }],
@@ -30,6 +33,11 @@ const demo: Snapshot = {
cli_version: '0.4.0-alpha.4',
cli_path: 'C:\\Program Files\\Hermes-Relay CLI\\hermes-relay.exe',
hardware_availability: { usb: true, adb: true, microphone: false, camera: false },
activity_screenshot_retention: { enabled: true, days: 7, count: 2, bytes: 842_000 },
computer_control_engine: {
selected: 'legacy', effective: 'legacy', available: false, state: 'not_installed',
foreground_escalation_enabled: false, message: 'CUA Driver is not installed. Legacy input remains active.'
},
pending_grants: [],
activity: [
{ ts: Date.now() - 110_000, tool: 'desktop_powershell', ok: true, summary: 'exit 0', request_detail: '{\n "script": "Get-Process | Select-Object -First 5"\n}', stdout: 'Handles NPM(K) PM(K) WS(K) CPU(s) Id ProcessName\n------- ------ ----- ----- ------ -- -----------\n 412 31 74248 98312 4.18 812 powershell' },
@@ -48,6 +56,8 @@ async function call<T>(command: string, args?: Record<string, unknown>): Promise
if (command === 'check_desktop_update') return { current: '0.4.0-alpha.3', up_to_date: true, ahead_of_latest: true, latest_version: '0.4.0-alpha.2', installed: false, needs_restart: false } as T
if (command === 'install_desktop_update') return { current: '0.4.0-alpha.3', up_to_date: true, ahead_of_latest: false, installed: true, needs_restart: true } as T
if (command === 'test_host_route') return { best: { label: 'LAN', url: 'ws://172.16.24.250:8767', reachable: true, elapsed_ms: 36, encrypted: false, security: 'Unencrypted relay connection' }, routes: [] } as T
if (command === 'computer_cua_status') return { installed: false, stale_path_shim: false, compatible: false, compatibility_reason: 'CUA Driver is not installed', supported_range: { minimum: '0.19.3', maximum_exclusive: '0.20.0' } } as T
if (command === 'computer_cua_health') return { state: 'degraded', checkedAt: new Date().toISOString(), overall: 'degraded', reason: 'UI Automation desktop enumeration exceeded 2000ms.', temporaryWindowsCompatibility: true } as T
return undefined as T
}
return invoke<T>(command, args)
@@ -116,11 +126,56 @@ function activityName(tool: string): string {
'host.access': 'Changed host access', 'host.pair': 'Pair host',
'client.revoke': 'Deauthorized client', 'grant.resolve': 'Resolved access request',
'daemon.start': 'Connected daemon', 'daemon.stop': 'Disconnected daemon',
'daemon.reconnecting': 'Connection interrupted', 'daemon.reconnected': 'Tunnel restored',
'daemon.disconnected': 'Tunnel disconnected', 'daemon.auth_failed': 'Connection failed',
'daemon.restart': 'Restarted daemon', 'startup.change': 'Changed startup setting'
}
return names[tool] ?? tool.replace(/^desktop[._]/, '').replaceAll('_', ' ').replaceAll('.', ' ').replace(/\b\w/g, value => value.toUpperCase())
}
function isComputerControl(entry: Activity): boolean {
return entry.backend === 'cua' || entry.backend === 'legacy_compat' || entry.tool.startsWith('desktop_computer_')
}
function controlActionLabel(entry: Activity): string {
return (entry.action ?? activityName(entry.tool)).replaceAll('_', ' ').replace(/\b\w/g, value => value.toUpperCase())
}
function controlVerificationLabel(value?: string): string {
return value === 'snapshot_captured' ? 'Post-action snapshot captured'
: value === 'failed' ? 'Verification failed'
: value ? value.replaceAll('_', ' ') : 'Not reported'
}
function formatBytes(bytes: number): string {
if (bytes < 1024) return `${bytes} B`
if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB`
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
}
type ActivityStep = { title: string; detail: string; state: 'done' | 'failed' | 'pending' }
function activitySteps(entry: Activity): ActivityStep[] {
if (isComputerControl(entry)) return [
{ title: 'Authorized session', detail: entry.control_session_id ? 'Authenticated control session' : 'Authenticated by Hermes', state: 'done' },
{ title: controlActionLabel(entry), detail: `${entry.backend === 'cua' ? 'CUA structured engine' : 'Windows input · Compatibility'} · ${entry.dispatch ?? 'background'}`, state: entry.ok ? 'done' : 'failed' },
{ title: 'Verification', detail: controlVerificationLabel(entry.verification), state: entry.verification === 'failed' ? 'failed' : entry.verification ? 'done' : 'pending' }
]
const category = activityCategory(entry)
if (category === 'system') {
const reconnecting = entry.tool === 'daemon.reconnecting'
return [
{ title: 'Connection state changed', detail: entry.summary ?? activityName(entry.tool), state: entry.ok ? 'done' : 'failed' },
...(reconnecting ? [{ title: 'Automatic retry', detail: 'Relay transport is retrying with backoff', state: 'pending' as const }] : [])
]
}
return [
{ title: 'Request received', detail: entry.args_preview ?? 'Validated local request', state: 'done' },
{ title: activityName(entry.tool), detail: entry.aborted ? 'Stopped before completion' : entry.error ?? entry.summary ?? 'Local execution', state: entry.ok ? 'done' : 'failed' },
{ title: 'Result recorded', detail: entry.ok ? (isNonZeroExit(entry) ? `Process exited ${activityExitCode(entry)}` : 'Evidence saved to local activity') : 'Failure details recorded', state: entry.ok ? 'done' : 'failed' }
]
}
function age(ts?: number): string {
if (!ts) return 'Not seen yet'
const seconds = Math.max(0, Math.floor(Date.now() / 1000) - ts)
@@ -196,7 +251,54 @@ function hostAccessLabel(host: Host): string {
}
export default function App() {
return isGrantWindow ? <GrantWindow /> : <ManagementApp />
return isGrantWindow ? <GrantWindow /> : isNoticeWindow ? <ConnectionNoticeWindow /> : isEvidenceWindow ? <EvidenceWindow /> : <ManagementApp />
}
type ConnectionNotice = { tone: 'connected' | 'warning' | 'offline'; title: string; detail: string }
function ConnectionNoticeWindow() {
const [notice, setNotice] = useState<ConnectionNotice | null>(null)
const hideTimer = useRef<number | null>(null)
useEffect(() => {
const receive = (event: Event) => {
const detail = (event as CustomEvent<ConnectionNotice>).detail
setNotice(detail)
if (hideTimer.current) window.clearTimeout(hideTimer.current)
hideTimer.current = window.setTimeout(() => void getCurrentWindow().hide(), detail.tone === 'warning' ? 6500 : 4200)
}
window.addEventListener('hermes-connection-notice', receive)
return () => { window.removeEventListener('hermes-connection-notice', receive); if (hideTimer.current) window.clearTimeout(hideTimer.current) }
}, [])
if (!notice) return null
return <div className={`connection-notice-shell ${notice.tone}`}>
<section className="connection-notice-card" role="status" aria-live="polite">
<span className="connection-notice-icon">{notice.tone === 'connected' ? <Check /> : notice.tone === 'warning' ? <RotateCcw /> : <Unplug />}</span>
<span><small>Hermes-Relay tunnel</small><strong>{notice.title}</strong><p>{notice.detail}</p></span>
<button className="connection-notice-open" onClick={() => call('open_management_from_notice')}>Open</button>
<button className="connection-notice-close" aria-label="Dismiss" onClick={() => getCurrentWindow().hide()}><X /></button>
</section>
</div>
}
function EvidenceWindow() {
const [evidenceId, setEvidenceId] = useState<string | null>(null)
const [source, setSource] = useState<string | null>(null)
const [error, setError] = useState<string | null>(null)
useEffect(() => {
const receive = (event: Event) => {
const id = (event as CustomEvent<{ evidenceId: string }>).detail.evidenceId
setEvidenceId(id); setSource(null); setError(null)
void call<string>('get_activity_screenshot', { evidenceId: id }).then(setSource).catch(value => setError(String(value)))
}
const close = (event: KeyboardEvent) => { if (event.key === 'Escape') void getCurrentWindow().hide() }
window.addEventListener('hermes-screenshot-evidence', receive)
window.addEventListener('keydown', close)
return () => { window.removeEventListener('hermes-screenshot-evidence', receive); window.removeEventListener('keydown', close) }
}, [])
return <div className="evidence-shell">
<header><span><Eye /><strong>Screenshot evidence</strong><small>Stored locally with this activity event</small></span><button aria-label="Close screenshot" onClick={() => getCurrentWindow().hide()}><X /></button></header>
<main>{source ? <img src={source} alt="Retained desktop screenshot" /> : error ? <div className="evidence-error"><AlertTriangle /><strong>Screenshot unavailable</strong><small>{error}</small></div> : <div className="evidence-loading"><LoaderCircle className="spin" /><span>{evidenceId ? 'Loading screenshot…' : 'Preparing viewer…'}</span></div>}</main>
</div>
}
function ManagementApp() {
@@ -251,11 +353,13 @@ function ManagementApp() {
finally { refreshInFlight.current = false }
}, [])
const daemonRetrying = Boolean(snapshot?.daemon.running && snapshot.daemon.state === 'reconnecting')
useEffect(() => {
refresh()
const timer = window.setInterval(refresh, connectionTransition ? 350 : 5000)
const timer = window.setInterval(refresh, connectionTransition ? 350 : daemonRetrying ? 1000 : 5000)
return () => window.clearInterval(timer)
}, [refresh, connectionTransition])
}, [refresh, connectionTransition, daemonRetrying])
useEffect(() => {
const close = (event: MouseEvent) => {
@@ -266,7 +370,11 @@ function ManagementApp() {
}, [])
const host = useMemo(() => snapshot?.hosts.find(item => item.url === selectedUrl) ?? null, [snapshot, selectedUrl])
const connected = Boolean(snapshot?.daemon.running && snapshot.daemon.state === 'connected' && host && (snapshot.daemon.configured_url ?? snapshot.daemon.url) === host.url)
const daemonTargetsHost = Boolean(host && (snapshot?.daemon.configured_url ?? snapshot?.daemon.url) === host.url)
const daemonActive = Boolean(snapshot?.daemon.running && daemonTargetsHost)
const connected = Boolean(daemonActive && snapshot?.daemon.state === 'connected')
const reconnecting = Boolean(daemonActive && snapshot?.daemon.state === 'reconnecting')
const retrySeconds = reconnecting && snapshot?.daemon.retry_at ? Math.max(0, snapshot.daemon.retry_at - Math.floor(Date.now() / 1000)) : null
useEffect(() => {
if (page !== 'host-detail' || !detailUrl) return
@@ -287,8 +395,8 @@ function ManagementApp() {
async function changeConnection() {
if (busy) return
const command = connected ? 'disconnect_daemon' : 'connect_daemon'
const transition = connected ? 'disconnecting' : 'connecting'
const command = daemonActive ? 'disconnect_daemon' : 'connect_daemon'
const transition = daemonActive ? 'disconnecting' : 'connecting'
setBusy(command)
setConnectionTransition(transition)
setError(null)
@@ -303,6 +411,16 @@ function ManagementApp() {
}
}
async function retryConnection() {
if (busy) return
setBusy('restart_daemon')
setConnectionTransition('connecting')
setError(null)
try { await call('restart_daemon'); await refresh() }
catch (e) { setError(String(e)) }
finally { setConnectionTransition(null); setBusy(null) }
}
async function testRoute(remote: string) {
setBusy('test_host_route')
setRouteTest(null)
@@ -418,7 +536,7 @@ function ManagementApp() {
<main className="content" ref={contentRef}>
{page === 'overview' && <>
<section className={`connection-route ${connected ? 'online' : 'offline'} ${connectionTransition ?? ''} ${snapshot.daemon.active_route === 'plugin_proxy' ? 'secure-link' : ''}`} aria-busy={connectionTransition !== null}>
<section className={`connection-route ${connected ? 'online' : reconnecting ? 'retrying' : 'offline'} ${connectionTransition ?? ''} ${snapshot.daemon.active_route === 'plugin_proxy' ? 'secure-link' : ''}`} aria-busy={connectionTransition !== null || reconnecting}>
{snapshot.hosts.length === 0 ?
<button className="empty-pair" onClick={() => openPair()}><Link2 /><span><strong>Pair host</strong><small>Connect this PC to a Hermes instance</small></span><ChevronRight /></button> :
<div className="route-grid" ref={selectorRef}>
@@ -445,10 +563,11 @@ function ManagementApp() {
const activeRole = snapshot.daemon.active_route ?? host?.endpoint_role ?? inferEndpointRole(snapshot.daemon.url ?? host?.url ?? '')
const security = describeTransportSecurity(snapshot.daemon.url ?? host?.url ?? '', activeRole)
return <div className={`route-status ${connected && !security.encrypted ? 'insecure' : ''}`} aria-live="polite" aria-atomic="true">
<strong>{connectionTransition === 'connecting' ? 'Connecting' : connectionTransition === 'disconnecting' ? 'Disconnecting' : connected ? 'Connected' : 'Disconnected'}</strong>
<strong>{connectionTransition === 'connecting' ? 'Connecting' : connectionTransition === 'disconnecting' ? 'Disconnecting' : reconnecting ? 'Reconnecting' : connected ? 'Connected' : 'Disconnected'}</strong>
{connected && <button className={`route-badge ${security.kind}`} aria-expanded={routeDetailsOpen} onClick={() => setRouteDetailsOpen(open => !open)}><ShieldCheck />{displayRouteLabel(activeRole ?? 'custom')}<ChevronDown /></button>}
{connectionTransition && <small>{connectionTransition === 'connecting' ? 'Starting daemon and opening relay tunnel' : 'Closing relay tunnel'}</small>}
{!connected && !connectionTransition && <small>Relay connection offline</small>}
{reconnecting && !connectionTransition && <><small>Attempt {snapshot.daemon.reconnect_attempt ?? 1}{retrySeconds !== null ? ` · retry in ${retrySeconds}s` : ' · retry scheduled'}</small><button className="retry-now" disabled={busy !== null} onClick={() => void retryConnection()}><RefreshCw /> Retry now</button></>}
{!connected && !reconnecting && !connectionTransition && <small>{snapshot.daemon.last_error ?? 'Relay connection offline'}</small>}
{connected && routeDetailsOpen && <aside className="route-detail-card"><div><ShieldCheck /><span><strong>{displayRouteLabel(activeRole ?? 'custom')}</strong><small>{security.detail}</small></span></div><dl><div><dt>Security</dt><dd>{security.label}</dd></div><div><dt>Endpoint</dt><dd title={snapshot.daemon.url ?? undefined}>{snapshot.daemon.url ?? 'Not reported'}</dd></div></dl><button className="route-test-button" disabled={busy === 'test_host_route'} onClick={() => host && testRoute(host.url)}>{busy === 'test_host_route' ? <LoaderCircle className="spin" /> : routeTest ? <RefreshCw /> : <ActivityIcon />}<span>{busy === 'test_host_route' ? <><strong>Testing connection…</strong><small>Checking every saved route</small></> : <><strong>{routeTest ? 'Test again' : 'Test connection'}</strong><small>Measure reachability and latency</small></>}</span></button>{routeTest && <div className={`route-test-result ${routeTest.best ? 'reachable' : 'unreachable'}`} aria-live="polite">{routeTest.best ? <><div className="route-test-summary"><span><Check /></span><strong>{routeTest.best.label} reachable</strong><em>{routeTest.best.elapsed_ms} ms</em></div><dl><div><dt>Protection</dt><dd className={routeTest.best.encrypted ? 'secure' : 'warning'}>{routeTest.best.security}</dd></div><div><dt>Tested endpoint</dt><dd title={routeTest.best.url}>{routeTest.best.url}</dd></div></dl><small>{Math.max(1, routeTest.routes?.length ?? 0)} saved route{(routeTest.routes?.length ?? 0) === 1 ? '' : 's'} checked</small></> : <div className="route-test-summary"><span><X /></span><strong>No route reachable</strong><em>Check host</em></div>}</div>}</aside>}
</div>
})()}
@@ -470,9 +589,9 @@ function ManagementApp() {
</button>
</section>}
<button className={`tunnel-button ${connectionTransition ? 'pending' : ''}`} disabled={busy !== null} aria-busy={connectionTransition !== null} onClick={() => void changeConnection()}>
{connectionTransition ? <LoaderCircle className="spin" /> : <Power />}
<span><strong>{connectionTransition === 'connecting' ? 'Connecting…' : connectionTransition === 'disconnecting' ? 'Disconnecting…' : connected ? 'Disconnect Tunnel' : 'Connect Tunnel'}</strong>{connectionTransition && <small>{connectionTransition === 'connecting' ? 'Waiting for the relay' : 'Stopping the local daemon'}</small>}</span>
<button className={`tunnel-button ${connectionTransition || reconnecting ? 'pending' : ''}`} disabled={busy !== null} aria-busy={connectionTransition !== null} onClick={() => void changeConnection()}>
{connectionTransition ? <LoaderCircle className="spin" /> : reconnecting ? <Unplug /> : <Power />}
<span><strong>{connectionTransition === 'connecting' ? 'Connecting…' : connectionTransition === 'disconnecting' ? 'Disconnecting…' : reconnecting ? 'Disconnect Tunnel' : connected ? 'Disconnect Tunnel' : 'Connect Tunnel'}</strong>{(connectionTransition || reconnecting) && <small>{connectionTransition === 'connecting' ? 'Waiting for the relay' : connectionTransition === 'disconnecting' ? 'Stopping the local daemon' : 'Automatic retry remains active'}</small>}</span>
</button>
<section className="activity-section">
@@ -489,7 +608,7 @@ function ManagementApp() {
{page === 'hosts' && <HostsPage hosts={snapshot.hosts} selected={host} onOpen={url => { setDetailUrl(url); setSelectedUrl(url); setPage('host-detail') }} onPair={() => openPair()} />}
{page === 'pair-host' && <PairHostPage initialUrl={pairInitialUrl} busy={busy === 'pair_host'} onBack={() => setPage('hosts')} onPair={pairHost} />}
{page === 'host-detail' && <HostDetailPage host={snapshot.hosts.find(item => item.url === detailUrl) ?? null} clients={clients} busy={busy !== null} onBack={() => setPage('hosts')} onConnect={connectHost} onRename={(remote, name) => action('rename_host', { remote, name })} onAccess={() => { setPolicyBack('host-detail'); setPage('access') }} onCapabilities={() => { setPolicyBack('host-detail'); setPage('capabilities') }} onRevoke={(remote, client) => setPending({ type: 'revoke', client, remote })} onRepair={host => setPending({ type: 'repair', host })} onForget={host => setPending({ type: 'forget', host })} />}
{page === 'settings' && <SettingsPage daemon={snapshot.daemon} startup={snapshot.startup_enabled} daemonAutostart={snapshot.daemon_autostart_enabled ?? false} activity={snapshot.activity} onAction={action} onStartup={value => action('set_startup', { enabled: value })} onDaemonAutostart={value => action('set_daemon_autostart', { enabled: value })} onHelp={() => setPage('help')} onViewActivity={() => { setActivityBack('settings'); setPage('activity') }} onOpenActivity={entry => { setSelectedActivity(entry); setActivityDetailBack('settings'); setPage('activity-detail') }} />}
{page === 'settings' && <SettingsPage daemon={snapshot.daemon} computerControl={snapshot.computer_control_engine ?? null} startup={snapshot.startup_enabled} daemonAutostart={snapshot.daemon_autostart_enabled ?? false} activity={snapshot.activity} screenshotRetention={snapshot.activity_screenshot_retention} onAction={action} onStartup={value => action('set_startup', { enabled: value })} onDaemonAutostart={value => action('set_daemon_autostart', { enabled: value })} onHelp={() => setPage('help')} onViewActivity={() => { setActivityBack('settings'); setPage('activity') }} onOpenActivity={entry => { setSelectedActivity(entry); setActivityDetailBack('settings'); setPage('activity-detail') }} />}
{page === 'help' && <HelpPage snapshot={snapshot} host={host} onBack={() => { setSelectedUrl(snapshot.active_url ?? null); setPage('settings') }} onAction={action} />}
{page === 'activity' && <ActivityPage entries={snapshot.activity} host={host} onBack={() => setPage(activityBack)} onClear={() => setPending({ type: 'clear-activity' })} onOpen={entry => { setSelectedActivity(entry); setActivityDetailBack('activity'); setPage('activity-detail') }} />}
{page === 'activity-detail' && <ActivityDetailPage entry={selectedActivity} host={host} onBack={() => setPage(activityDetailBack)} />}
@@ -623,12 +742,14 @@ function ActivityList({ entries, host, onOpen }: { entries: Activity[]; host: Ho
const warning = isNonZeroExit(entry)
const key = entry.request_id ?? `${entry.ts}-${i}`
const Icon = category === 'command' ? TerminalSquare : category === 'files' ? FileText : category === 'screen' ? Eye : category === 'input' ? MousePointer2 : category === 'devices' ? Usb : category === 'system' ? LogOut : ActivityIcon
const detail = entry.error ?? entry.summary ?? (entry.aborted ? 'Request aborted' : 'Completed')
const computerControl = isComputerControl(entry)
const target = entry.target_app ?? entry.target_title
const detail = entry.error ?? (computerControl ? `${entry.backend === 'cua' ? 'CUA' : 'Compatibility'} · ${entry.dispatch ?? 'background'}${target ? ` · ${target}` : ''}` : entry.summary) ?? (entry.aborted ? 'Request aborted' : 'Completed')
const eventHost = entry.host_url ? displayHost(entry.host_url) : host?.name ?? 'Local daemon'
return <article className="activity-item" key={key}>
<button className="activity-row" disabled={!onOpen} onClick={() => onOpen?.(entry)}>
<span className={`activity-icon ${attention || warning ? 'amber' : category === 'system' ? 'green' : 'violet'}`}><Icon /></span>
<span className="activity-copy"><strong>{activityName(entry.tool)}</strong><small className={attention ? 'attention' : warning ? 'warning' : ''}>{detail} · {eventHost}</small></span>
<span className="activity-copy"><strong>{computerControl ? controlActionLabel(entry) : activityName(entry.tool)}</strong><small className={attention ? 'attention' : warning ? 'warning' : ''}>{detail} · {eventHost}</small></span>
<span className="activity-tail"><time>{formatTime(entry.ts)}</time>{onOpen && <ChevronRight />}</span>
</button>
</article>
@@ -663,22 +784,29 @@ function ActivityPage({ entries, host, onBack, onClear, onOpen }: { entries: Act
}
function ActivityDetailPage({ entry, host, onBack }: { entry: Activity | null; host: Host | null; onBack: () => void }) {
const [evidenceError, setEvidenceError] = useState<string | null>(null)
if (!entry) return <section className="page-panel"><button className="back-button" onClick={onBack}><ArrowLeft /> Back to Activity</button><div className="large-empty"><ActivityIcon /><h2>Event unavailable</h2></div></section>
const attention = needsAttention(entry)
const warning = isNonZeroExit(entry)
const status = entry.aborted ? 'Aborted' : !entry.ok ? 'Failed' : warning ? `Exit ${activityExitCode(entry)}` : 'Completed'
const eventHost = entry.host_url ? displayHost(entry.host_url) : host?.name ?? 'Local daemon'
const computerControl = isComputerControl(entry)
const steps = activitySteps(entry)
const blocks = [
['Request', entry.request_detail ?? entry.args_preview, entry.request_truncated],
['Standard output', entry.stdout, entry.stdout_truncated],
['Standard error', entry.stderr, entry.stderr_truncated],
['Result', entry.result_detail, entry.result_truncated],
['Error', entry.error, false]
['Result', entry.result_detail, entry.result_truncated]
] as const
return <section className="page-panel activity-detail-page">
<button className="back-button" onClick={onBack}><ArrowLeft /> Back to Activity</button>
<div className="activity-detail-title"><span className={`activity-icon ${attention || warning ? 'amber' : 'violet'}`}><TerminalSquare /></span><span><p>{activityCategory(entry)}</p><h1>{activityName(entry.tool)}</h1><small>{eventHost}</small></span></div>
<div className="activity-detail-title"><span className={`activity-icon ${attention || warning ? 'amber' : 'violet'}`}>{computerControl ? <MousePointerClick /> : <TerminalSquare />}</span><span><p>{computerControl ? 'Computer control' : activityCategory(entry)}</p><h1>{computerControl ? controlActionLabel(entry) : activityName(entry.tool)}</h1><small>{eventHost}</small></span></div>
<dl className="activity-detail-meta"><div><dt>Status</dt><dd className={attention ? 'attention' : warning ? 'warning' : 'success'}>{status}</dd></div><div><dt>When</dt><dd>{formatDateTime(entry.ts)}</dd></div><div><dt>Duration</dt><dd>{formatDuration(entry.duration_ms)}</dd></div></dl>
<section className="control-timeline" aria-label="Event timeline">{steps.map((step, index) => <div className={step.state} key={`${step.title}-${index}`}><i /><span><strong>{step.title}</strong><small>{step.detail}</small></span></div>)}</section>
{computerControl && (entry.target_app || entry.target_title || entry.target_pid || entry.target_window_id) && <dl className="control-target"><div><dt>Application</dt><dd>{entry.target_app ?? 'Not reported'}</dd></div><div><dt>Window</dt><dd title={entry.target_title}>{entry.target_title ?? 'Not reported'}</dd></div><div><dt>Target</dt><dd>{entry.target_pid ? `PID ${entry.target_pid}` : 'PID —'} · {entry.target_window_id ? `Window ${entry.target_window_id}` : 'Window —'}</dd></div></dl>}
{entry.error && <aside className="activity-error-callout" role="alert"><AlertTriangle /><span><strong>{entry.aborted ? 'Action stopped' : 'Action failed'}</strong><small>{entry.error}</small></span></aside>}
{entry.screenshot_evidence_id && <button className="screenshot-evidence-card" onClick={() => { setEvidenceError(null); void call('present_activity_screenshot', { evidenceId: entry.screenshot_evidence_id }).catch(error => setEvidenceError(String(error))) }}><span><Eye /><strong>Screenshot captured</strong><small>{entry.screenshot_width && entry.screenshot_height ? `${entry.screenshot_width} × ${entry.screenshot_height} PNG` : 'Retained local evidence'}</small></span><em><Maximize2 /> View larger</em></button>}
{evidenceError && <aside className="activity-error-callout"><AlertTriangle /><span><strong>Screenshot unavailable</strong><small>{evidenceError}</small></span></aside>}
<div className="activity-output-list">{blocks.filter(([, value]) => value).map(([label, value, truncated]) => <section key={label}><header><strong>{label}</strong>{truncated && <em>Truncated</em>}</header><pre>{value}</pre></section>)}</div>
{entry.request_id && <div className="activity-request-id">Request ID {entry.request_id}</div>}
</section>
@@ -783,10 +911,32 @@ function HostDetailPage({ host, clients, busy, onBack, onConnect, onRename, onAc
</section>
}
function SettingsPage({ daemon, startup, daemonAutostart, activity, onAction, onStartup, onDaemonAutostart, onHelp, onViewActivity, onOpenActivity }: { daemon: Snapshot['daemon']; startup: boolean; daemonAutostart: boolean; activity: Activity[]; onAction: (name: string, args?: Record<string, unknown>) => Promise<unknown>; onStartup: (value: boolean) => void; onDaemonAutostart: (value: boolean) => void; onHelp: () => void; onViewActivity: () => void; onOpenActivity: (entry: Activity) => void }) {
function SettingsPage({ daemon, computerControl, startup, daemonAutostart, activity, screenshotRetention, onAction, onStartup, onDaemonAutostart, onHelp, onViewActivity, onOpenActivity }: { daemon: Snapshot['daemon']; computerControl: Snapshot['computer_control_engine']; startup: boolean; daemonAutostart: boolean; activity: Activity[]; screenshotRetention: Snapshot['activity_screenshot_retention']; onAction: (name: string, args?: Record<string, unknown>) => Promise<unknown>; onStartup: (value: boolean) => void; onDaemonAutostart: (value: boolean) => void; onHelp: () => void; onViewActivity: () => void; onOpenActivity: (entry: Activity) => void }) {
const [update, setUpdate] = useState<UpdateReport | null>(null)
const [updateBusy, setUpdateBusy] = useState<'check' | 'install' | null>(null)
const [updateError, setUpdateError] = useState<string | null>(null)
const [cuaManagement, setCuaManagement] = useState<CuaManagementStatus | null>(null)
const [cuaHealth, setCuaHealth] = useState<CuaHealthStatus | null>(null)
const [cuaBusy, setCuaBusy] = useState<'status' | 'health' | 'install' | 'check' | 'update' | null>(null)
const [cuaError, setCuaError] = useState<string | null>(null)
const cuaOperation = useCallback(async (operation: 'status' | 'install' | 'check' | 'update') => {
setCuaBusy(operation)
try {
setCuaManagement(await call<CuaManagementStatus>(operation === 'status' ? 'computer_cua_status' : operation === 'install' ? 'computer_cua_install' : operation === 'check' ? 'computer_cua_check_update' : 'computer_cua_update'))
setCuaError(null)
} catch (error) { setCuaError(String(error).replace(/^Error:\s*/i, '')) }
finally { setCuaBusy(null) }
}, [])
const recheckCuaHealth = useCallback(async () => {
setCuaBusy('health')
try {
setCuaHealth(await call<CuaHealthStatus>('computer_cua_health'))
setCuaError(null)
} catch (error) { setCuaError(String(error).replace(/^Error:\s*/i, '')) }
finally { setCuaBusy(null) }
}, [])
const checkUpdate = useCallback(async () => {
setUpdateBusy('check')
@@ -803,6 +953,7 @@ function SettingsPage({ daemon, startup, daemonAutostart, activity, onAction, on
}, [])
useEffect(() => { void checkUpdate() }, [checkUpdate])
useEffect(() => { void cuaOperation('status') }, [cuaOperation])
const updateSummary = updateError
? updateError
@@ -816,12 +967,44 @@ function SettingsPage({ daemon, startup, daemonAutostart, activity, onAction, on
? `${update.current} → ${update.latest_version} available`
: 'Check the desktop release channel.'
const cuaReady = computerControl?.available === true && computerControl.state === 'ready'
const engineState = computerControl?.state ?? 'not_installed'
const engineLabel = engineState === 'not_installed' ? 'Not installed' : engineState === 'incompatible' ? 'Incompatible' : engineState === 'degraded' ? 'Degraded' : engineState === 'ready' ? 'Ready' : 'Unavailable'
const engineDetail = computerControl?.message
?? (engineState === 'not_installed' ? 'CUA Driver is not installed. Windows input remains available for compatibility.'
: engineState === 'incompatible' ? 'The installed CUA Driver version is not compatible with this CLI.'
: engineState === 'degraded' ? 'CUA Driver reported a health problem. Windows compatibility input remains active.'
: engineState === 'ready' ? `CUA Driver ${computerControl?.version ?? ''} is compatible and healthy.`.trim()
: 'CUA Driver status could not be verified. Hermes uses the safe fallback.')
const effectiveEngine = computerControl?.effective === 'cua' ? 'CUA Driver' : 'Windows input'
const engineRole = computerControl?.effective === 'cua' ? 'Preferred structured engine' : 'Compatibility'
const activeSessions = computerControl?.active_sessions ?? 0
const activeBackend = computerControl?.active_backend === 'cua' ? 'CUA active'
: computerControl?.active_backend === 'legacy_compat' ? 'Compatibility active'
: computerControl?.active_backend === 'mixed' ? 'Mixed backends' : 'Idle'
const healthLabel = cuaHealth?.state === 'healthy' ? 'Healthy' : cuaHealth?.state === 'degraded' ? 'Degraded' : cuaHealth?.state === 'error' ? 'Check failed' : 'Not checked'
const healthDetail = cuaHealth?.reason
?? (cuaHealth?.state === 'healthy'
? 'The latest explicit accessibility check passed.'
: 'Optional diagnostic; it does not disable the runtime while the temporary Windows workaround is active.')
return <section className="page-panel settings-page"><div className="page-title"><div><p>Local management</p><h1>Settings</h1></div></div>
<div className="settings-group"><h2>Relay daemon</h2><div className="settings-card"><div className="setting-row"><span><strong>Daemon status</strong><small>{daemon.running ? `${daemon.state} · ${daemon.privilege ?? 'user'}` : 'Stopped'}</small></span><button className="compact-button" onClick={() => onAction('restart_daemon')}><RefreshCw /> Restart</button></div><div className="setting-row"><span><strong>{daemon.privilege === 'administrator' ? 'Administrator mode' : 'User mode'}</strong><small>{daemon.privilege === 'administrator' ? 'Remote actions inherit elevated rights.' : 'Recommended for normal operation.'}</small></span><button className={`compact-button privilege-action ${daemon.privilege === 'administrator' ? '' : 'admin-action'}`} onClick={() => onAction(daemon.privilege === 'administrator' ? 'restart_daemon_as_user' : 'restart_daemon_as_administrator')}>{daemon.privilege === 'administrator' ? 'Return to user mode' : 'Restart as Administrator…'}</button></div><label className="setting-row toggle-row"><span><strong>Start UI at sign-in</strong><small>Launch the tray after you sign in.</small></span><input type="checkbox" checked={startup} onChange={e => onStartup(e.target.checked)} /><i /></label><label className="setting-row toggle-row"><span><strong>Start daemon with UI</strong><small>Connect remote access when the tray starts.</small></span><input type="checkbox" checked={daemonAutostart} onChange={e => onDaemonAutostart(e.target.checked)} /><i /></label></div></div>
<div className="settings-group"><h2>Computer control</h2><div className={`settings-card engine-card engine-${engineState}`}>
<div className="engine-status"><span className="setting-icon"><MousePointerClick /></span><span><strong>{effectiveEngine}</strong><small>{engineRole} · {engineDetail}</small></span><em>{engineLabel}</em></div>
{cuaReady && <div className="engine-options">
<div className="engine-choice"><span><strong>Control engine</strong><small>CUA is preferred; Windows input is the compatibility fallback.</small></span><div role="radiogroup" aria-label="Computer control engine"><button role="radio" aria-checked={computerControl?.selected === 'cua'} className={computerControl?.selected === 'cua' ? 'active' : ''} onClick={() => onAction('set_computer_control_engine', { engine: 'cua' })}>CUA</button><button role="radio" aria-checked={computerControl?.selected !== 'cua'} className={computerControl?.selected !== 'cua' ? 'active' : ''} onClick={() => onAction('set_computer_control_engine', { engine: 'legacy' })}>Compatibility</button></div></div>
<div className="engine-live"><span><strong>{activeSessions}</strong><small>Active session{activeSessions === 1 ? '' : 's'}</small></span><em className={activeSessions ? 'active' : ''}><i /> {activeBackend}</em></div>
<label className="setting-row toggle-row"><span><strong>Animated agent cursor</strong><small>Labeled · smooth glide · click pulse. It does not move your physical mouse.</small></span><input type="checkbox" disabled={computerControl?.selected !== 'cua'} checked={computerControl?.selected === 'cua' && computerControl.cursor_enabled === true} onChange={e => onAction('set_cua_cursor_enabled', { enabled: e.target.checked })} /><i /></label>
<div className="setting-row background-only"><span><strong>Window interaction</strong><small>CUA actions stay in the background and never bring an app forward.</small></span><em>Background only</em></div>
<div className="setting-row cua-health"><span><strong>Accessibility health</strong><small>{healthDetail}</small></span><div><em className={`health-${cuaHealth?.state ?? 'unchecked'}`}>{healthLabel}</em><button className="compact-button" disabled={cuaBusy !== null} onClick={() => void recheckCuaHealth()}>{cuaBusy === 'health' ? <LoaderCircle className="spin" /> : <RefreshCw />} Recheck</button></div></div>
</div>}
<div className="cua-maintenance"><span><strong>{cuaManagement?.installed ? `CUA Driver ${cuaManagement.current_version ?? ''}`.trim() : 'CUA Driver'}</strong><small>{cuaError ?? cuaManagement?.update?.error ?? cuaManagement?.compatibility_reason ?? (cuaManagement?.update?.update_available ? `${cuaManagement.update.latest_version} available` : cuaManagement?.installed ? 'Installed from the verified upstream release.' : 'Install the verified compatible driver explicitly.')}</small></span><div>{!cuaManagement?.installed ? <button disabled={cuaBusy !== null} onClick={() => void cuaOperation('install')}>{cuaBusy === 'install' ? <LoaderCircle className="spin" /> : <Download />} Install</button> : cuaManagement.update?.update_available && cuaManagement.update.compatible ? <button disabled={cuaBusy !== null} onClick={() => void cuaOperation('update')}>{cuaBusy === 'update' ? <LoaderCircle className="spin" /> : <Download />} Update</button> : <button disabled={cuaBusy !== null} onClick={() => void cuaOperation('check')}>{cuaBusy === 'check' || cuaBusy === 'status' ? <LoaderCircle className="spin" /> : <RefreshCw />} Check</button>}</div></div>
</div><p className="group-help engine-help"><ShieldCheck /> CUA is the preferred structured engine. Hermes permissions, grants, audit, and emergency stop remain in control.</p></div>
<div className="settings-group"><h2>CLI & diagnostics</h2><div className="settings-card quick-action-grid"><button onClick={() => onAction('open_terminal')}><TerminalSquare /><span>Open terminal</span></button><button onClick={() => onAction('open_cli_terminal')}><Bot /><span>Open Hermes CLI</span></button><button onClick={() => onAction('open_logs')}><FolderOpen /><span>View daemon log</span></button><button onClick={() => onAction('run_diagnostics')}><ActivityIcon /><span>Run diagnostics</span></button></div></div>
<div className="settings-group"><h2>Updates</h2><div className={`settings-card update-card ${updateError ? 'error' : update?.ahead_of_latest ? 'ahead' : update?.up_to_date ? 'current' : ''}`}><div className="setting-row update-row"><span><strong>Hermes-Relay CLI UI</strong><small>{updateSummary}</small></span>{update && !update.up_to_date && !update.ahead_of_latest && !update.installed ? <button className="compact-button update-button" disabled={updateBusy !== null} onClick={installUpdate}>{updateBusy === 'install' ? <LoaderCircle className="spin" /> : <Download />} Install</button> : <button className="compact-button" disabled={updateBusy !== null} onClick={checkUpdate}>{updateBusy === 'check' ? <LoaderCircle className="spin" /> : <RefreshCw />} Check</button>}</div></div><p className="group-help update-help">Updates the management UI and CLI together, then restarts the tray automatically.</p></div>
<button className="about-link" onClick={onHelp}><span className="setting-icon"><Info /></span><span><strong>Help & About</strong><small>Versions, documentation and troubleshooting.</small></span><ChevronRight /></button>
<div className="settings-group"><div className="settings-group-heading"><h2>Activity</h2><button onClick={onViewActivity}>View all <ChevronRight /></button></div><p className="group-help">Recent remote actions recorded on this PC.</p><div className="settings-card padded"><ActivityList entries={activity.slice(-3).reverse()} host={null} onOpen={onOpenActivity} /></div></div>
<div className="settings-group"><div className="settings-group-heading"><h2>Activity</h2><button onClick={onViewActivity}>View all <ChevronRight /></button></div><p className="group-help">Recent remote actions recorded on this PC.</p><div className="settings-card activity-retention-card"><div className="setting-row"><span><strong>Screenshot evidence</strong><small>{screenshotRetention.count} retained · {formatBytes(screenshotRetention.bytes)} · stored only on this PC</small></span><div className="retention-options" role="radiogroup" aria-label="Screenshot evidence retention">{([{ label: 'Off', enabled: false, days: 7 }, { label: '1d', enabled: true, days: 1 }, { label: '7d', enabled: true, days: 7 }, { label: '30d', enabled: true, days: 30 }] as const).map(option => <button key={option.label} role="radio" aria-checked={screenshotRetention.enabled === option.enabled && (!option.enabled || screenshotRetention.days === option.days)} className={screenshotRetention.enabled === option.enabled && (!option.enabled || screenshotRetention.days === option.days) ? 'active' : ''} onClick={() => onAction('set_activity_screenshot_retention', { enabled: option.enabled, days: option.days })}>{option.label}</button>)}</div></div></div><div className="settings-card padded"><ActivityList entries={activity.slice(-3).reverse()} host={null} onOpen={onOpenActivity} /></div></div>
</section>
}
@@ -834,6 +1017,7 @@ function HelpPage({ snapshot, host, onBack, onAction }: { snapshot: Snapshot; ho
return <section className="page-panel help-page"><button className="back-button" onClick={onBack}><ArrowLeft /> Back to Settings</button><div className="about-hero"><img src={logo} alt="" /><span><p>Desktop companion</p><h1>Hermes-Relay CLI UI</h1><small>Compact control for this PC</small></span></div>
<div className="settings-group"><h2>About</h2><div className="settings-card about-facts"><dl><div><dt>UI version</dt><dd>{snapshot.ui_version ?? 'Unknown'}</dd></div><div><dt>CLI version</dt><dd>{snapshot.cli_version ?? 'Unknown'}</dd></div><div><dt>CLI path</dt><dd title={snapshot.cli_path ?? undefined}>{snapshot.cli_path ?? 'Not reported'}</dd></div><div><dt>Relay server</dt><dd>{host?.server_version ?? 'Not connected'}</dd></div></dl></div></div>
<div className="settings-group"><h2>Get help</h2><div className="settings-card management-links">{links.map(([label, url]) => <button key={url} onClick={() => onAction('open_external_url', { url })}><span><strong>{label}</strong></span><ExternalLink /></button>)}</div></div>
<div className="settings-group"><h2>Computer control</h2><div className="settings-card help-note"><ShieldCheck /><span><strong>CUA Driver is an optional control engine</strong><small>When compatible and healthy, it can use app-aware background control and separate animated agent cursors. These are visual overlays—not additional Windows hardware pointers. Foreground switching is not available; control remains background only. Full Access never bypasses targeting, sensitive-surface rules, audit, or emergency stop.</small></span></div></div>
<button className="diagnostic-action" onClick={() => onAction('run_diagnostics')}><ActivityIcon /><span><strong>Run diagnostics</strong><small>Check the daemon, installation and active relay.</small></span><ChevronRight /></button>
</section>
}
+127
View File
@@ -68,6 +68,8 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
.route-endpoint small { font-size: 10px; font-weight: 550; white-space: nowrap; }
.route-link { position: relative; z-index: 0; display: block; height: 1px; border-top: 1.5px dashed #4fdf72; opacity: .9; }
.connection-route.offline .route-link { border-color: #66717d; opacity: .55; }
.connection-route.retrying .route-link { border-color: var(--amber); opacity: .7; animation: retry-link 1.4s ease-in-out infinite; }
@keyframes retry-link { 50% { opacity: .28; } }
.route-traffic { position: absolute; z-index: 1; pointer-events: none; left: 49px; right: 49px; top: 29px; height: 1px; overflow: hidden; }
.packet { position: absolute; top: -2px; left: 0; width: 9px; height: 5px; border-radius: 2px; background: #a3f9b2; box-shadow: 0 0 4px #72ed89, 0 0 10px #52df70; opacity: 0; }
.packet::after { content: ''; position: absolute; inset: 1px 2px; border-radius: 1px; background: #effff2; }
@@ -107,6 +109,10 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
.connection-route.offline .route-status strong { color: #8d98a3; }
.connection-route.connecting .route-status strong { color: var(--violet); }
.connection-route.disconnecting .route-status strong { color: var(--amber); }
.connection-route.retrying .route-status > strong { color: var(--amber); }
.retry-now { border: 0; background: transparent; color: #d3a84f; padding: 2px 5px; display: inline-flex; align-items: center; gap: 4px; font-size: 9.5px; cursor: pointer; }
.retry-now:hover { color: #ffd77b; }
.retry-now svg { width: 11px; height: 11px; }
.connection-route.connecting .route-link { opacity: .9; animation-duration: .65s; }
.connection-route.disconnecting .route-link, .connection-route.disconnecting .route-traffic { opacity: .38; transition: opacity .18s ease; }
.route-status span, .route-status small { color: var(--muted); font-size: 11px; }
@@ -283,6 +289,34 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
.activity-output-list header strong { font-size: 10px; font-weight: 550; }
.activity-output-list header em { color: var(--amber); font-size: 8px; font-style: normal; text-transform: uppercase; letter-spacing: .5px; }
.activity-output-list pre { max-height: 155px; margin: 0; padding: 9px; overflow: auto; color: #cdd2d7; font: 10px/1.45 'Cascadia Mono', Consolas, monospace; white-space: pre-wrap; overflow-wrap: anywhere; user-select: text; }
.control-timeline { margin: 0 0 10px; border: 1px solid var(--line); border-radius: 7px; background: #0d151c; padding: 8px 10px; display: grid; }
.control-timeline > div { min-height: 39px; position: relative; display: grid; grid-template-columns: 14px 1fr; gap: 8px; }
.control-timeline > div:not(:last-child)::after { content: ''; position: absolute; left: 5px; top: 15px; bottom: -2px; border-left: 1px dashed #4b5864; }
.control-timeline i { width: 11px; height: 11px; margin-top: 3px; border: 2px solid #53606c; border-radius: 50%; background: #0d151c; z-index: 1; }
.control-timeline .done i { border-color: var(--green); box-shadow: 0 0 6px #50e57755; }
.control-timeline .failed i { border-color: var(--amber); }
.control-timeline span { display: grid; align-content: start; gap: 1px; }
.control-timeline strong { font-size: 10.5px; font-weight: 560; }
.control-timeline small { color: var(--muted); font-size: 9.5px; overflow-wrap: anywhere; }
.control-timeline .pending i { border-color: #6d7782; border-style: dashed; }
.activity-error-callout { margin: 0 0 10px; padding: 9px 10px; border: 1px solid #77502c; border-radius: 7px; background: #2b1b0f; color: #f4c878; display: flex; gap: 8px; align-items: flex-start; }
.activity-error-callout > svg { width: 17px; flex: 0 0 17px; margin-top: 1px; }
.activity-error-callout span { min-width: 0; display: grid; gap: 2px; }
.activity-error-callout strong { font-size: 11px; }
.activity-error-callout small { color: #ddb986; font-size: 10px; line-height: 1.35; user-select: text; overflow-wrap: anywhere; }
.screenshot-evidence-card { width: 100%; min-height: 53px; margin: 0 0 10px; border: 1px solid #4c3d69; border-radius: 8px; background: linear-gradient(110deg, #171329, #101922); padding: 8px 10px; display: flex; justify-content: space-between; align-items: center; text-align: left; cursor: pointer; }
.screenshot-evidence-card:hover { border-color: #8e62d5; background: linear-gradient(110deg, #201638, #111b24); }
.screenshot-evidence-card > span { min-width: 0; display: grid; grid-template-columns: 25px 1fr; grid-template-rows: auto auto; column-gap: 7px; }
.screenshot-evidence-card > span svg { grid-row: 1 / 3; width: 20px; color: var(--violet); align-self: center; }
.screenshot-evidence-card strong { font-size: 11.5px; }
.screenshot-evidence-card small { color: var(--muted); font-size: 9.5px; }
.screenshot-evidence-card em { color: #c39aff; font-size: 10px; font-style: normal; display: flex; align-items: center; gap: 4px; white-space: nowrap; }
.screenshot-evidence-card em svg { width: 13px; }
.control-target { margin: 0 0 10px; display: grid; grid-template-columns: .8fr 1.2fr 1fr; border: 1px solid var(--line); border-radius: 7px; overflow: hidden; }
.control-target div { min-width: 0; padding: 7px 8px; border-right: 1px solid var(--line); display: grid; gap: 2px; }
.control-target div:last-child { border: 0; }
.control-target dt { color: var(--faint); font-size: 8px; text-transform: uppercase; letter-spacing: .55px; }
.control-target dd { margin: 0; color: #c8cdd2; font-size: 9.5px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.admin-warning { min-height: 44px; border: 1px solid #5b4615; border-radius: 6px; background: #3c2d0d3d; margin-top: 11px; padding: 0 12px; display: flex; align-items: center; gap: 11px; font-size: 12px; }
.admin-warning svg { color: var(--amber); width: 22px; }
.admin-warning span { flex: 1; }
@@ -445,6 +479,60 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
.settings-group h2 span { color: var(--faint); font-weight: 400; margin-left: 4px; }
.group-help { margin: -2px 0 8px; }
.settings-card { border: 1px solid var(--line); border-radius: 8px; background: #111922a8; overflow: hidden; }
.engine-card { position: relative; }
.engine-card::before { content: ''; position: absolute; inset: 0 auto 0 0; width: 2px; background: var(--faint); }
.engine-card.engine-ready::before { background: var(--green); box-shadow: 0 0 10px #50e57788; }
.engine-card.engine-degraded::before, .engine-card.engine-incompatible::before { background: var(--amber); }
.engine-status { min-height: 67px; padding: 10px 11px 10px 13px; display: flex; align-items: center; gap: 9px; }
.engine-status > span:nth-child(2) { min-width: 0; flex: 1; display: grid; gap: 3px; }
.engine-status strong { font-size: 13px; font-weight: 570; }
.engine-status small { color: var(--muted); font-size: 10px; line-height: 1.35; }
.engine-status em { flex: 0 0 auto; align-self: flex-start; margin-top: 2px; border: 1px solid #46515d; border-radius: 10px; color: var(--muted); padding: 2px 6px; font-size: 8.5px; font-style: normal; text-transform: uppercase; letter-spacing: .5px; }
.engine-ready .engine-status em { border-color: #346443; color: var(--green); background: #163c2524; }
.engine-degraded .engine-status em, .engine-incompatible .engine-status em { border-color: #705528; color: var(--amber); background: #251e12; }
.engine-options { border-top: 1px solid var(--line); }
.engine-choice { min-height: 68px; padding: 9px 11px 9px 13px; display: flex; align-items: center; gap: 10px; border-bottom: 1px solid var(--line); }
.engine-choice > span { min-width: 0; flex: 1; display: grid; gap: 3px; }
.engine-choice strong { font-size: 12.5px; font-weight: 550; }
.engine-choice small { color: var(--muted); font-size: 9.5px; line-height: 1.25; }
.engine-choice > div { flex: 0 0 auto; display: grid; grid-template-columns: 1fr 1fr; border: 1px solid #46515d; border-radius: 6px; overflow: hidden; }
.engine-choice button { min-width: 54px; height: 28px; border: 0; border-right: 1px solid #46515d; background: #121b24; color: var(--muted); font-size: 10px; cursor: pointer; }
.engine-choice button:last-child { border-right: 0; }
.engine-choice button.active { background: #6137bd; color: #fff; }
.engine-live { min-height: 38px; padding: 6px 11px 6px 13px; border-bottom: 1px solid var(--line); display: flex; align-items: center; justify-content: space-between; }
.engine-live > span { display: flex; align-items: baseline; gap: 5px; }
.engine-live strong { font-size: 15px; color: #dfe4e9; }
.engine-live small { color: var(--muted); font-size: 9.5px; }
.engine-live em { color: var(--faint); font-size: 8.5px; font-style: normal; text-transform: uppercase; letter-spacing: .45px; }
.engine-live em i { display: inline-block; width: 5px; height: 5px; border-radius: 50%; background: var(--faint); margin-right: 3px; }
.engine-live em.active { color: var(--green); }
.engine-live em.active i { background: var(--green); box-shadow: 0 0 6px var(--green); }
.engine-options .setting-row { min-height: 64px; padding-left: 13px; }
.engine-options .setting-row strong { font-size: 12.5px; }
.engine-options .setting-row small { max-width: 250px; font-size: 9.5px; line-height: 1.3; }
.background-only { background: #5c41150f; }
.background-only em { flex: 0 0 auto; color: #b5bdc6; font-size: 9px; font-style: normal; text-transform: uppercase; letter-spacing: .45px; }
.cua-health { background: #101820; }
.cua-health > div { flex: 0 0 auto; display: flex; align-items: center; gap: 7px; }
.cua-health em { font-size: 8.5px; font-style: normal; text-transform: uppercase; letter-spacing: .4px; color: var(--faint); }
.cua-health em.health-healthy { color: var(--green); }
.cua-health em.health-degraded, .cua-health em.health-error { color: var(--amber); }
.cua-health .compact-button { min-width: 68px; height: 27px; padding: 4px 7px; font-size: 9px; }
.cua-health .compact-button svg { width: 11px; height: 11px; }
.cua-maintenance { min-height: 49px; padding: 7px 10px 7px 13px; border-top: 1px solid var(--line); display: flex; align-items: center; gap: 8px; }
.cua-maintenance > span { min-width: 0; flex: 1; display: grid; gap: 2px; }
.cua-maintenance strong { font-size: 10.5px; font-weight: 560; }
.cua-maintenance small { color: var(--muted); font-size: 9px; line-height: 1.25; }
.cua-maintenance button { min-width: 60px; height: 27px; border: 1px solid #4c5762; border-radius: 5px; background: #17212a; color: #dce1e6; display: flex; align-items: center; justify-content: center; gap: 4px; font-size: 9.5px; cursor: pointer; }
.cua-maintenance button:disabled { opacity: .55; cursor: wait; }
.cua-maintenance button svg { width: 12px; height: 12px; }
.engine-help { display: flex; align-items: flex-start; gap: 5px; margin: 6px 2px 0; color: var(--faint); font-size: 9.5px; line-height: 1.35; }
.engine-help svg { width: 13px; height: 13px; flex: 0 0 13px; color: var(--violet); }
.help-note { min-height: 92px; padding: 11px; display: flex; align-items: flex-start; gap: 9px; }
.help-note > svg { width: 18px; flex: 0 0 18px; color: var(--violet); }
.help-note span { display: grid; gap: 4px; }
.help-note strong { font-size: 12px; }
.help-note small { color: var(--muted); font-size: 10px; line-height: 1.45; }
.settings-card.padded { padding: 0 11px; }
.activity-panel { display: grid; gap: 8px; }
.activity-summary { display: grid; grid-template-columns: 1fr 1fr auto; gap: 7px; align-items: stretch; }
@@ -466,6 +554,11 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
.clear-activity:disabled { opacity: .45; cursor: default; }
.activity-card { padding: 0; }
.activity-card .empty-state { min-height: 90px; }
.activity-retention-card { margin-bottom: 8px; }
.activity-retention-card .setting-row { align-items: center; }
.retention-options { flex: 0 0 auto; display: grid; grid-template-columns: repeat(4, auto); gap: 2px; padding: 2px; border: 1px solid #35414c; border-radius: 7px; background: #0b1218; }
.retention-options button { min-width: 29px; height: 24px; padding: 0 5px; border: 0; border-radius: 5px; background: transparent; color: var(--muted); font-size: 9.5px; cursor: pointer; }
.retention-options button.active { background: #7042d6; color: white; box-shadow: inset 0 1px #c9a5ff33; }
.settings-group-heading { display: flex; align-items: center; justify-content: space-between; }
.settings-group-heading h2 { margin-bottom: 7px; }
.settings-group-heading button { border: 0; background: transparent; color: var(--violet); display: flex; align-items: center; gap: 2px; padding: 0 0 7px; font-size: 11.5px; cursor: pointer; }
@@ -617,6 +710,40 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
.grant-actions button:hover:not(:disabled) { filter: brightness(1.1); }
.grant-actions button:disabled { opacity: .6; cursor: wait; }
.connection-notice-shell { position: fixed; inset: 0; padding: 7px; display: grid; align-items: stretch; animation: notice-in .2s cubic-bezier(.2,.8,.2,1); }
@keyframes notice-in { from { opacity: 0; transform: translateY(12px) scale(.98); } }
.connection-notice-card { position: relative; border: 1px solid #3c4854; border-radius: 12px; background: #101922f5; box-shadow: 0 15px 38px #000b, inset 0 1px #ffffff08; padding: 13px 56px 12px 13px; display: grid; grid-template-columns: 38px minmax(0,1fr); gap: 10px; align-items: center; overflow: hidden; }
.connection-notice-card::before { content: ''; position: absolute; inset: 0 auto 0 0; width: 3px; background: var(--green); box-shadow: 0 0 13px var(--green); }
.connection-notice-shell.warning .connection-notice-card::before { background: var(--amber); box-shadow: 0 0 13px var(--amber); }
.connection-notice-shell.offline .connection-notice-card::before { background: #8c96a0; box-shadow: none; }
.connection-notice-icon { width: 36px; height: 36px; border-radius: 10px; background: #21432b; color: var(--green); display: grid; place-items: center; }
.connection-notice-shell.warning .connection-notice-icon { background: #493817; color: var(--amber); }
.connection-notice-shell.offline .connection-notice-icon { background: #29323b; color: #aeb7c0; }
.connection-notice-icon svg { width: 21px; }
.connection-notice-card > span:nth-child(2) { min-width: 0; display: grid; gap: 1px; }
.connection-notice-card small { color: var(--violet); font-size: 8.5px; text-transform: uppercase; letter-spacing: .8px; }
.connection-notice-card strong { font-size: 13px; }
.connection-notice-card p { margin: 1px 0 0; color: var(--muted); font-size: 10px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
.connection-notice-open { position: absolute; right: 12px; bottom: 12px; border: 0; background: transparent; color: #c59aff; font-size: 9.5px; cursor: pointer; }
.connection-notice-close { position: absolute; right: 8px; top: 8px; width: 25px; height: 25px; border: 0; border-radius: 6px; background: transparent; color: #8f99a4; display: grid; place-items: center; cursor: pointer; }
.connection-notice-close:hover { color: white; background: #ffffff0c; }
.connection-notice-close svg { width: 14px; }
.evidence-shell { position: fixed; inset: 0; border: 1px solid #3a4652; border-radius: 11px; background: #0b1218; box-shadow: 0 24px 70px #000d; display: grid; grid-template-rows: 52px minmax(0,1fr); overflow: hidden; }
.evidence-shell > header { border-bottom: 1px solid var(--line); background: #111a23; padding: 0 13px 0 16px; display: flex; align-items: center; justify-content: space-between; }
.evidence-shell > header > span { display: grid; grid-template-columns: 28px auto; grid-template-rows: auto auto; column-gap: 8px; }
.evidence-shell > header svg { grid-row: 1 / 3; align-self: center; width: 20px; color: var(--violet); }
.evidence-shell > header strong { font-size: 13px; }
.evidence-shell > header small { color: var(--muted); font-size: 9.5px; }
.evidence-shell > header button { width: 34px; height: 34px; border: 0; border-radius: 7px; background: transparent; color: #aab2bb; display: grid; place-items: center; cursor: pointer; }
.evidence-shell > header button:hover { color: white; background: #c42b1c; }
.evidence-shell > main { min-width: 0; min-height: 0; padding: 14px; display: grid; place-items: center; background: radial-gradient(circle at center, #18232c, #080e13 68%); }
.evidence-shell img { max-width: 100%; max-height: 100%; object-fit: contain; border-radius: 5px; box-shadow: 0 8px 30px #000a; user-select: none; }
.evidence-loading, .evidence-error { color: var(--muted); display: grid; place-items: center; gap: 8px; text-align: center; }
.evidence-error svg { color: var(--amber); width: 30px; }
.evidence-error strong { color: var(--ink); font-size: 14px; }
.evidence-error small { max-width: 420px; font-size: 11px; }
@media (max-width: 410px) {
.content { padding-left: 20px; padding-right: 20px; }
.access-control button { gap: 5px; }
+66 -1
View File
@@ -23,17 +23,76 @@ export interface DaemonStatus {
privilege?: string | null
username?: string | null
updated_at?: number | null
last_event?: string | null
reconnect_attempt?: number | null
retry_at?: number | null
last_error?: string | null
}
export interface ComputerControlEngine {
selected: 'legacy' | 'cua'
effective: 'legacy' | 'cua' | 'unavailable'
available: boolean
state: 'not_installed' | 'incompatible' | 'degraded' | 'ready' | 'error'
version?: string | null
health?: string | null
path?: string | null
cursor_enabled?: boolean
foreground_escalation_enabled: boolean
active_sessions?: number
active_backend?: 'cua' | 'legacy_compat' | 'mixed' | 'idle' | null
last_action?: {
action?: string | null
target_app?: string | null
target_title?: string | null
target_pid?: number | null
target_window_id?: number | null
verification?: string | null
occurred_at?: number | string | null
} | null
message?: string | null
}
export interface CuaManagementStatus {
installed: boolean
canonical_path?: string | null
discovered_path?: string | null
stale_path_shim: boolean
current_version?: string | null
compatible: boolean
compatibility_reason?: string | null
supported_range: { minimum: string; maximum_exclusive: string }
update?: { latest_version?: string; update_available: boolean; compatible: boolean; error?: string; release_notes_url?: string }
operation?: { kind: 'install' | 'update'; state: 'completed'; version: string }
}
export interface CuaHealthStatus {
state: 'healthy' | 'degraded' | 'error'
checkedAt: string
overall?: string
reason?: string
temporaryWindowsCompatibility: true
}
export interface Activity {
ts: number
kind?: 'tool.completed' | 'management.completed'
kind?: 'tool.completed' | 'management.completed' | 'connection.state'
tool: string
category?: 'command' | 'files' | 'screen' | 'input' | 'devices' | 'system' | 'other'
ok: boolean
aborted?: boolean
request_id?: string
host_url?: string
backend?: string
dispatch?: string
control_session_id?: string
target_app?: string
target_title?: string
target_pid?: number
target_window_id?: number
action?: string
verification?: string
phase?: string
duration_ms?: number
exit_code?: number
summary?: string
@@ -47,6 +106,10 @@ export interface Activity {
stderr_truncated?: boolean
result_truncated?: boolean
error?: string
screenshot_evidence_id?: string
screenshot_mime_type?: 'image/png'
screenshot_width?: number
screenshot_height?: number
}
export interface Snapshot {
@@ -54,6 +117,7 @@ export interface Snapshot {
active_url?: string | null
daemon: DaemonStatus
activity: Activity[]
activity_screenshot_retention: { enabled: boolean; days: number; count: number; bytes: number }
pending_grants: PendingGrantRequest[]
startup_enabled: boolean
daemon_autostart_enabled?: boolean
@@ -61,6 +125,7 @@ export interface Snapshot {
cli_version?: string | null
cli_path?: string | null
hardware_availability: { usb: boolean; adb: boolean; microphone: boolean; camera: boolean }
computer_control_engine?: ComputerControlEngine | null
}
export interface PendingGrantRequest {
+151
View File
@@ -270,6 +270,8 @@ not require an API endpoint or API bearer when dashboard chat is ready.
>
> **Decision (2026-06-29) — unified-session "Threads", not a separate surface:** the proactive agent↔phone conversation is **not** a separate app lane/tab/segment. It is a **source-tagged session inside the one Chat surface** — a **Thread** (`source=phone`). The three things distinguishing a Thread from a normal gateway chat are *session properties*, not a separate UI: (a) the agent can initiate a turn, (b) it rides the relay `proactive` transport and is relay-gated, (c) it's a standing/named DM. **Scrollback = the gateway session store** (same read path Chat uses); **live receive = the relay `proactive` push** (→ notification); **send = `proactive.reply`**. The local `ProactiveInboxStore` is demoted to a live-push cache + outbox (no parallel history). The Thread capability is surfaced in the **connection best-path/capability UI** (a relay-tier capability, like terminal/bridge/voice) and as a clean **Threads** entry (thread-spool icon, NOT a phone glyph) pinned atop the session drawer when active — never a connection-wizard step. Degrades cleanly: no relay plugin → no `source=phone` sessions → Chat is unchanged (standard-path-safe). This **supersedes the earlier "separate Agent lane / 4th nav segment" sketch** and folds in the "show chat source/platform attribution in Chat" goal in one stroke.
>
> **Outbound-first refinement (2026-08-14):** an outbound platform send does not itself create a gateway session; the `source=phone` session is created only when the phone first replies. Android groups the bounded proactive cache by connection + `chat_id` and renders a provisional Thread during that gap. A notification tap opens that exact provisional Thread, and the first reply uses the existing phone-platform path before promoting the view to the real gateway session. Once `/phone/threads` reports the mapping, the provisional row disappears. This cache is a bootstrap transcript, not a second long-term history store.
>
> **Why unified, not separate:** a separate "agent chat" tab is redundant — a Thread is just a chat session the agent can also start. One Chat surface (sessions tagged by source) matches the Discord/messaging-app model the product targets. **Two distinct senses of "gateway" to keep straight:** the *messaging gateway / platform layer* (`gateway/platforms/*` — phone/Discord/Slack as platforms; this is the Thread's `source`) vs. the *dashboard gateway transport* (`/api/ws` tui_gateway — how live chat bytes flow). A Thread is defined by its **platform/source**, not its transport; the two are orthogonal.
**Original research (2026-04-07, retained as lineage — the `mobile:<device_id>` syntax and the ~16-file upstream-fork registration below are SUPERSEDED by the no-fork plugin path):**
@@ -3041,3 +3043,152 @@ Hermes authorization remains local and the vanilla upstream path remains
independent. Implementing the client requires a TLS-over-broker byte-stream
adapter; advertising the candidate is forbidden until that adapter and the
cross-platform acceptance gates are complete.
---
## ADR 56 — Structured desktop control may use CUA Driver behind Hermes policy
**Status:** Accepted for phased implementation (2026-08-13).
**Implementation note (2026-08-13).** The first Windows integration keeps the
public `desktop_computer_*` contract and adds canonical runtime discovery,
manifest/version/tool/permission/health checks, an allowlisted CUA adapter,
server-owned control-session envelopes, per-session grants, sensitive-target
preflight, single-use snapshot tokens, pre/action/post snapshot flow, and local
CLI/UI engine status. CUA actions are background-only in this phase; foreground
escalation remains reserved and reports disabled. The optional driver is not
bundled or updated by Hermes-Relay, and Hermes forces driver telemetry off for
every child process it starts. The legacy engine remains the default and the
fail-closed fallback while the live Windows acceptance and remaining scope,
redaction, and grant-bridge hardening gates tracked in `TODO.md` stay open.
**Second-phase refinement (2026-08-13).** CUA is the preferred/default setting
for new structured-control sessions; the original Windows input path is named
`legacy_compat` and remains available only as an explicit compatibility choice
or a fail-closed pre-session fallback when CUA is unavailable. The backend is
selected once per authenticated control session and cannot change mid-session.
Window-scoped snapshots/actions use CUA, while the existing read-only full-
display screenshot remains a separate `system_capture` path. Local audit and
the UI activity timeline expose bounded high-level fields—backend, dispatch,
control session, target app/window identifiers, action, phase, and verification
state—while omitting accessibility text, screenshots, entered values, and raw
driver responses.
Hermes now owns an explicit Windows lifecycle surface without bundling the
driver: `computer-use cua status|health|install|check-update|update`, with `--yes`
required for install/update. Mutations accept only supported upstream releases
(`>=0.19.3 <0.20.0`), verify the `trycua/cua` product/version manifest and
installer SHA-256 before running a temporary installer under a sanitized
environment, then verify the canonical `packages/current` binary, manifest,
version, path, and permission mode. Accessibility health is an explicit
diagnostic. There is no automatic install or
update.
**Context.** The first Windows input backend uses PowerShell, `SetCursorPos`,
`mouse_event`, and `SendKeys`. It can move the operator's physical pointer and
depends on foreground focus. CUA Driver provides target-process/window UI
Automation, accessibility snapshots, background input, and session-scoped
animated agent cursors without moving the physical pointer. Its full local tool
surface also includes foreground activation, arbitrary application control,
recording, configuration, and update operations that exceed Hermes-Relay's
screen/input capability.
**Decision.** Adopt CUA Driver only as an optional, version-pinned structured
computer-control backend behind the existing `desktop_computer_*` contract.
Hermes remains the outer authority for Relay authentication, target-device
routing, per-host capability policy, local grants, emergency stop, and audit.
The daemon translates an allowlisted Hermes schema to CUA operations; it never
forwards arbitrary CUA tool names or JSON.
Background dispatch is the default. A `background_unavailable` result returns
to the caller instead of silently foregrounding another application. Foreground
dispatch, application launch/termination, JavaScript execution, recording,
replay, configuration, and driver updates require separate, explicit local
authority. Full Access may bypass ordinary task prompts, but never authenticated
targeting, sensitive-surface blocks, UAC/session boundaries, audit, emergency
stop, runtime validation, or per-action failures.
Every semantic element action requires a fresh pre-action window snapshot, an
opaque element token bound to its snapshot generation, authenticated principal,
grant, PID, and window, followed by a post-action snapshot that records whether
the expected state changed. Grant scope and sensitive-surface policy apply to
both accessibility text and pixels.
Multiple animated pointers are virtual agent overlays, not additional Windows
hardware cursors. Relay must first attach a server-owned authenticated control
session identity—at minimum the Relay session, requester device, chat/run,
target device, and request—to each command. The desktop derives cursor identity
locally and owns one bounded CUA session per active control session. Model
arguments cannot choose or share cursor IDs. Grant expiry, cancellation,
disconnect, re-pair, policy downgrade, emergency stop, desktop lock/session
change, or daemon shutdown ends the corresponding driver session immediately.
CUA runs in the interactive user's logon session, never Windows Session 0.
Initial packaging remains optional and resolves the canonical installed package
rather than an untrusted PATH entry. Readiness uses the driver's live manifest,
schema, tool surface, daemon status, and permission mode and fails closed on an
absent or incompatible backend. Telemetry and driver updates remain explicit
operator choices.
**Temporary Windows implementation note (2026-08-14).** Until
`trycua/cua#3103` is fixed in the supported driver range, the whole-desktop
`health_report` is not a session-start gate: its fixed UIA timeout can report a
false degradation and leave the driver temporarily busy. Operators can re-run
that diagnostic from the CLI or UI. Structured actions retain their existing
target, grant, snapshot, timeout, and fail-closed checks. Remove this exception
when the upstream probe is bounded and cannot poison later actions.
**Consequences.** Hermes can gain background, element-aware control and clean
per-agent animated cursors without replacing its Relay protocol or permission
model. The existing PowerShell/User32 backend remains a compatibility fallback
until authenticated control-session identity, scoped grants, sensitive-surface
enforcement, and lifecycle tests ship. Raw command execution remains an honest
escape hatch: strong computer-control isolation is meaningful only when command
execution is disabled or the host is already fully trusted.
---
## ADR 57 — Official Desktop Relay UI is a lazy, unified-package runtime plugin
**Status:** Accepted (2026-08-14).
**Context.** Official Hermes Desktop now discovers a regular agent plugin's
`desktop/plugin.js` and loads it through `@hermes/plugin-sdk`. The SDK provides
profile-aware `ctx.rest()` access to the same `plugin_api.py` namespace used by
the web Dashboard, plus native pane, sidebar, status-bar, palette, close,
reveal, drag, dock, i18n, query, and unload lifecycles. Registering a pane also
adopts it into the live layout, which would violate Hermes-Relay's requirement
that plugin load and application lifecycle events never open or focus Relay.
**Decision.** Ship `plugin/desktop/plugin.js` beside `plugin.yaml` and the
existing Dashboard half. Keep the Desktop half opt-in and use only public SDK
imports. At registration time contribute three labeled open actions, but no
pane and no network work. The first explicit action registers one dismissible
management pane and calls the plugin-scoped `ctx.panes.reveal()` method.
Subsequent actions reveal the same pane. The host owns close, focus, drag,
docking, enable state, hot reload, and disposer execution.
The pane reads and mutates the existing Relay Dashboard backend through
`ctx.rest()` only. It maintains no server cache, starts no timer/socket/polling
loop, sends no notification, and makes no request until the user opens a view.
Every query key contains the active Desktop profile, while the SDK binds the
request to that profile's authenticated backend namespace. Pairing, revocation,
and remote-access actions stay user initiated; destructive or host-changing
actions require an additional in-pane confirmation.
**Security boundary.** Runtime Desktop plugins are trusted local ESM with full
renderer authority; upstream provides error isolation, not a sandbox. Relay
therefore ships fixed reviewed UI code only. It does not load generated ESM,
Kotlin, Python, remote modules, or arbitrary action schemas, and it never stores
tokens, keys, QR secrets, media paths, or duplicated Relay state. The existing
Relay-gated declarative Android Plugin Studio remains a separate host-rendered,
digest-approved capability model.
**Consequences.** One plugin install now exposes the Relay-specific management
role in both the web Dashboard and official Desktop without patching Hermes or
replacing the standalone Relay CLI/tray. The SDK has no public programmatic
pane-coordinate API and its agent `focus_pane` surface excludes contributed
IDs, so movement remains native drag/dock and agent-driven reveal remains
unsupported. Physical Desktop certification remains required for multi-window,
named-profile, remote/SSH-mapped profile, close/reopen, drag/dock, hot-reload,
and renderer-log behavior.
+29 -1
View File
@@ -387,7 +387,8 @@ Sources: `plugin/relay/channels/notifications.py`, `app/src/main/kotlin/.../noti
| Type | Direction | Payload |
|------|-----------|---------|
| `desktop.command` | Server → Client | `{request_id, tool, args}` |
| `desktop.command` | Server → Client | `{request_id, tool, args, control_session?}` |
| `desktop.control_session_end` | Server → Client | `{version: 1, id, target_device_id, reason?}` |
| `desktop.response` | Client → Server | `{request_id, ok: true, result}` or `{request_id, ok: false, error}` |
| `desktop.status` | Client → Server | `{advertised_tools, device_id, host, platform, version, computer_use?}` |
| `desktop.workspace` | Client → Server | Workspace context snapshot |
@@ -403,6 +404,33 @@ pending request to the selected WebSocket, ignores responses from other PCs,
and includes resolved target metadata in the response. `/desktop/health` lists
the connected targets.
For `desktop_computer_*`, current relays can attach a server-owned
`control_session` object with `version`, opaque `id`, `request_id`,
`requester_device_id`, `target_device_id`, and `run_id`, plus optional
`chat_session_id` and `profile`. The desktop client validates the version and
all required fields, requires the embedded request and target to match the
outer command and local installation, and never reads identity from tool
arguments. A malformed supplied binding is rejected. Omission remains
backward-compatible for legacy computer handlers, but identity-sensitive CUA
operations fail closed until the relay supplies an authenticated binding.
The plugin's standard tool route is loopback-only. Executor context is passed
in internal headers and is trusted only inside the existing same-user host
boundary; any local process running as that user can reach the same boundary.
When a paired bearer is present, the relay derives `requester_device_id` from
the authenticated plugin session instead. The model-facing tool wrapper drops
reserved identity arguments, and the relay—not the caller—mints the opaque
control-session ID and binds each selected target and request.
The relay ends a control authority when the same authenticated requester/chat/
profile starts a different run for that target, after 15 minutes of inactivity,
on bounded capacity eviction, or when the target desktop disconnects. For the
first three cases it emits `desktop.control_session_end`; the desktop accepts
that event only from its attached relay channel, requires the target to match
its stable local device ID and the opaque ID to match an active authority, then
revokes that authority's grants and closes its exact CUA cursor/MCP session.
Desktop disconnect and local cancellation independently revoke local state.
Per-host Structured access withholds `desktop_terminal`,
`desktop_powershell`, `desktop_spawn_detached`, and `desktop_job_start` from
`advertised_tools`. Brokered USB tools use the same request/response envelope:
+29
View File
@@ -1046,6 +1046,7 @@ Current Android dependency versions. Source of truth is `gradle/libs.versions.to
| **Plugin system** | `register_tool()` via `ctx` for `android_*` and `desktop_*` tools |
| **Relay plugin** | `hermes pair`, `hermes relay start`, `hermes relay doctor`, `hermes relay compat`, dashboard `/relay` plugin tab |
| **Dashboard plugin** | Lives at `plugin/dashboard/`; see §10.1 below |
| **Official Desktop plugin** | Unified-package `plugin/desktop/plugin.js`; official `@hermes/plugin-sdk` only; see §10.2 below |
### 10.1 Dashboard plugin
@@ -1073,6 +1074,34 @@ Hermes-Relay ships a hermes-agent Dashboard Plugin that surfaces relay-specific
**Frontend.** Source under `plugin/dashboard/src/` (JSX + esbuild), committed pre-built IIFE at `plugin/dashboard/dist/index.js` (~16 KB minified). Uses the dashboard's `window.__HERMES_PLUGIN_SDK__` global for React + shadcn primitives + `fetchJSON()` — no external HTTP library, no bundled React. See ADR 19 in [`docs/decisions.md`](decisions.md) for the architectural rationale.
### 10.2 Official Desktop plugin
The same installable `hermes-relay` plugin folder includes
`plugin/desktop/plugin.js`, discovered through the upstream unified-package path
`$HERMES_HOME/plugins/hermes-relay/desktop/plugin.js`. It imports only
`@hermes/plugin-sdk`, React, and the React JSX runtime. Backend access uses the
SDK's profile-aware `ctx.rest()` door, so Desktop and the web Dashboard share
the existing `/api/plugins/hermes-relay/*` backend without copying state or
introducing a second control plane.
The Desktop half is opt-in. Enabling or loading it registers only labeled
sidebar, status-bar, and command-palette entry points. The management pane is
registered lazily after one of those explicit actions, then restored and
focused with `ctx.panes.reveal()`. Startup, reconnect, profile change, hot
reload, update, navigation restoration, and background events never reveal or
focus it. Closing uses the official dismissible-pane lifecycle; moving and
docking use the host's native drag targets. The current SDK does not expose
programmatic move coordinates or agent-driven focus for contributed pane IDs,
so Hermes-Relay does not emulate either with private layout or Electron hooks.
The pane provides four manually refreshed views: Relay management and pairing,
bridge activity, sanitized media metadata, and remote access. Mutations require
explicit labeled actions; session revocation and remote-access changes add an
in-pane confirmation. The plugin has no timers, notifications, background
polling, arbitrary renderer code generation, telemetry, or direct external
networking. Query keys include the active profile and `ctx.rest()` supplies the
matching authenticated backend scope.
---
## Related
+1 -1
View File
@@ -3,7 +3,7 @@
"label": "Relay",
"description": "Paired devices, bridge activity, media inspection, and remote access for hermes-relay",
"icon": "Activity",
"version": "1.7.0",
"version": "1.8.0",
"tab": {
"path": "/relay",
"position": "after:skills"
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "hermes-relay-dashboard",
"version": "1.7.0",
"version": "1.8.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "hermes-relay-dashboard",
"version": "1.7.0",
"version": "1.8.0",
"devDependencies": {
"esbuild": "^0.25.12",
"qrcode": "^1.5.4"
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "hermes-relay-dashboard",
"version": "1.7.0",
"version": "1.8.0",
"private": true,
"description": "Hermes-Relay dashboard plugin frontend (IIFE bundle). Loaded verbatim by the hermes-agent dashboard via the Plugin SDK global.",
"scripts": {
+36
View File
@@ -0,0 +1,36 @@
# Hermes-Relay plugin for official Hermes Desktop
This is the Desktop half of the unified `hermes-relay` plugin package. Official
Hermes Desktop discovers it at `plugins/hermes-relay/desktop/plugin.js`, beside
the existing Python and web Dashboard halves.
The plugin is opt-in and never opens itself. Enabling or loading it only adds
three labeled native entry points: **Relay** in the sidebar and status bar, and
**Hermes Relay: Open** in the command palette. One of those explicit actions
registers, restores, and focuses the management pane. The pane uses the host's
native tab close and drag/dock affordances.
All backend calls use the official profile-aware `ctx.rest()` namespace and the
existing `dashboard/plugin_api.py` routes. The pane keeps no server state, does
not poll, does not notify, and does not perform network work while closed.
## SDK baseline
Implemented against upstream Hermes Desktop source
`0296c7d8aa1d1cdb7bec8522031ca6b14eadc604` (2026-08-14), using only:
- `PluginContext.register`, `registerMany`, `onDispose`, `rest`, `os`, and
`panes.reveal`
- pane, sidebar, status-bar, and palette contribution areas
- the exported React/query/i18n/UI-kit surface
The SDK currently has no public API for programmatic pane move coordinates or
for agent-driven focus of contributed pane IDs. Users move/dock the pane with
native drag targets; the plugin does not use private layout or Electron hooks.
## Test
```bash
cd plugin/desktop
npm test
```
+8
View File
@@ -0,0 +1,8 @@
{
"name": "hermes-relay-desktop-plugin",
"private": true,
"type": "module",
"scripts": {
"test": "node --test --experimental-loader ./test/loader.mjs test/*.test.mjs"
}
}
+576
View File
@@ -0,0 +1,576 @@
import {
Badge,
Button,
Codicon,
EmptyState,
ErrorState,
Input,
PANES_AREA,
PALETTE_AREA,
SIDEBAR_NAV_AREA,
STATUSBAR_AREAS,
StatusDot,
Tip,
host,
useMutation,
usePluginI18n,
useQuery,
useQueryClient,
useValue
} from '@hermes/plugin-sdk'
import { useState } from 'react'
import { Fragment, jsx, jsxs } from 'react/jsx-runtime'
const PLUGIN_ID = 'hermes-relay'
const messages = {
en: {
'action.cancel': 'Cancel',
'action.confirm': 'Confirm',
'action.copy': 'Copy invite',
'action.open': 'Open Hermes Relay',
'action.pair': 'Pair new device',
'action.refresh': 'Refresh',
'activity.empty': 'No recent bridge activity.',
'activity.title': 'Bridge activity',
'management.empty': 'No paired devices.',
'management.title': 'Relay management',
'media.empty': 'No active media deliveries.',
'media.title': 'Media',
'nav.label': 'Relay',
'pairing.copyFailed': 'The Desktop clipboard bridge is unavailable. Select and copy the invite manually.',
'pairing.expires': 'One-time invite. Keep it private and use it before it expires.',
'pairing.title': 'Pairing invite',
'pane.description': 'Profile-scoped views of the existing Hermes-Relay service.',
'pane.title': 'Hermes Relay',
'remote.disable': 'Disable Tailscale serving',
'remote.enable': 'Enable Tailscale serving',
'remote.probe': 'Probe URL',
'remote.save': 'Save public URL',
'remote.title': 'Remote access',
'session.revoke': 'Revoke',
'session.revokeConfirm': 'Revoke this device session? The device must pair again.',
'status.detail': 'Open the profile-scoped Relay management pane',
'status.label': 'Relay'
}
}
export function profileQueryKey(profile, resource) {
return [PLUGIN_ID, profile || 'default', resource]
}
function text(value, fallback = '—') {
if (value === null || value === undefined || value === '') return fallback
if (typeof value === 'object') return JSON.stringify(value)
return String(value)
}
function list(value, key) {
if (Array.isArray(value)) return value
if (value && Array.isArray(value[key])) return value[key]
return []
}
function friendlyError(error) {
if (!error) return 'Unknown error'
if (error instanceof Error) return error.message
return String(error)
}
function formatTime(value) {
if (!value) return '—'
const numeric = Number(value)
const millis = Number.isFinite(numeric) && numeric < 10_000_000_000 ? numeric * 1000 : numeric
const date = new Date(Number.isFinite(millis) ? millis : value)
return Number.isNaN(date.getTime()) ? text(value) : date.toLocaleString()
}
function Field({ label, value }) {
return jsxs('div', {
className: 'min-w-0 rounded-md border border-(--ui-stroke-tertiary) px-3 py-2',
children: [
jsx('div', {
className: 'text-[0.625rem] font-medium uppercase tracking-wide text-(--ui-text-quaternary)',
children: label
}),
jsx('div', {
className: 'mt-1 break-words text-xs text-(--ui-text-secondary)',
children: text(value)
})
]
})
}
function SectionHeader({ action, description, title }) {
return jsxs('header', {
className: 'mb-4 flex flex-wrap items-start justify-between gap-3',
children: [
jsxs('div', {
children: [
jsx('h2', { className: 'text-sm font-semibold text-(--ui-text-primary)', children: title }),
description
? jsx('p', { className: 'mt-1 text-xs leading-5 text-(--ui-text-tertiary)', children: description })
: null
]
}),
action
]
})
}
function QueryState({ empty, query, render }) {
if (query.isPending) {
return jsx('p', { className: 'py-8 text-xs text-(--ui-text-tertiary)', role: 'status', children: 'Loading…' })
}
if (query.error) {
return jsx(ErrorState, { description: friendlyError(query.error), title: 'Relay unavailable' })
}
if (empty(query.data)) {
return jsx(EmptyState, { description: '', title: 'Nothing to show' })
}
return render(query.data)
}
function createManagement(ctx) {
return function Management() {
const t = usePluginI18n(PLUGIN_ID)
const profile = useValue(host.state.profile)
const client = useQueryClient()
const [confirmPrefix, setConfirmPrefix] = useState(null)
const [pairing, setPairing] = useState(null)
const [copyError, setCopyError] = useState(null)
const overviewKey = profileQueryKey(profile, 'overview')
const sessionsKey = profileQueryKey(profile, 'sessions')
const overview = useQuery({ queryKey: overviewKey, queryFn: () => ctx.rest('/overview'), retry: false })
const sessions = useQuery({ queryKey: sessionsKey, queryFn: () => ctx.rest('/sessions'), retry: false })
const refresh = () => {
void client.invalidateQueries({ queryKey: overviewKey })
void client.invalidateQueries({ queryKey: sessionsKey })
}
const pair = useMutation({
mutationFn: () => ctx.rest('/pairing', { method: 'POST', body: { mode: 'auto' } }),
onSuccess: result => {
setCopyError(null)
setPairing(result)
}
})
const revoke = useMutation({
mutationFn: prefix => ctx.rest(`/sessions/${encodeURIComponent(prefix)}`, { method: 'DELETE' }),
onSuccess: () => {
setConfirmPrefix(null)
void client.invalidateQueries({ queryKey: sessionsKey })
}
})
const rows = list(sessions.data, 'sessions')
const copyInvite = async () => {
const invite = pairing?.pairing_url || pairing?.qr_payload
if (!invite) return
if (!(await ctx.os.writeClipboard(invite))) setCopyError(t('pairing.copyFailed'))
}
return jsxs('section', {
children: [
jsx(SectionHeader, {
title: t('management.title'),
description: `Profile: ${profile || 'default'}`,
action: jsxs('div', {
className: 'flex gap-2',
children: [
jsx(Button, { size: 'sm', variant: 'outline', onClick: refresh, children: t('action.refresh') }),
jsx(Button, {
size: 'sm',
disabled: pair.isPending,
onClick: () => pair.mutate(),
children: pair.isPending ? 'Creating…' : t('action.pair')
})
]
})
}),
overview.error
? jsx(ErrorState, { title: 'Relay unavailable', description: friendlyError(overview.error) })
: overview.data
? jsxs('div', {
className: 'mb-5 grid gap-2 sm:grid-cols-2 lg:grid-cols-4',
children: [
jsx(Field, { label: 'Health', value: overview.data.health || overview.data.status }),
jsx(Field, { label: 'Version', value: overview.data.version }),
jsx(Field, { label: 'Uptime (seconds)', value: overview.data.uptime_seconds }),
jsx(Field, { label: 'Connected', value: overview.data.connected_clients ?? overview.data.active_sessions })
]
})
: null,
pair.error ? jsx(ErrorState, { title: 'Pairing failed', description: friendlyError(pair.error) }) : null,
pairing
? jsxs('div', {
className: 'mb-5 rounded-md border border-(--ui-accent) p-3',
children: [
jsxs('div', {
className: 'flex flex-wrap items-center justify-between gap-2',
children: [
jsxs('div', {
children: [
jsx('h3', { className: 'text-xs font-semibold text-(--ui-text-primary)', children: t('pairing.title') }),
jsx('p', { className: 'mt-1 text-xs text-(--ui-text-tertiary)', children: t('pairing.expires') })
]
}),
jsx(Button, { size: 'sm', variant: 'outline', onClick: copyInvite, children: t('action.copy') })
]
}),
pairing.code
? jsx('div', { className: 'mt-3 font-mono text-xl tracking-widest', children: pairing.code })
: null,
jsx('div', {
className: 'mt-2 select-all break-all font-mono text-[0.6875rem] text-(--ui-text-tertiary)',
children: pairing.pairing_url || pairing.qr_payload || 'Invite created'
}),
copyError ? jsx('p', { className: 'mt-2 text-xs text-destructive', children: copyError }) : null
]
})
: null,
revoke.error ? jsx(ErrorState, { title: 'Revoke failed', description: friendlyError(revoke.error) }) : null,
jsx(QueryState, {
query: sessions,
empty: data => list(data, 'sessions').length === 0,
render: () => jsxs('div', {
className: 'overflow-hidden rounded-md border border-(--ui-stroke-tertiary)',
children: rows.map((session, index) => {
const prefix = session.token_prefix || session.prefix || session.id || `session-${index}`
const confirming = confirmPrefix === prefix
return jsxs('div', {
className: 'flex flex-wrap items-center justify-between gap-3 border-b border-(--ui-stroke-tertiary) px-3 py-3 last:border-b-0',
children: [
jsxs('div', {
className: 'min-w-0',
children: [
jsxs('div', {
className: 'flex flex-wrap items-center gap-2',
children: [
jsx('span', { className: 'font-medium text-xs text-(--ui-text-primary)', children: session.device_name || session.name || prefix }),
jsx(Badge, { children: session.device_type || session.client_type || 'device' })
]
}),
jsx('p', {
className: 'mt-1 text-[0.6875rem] text-(--ui-text-tertiary)',
children: `${prefix} · paired ${formatTime(session.paired_at)}`
})
]
}),
confirming
? jsxs('div', {
className: 'flex items-center gap-2',
children: [
jsx('span', { className: 'max-w-52 text-right text-[0.6875rem] text-(--ui-text-tertiary)', children: t('session.revokeConfirm') }),
jsx(Button, { size: 'sm', variant: 'ghost', onClick: () => setConfirmPrefix(null), children: t('action.cancel') }),
jsx(Button, { size: 'sm', disabled: revoke.isPending, onClick: () => revoke.mutate(prefix), children: t('action.confirm') })
]
})
: jsx(Button, { size: 'sm', variant: 'outline', onClick: () => setConfirmPrefix(prefix), children: t('session.revoke') })
]
}, prefix)
})
})
})
]
})
}
}
function createActivity(ctx) {
return function Activity() {
const t = usePluginI18n(PLUGIN_ID)
const profile = useValue(host.state.profile)
const query = useQuery({
queryKey: profileQueryKey(profile, 'bridge-activity'),
queryFn: () => ctx.rest('/bridge-activity?limit=100'),
retry: false
})
return jsxs('section', {
children: [
jsx(SectionHeader, {
title: t('activity.title'),
description: 'Recent Relay-owned bridge commands. No live subscription or background polling.'
}),
jsx(QueryState, {
query,
empty: data => list(data, 'activity').length === 0,
render: data => jsx('div', {
className: 'space-y-2',
children: list(data, 'activity').map((item, index) => jsxs('div', {
className: 'rounded-md border border-(--ui-stroke-tertiary) px-3 py-2',
children: [
jsxs('div', {
className: 'flex flex-wrap items-center gap-2',
children: [
jsx(StatusDot, { tone: item.ok === false || item.error ? 'bad' : 'good' }),
jsx('span', { className: 'text-xs font-medium text-(--ui-text-primary)', children: item.command || item.type || item.action || 'Bridge command' }),
jsx('span', { className: 'ml-auto text-[0.6875rem] text-(--ui-text-quaternary)', children: formatTime(item.timestamp || item.created_at || item.at) })
]
}),
jsx('p', { className: 'mt-1 break-words text-[0.6875rem] text-(--ui-text-tertiary)', children: text(item.summary || item.result || item.error || item.status) })
]
}, item.id || `${index}`))
})
})
]
})
}
}
function createMedia(ctx) {
return function Media() {
const t = usePluginI18n(PLUGIN_ID)
const profile = useValue(host.state.profile)
const query = useQuery({
queryKey: profileQueryKey(profile, 'media'),
queryFn: () => ctx.rest('/media'),
retry: false
})
return jsxs('section', {
children: [
jsx(SectionHeader, {
title: t('media.title'),
description: 'Sanitized Relay media metadata. Filesystem paths and bearer tokens are not requested.'
}),
jsx(QueryState, {
query,
empty: data => list(data, 'media').length === 0,
render: data => jsx('div', {
className: 'grid gap-2 sm:grid-cols-2',
children: list(data, 'media').map((item, index) => jsxs('div', {
className: 'rounded-md border border-(--ui-stroke-tertiary) px-3 py-3',
children: [
jsx('div', { className: 'truncate text-xs font-medium text-(--ui-text-primary)', children: item.filename || item.name || `Media ${index + 1}` }),
jsx('p', { className: 'mt-1 text-[0.6875rem] text-(--ui-text-tertiary)', children: `${text(item.content_type || item.mime_type)} · ${text(item.size_bytes || item.size, 'unknown size')}` }),
jsx('p', { className: 'mt-1 text-[0.6875rem] text-(--ui-text-quaternary)', children: `expires ${formatTime(item.expires_at)}` })
]
}, item.id || item.token_prefix || `${index}`))
})
})
]
})
}
}
function createRemoteAccess(ctx) {
return function RemoteAccess() {
const t = usePluginI18n(PLUGIN_ID)
const profile = useValue(host.state.profile)
const client = useQueryClient()
const key = profileQueryKey(profile, 'remote-access')
const status = useQuery({ queryKey: key, queryFn: () => ctx.rest('/remote-access/status'), retry: false })
const [url, setUrl] = useState('')
const [confirmAction, setConfirmAction] = useState(null)
const [result, setResult] = useState(null)
const refresh = () => void client.invalidateQueries({ queryKey: key })
const mutate = useMutation({
mutationFn: async action => {
if (action === 'enable') return ctx.rest('/remote-access/tailscale/enable', { method: 'POST', body: { stack: true } })
if (action === 'disable') return ctx.rest('/remote-access/tailscale/disable', { method: 'POST', body: { stack: true } })
if (action === 'save') return ctx.rest('/remote-access/public-url', { method: 'PUT', body: { url: url.trim() || null } })
if (action === 'probe') return ctx.rest('/remote-access/probe', { method: 'POST', body: { candidates: [url.trim()] } })
throw new Error('Unknown remote access action')
},
onSuccess: value => {
setResult(value)
setConfirmAction(null)
refresh()
}
})
const actionButton = (action, label, destructive = false) => confirmAction === action
? jsxs('span', {
className: 'inline-flex items-center gap-1',
children: [
jsx(Button, { size: 'sm', variant: 'ghost', onClick: () => setConfirmAction(null), children: t('action.cancel') }),
jsx(Button, { size: 'sm', disabled: mutate.isPending, onClick: () => mutate.mutate(action), children: t('action.confirm') })
]
})
: jsx(Button, { size: 'sm', variant: destructive ? 'outline' : 'outline', onClick: () => setConfirmAction(action), children: label })
return jsxs('section', {
children: [
jsx(SectionHeader, {
title: t('remote.title'),
description: 'All changes require a labeled action and a second confirmation.'
}),
jsx(QueryState, {
query: status,
empty: data => !data,
render: data => jsxs(Fragment, {
children: [
jsxs('div', {
className: 'grid gap-2 sm:grid-cols-2',
children: [
jsx(Field, { label: 'Tailscale', value: data.tailscale?.state || (data.tailscale?.enabled ? 'enabled' : 'disabled') }),
jsx(Field, { label: 'Public URL', value: data.public?.url || data.public_url }),
jsx(Field, { label: 'Secure Link', value: data.secure_link?.state || (data.secure_link?.enabled ? 'enabled' : 'disabled') }),
jsx(Field, { label: 'Upstream helper', value: data.upstream_canonical ? 'available' : 'not detected' })
]
}),
jsxs('div', {
className: 'mt-4 flex flex-wrap gap-2',
children: [
actionButton('enable', t('remote.enable')),
actionButton('disable', t('remote.disable'), true),
jsx(Button, { size: 'sm', variant: 'ghost', onClick: refresh, children: t('action.refresh') })
]
})
]
})
}),
jsxs('div', {
className: 'mt-5 rounded-md border border-(--ui-stroke-tertiary) p-3',
children: [
jsx('label', { className: 'text-xs font-medium text-(--ui-text-secondary)', htmlFor: 'hermes-relay-public-url', children: 'Public relay URL' }),
jsx(Input, {
id: 'hermes-relay-public-url',
className: 'mt-2',
placeholder: 'https://relay.example.com',
value: url,
onChange: event => setUrl(event.target.value)
}),
jsxs('div', {
className: 'mt-2 flex flex-wrap gap-2',
children: [
actionButton('save', t('remote.save')),
actionButton('probe', t('remote.probe'))
]
})
]
}),
mutate.error ? jsx(ErrorState, { title: 'Remote access action failed', description: friendlyError(mutate.error) }) : null,
result ? jsx('pre', { className: 'mt-3 max-h-48 overflow-auto whitespace-pre-wrap rounded-md border border-(--ui-stroke-tertiary) p-3 text-[0.6875rem] text-(--ui-text-tertiary)', children: JSON.stringify(result, null, 2) }) : null
]
})
}
}
function createRelayPane(ctx) {
const Management = createManagement(ctx)
const Activity = createActivity(ctx)
const Media = createMedia(ctx)
const RemoteAccess = createRemoteAccess(ctx)
return function RelayPane() {
const t = usePluginI18n(PLUGIN_ID)
const [tab, setTab] = useState('management')
const tabs = [
['management', 'server', t('management.title')],
['activity', 'pulse', t('activity.title')],
['media', 'file-media', t('media.title')],
['remote', 'remote-explorer', t('remote.title')]
]
const content = tab === 'activity'
? jsx(Activity, {})
: tab === 'media'
? jsx(Media, {})
: tab === 'remote'
? jsx(RemoteAccess, {})
: jsx(Management, {})
return jsxs('div', {
className: 'flex h-full min-h-0 flex-col',
children: [
jsxs('header', {
className: 'border-b border-(--ui-stroke-tertiary) px-4 py-3',
children: [
jsxs('div', {
className: 'flex items-center gap-2',
children: [
jsx(Codicon, { name: 'radio-tower', size: '1rem' }),
jsx('h1', { className: 'text-sm font-semibold text-(--ui-text-primary)', children: t('pane.title') })
]
}),
jsx('p', { className: 'mt-1 text-xs text-(--ui-text-tertiary)', children: t('pane.description') }),
jsx('nav', {
'aria-label': 'Hermes Relay sections',
className: 'mt-3 flex flex-wrap gap-1',
children: tabs.map(([id, icon, label]) => jsx(Button, {
size: 'sm',
variant: tab === id ? 'secondary' : 'ghost',
onClick: () => setTab(id),
children: jsxs(Fragment, { children: [jsx(Codicon, { name: icon, size: '0.75rem' }), label] })
}, id))
})
]
}),
jsx('main', { className: 'min-h-0 flex-1 overflow-y-auto p-4', children: content })
]
})
}
}
function RelayStatus({ open }) {
const t = usePluginI18n(PLUGIN_ID)
return jsx(Tip, {
label: t('status.detail'),
children: jsxs('button', {
type: 'button',
className: 'inline-flex h-full items-center gap-1.5 px-1.5 text-[0.6875rem] text-(--ui-text-tertiary) hover:bg-(--chrome-action-hover) hover:text-(--ui-text-primary)',
onClick: open,
children: [jsx(StatusDot, { tone: 'muted' }), jsx('span', { children: t('status.label') })]
})
})
}
const plugin = {
id: PLUGIN_ID,
name: 'Hermes Relay',
description: 'Profile-scoped Hermes-Relay management through the official Desktop Plugin SDK.',
defaultEnabled: false,
register(ctx) {
ctx.i18n.register(messages)
const RelayPane = createRelayPane(ctx)
let paneRegistered = false
const open = () => {
if (!paneRegistered) {
ctx.register({
id: 'management',
area: PANES_AREA,
title: ctx.i18n.t('pane.title'),
data: {
closeBehavior: 'dismiss',
placement: 'right',
dock: { pane: 'workspace', pos: 'right' },
width: 'min(420px, 42vw)'
},
render: () => jsx(RelayPane, {})
})
paneRegistered = true
}
ctx.panes.reveal('management')
}
ctx.onDispose(() => {
paneRegistered = false
})
ctx.registerMany([
{
id: 'nav',
area: SIDEBAR_NAV_AREA,
order: 75,
data: { codicon: 'radio-tower', label: ctx.i18n.t('nav.label'), onSelect: open }
},
{
id: 'status',
area: STATUSBAR_AREAS.right,
order: 110,
render: () => jsx(RelayStatus, { open })
},
{
id: 'open',
area: PALETTE_AREA,
data: {
id: 'hermes-relay.open',
label: ctx.i18n.t('action.open'),
keywords: ['relay', 'phone', 'paired devices', 'bridge', 'media', 'remote access'],
run: open
}
}
])
}
}
export default plugin
+40
View File
@@ -0,0 +1,40 @@
const sdk = `
export const PANES_AREA = 'panes'
export const SIDEBAR_NAV_AREA = 'sidebar.nav'
export const STATUSBAR_AREAS = { right: 'statusBar.right' }
export const PALETTE_AREA = 'palette'
export const Badge = 'Badge'
export const Button = 'Button'
export const Codicon = 'Codicon'
export const EmptyState = 'EmptyState'
export const ErrorState = 'ErrorState'
export const Input = 'Input'
export const StatusDot = 'StatusDot'
export const Tip = 'Tip'
export const host = { state: { profile: { get: () => 'default' } }, notify: () => {} }
export const useMutation = () => ({ mutate: () => {}, isPending: false })
export const usePluginI18n = () => key => key
export const useQuery = () => ({ data: null, error: null, isPending: false, refetch: () => {} })
export const useQueryClient = () => ({ invalidateQueries: () => {} })
export const useValue = atom => atom.get()
`
const react = `export const useState = initial => [initial, () => {}]`
const jsx = `
export const Fragment = Symbol.for('fixture.fragment')
export const jsx = (type, props) => ({ type, props: props || {} })
export const jsxs = jsx
`
export async function resolve(specifier, context, nextResolve) {
if (specifier === '@hermes/plugin-sdk') {
return { shortCircuit: true, url: `data:text/javascript,${encodeURIComponent(sdk)}` }
}
if (specifier === 'react') {
return { shortCircuit: true, url: `data:text/javascript,${encodeURIComponent(react)}` }
}
if (specifier === 'react/jsx-runtime') {
return { shortCircuit: true, url: `data:text/javascript,${encodeURIComponent(jsx)}` }
}
return nextResolve(specifier, context)
}
+127
View File
@@ -0,0 +1,127 @@
import assert from 'node:assert/strict'
import { readFile } from 'node:fs/promises'
import { dirname, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
import test from 'node:test'
import plugin, { profileQueryKey } from '../plugin.js'
const here = dirname(fileURLToPath(import.meta.url))
function harness() {
const contributions = []
const disposed = []
const revealed = []
const cleanups = []
const ctx = {
i18n: {
register() {
const dispose = () => disposed.push('i18n')
cleanups.push(dispose)
return dispose
},
t: key => key
},
onDispose(fn) {
cleanups.push(fn)
},
os: { writeClipboard: async () => true },
panes: { reveal: id => revealed.push(id) },
register(contribution) {
contributions.push(contribution)
const dispose = () => disposed.push(contribution.id)
cleanups.push(dispose)
return dispose
},
registerMany(batch) {
contributions.push(...batch)
const dispose = () => disposed.push(...batch.map(item => item.id))
cleanups.push(dispose)
return dispose
},
rest: async () => ({})
}
return { cleanups, contributions, ctx, disposed, revealed }
}
test('ships opt-in and registers no pane or network work at load', () => {
const state = harness()
let restCalls = 0
state.ctx.rest = async () => {
restCalls += 1
return {}
}
plugin.register(state.ctx)
assert.equal(plugin.defaultEnabled, false)
assert.deepEqual(state.contributions.map(item => item.id), ['nav', 'status', 'open'])
assert.equal(state.contributions.some(item => item.area === 'panes'), false)
assert.equal(state.revealed.length, 0)
assert.equal(restCalls, 0)
})
test('startup, reconnect, profile-change, and reload registration never reveal the pane', () => {
for (const lifecycle of ['startup', 'reconnect', 'profile-change', 'reload']) {
const state = harness()
plugin.register(state.ctx)
assert.equal(state.contributions.some(item => item.area === 'panes'), false, lifecycle)
assert.deepEqual(state.revealed, [], lifecycle)
}
})
test('every labeled entry point opens lazily and repeated opens reuse the pane', () => {
const state = harness()
plugin.register(state.ctx)
const nav = state.contributions.find(item => item.id === 'nav')
const status = state.contributions.find(item => item.id === 'status')
const command = state.contributions.find(item => item.id === 'open')
nav.data.onSelect()
const statusElement = status.render()
const renderedStatus = statusElement.type(statusElement.props)
renderedStatus.props.children.props.onClick()
command.data.run()
const panes = state.contributions.filter(item => item.area === 'panes')
assert.equal(panes.length, 1)
assert.equal(panes[0].data.closeBehavior, 'dismiss')
assert.equal(panes[0].data.placement, 'right')
assert.deepEqual(state.revealed, ['management', 'management', 'management'])
})
test('unload disposes entry points, lazy pane, locale bundle, and module state', () => {
const state = harness()
plugin.register(state.ctx)
state.contributions.find(item => item.id === 'nav').data.onSelect()
for (const cleanup of [...state.cleanups].reverse()) cleanup()
assert.ok(state.disposed.includes('management'))
assert.ok(state.disposed.includes('nav'))
assert.ok(state.disposed.includes('i18n'))
})
test('query keys isolate cached backend state by active profile', () => {
assert.deepEqual(profileQueryKey('default', 'overview'), ['hermes-relay', 'default', 'overview'])
assert.notDeepEqual(profileQueryKey('work', 'sessions'), profileQueryKey('personal', 'sessions'))
})
test('unified package uses only runtime-plugin imports and contains no auto-open primitive', async () => {
const source = await readFile(resolve(here, '..', 'plugin.js'), 'utf8')
const imports = [...source.matchAll(/from\s+['\"]([^'\"]+)['\"]/g)].map(match => match[1])
assert.deepEqual([...new Set(imports)].sort(), ['@hermes/plugin-sdk', 'react', 'react/jsx-runtime'])
assert.equal(/setInterval|setTimeout|host\.navigate|window\.location|focus\s*\(/.test(source), false)
assert.match(source, /const open = \(\) => \{/)
assert.match(source, /paneRegistered = true/)
})
test('desktop half is bundled beside the existing dashboard half', async () => {
const pluginRoot = resolve(here, '..', '..')
const manifest = await readFile(resolve(pluginRoot, 'plugin.yaml'), 'utf8')
const dashboard = await readFile(resolve(pluginRoot, 'dashboard', 'manifest.json'), 'utf8')
const desktop = await readFile(resolve(pluginRoot, 'desktop', 'plugin.js'), 'utf8')
assert.match(manifest, /^name:\s+hermes-relay/m)
assert.equal(JSON.parse(dashboard).name, 'hermes-relay')
assert.match(desktop, /id:\s*PLUGIN_ID/)
})
+1 -1
View File
@@ -1,6 +1,6 @@
name: hermes-relay
manifest_version: 1
version: 1.7.0
version: 1.8.0
description: "Hermes-Relay plugin for QR pairing, relay sessions, dashboard management, remote desktop/phone tooling, and optional legacy compatibility diagnostics. Standard chat, Manage, and dashboard voice remain vanilla upstream Hermes surfaces."
author: Axiom Labs
# All three are OPTIONAL — only needed if you use the relay's extra /
+1 -1
View File
@@ -19,7 +19,7 @@ See ``plugin/relay/server.py`` for the aiohttp server,
# Desktop releases use desktop/package.json and desktop-v* tags. The /health endpoint
# reports this plugin version, and stale values make live diagnosis harder than
# it should be.
__version__ = "1.7.0"
__version__ = "1.8.0"
from .server import create_app, main # noqa: E402 — must come after __version__
+149 -1
View File
@@ -22,7 +22,7 @@ Two jobs over the same WSS ``desktop`` envelope stream:
client auth re-advertises if the relay restarts.
Wire envelopes (frozen — do not rename fields):
* ``desktop.command`` — server → client: ``{request_id, tool, args}``
* ``desktop.command`` — server → client: ``{request_id, tool, args, control_session?}``
* ``desktop.response`` — client → server: ``{request_id, status, result}``
* ``desktop.status`` — client → server: ``{advertised_tools, host?, platform?, cwd?, ...}``
* ``desktop.workspace`` — client → server: opaque dict (``cwd`` / ``git_root`` / ``git_branch`` / ...)
@@ -58,12 +58,33 @@ _REDACT_KEYS = frozenset({"password", "token", "secret", "otp", "bearer", "api_k
# Cap for the recent-commands ring buffer.
RECENT_COMMANDS_MAX = 100
CONTROL_SESSIONS_MAX = 256
CONTROL_SESSION_IDLE_SECONDS = 900.0
class DesktopError(Exception):
"""Raised when a desktop command cannot be dispatched or times out."""
@dataclass(frozen=True)
class DesktopRequesterContext:
"""Trusted caller context supplied by the loopback HTTP boundary."""
requester_device_id: str | None = None
chat_session_id: str | None = None
run_id: str | None = None
profile: str | None = None
@dataclass
class _ControlSessionRecord:
session_id: str
touched_at: float
target_ws: web.WebSocketResponse
target_device_id: str
expiry_task: asyncio.Task[None]
def _redact_args(value: Any) -> Any:
"""Return a copy of ``value`` with sensitive-key values replaced by
``"[redacted]"``. Recurses into nested dicts and lists.
@@ -190,6 +211,108 @@ class DesktopHandler:
# registry. Cleared per-ws in :meth:`detach_ws` so disconnected
# clients don't leak session structs.
self._sessions: dict[web.WebSocketResponse, DesktopSession] = {}
# Stable opaque ids for active computer-control runs. Callers provide
# only trusted context fields; this handler always owns the id and
# binds every emitted identity to the selected target + request id.
self._control_sessions: dict[tuple[str, str, str, str, str], _ControlSessionRecord] = {}
async def _send_control_session_end(
self,
record: _ControlSessionRecord,
reason: str,
) -> None:
if record.target_ws.closed:
return
await record.target_ws.send_str(json.dumps({
"channel": "desktop",
"type": "desktop.control_session_end",
"payload": {
"version": 1,
"id": record.session_id,
"target_device_id": record.target_device_id,
"reason": reason[:256],
},
}))
async def _expire_control_session(
self,
key: tuple[str, str, str, str, str],
session_id: str,
) -> None:
try:
while True:
await asyncio.sleep(CONTROL_SESSION_IDLE_SECONDS)
record = self._control_sessions.get(key)
if record is None or record.session_id != session_id:
return
if time.monotonic() - record.touched_at < CONTROL_SESSION_IDLE_SECONDS:
continue
self._control_sessions.pop(key, None)
await self._send_control_session_end(record, "control session idle timeout")
return
except asyncio.CancelledError:
return
async def _control_session(
self,
*,
request_id: str,
target_device_id: str,
target_ws: web.WebSocketResponse,
requester: DesktopRequesterContext | None,
) -> dict[str, Any] | None:
context = requester or DesktopRequesterContext()
# A server-owned UUID is not useful authority by itself. Require an
# executor-authenticated requester and run before advertising the
# identity; older/unauthenticated callers stay in compatibility mode.
if not context.requester_device_id or not context.run_id:
return None
now = time.monotonic()
key = (
context.requester_device_id,
context.chat_session_id or "",
context.run_id or "",
context.profile or "",
target_device_id or "legacy-desktop",
)
# One requester/chat/profile/target may own only one active run. End
# the prior authority before minting a replacement for a new run.
transitions = [
(other_key, record)
for other_key, record in self._control_sessions.items()
if other_key[0] == key[0] and other_key[1] == key[1]
and other_key[3] == key[3] and other_key[4] == key[4]
and other_key[2] != key[2]
]
for other_key, record in transitions:
self._control_sessions.pop(other_key, None)
record.expiry_task.cancel()
await self._send_control_session_end(record, "requester run changed")
current = self._control_sessions.get(key)
if current is None:
if len(self._control_sessions) >= CONTROL_SESSIONS_MAX:
oldest = min(self._control_sessions, key=lambda item: self._control_sessions[item].touched_at)
evicted = self._control_sessions.pop(oldest)
evicted.expiry_task.cancel()
await self._send_control_session_end(evicted, "control session capacity eviction")
session_id = f"control-{uuid.uuid4()}"
task = asyncio.create_task(self._expire_control_session(key, session_id))
self._control_sessions[key] = _ControlSessionRecord(
session_id, now, target_ws, key[4], task
)
else:
session_id = current.session_id
current.touched_at = now
return {
"version": 1,
"id": session_id,
"request_id": request_id,
"requester_device_id": key[0],
"target_device_id": key[4],
**({"chat_session_id": context.chat_session_id} if context.chat_session_id else {}),
**({"run_id": context.run_id} if context.run_id else {}),
**({"profile": context.profile} if context.profile else {}),
}
# ── Envelope dispatch ────────────────────────────────────────────────
@@ -389,6 +512,7 @@ class DesktopHandler:
method: str,
args: dict[str, Any] | None = None,
device: str | None = None,
requester: DesktopRequesterContext | None = None,
) -> dict[str, Any]:
"""Dispatch a ``desktop_*`` tool call to the connected client.
@@ -433,6 +557,15 @@ class DesktopHandler:
else None
),
}
if method.startswith("desktop_computer_"):
control_session = await self._control_session(
request_id=request_id,
target_device_id=str(command_payload["target_device_id"] or "legacy-desktop"),
target_ws=ws,
requester=requester,
)
if control_session is not None:
command_payload["control_session"] = control_session
logger.info(
"desktop >>> %s args=%s",
method,
@@ -736,6 +869,18 @@ class DesktopHandler:
"desktop: session detached (had workspace from %s)",
session.workspace_context.get("hostname", "?"),
)
if session is not None:
target_device_id = (
session.device_id
or str(session.client_status.get("device_id", "") or "").strip()
)
if target_device_id:
removed = [
key for key in self._control_sessions if key[4] == target_device_id
]
for key in removed:
record = self._control_sessions.pop(key)
record.expiry_task.cancel()
# Tool-routing cleanup — only if this ws was the latched client.
if self.client_ws is ws:
@@ -774,6 +919,9 @@ class DesktopHandler:
self.client_ws = None
self.advertised_tools = set()
self._sessions.clear()
for record in self._control_sessions.values():
record.expiry_task.cancel()
self._control_sessions.clear()
await self._fail_pending("Relay server shutting down")
+33 -1
View File
@@ -1273,8 +1273,40 @@ async def handle_desktop_dispatch(request: web.Request) -> web.Response:
device = args.pop("device", None) or args.pop("device_id", None)
if not isinstance(device, str):
device = None
# These headers are emitted by the in-process Hermes tool wrapper and are
# not part of the model-facing schema. The route is loopback-only; the
# Relay still mints the opaque id and binds target/request fields itself.
from .channels.desktop import DesktopRequesterContext
def _context_header(name: str) -> str | None:
value = request.headers.get(name, "").strip()
if not value:
return None
return "".join(ch for ch in value if ch.isprintable())[:256] or None
chat_session_id = _context_header("X-Hermes-Relay-Chat-Session")
requester_device_id: str | None = None
auth_header = request.headers.get("Authorization", "")
if auth_header.startswith("Bearer "):
# A supplied bearer must validate; never downgrade a bad credential to
# loopback trust. This yields the real paired requester device.
_server, authenticated_session = _require_bearer_session(request)
requester_device_id = authenticated_session.device_id or None
elif chat_session_id:
# Standard plugin calls are loopback-only. Bind their principal to the
# executor-owned Hermes session rather than inventing a host identity.
# Same-user localhost processes are inside this existing trust boundary.
requester_device_id = f"hermes-agent:{chat_session_id}"
requester = DesktopRequesterContext(
requester_device_id=requester_device_id,
chat_session_id=chat_session_id,
run_id=_context_header("X-Hermes-Relay-Run-Id"),
profile=_context_header("X-Hermes-Relay-Profile"),
)
try:
result = await server.desktop.handle_command(tool_name, args, device=device)
result = await server.desktop.handle_command(
tool_name, args, device=device, requester=requester
)
return web.json_response(result)
except Exception as exc: # DesktopError or asyncio.TimeoutError
msg = str(exc) or exc.__class__.__name__
+260 -2
View File
@@ -9,7 +9,11 @@ from dataclasses import dataclass
from typing import Any
from unittest.mock import AsyncMock, Mock, patch
from plugin.relay.channels.desktop import DesktopError, DesktopHandler
from plugin.relay.channels.desktop import (
DesktopError,
DesktopHandler,
DesktopRequesterContext,
)
from plugin.relay.server import handle_desktop_dispatch
from plugin.tools import desktop_tool
@@ -35,7 +39,11 @@ def _status(host: str) -> dict[str, Any]:
"type": "desktop.status",
"payload": {
"host": host,
"advertised_tools": ["desktop_powershell", "desktop_read_file"],
"advertised_tools": [
"desktop_powershell",
"desktop_read_file",
"desktop_computer_snapshot",
],
},
}
@@ -119,6 +127,30 @@ class DesktopMultiDeviceTests(unittest.IsolatedAsyncioTestCase):
self.assertEqual(post.call_args.kwargs["json"]["device"], "desktop-1")
self.assertEqual(post.call_args.kwargs["json"]["script"], "'ok'")
async def test_tool_dispatch_uses_executor_context_and_drops_model_identity(self) -> None:
response = Mock(status_code=200)
response.json.return_value = {"ok": True, "result": {}}
with patch.object(desktop_tool.requests, "post", return_value=response) as post:
desktop_tool._HANDLERS["desktop_computer_action"](
{
"action": "click",
"x": 10,
"y": 20,
"control_session": {"id": "model-forged"},
"control_context": {"run_id": "model-forged"},
},
session_id="chat-authenticated",
task_id="run-authenticated",
profile="default",
)
body = post.call_args.kwargs["json"]
headers = post.call_args.kwargs["headers"]
self.assertNotIn("control_session", body)
self.assertNotIn("control_context", body)
self.assertEqual(headers["X-Hermes-Relay-Chat-Session"], "chat-authenticated")
self.assertEqual(headers["X-Hermes-Relay-Run-Id"], "run-authenticated")
self.assertEqual(headers["X-Hermes-Relay-Profile"], "default")
async def test_http_dispatch_strips_selector_before_client_forwarding(self) -> None:
desktop = Mock()
desktop.handle_command = AsyncMock(return_value={"ok": True, "result": {}})
@@ -126,6 +158,7 @@ class DesktopMultiDeviceTests(unittest.IsolatedAsyncioTestCase):
remote="127.0.0.1",
app={"server": Mock(desktop=desktop)},
match_info={"tool_name": "desktop_powershell"},
headers={},
)
request.json = AsyncMock(
return_value={"script": "'ok'", "device": "desktop-1"}
@@ -138,8 +171,233 @@ class DesktopMultiDeviceTests(unittest.IsolatedAsyncioTestCase):
"desktop_powershell",
{"script": "'ok'"},
device="desktop-1",
requester=DesktopRequesterContext(),
)
async def test_http_dispatch_binds_loopback_executor_context(self) -> None:
desktop = Mock()
desktop.handle_command = AsyncMock(return_value={"ok": True, "result": {}})
request = Mock(
remote="127.0.0.1",
app={"server": Mock(desktop=desktop)},
match_info={"tool_name": "desktop_computer_snapshot"},
headers={
"X-Hermes-Relay-Chat-Session": "chat-1",
"X-Hermes-Relay-Run-Id": "run-1",
"X-Hermes-Relay-Profile": "default",
},
)
request.json = AsyncMock(return_value={"device": "desktop-1"})
response = await handle_desktop_dispatch(request)
self.assertEqual(response.status, 200)
desktop.handle_command.assert_awaited_once_with(
"desktop_computer_snapshot",
{},
device="desktop-1",
requester=DesktopRequesterContext(
requester_device_id="hermes-agent:chat-1",
chat_session_id="chat-1",
run_id="run-1",
profile="default",
),
)
async def test_http_dispatch_prefers_authenticated_paired_requester(self) -> None:
desktop = Mock()
desktop.handle_command = AsyncMock(return_value={"ok": True, "result": {}})
server = Mock(desktop=desktop)
request = Mock(
remote="127.0.0.1",
app={"server": server},
match_info={"tool_name": "desktop_computer_snapshot"},
headers={
"Authorization": "Bearer authenticated-token",
"X-Hermes-Relay-Chat-Session": "chat-1",
"X-Hermes-Relay-Run-Id": "run-1",
},
)
request.json = AsyncMock(return_value={"device": "desktop-1"})
paired_session = Mock(device_id="paired-phone-1")
with patch(
"plugin.relay.server._require_bearer_session",
return_value=(server, paired_session),
):
response = await handle_desktop_dispatch(request)
self.assertEqual(response.status, 200)
requester = desktop.handle_command.await_args.kwargs["requester"]
self.assertEqual(requester.requester_device_id, "paired-phone-1")
self.assertEqual(requester.chat_session_id, "chat-1")
self.assertEqual(requester.run_id, "run-1")
async def test_computer_control_session_is_server_owned_stable_and_request_bound(self) -> None:
handler, office, _laptop = await _register_two()
requester = DesktopRequesterContext(
requester_device_id="paired-agent-1",
chat_session_id="chat-1",
run_id="run-1",
profile="default",
)
async def dispatch_once() -> dict[str, Any]:
task = asyncio.create_task(
handler.handle_command(
"desktop_computer_snapshot",
{},
device="desktop-1",
requester=requester,
)
)
await asyncio.sleep(0)
payload = office.sent[-1]["payload"]
await handler.handle(
office, # type: ignore[arg-type]
{
"channel": "desktop",
"type": "desktop.response",
"payload": {"request_id": payload["request_id"], "ok": True, "result": {}},
},
)
await asyncio.wait_for(task, timeout=1)
return payload
first = await dispatch_once()
second = await dispatch_once()
first_control = first["control_session"]
second_control = second["control_session"]
self.assertEqual(first_control["version"], 1)
self.assertRegex(first_control["id"], r"^control-[0-9a-f-]+$")
self.assertEqual(first_control["id"], second_control["id"])
self.assertNotEqual(first_control["request_id"], second_control["request_id"])
self.assertEqual(first_control["request_id"], first["request_id"])
self.assertEqual(first_control["requester_device_id"], "paired-agent-1")
self.assertEqual(first_control["target_device_id"], "desktop-1")
self.assertEqual(first_control["chat_session_id"], "chat-1")
self.assertEqual(first_control["run_id"], "run-1")
self.assertEqual(len(handler._control_sessions), 1)
await handler.detach_ws(office, "test disconnect") # type: ignore[arg-type]
self.assertEqual(handler._control_sessions, {})
async def test_computer_command_omits_identity_without_authoritative_run(self) -> None:
handler, office, _laptop = await _register_two()
task = asyncio.create_task(
handler.handle_command(
"desktop_computer_snapshot",
{},
device="desktop-1",
requester=DesktopRequesterContext(requester_device_id="paired-agent-1"),
)
)
await asyncio.sleep(0)
payload = office.sent[-1]["payload"]
self.assertNotIn("control_session", payload)
await handler.handle(
office, # type: ignore[arg-type]
{
"channel": "desktop",
"type": "desktop.response",
"payload": {"request_id": payload["request_id"], "ok": True, "result": {}},
},
)
await asyncio.wait_for(task, timeout=1)
async def test_new_requester_run_ends_prior_control_session(self) -> None:
handler, office, _laptop = await _register_two()
async def dispatch(run_id: str) -> dict[str, Any]:
task = asyncio.create_task(handler.handle_command(
"desktop_computer_snapshot", {}, device="desktop-1",
requester=DesktopRequesterContext(
requester_device_id="paired-agent-1",
chat_session_id="chat-1",
run_id=run_id,
profile="default",
),
))
await asyncio.sleep(0)
command = next(
item for item in reversed(office.sent)
if item["type"] == "desktop.command"
)
await handler.handle(office, { # type: ignore[arg-type]
"channel": "desktop", "type": "desktop.response",
"payload": {"request_id": command["payload"]["request_id"], "ok": True, "result": {}},
})
await task
return command["payload"]
first = await dispatch("run-1")
second = await dispatch("run-2")
ended = next(item for item in office.sent if item["type"] == "desktop.control_session_end")
self.assertEqual(ended["payload"]["version"], 1)
self.assertEqual(ended["payload"]["id"], first["control_session"]["id"])
self.assertEqual(ended["payload"]["target_device_id"], "desktop-1")
self.assertEqual(ended["payload"]["reason"], "requester run changed")
self.assertNotEqual(first["control_session"]["id"], second["control_session"]["id"])
async def test_idle_control_session_emits_end_event(self) -> None:
with patch("plugin.relay.channels.desktop.CONTROL_SESSION_IDLE_SECONDS", 0.01):
handler, office, _laptop = await _register_two()
task = asyncio.create_task(handler.handle_command(
"desktop_computer_snapshot", {}, device="desktop-1",
requester=DesktopRequesterContext(
requester_device_id="paired-agent-1", run_id="run-1"
),
))
await asyncio.sleep(0)
command = office.sent[-1]
await handler.handle(office, { # type: ignore[arg-type]
"channel": "desktop", "type": "desktop.response",
"payload": {"request_id": command["payload"]["request_id"], "ok": True, "result": {}},
})
await task
await asyncio.sleep(0.03)
ended = next(item for item in office.sent if item["type"] == "desktop.control_session_end")
self.assertEqual(ended["payload"]["id"], command["payload"]["control_session"]["id"])
self.assertEqual(ended["payload"]["reason"], "control session idle timeout")
async def test_control_session_capacity_eviction_emits_end_event(self) -> None:
with patch("plugin.relay.channels.desktop.CONTROL_SESSIONS_MAX", 1):
handler, office, _laptop = await _register_two()
async def dispatch(requester_device_id: str) -> dict[str, Any]:
task = asyncio.create_task(handler.handle_command(
"desktop_computer_snapshot", {}, device="desktop-1",
requester=DesktopRequesterContext(
requester_device_id=requester_device_id,
run_id="run-1",
),
))
await asyncio.sleep(0)
command = next(
item for item in reversed(office.sent)
if item["type"] == "desktop.command"
)
await handler.handle(office, { # type: ignore[arg-type]
"channel": "desktop", "type": "desktop.response",
"payload": {
"request_id": command["payload"]["request_id"],
"ok": True,
"result": {},
},
})
await task
return command["payload"]
first = await dispatch("paired-agent-1")
await dispatch("paired-agent-2")
ended = next(
item for item in office.sent
if item["type"] == "desktop.control_session_end"
)
self.assertEqual(ended["payload"]["version"], 1)
self.assertEqual(ended["payload"]["id"], first["control_session"]["id"])
self.assertEqual(ended["payload"]["target_device_id"], "desktop-1")
self.assertEqual(ended["payload"]["reason"], "control session capacity eviction")
async def test_untargeted_command_fails_closed_with_multiple_pcs(self) -> None:
handler, _office, _laptop = await _register_two()
with self.assertRaisesRegex(DesktopError, "pass device or device_id explicitly"):
+70 -2
View File
@@ -78,6 +78,34 @@ import requests
_DESKTOP_TARGET: ContextVar[str | None] = ContextVar("desktop_target", default=None)
_DESKTOP_CALL_CONTEXT: ContextVar[dict[str, str]] = ContextVar(
"desktop_call_context", default={}
)
_CONTROL_CONTEXT_HEADERS = {
"chat_session_id": "X-Hermes-Relay-Chat-Session",
"run_id": "X-Hermes-Relay-Run-Id",
"profile": "X-Hermes-Relay-Profile",
}
def _bounded_context_value(value: Any, limit: int = 256) -> str | None:
if not isinstance(value, str):
return None
cleaned = "".join(ch for ch in value.strip() if ch.isprintable())
return cleaned[:limit] or None
def _trusted_call_context(kwargs: dict[str, Any]) -> dict[str, str]:
"""Extract executor-owned identity without consulting model arguments."""
values = {
"chat_session_id": _bounded_context_value(kwargs.get("session_id")),
"run_id": _bounded_context_value(kwargs.get("task_id")),
"profile": _bounded_context_value(kwargs.get("profile"))
or _bounded_context_value(os.getenv("HERMES_PROFILE")),
}
return {key: value for key, value in values.items() if value is not None}
# ── Config ────────────────────────────────────────────────────────────────────
@@ -115,11 +143,16 @@ def _post(path: str, payload: dict, *, timeout: Optional[float] = None) -> dict:
target = _DESKTOP_TARGET.get()
if target:
payload["device"] = target
headers = _auth_headers()
for field, value in _DESKTOP_CALL_CONTEXT.get().items():
header = _CONTROL_CONTEXT_HEADERS.get(field)
if header:
headers[header] = value
try:
r = requests.post(
f"{_relay_url()}{path}",
json=payload,
headers=_auth_headers(),
headers=headers,
timeout=_timeout() if timeout is None else timeout,
)
except requests.RequestException as exc:
@@ -588,8 +621,12 @@ def desktop_computer_screenshot(
include_cursor: bool = True,
redact_sensitive: bool = True,
save_to: Optional[str] = None,
pid: Optional[int] = None,
window_id: Optional[int] = None,
query: Optional[str] = None,
include_screenshot: bool = True,
) -> str:
"""[EXPERIMENTAL] Capture a desktop screenshot in observe mode."""
"""[EXPERIMENTAL] Capture a display or a structured CUA window snapshot."""
payload: dict[str, Any] = {
"display": display,
"include_cursor": bool(include_cursor),
@@ -599,6 +636,15 @@ def desktop_computer_screenshot(
payload["region"] = region
if save_to is not None:
payload["save_to"] = save_to
for key, value in {
"pid": pid,
"window_id": window_id,
"query": query,
}.items():
if value is not None:
payload[key] = value
if pid is not None or window_id is not None or not include_screenshot:
payload["include_screenshot"] = bool(include_screenshot)
data = _post("/desktop/desktop_computer_screenshot", payload)
return json.dumps(data)
@@ -1160,6 +1206,10 @@ _SCHEMAS: dict[str, dict[str, Any]] = {
"type": "string",
"description": "Optional path on the desktop client to save the PNG instead of returning bytes.",
},
"pid": {"type": "integer", "minimum": 1},
"window_id": {"type": "integer", "minimum": 1},
"query": {"type": "string", "description": "Optional accessibility-tree projection query."},
"include_screenshot": {"type": "boolean", "default": True},
},
},
},
@@ -1189,6 +1239,10 @@ _SCHEMAS: dict[str, dict[str, Any]] = {
"type",
"type_text",
"wait",
"click_element",
"set_value",
"press_key",
"scroll_element",
],
},
"coordinate": {
@@ -1227,6 +1281,13 @@ _SCHEMAS: dict[str, dict[str, Any]] = {
"type": "string",
"description": "Human-readable reason for the requested action.",
},
"pid": {"type": "integer", "minimum": 1},
"window_id": {"type": "integer", "minimum": 1},
"snapshot_token": {"type": "string"},
"snapshot_generation": {"type": "string"},
"value": {"type": "string"},
"direction": {"type": "string", "enum": ["up", "down", "left", "right"]},
"amount": {"type": "integer", "minimum": 1, "maximum": 20},
},
"required": ["action"],
"additionalProperties": False,
@@ -1333,10 +1394,17 @@ def _targeted_handler(handler: Any) -> Any:
def invoke(args: dict[str, Any], **kwargs: Any) -> Any:
call_args = dict(args)
target = call_args.pop("device", None) or call_args.pop("device_id", None)
# Identity is never a model-owned argument. Drop every reserved shape
# before dispatch, then obtain authoritative context from Hermes'
# executor kwargs instead.
call_args.pop("control_session", None)
call_args.pop("control_context", None)
token = _DESKTOP_TARGET.set(str(target).strip() if target else None)
context_token = _DESKTOP_CALL_CONTEXT.set(_trusted_call_context(kwargs))
try:
return handler(call_args, **kwargs)
finally:
_DESKTOP_CALL_CONTEXT.reset(context_token)
_DESKTOP_TARGET.reset(token)
return invoke
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hermes-relay"
version = "1.7.0"
version = "1.8.0"
description = "Hermes-Relay plugin — Android device control toolset, QR pairing CLI, and WSS relay server for hermes-agent"
requires-python = ">=3.11"
dependencies = [
+11
View File
@@ -129,6 +129,17 @@ The Windows one-liner installs the checksum-verified CLI and UI bundle by defaul
The CLI section can **Open terminal** at a normal prompt, **Open Hermes CLI** directly into the paired Hermes TUI, **View daemon log**, or **Run diagnostics**. Updates manage the CLI and UI bundle together. **Help & About** shows the UI, CLI, and connected Relay versions and links to the [desktop documentation](https://hermes-relay.dev/docs/desktop/), [troubleshooting guide](https://hermes-relay.dev/docs/desktop/troubleshooting/), [release notes](https://github.com/Codename-11/hermes-relay/releases?q=desktop), logs, and diagnostics. External links open in the default browser.
**Settings → Computer control** reports whether the preferred CUA Driver engine
is absent, incompatible, degraded, or ready. A ready runtime handles structured
actions against a named
window in the background, and display a separate animated virtual cursor for
each agent control session without moving the physical mouse. CUA remains
optional: Windows input is the explicit compatibility backend, this release
keeps foreground escalation disabled, and the chosen backend cannot change
during an active control session. The same card provides explicit **Install**,
**Check**, and **Update** actions; none run automatically.
See [Computer-use engines](./tools.md#computer-use-engines).
## Why both shell AND chat modes?
They're not the same thing:
+39
View File
@@ -60,6 +60,45 @@ hermes-relay ui # same as: hermes-relay ui open
hermes-relay ui status
```
### Preferred CUA Driver
CUA Driver is an optional Windows computer-control engine and is **not** bundled
with either Hermes-Relay installer surface. Hermes resolves only the canonical
upstream package at
`%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe`; it does not use an
arbitrary PATH shim.
Hermes exposes the verified upstream lifecycle only after an explicit local
choice. Read-only status and update checks never install anything:
```powershell
hermes-relay computer-use cua status
hermes-relay computer-use cua health
hermes-relay computer-use cua check-update
hermes-relay computer-use cua install --yes
hermes-relay computer-use cua update --yes
```
Install is pinned to the minimum compatible release. Update first asks the
installed driver's native update service which release is current, then refuses
to apply it if it falls outside Hermes-Relay's supported range (`>=0.19.3,
<0.20.0`). Hermes downloads the versioned GitHub release manifest and installer,
checks the manifest repository/product/version and the installer's SHA-256, and
then verifies the canonical binary path, version, driver manifest, required
tools, and permission mode. Accessibility health remains an explicit recheck
while the temporary Windows compatibility workaround is active. These are release-metadata and
checksum integrity checks, not a Windows publisher signature.
The UI provides the same explicit **Install**, **Check**, and **Update** actions
under **Settings → Computer control**. When the report says CUA is ready it is
the preferred structured engine; `hermes-relay computer-use engine legacy`
selects the original Windows input path as an explicit compatibility backend.
Engine changes apply only to new control sessions. Hermes does not
silently install or update CUA Driver, and `hermes-relay update` manages only
the Hermes-Relay CLI/UI bundle. Hermes also disables CUA telemetry for every
driver process it starts; telemetry is not silently enabled as part of pairing,
Full Access, or engine selection.
### SmartScreen warning on first launch
The binaries are unsigned during the experimental phase. Windows may show "Windows protected your PC" for the installer or first launch. Click **More info → Run anyway**. For a CLI-only install, you can also pre-allow the executable from PowerShell:
+24
View File
@@ -292,10 +292,26 @@ hermes-relay computer-use enable # confirm and persist enablement
hermes-relay computer-use enable --yes # explicit non-interactive confirmation
hermes-relay computer-use cancel # cancel the active task-scoped grant
hermes-relay computer-use disable # disable and request active-grant cancellation
hermes-relay computer-use engine cua # prefer structured CUA for new sessions
hermes-relay computer-use engine legacy # explicit Windows-input compatibility
hermes-relay computer-use cursor on # virtual per-session cursor
hermes-relay computer-use cua status
hermes-relay computer-use cua health # recheck accessibility health
hermes-relay computer-use cua check-update
hermes-relay computer-use cua install --yes
hermes-relay computer-use cua update --yes
```
Enablement is stored in `~/.hermes/desktop-settings.json`. Restart the daemon after changing it; the Windows tray does that automatically while preserving whether the daemon is running as User or Administrator. `--experimental-computer-use` is a one-process enable override, and `--no-computer-use` always suppresses advertisement for that invocation.
CUA is the preferred/default structured backend; Windows input is an explicit
compatibility choice. Backend selection is fixed when a control session starts,
so an engine setting change affects only new sessions. CUA lifecycle mutations
require `--yes` and never run automatically. Hermes verifies the upstream
release manifest and installer checksum, runs the installer with a sanitized
environment, and accepts only `>=0.19.3 <0.20.0` under the canonical
`%USERPROFILE%\.cua-driver\packages\current` package.
## `hermes-relay grants`
Review assist/control requests written by a headless daemon to the local grant bridge:
@@ -317,8 +333,16 @@ Show what the remote agent has run on **this** machine through the desktop tools
hermes-relay audit # last 50 desktop-tool calls
hermes-relay audit --limit 20 # fewer
hermes-relay audit --json # raw entries for scripting
hermes-relay audit screenshots --json
hermes-relay audit screenshots on --days 7 --yes
hermes-relay audit screenshots off --yes
```
Screenshot evidence is local and separate from the JSONL log. Retention defaults
to seven days and is capped at 20 PNG files and 10 MB per file. Choose 1, 7, or
30 days, or turn it off; disabling retention removes existing screenshot
evidence without deleting the remaining activity history.
```
Desktop-tool activity (3 most recent)
+90 -1
View File
@@ -221,6 +221,88 @@ Use `hermes-relay computer-use status` to see the persisted preference, daemon p
Input injection is currently **Windows-only**; `status` / `screenshot` work cross-platform.
### Computer-use engines
The `desktop_computer_*` contract stays the same regardless of the local engine.
On Windows you can choose:
| Engine | Behavior |
|--------|----------|
| **CUA Driver** | Preferred/default structured engine. Targets a named PID and window through UI Automation and background dispatch, and can show an animated virtual cursor without moving the physical pointer. |
| **Windows input** | Explicit compatibility backend. Uses the original Windows input path and can depend on foreground focus or move the physical pointer for some actions. |
Check the detected runtime and current selection from the CLI:
```powershell
hermes-relay computer-use status --json
hermes-relay computer-use engine cua
hermes-relay computer-use cursor on
hermes-relay computer-use cua status
hermes-relay computer-use cua health
hermes-relay computer-use cua check-update
hermes-relay computer-use cua install --yes
hermes-relay computer-use cua update --yes
```
The management UI exposes the same choices under **Settings → Computer
control**. Selecting `cua` succeeds only when Hermes finds the canonical runtime
at `%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe` and verifies its
supported version, manifest identity, allowlisted tools, and non-unrestricted
permission mode. Accessibility health is an explicit diagnostic while the
temporary Windows workaround for trycua/cua#3103 is active. Hermes does not trust whichever `cua-driver.exe`
happens to appear first on `PATH`. If the runtime is missing, incompatible, or
degraded, the UI explains why and a new control session can use Windows input
compatibility. Backend selection is made once per authenticated control session;
changing the preference never switches an active session underneath an action.
CUA sessions follow Hermes control sessions. Hermes derives the driver session
identity locally; an agent cannot choose or share a cursor ID. Each action uses
a fresh target-window snapshot, and element handles are wrapped in a short-lived,
single-use Hermes token bound to the control authority, grant, PID, window, and
snapshot generation. The handler returns a post-action snapshot so the caller
can verify whether the expected state actually changed. Grant expiry, cancellation, disconnect, policy change,
emergency stop, or daemon shutdown revokes the associated local state.
**Background is mandatory for CUA.** A target that cannot accept background input
returns a failure instead of silently bringing itself forward. The foreground
escalation control is reserved for a later explicit-authority path; this release
always reports it off, even if an older preview saved the preference, and Full
Access cannot enable it. The animated agent cursor is optional. It is a visual overlay
with a distinct session identity—not another Windows hardware pointer—and the
operator's physical cursor remains untouched by CUA actions.
CUA Driver is a separate optional dependency. Hermes-Relay does not bundle or
silently install/update it. `computer-use cua status|check-update` are
read-only; `install|update` require explicit `--yes` confirmation. Hermes uses
the canonical upstream package, validates versioned GitHub release metadata and
installer SHA-256, and refuses updates outside `>=0.19.3,<0.20.0`. These checks
do not claim a Windows publisher signature. Hermes executes the verified
temporary installer under a sanitized environment, then validates the canonical
`packages/current` binary, driver manifest, and permission mode. The UI and
`computer-use cua health` can recheck accessibility health without changing
the selected backend. Hermes forces CUA telemetry off for
its child invocations; any future telemetry opt-in belongs to the local
operator. `hermes-relay update` continues to manage only the Hermes-Relay CLI
and Windows UI.
Window-scoped snapshots and element actions use the selected backend. A normal
full-display screenshot remains on the separate read-only `system_capture` path;
using it does not switch the input backend. The local audit and UI Activity
timeline show bounded high-level evidence—backend, background dispatch, control
session, target app/window identifiers, action, phase, and verification state.
Accessibility text, screenshot bytes, entered values, and raw driver responses
are redacted from the activity drilldown.
::: warning CUA does not sandbox raw commands
CUA's PID/window and snapshot boundaries apply to structured computer-control
actions. If Commands, PowerShell, or terminal execution is allowed, the agent
can still invoke ordinary OS automation through that broader trusted path.
Disable raw command access when scoped computer control is part of your threat
model. Full Access never bypasses authenticated targeting, sensitive-surface
checks, UAC or Windows-session boundaries, audit, health checks, or emergency
stop.
:::
## Diagnosing routing
If the agent says "desktop_terminal is not available" or calls time out immediately:
@@ -278,7 +360,14 @@ If `connected: true` but the agent still says the tool is missing:
`hermes-relay daemon` runs the WSS connection + tool router headless, so the agent can reach your machine while you're in another window or VS Code or off making coffee. Use `hermes-relay daemon start` to run it in the **background** (no console window, survives closing the terminal), `daemon status` to check it, and `daemon stop` to stop it. See [Subcommands → daemon](./subcommands.md#hermes-relay-daemon) for full lifecycle/log details.
Want to see what the agent actually ran on your machine? `hermes-relay audit` lists recent `desktop_*` activity from a local log. The management UI previews the latest three events and opens each event into bounded request, stdout, stderr, result, exit, timing, and truncation details. Sensitive request inputs are excluded.
Want to see what the agent actually ran on your machine? `hermes-relay audit` lists recent `desktop_*` activity from a local log. The management UI previews the latest three events and opens each event into a lifecycle stepper plus bounded request, stdout, stderr, result, exit, timing, and truncation details. Errors have a dedicated failure panel. Screenshot events may retain bounded local evidence for a larger viewer; Settings controls Off/1-day/7-day/30-day retention and clearing Activity removes it. Sensitive request inputs remain excluded from the JSON audit log.
The daemon records one interruption event when automatic reconnect begins and a
recovery event when it succeeds rather than adding one row per backoff attempt.
The Overview shows retry attempt/timing and supports **Retry now** or an explicit
disconnect. When the management UI is hidden, connection loss and restoration
use the same compact local-card language as permission requests; no duplicate
card appears while the UI is already open.
`daemon start` covers "background, this session." On Windows, **Start UI at sign-in** registers the optional tray as a per-user login entry. **Start daemon with UI** separately opts into connecting remote access when the tray launches and defaults off for existing installs. Neither is a Windows service. For Linux/macOS or a machine-level lifetime, wrap foreground `hermes-relay daemon` with your service manager of choice.
+36
View File
@@ -84,6 +84,42 @@ hermes-relay computer-use cancel
Switching the host to **Restricted** or using emergency stop also requests cancellation. Without a local approval response, a headless request times out and input remains blocked.
## CUA Driver is unavailable, incompatible, or degraded
Inspect the machine-local engine report:
```powershell
hermes-relay computer-use status --json
```
- **Not installed** means the canonical package was not found at
`%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe`. A PATH-only shim
is intentionally ignored.
- **Incompatible** means the executable, manifest, supported version, required
tool set, or permission mode did not match the Hermes adapter contract.
- **Accessibility health** is a separate, explicit diagnostic on Windows. Use
**Recheck** in the UI or run `hermes-relay computer-use cua health`. A
degraded result does not disable the runtime while the temporary workaround
for the upstream fixed-timeout issue is active; canonical runtime checks and
each structured action still fail closed. Confirm the driver is running in
the interactive user's logon session rather than Session 0 before deeper
diagnosis.
CUA is preferred for new structured-control sessions. If its executable,
manifest, required tool set, daemon status, or safe permission mode is not
ready before a session begins, Hermes can select the Windows input compatibility backend;
it never changes backend in the middle of a control session. Re-check with
`hermes-relay computer-use cua status`, repair explicitly with
`computer-use cua install --yes`, or use `computer-use cua check-update` followed
by `computer-use cua update --yes`. The UI exposes matching Install, Check, and
Update actions. None run automatically.
If a CUA action reports that background delivery is unavailable, Hermes does
not silently foreground the target. Use another structured action or complete
that step manually. **Allow foreground escalation** is reserved in this release:
runtime status remains off even if an older preview saved the preference, and
Full Access does not turn it on.
## `auth timed out after 15000ms`
The relay subprocess takes 15–30 seconds on first attach because Hermes initializes the full agent. Bump the timeout for slow first connects:
+17 -2
View File
@@ -1,6 +1,8 @@
# Dashboard Plugin
# Dashboard and official Desktop plugins
A hermes-agent dashboard plugin that surfaces relay-specific state in the gateway's web UI. Paired devices, bridge command history, push delivery (future), and active inbound-media tokens — all in one "Relay" tab, no SSH required.
Hermes-Relay surfaces its Relay-specific state in the web Dashboard and, when
explicitly enabled, in official Hermes Desktop. Both use the same existing
profile-scoped backend; neither creates a second Relay service or state store.
## What It Is
@@ -20,6 +22,19 @@ The plugin is a thin observer — it never modifies state, never writes to your
Nothing. The dashboard plugin renders in your browser against the Hermes server — the phone is the subject of observation, not a participant.
**For official Hermes Desktop:** use a build with the runtime Plugin SDK and
unified-package discovery. The regular Hermes-Relay plugin install already
places `desktop/plugin.js` beside the Dashboard half. Open **Settings →
Plugins**, enable **Hermes Relay**, then use the labeled **Relay** sidebar or
status-bar item, or **Hermes Relay: Open** in the command palette.
Enabling or loading the Desktop plugin does not open its pane. App startup,
reconnect, profile changes, navigation restoration, updates, and background
events also leave it closed. After an explicit open, close and move/dock work
through the native Desktop pane controls. Desktop currently offers no supported
agent command for focusing contributed panes and no programmatic move-coordinate
API; Hermes-Relay does not use private hooks to imitate those features.
## Accessing the Dashboard
Open the hermes-agent dashboard in your browser (default: `http://localhost:<dashboard_port>`). The **Relay** tab sits between Skills and whatever you have next in your nav order — click it and you land on the four-tab shell.