Compare commits
25
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5100c524f6 | ||
|
|
c609ae867f | ||
|
|
107f8c7720 | ||
|
|
8963e4fafd | ||
|
|
5d9624e2ef | ||
|
|
4b063d3fd7 | ||
|
|
9b9d7b654c | ||
|
|
a26e17e72c | ||
|
|
a3a6a9bb13 | ||
|
|
169bd09559 | ||
|
|
45d631e7ac | ||
|
|
eb6a6c95d2 | ||
|
|
3b102663c2 | ||
|
|
0e5fc4c606 | ||
|
|
c3189f2cbb | ||
|
|
0d6c3bd6b0 | ||
|
|
06d88ad40a | ||
|
|
8ae5b3fbc2 | ||
|
|
39b7a8f108 | ||
|
|
af6e167692 | ||
|
|
274bd6ae98 | ||
|
|
9fc55b379a | ||
|
|
559a0ffdc8 | ||
|
|
75bcd9180f | ||
|
|
9c995a443d |
@@ -20,6 +20,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this
|
||||
- **Android keeps profile management and retained automation truthful.** Custom Endpoint list and mutation routes now follow the selected Hermes profile, while completed one-shot cron jobs show their retained outcome and expose only valid Runs/Delete actions.
|
||||
- **Android and Relay recover more generated media reliably.** Android accepts upstream-valid wrapped, punctuated, adjacent, spaced, and Windows `MEDIA:` markers without consuming fenced examples, and Relay translates Docker-visible workspace, home, cache, and configured-mount paths before applying its existing credential, sandbox, and size checks.
|
||||
|
||||
## [1.8.0] - 2026-08-14
|
||||
|
||||
### Added
|
||||
|
||||
- **Official Hermes Desktop can surface Relay through its supported runtime Plugin SDK.** The unified plugin package now includes an opt-in, profile-scoped Desktop pane for Relay status, paired devices, bridge activity, media, pairing, revocation, and remote-access management. Loading, startup, reconnects, profile changes, and updates never open it; only labeled sidebar, status-bar, or command-palette actions register and reveal the movable native pane.
|
||||
|
||||
## [0.4.0-beta.2] - 2026-08-14
|
||||
|
||||
### Added
|
||||
|
||||
- **Desktop Activity now keeps inspectable local evidence.** Commands, files, devices, connection lifecycle, and computer control share a truthful event stepper with dedicated failure details; screenshot events can retain bounded local PNG evidence and open it in a larger borderless viewer. Settings controls retention as Off, 1 day, 7 days, or 30 days and shows local file usage.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Tunnel state stays responsive through interruption and retry.** The CLI UI distinguishes connected, reconnecting, and stopped states, exposes retry attempt/timing and a Retry now action, records connection failures and recovery in Activity, and shows compact connection cards only while the main UI is hidden.
|
||||
- **Windows CUA readiness no longer depends on the flaky whole-desktop health scan.** Hermes-Relay verifies the canonical runtime, manifest, required tools, daemon, and safe permission mode before starting structured sessions, while accessibility health remains an explicit CLI/UI diagnostic that can be rechecked without forcing the compatibility backend. This temporary workaround is scoped to the upstream fixed-timeout issue and keeps individual actions fail-closed.
|
||||
|
||||
## [0.4.0-beta.1] - 2026-08-14
|
||||
|
||||
### Added
|
||||
|
||||
- **CUA Driver is the preferred Windows structured-control engine.** New local settings prefer a verified CUA runtime for window-targeted background actions, fresh snapshot tokens, and optional per-session animated agent cursors without moving the physical pointer; Windows Input is the explicit compatibility backend and backend choice is fixed for each control session. Full-display observation remains on the read-only system capture path. CLI and UI can explicitly install, check, or update the canonical CUA package after verifying the upstream release manifest and installer checksum; nothing is bundled or updated automatically, driver telemetry stays off for Hermes sessions, and activity records contain only bounded, redacted control metadata.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Windows bundle updates fail closed when installed processes retain a binary lock.** Setup waits for the invoking CLI, quiesces the tray and its short-lived CLI children, checks every payload extraction before writing release metadata, preserves custom install directories, and returns a failure instead of reporting a mixed-version installation.
|
||||
- **CUA readiness follows the published driver contract.** Hermes accepts the documented `ok` health state, distinguishes an installed-but-degraded runtime from a missing installation, and constructs trusted Windows installer paths consistently across verification environments.
|
||||
|
||||
## [1.7.0] - 2026-08-13
|
||||
|
||||
### Added
|
||||
|
||||
+9
-12
@@ -1,30 +1,27 @@
|
||||
# Hermes-Relay CLI v__VERSION__
|
||||
|
||||
**Release Date:** 2026-08-13
|
||||
**Release Date:** 2026-08-14
|
||||
|
||||
This alpha makes the compact **Hermes-Relay CLI UI** responsive during connection changes, expands host and capability management, and presents live route security and diagnostics without turning the tray into a full desktop client.
|
||||
This beta makes connection recovery and Activity evidence inspectable in the compact management UI, and keeps the preferred CUA control engine usable when its upstream whole-desktop accessibility probe times out.
|
||||
|
||||
**Experimental phase.** Assets remain unsigned, so Windows SmartScreen and macOS Gatekeeper may warn on first launch. Standalone CLI binaries ship for Windows x64, Linux x64, and macOS x64/arm64; the management tray is Windows-only.
|
||||
**Beta phase.** Assets remain unsigned, so Windows SmartScreen and macOS Gatekeeper may warn on first launch. Standalone CLI binaries ship for Windows x64, Linux x64, and macOS x64/arm64; the management UI is Windows-only.
|
||||
|
||||
## What's changed
|
||||
|
||||
### Added
|
||||
|
||||
- **Host management hub.** Each paired Hermes host has local identity, pairing facts, access and capability controls, authorized-client revocation, re-pairing, and guarded removal.
|
||||
- **Evidence-first activity.** Overview shows the latest three events and detailed views retain bounded command, output, exit, duration, and truncation evidence.
|
||||
- **Live route details.** The Agent-to-PC path shows route type, encryption state, endpoint, packet motion, and a connection test with reachability and latency.
|
||||
- **Inspectable Activity evidence.** Commands, files, device work, connection lifecycle, and computer control share a consistent event stepper with dedicated failure details.
|
||||
- **Optional screenshot retention.** Screenshot events can keep bounded local PNG evidence for Off, 1 day, 7 days, or 30 days and open it in a larger borderless viewer. Evidence stays outside the JSON activity log.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Connection lifecycle stays responsive.** Connect, disconnect, and snapshot work run outside the UI thread with immediate transition feedback and single-flight live polling.
|
||||
- **Access presets are explicit.** Restricted, Ask Every Time, Standard, Full Access, and Custom map visibly onto individual command, file, screen/input, and hardware capabilities.
|
||||
- **Tailscale is the recommended remote route.** Direct TLS and Hermes Secure Link remain supported; Hermes Reach is marked experimental and stays below supported routes.
|
||||
- **Connection state is live and actionable.** The UI distinguishes connected, reconnecting, and stopped states, shows retry timing, and offers Retry now without freezing the popup.
|
||||
- **Connection notices stay out of the way.** Compact connect, disconnect, and reconnect cards appear only while the main management UI is hidden.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Legacy LAN and Tailscale routes no longer appear as Custom VPN.** Route testing infers generic saved roles from the endpoint and reports the correct network path.
|
||||
- **PowerShell output remains complete.** Scalar, pipeline, JSON, native output, errors, exit status, and truncation metadata return reliably through desktop RPC.
|
||||
- **Tray placement follows the real notification area.** Responsive geometry uses the tray monitor and DPI and remains anchored above the icon.
|
||||
- **CUA readiness no longer depends on the flaky global accessibility scan.** Hermes verifies the canonical runtime, required tools, daemon, and safe permission mode before structured control; explicit accessibility health remains available for diagnosis and individual actions still fail closed.
|
||||
- **Connection errors retain useful context.** Activity records bounded retry and recovery evidence without flooding one event per backoff attempt.
|
||||
|
||||
## Install
|
||||
|
||||
|
||||
+6
-12
@@ -1,8 +1,8 @@
|
||||
# Hermes-Relay-Server v__VERSION__
|
||||
|
||||
**Release Date:** August 13, 2026
|
||||
**Release Date:** August 14, 2026
|
||||
|
||||
This release adds an operator-owned secure ingress path, promotes Tailscale as the easiest supported remote route, and introduces Hermes Reach as a disabled-by-default experimental broker for outbound-only environments.
|
||||
This release adds an official, opt-in Relay pane for Hermes Desktop through the supported runtime Plugin SDK. It keeps Relay management profile-scoped and user-invoked without opening a pane during startup, reconnects, profile changes, or plugin updates.
|
||||
|
||||
Standard chat, session history, and Vanilla Hermes voice remain upstream-owned and do not require this plugin.
|
||||
|
||||
@@ -10,19 +10,13 @@ Standard chat, session history, and Vanilla Hermes voice remain upstream-owned a
|
||||
|
||||
### Added
|
||||
|
||||
- **Hermes Secure Link.** A pinned-TLS ingress can expose Relay, API, and Dashboard namespaces through one self-hosted endpoint while retaining each service's native authentication.
|
||||
- **Experimental Hermes Reach.** An optional self-hosted rendezvous broker forwards opaque inner Secure Link TLS records, with hashed credentials, replay protection, bounded streams, persistence, and revocation.
|
||||
- **Remote-access status.** Dashboard and pairing metadata distinguish reachability from transport protection and show supported services without exposing certificate pins.
|
||||
- **Official Hermes Desktop pane.** The unified plugin package registers a movable native pane for Relay status, paired devices, bridge activity, media, pairing, revocation, and remote-access management.
|
||||
- **Explicit entry points.** Labeled sidebar, status-bar, and command-palette actions register and reveal the pane lazily; repeated opens reuse the same surface.
|
||||
- **Profile-scoped state.** Cached Relay state follows the active Hermes profile and is disposed cleanly when the plugin unloads.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Tailscale is recommended for remote access.** Tailscale Serve remains the simplest supported path; direct TLS and Secure Link are self-hosted alternatives, while Reach stays advanced and experimental.
|
||||
- **Pairing carries reviewed transport trust.** QR payloads include the Secure Link endpoint and pin before the first network request; rotation requires explicit re-pairing.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Route credentials remain scoped and revocable.** Reach credentials are issued only through trusted Secure Link ingress, replaced atomically per Relay session, bounded by session expiry, and removed on revocation.
|
||||
- **Proxy namespaces preserve credential isolation.** API and Dashboard headers, cookies, redirects, methods, sizes, timeouts, and loopback authority are constrained independently.
|
||||
- **Plugin loading stays passive.** Loading, startup, reconnects, profile changes, and updates never reveal the pane or perform pane-owned network work.
|
||||
|
||||
## Install / update
|
||||
|
||||
|
||||
@@ -219,6 +219,14 @@ It pairs against the **same relay and credential store** as the Android app —
|
||||
|
||||
On Windows, the default installer adds the optional compact **Hermes-Relay CLI UI** tray popup for host selection and pairing, connection and daemon state, per-host Ask/Trusted/Full Access, local grant dialogs, authorized-client revocation, activity, settings, and emergency stop. It is a management surface only—chat, TUI, plugins, voice, and agent sessions remain CLI/upstream concerns.
|
||||
|
||||
Structured Windows computer control prefers a compatible local CUA Driver
|
||||
runtime for window-targeted background actions and virtual per-session agent
|
||||
cursors. It remains behind Hermes host policy, grants, targeting, audit, and
|
||||
emergency stop; Windows input is an explicit compatibility backend. CUA is not
|
||||
bundled or updated automatically, but the local CLI/UI can explicitly install,
|
||||
check, or update its verified canonical package. It is never exposed as a raw
|
||||
remote tool surface. See the [desktop tools guide](https://hermes-relay.dev/docs/desktop/tools.html#computer-use-engines).
|
||||
|
||||
- **Docs:** [CLI guide](https://hermes-relay.dev/docs/desktop/) · [`desktop/README.md`](desktop/README.md)
|
||||
- **AI-agent setup recipe:** `/hermes-relay-desktop-setup`
|
||||
|
||||
|
||||
@@ -6,6 +6,27 @@ For shipped work, see `DEVLOG.md`. For architectural decisions, see `docs/decisi
|
||||
|
||||
---
|
||||
|
||||
## Certify the official Desktop Relay plugin
|
||||
|
||||
The unified `plugin/desktop/plugin.js` implementation is covered by source-level
|
||||
SDK contract, packaging, explicit-open, no-auto-open, close, unload, and profile
|
||||
cache-isolation tests. A physical official Hermes Desktop session is still
|
||||
required before calling the UX live-certified:
|
||||
|
||||
- Test default and named local profiles, ordinary authenticated remote mode,
|
||||
and SSH mode with differently named local/remote profile mapping.
|
||||
- In two full app windows, prove enabling, registration, explicit open,
|
||||
requests, close/reopen, hot reload, and disable/unload remain window-local.
|
||||
- Prove startup, reconnect, profile change, layout restore/reset, update, and
|
||||
background events never open or focus Relay.
|
||||
- Drag and dock the pane across native zones, close it, reopen it from all three
|
||||
labeled actions, and verify no private-hook fallback is needed.
|
||||
- Exercise Relay running/unreachable, zero/one/multiple devices, pairing,
|
||||
revocation, bridge activity, media, remote access, and renderer error logging
|
||||
without exposing credentials, pairing payloads, filesystem paths, or tokens.
|
||||
|
||||
---
|
||||
|
||||
## Structured desktop hardware capabilities
|
||||
|
||||
Structured access and per-host USB policy now ship with typed, serial-bound ADB
|
||||
@@ -935,7 +956,7 @@ Phase 1 (end-to-end spine) shipped on `Codename-11/phone-platform` — `send_mes
|
||||
|
||||
- **Outbound buffering — ✅ relay-side DONE (2026-06-29).** `ProactiveChannel.push()` now queues agent→phone messages in a bounded deque (drop-oldest, 24 h TTL) when no phone is subscribed and returns `{queued: true}` (not 503); `_flush_outbound` delivers FIFO on the next subscribe (stale pruned). Inspect/cancel via `peek_outbound`/`cancel_outbound` + loopback `GET`/`DELETE /phone/outbound`. **UI surfacing of the queued state** (host-side, since the queue exists while the phone is OFFLINE): (a) ✅ **desktop CLI `relay queue` / `relay queue --clear` / `--cancel <id>` DONE (2026-06-29)** over the new endpoints (loopback-only — run on the relay host); a dashboard Relay-tab view is the optional GUI equivalent; (b) **remaining** — in the threaded agent surface, mark messages that arrived-while-away, and show the user's OWN pending replies (the Phase 3 reply queue) with a sending/Cancel affordance — that's where phone-side "queued + cancel" belongs.
|
||||
- **Threads surface (unified-session model — see ADR 12 + the Refinement above).** Build order, each shippable: **(1)** source tags in the session drawer (`source=phone` → clean **Threads** chip + thread-spool icon, NOT a phone glyph) — also delivers the "source attribution in Chat" goal; **(2)** open a Thread in Chat from its session-store history (reuse the existing message-history path); **(3)** route the live `proactive` push into the session view + notification + unread, demoting `ProactiveInboxStore` to cache/outbox; **(4)** reply from the Chat composer via `proactive.reply` + persist the user turn + local `Sending/Queued/Failed` status — **MVP**; **(5)** a **Threads capability row** in the best-path UI + a pinned **Threads** entry atop the drawer (thread-spool icon, shown only when relay-paired + opted-in) + retire `HermesInboxScreen`, re-point the notification deep-link + Settings "View messages"; **(6)** outbox/retry on reconnect; **(7)** relay `proactive.reply.ack` (honest Delivered) + `proactive.cancel`; **(8)** multi-thread `chat_id` (named/project Threads). **Verify gate before (1):** confirm the app's session-list/history path surfaces a `source=phone` session cleanly (upstream `session.list` returns all sources flat, so it should — but check whether the drawer currently filters it out). Honesty call: do NOT show "Delivered" until (7) lands (can't confirm it client-side before the ack).
|
||||
- **Status (2026-06-29, implemented UNBUILT — verify in Studio):** **CODE-COMPLETE on `dev`:** slice **1** (drawer source tags + `ThreadSpoolGlyph` + Threads filter), **2** (open a Thread from history — free via the existing `loadSessionHistory` path), **3-parse** (carry `reply_to` on `ProactiveMessage`), **4** (composer reply in a `source=phone` session routes over `proactive.reply`; `MessageDeliveryStatus` SENDING→DELIVERED/FAILED on the bubble), **5** (Threads capability row in `SessionPathCard` + `threadsCapabilityActive` drawer wiring), **7** (relay `proactive.reply.ack` + `proactive.cancel` — 25/25 `unittest` green — and client ack handling). **DONE since (2026-06-29, built + on phone):** live **in-thread reply rendering** (an agent reply lands in the open Thread as an ASSISTANT bubble, suppressing the notification/inbox — `injectIntoThread`); **user-created named Threads** ("+ New Thread"); **retire `HermesInboxScreen`** (deleted; route + nav removed; notification tap + Settings "View messages" re-pointed to Chat; surface renamed "Hermes messages" → **"Threads"**); relay slice-7 ack/cancel **DEPLOYED** to the host so **"Delivered" is live**. **DEFERRED (reasons):** per-session **unread badge**; **outbox/retry** (needs multiplexer connection-state); **exact-Thread deep-link** from the notification (opens Chat today, not the specific thread — needs select-session-on-entry); **remove the now-orphaned `ProactiveInboxStore`** (viewer-less write-only log); **agent-initiated** named Threads (upstream `send_message` thread param). On-device verifies for the create-flow: fresh-`chat_id` auto-create, the `…:dm:<chat_id>` id form, `renameSession` on a phone session.
|
||||
- **Status (2026-06-29, implemented UNBUILT — verify in Studio):** **CODE-COMPLETE on `dev`:** slice **1** (drawer source tags + `ThreadSpoolGlyph` + Threads filter), **2** (open a Thread from history — free via the existing `loadSessionHistory` path), **3-parse** (carry `reply_to` on `ProactiveMessage`), **4** (composer reply in a `source=phone` session routes over `proactive.reply`; `MessageDeliveryStatus` SENDING→DELIVERED/FAILED on the bubble), **5** (Threads capability row in `SessionPathCard` + `threadsCapabilityActive` drawer wiring), **7** (relay `proactive.reply.ack` + `proactive.cancel` — 25/25 `unittest` green — and client ack handling). **DONE since (2026-06-29, built + on phone):** live **in-thread reply rendering**; **user-created named Threads** ("+ New Thread"); **retire `HermesInboxScreen`**; relay slice-7 ack/cancel **DEPLOYED** to the host so **"Delivered" is live**. **DONE (2026-08-14):** notification taps survive cold start and open the exact `chat_id`; agent-initiated outbound messages appear as connection-scoped provisional Threads backed by the bounded proactive store, then promote to the real `source=phone` session after the first reply. **DEFERRED:** per-session **unread badge**; **outbox/retry** (needs multiplexer connection-state); **agent-initiated** named Threads (upstream `send_message` thread param). On-device verifies for the create-flow: fresh-`chat_id` auto-create, the `…:dm:<chat_id>` id form, `renameSession` on a phone session.
|
||||
- **User-created Threads (slice 8, Discord-style) — CODE-COMPLETE on `dev` (built + installed 2026-06-29; on-device behavior pending).** "+ New Thread" in the drawer's Threads view → name dialog → `ChatViewModel.startNewThread` mints a fresh `chat_id`; the first composer message opens it over `proactive.reply` (gateway auto-creates the `source=phone` session) → `switchToCreatedThread` polls + switches to the real session + applies the name. Existing-thread replies route by the `chat_id` parsed from the session id (`…:dm:<chat_id>`; opaque id → home fallback). **On-device verifies:** (1) a fresh-`chat_id` no-`reply_to` inbound creates a new `source=phone` session; (2) the phone session id carries the `…:dm:<chat_id>` form the client parses; (3) `renameSession` titles a phone session. **Remaining slice-8:** AGENT-initiated named Threads (the upstream `send_message` thread/chat_id param so the agent can open its own named Threads).
|
||||
- **`chat_id` not exposed by `/api/sessions` (root cause of the 2026-06-29 on-device create-flow bugs — fixed client-side).** Confirmed on the host: a phone session's `id` is a timestamp (e.g. `20260629_204755_94f391d6`); the real `chat_id` lives in the `session_key` (`agent:main:phone:dm:<chat_id>`) and a `chat_id` column — but `/api/sessions` returns **neither `chat_id` nor `session_key`**, only `source` + the timestamp `id`. So the client could not map a session ↔ its `chat_id`, which broke create-thread switch/rename + reply routing + in-thread injection. **Client workaround shipped:** find a created thread by session-list **diff** (the new `source=phone` session), keep an in-memory `sessionId → chat_id` map (learned at creation + from incoming `phone.message`s) for reply routing, and inject by source (+ learned chat_id) rather than a parsed id. **Limitation:** for a thread the app didn't create *this* session (agent-created, another device, or after an app restart) `chat_id` is unknown until a message arrives while viewing it → its replies fall back to the home channel until then. **RESOLVED via the plugin (2026-06-29, per upstream-or-plugin policy):** the relay now exposes `GET /phone/threads` (`plugin/relay/session_store.py` reads the gateway store read-only → `[{session_id, chat_id, title}]`; `server.py` `handle_phone_threads`, bearer for the app / loopback for diag; 5 unit tests). The app (`RelayHttpClient.fetchPhoneThreads` → `ConnectionViewModel.phoneThreadChatIds` on every `auth.ok` → `ChatViewModel.seedThreadChatIds`, authoritative over the learned map) now routes replies correctly for **any** Thread — incl. ones it didn't create + after restart. Deployed + verified live. **Still-nice-to-have (lower priority): the upstream PR** to add `chat_id`/`session_key` to `/api/sessions` (the standard-path proper fix; the relay route then becomes redundant + the client prefers upstream when present).
|
||||
- **Threads as named/project conversations (Discord-parity — folds into multi-thread #8).** A stable *named* `chat_id` per project = a persistent, agent-reachable project Thread (Discord named-thread parity for "persist a session for a project"). Enables: the agent **opening** a new named Thread for a background job/topic (a relay/gateway "open thread" affordance + a `send_message`-adjacent tool); cron/job updates landing in their own Thread; and replying to a Thread from any surface (desktop CLI / dashboard) since it is just a gateway session. Also evaluate per-Thread profile binding (a project Thread uses the "work" profile — ties to profile=contact).
|
||||
@@ -964,7 +985,7 @@ The gateway-platform model is the *correct + sufficient architecture* (the phone
|
||||
session store; the relay buffer is only the live/offline-delivery layer, not a
|
||||
parallel history database.
|
||||
- **Profile = contact mapping (new idea, fold in).** Multiple Hermes **profiles** (distinct agent personas/configs) could each be a distinct thread *source*/"contact" — DMing different agents. Maps cleanly onto the per-thread `chat_id` + source-attribution work; lets the app feel like a contact list of agents.
|
||||
- **Per-thread notification controls + deep-link (Discord-parity affordances).** Per-thread notification channels, mute/DND/quiet-hours (Phase 3 partially), and a notification that **deep-links into the exact thread** (tap → land in that conversation) so dipping in/out while multitasking is frictionless.
|
||||
- **Per-thread notification controls (Discord-parity affordances).** Exact-thread notification deep-linking is shipped. Remaining: per-thread notification channels and mute/DND/quiet-hours controls (Phase 3 partially).
|
||||
- **Agent-initiated rich content.** Agent → phone thread with **images/cards** (relay media infra + `InboundAttachmentCard`/`HermesCardBubble` already exist on the chat side — reuse). Inbound (phone → agent) reply media stays deferred (text-first), but outbound rich content is low-cost parity.
|
||||
- **In-thread "agent is working" indicator.** A typing/working state in the thread while the agent thinks/runs tools (Discord typing-dots parity) — the chat surface already has thinking indicators to reuse.
|
||||
|
||||
@@ -1230,6 +1251,25 @@ When the answer becomes clearer, this section becomes either an ADR in `docs/dec
|
||||
|
||||
## Smaller deferred items
|
||||
|
||||
- **Certify the preferred CUA Driver backend (ADR 56).** The canonical-runtime
|
||||
probe, bounded adapter, server-owned control-session envelope, per-session
|
||||
grant state, local engine/status controls, telemetry-off process environment,
|
||||
and Hermes snapshot-token primitives now exist. Before graduating the engine,
|
||||
finish end-to-end enforcement of app/display/folder scopes and sensitive
|
||||
pixel/accessibility denial or redaction, harden the grant-bridge ACL and nonce
|
||||
lifecycle, and complete live Windows certification proving the physical cursor and
|
||||
foreground app stay unchanged, stale or cross-window tokens fail, two remote
|
||||
control sessions receive isolated animated cursors, and foreground escalation
|
||||
never happens implicitly. Exercise revoke on grant expiry, disconnect,
|
||||
re-pair, policy downgrade, emergency stop, Windows-session change, and daemon
|
||||
shutdown. The explicit local CUA install/update surface now verifies upstream
|
||||
manifest identity and installer SHA-256; add Windows publisher verification
|
||||
when upstream signs the installer. Keep raw CUA tools, configuration,
|
||||
recording, replay, and JavaScript outside the remote agent surface.
|
||||
Remove the temporary Windows readiness/health split once
|
||||
[trycua/cua#3103](https://github.com/trycua/cua/issues/3103) ships in the
|
||||
supported CUA range; restore a mandatory health gate only if the upstream
|
||||
probe is bounded and cannot leave UI Automation falsely busy.
|
||||
- **MediaProjection consent flow** — wired in MainActivity (2026-04-12), needs end-to-end test on a real device
|
||||
- **WorkManager upgrade for auto-disable timer** — currently a coroutine `Job + delay()` in `AutoDisableWorker.kt`; documented at top of file. Upgrade when androidx.work joins the classpath
|
||||
- **Wave 3 voice-bridge multi-turn confirmation** — currently a 5s TTS countdown with cancel; conversational confirmation is the follow-up
|
||||
|
||||
@@ -34,6 +34,8 @@ data class ProactiveInboxEntry(
|
||||
* field).
|
||||
*/
|
||||
val chatId: String? = null,
|
||||
/** Owning saved connection. Null only for entries written by older builds. */
|
||||
val connectionId: String? = null,
|
||||
)
|
||||
|
||||
private val Context.proactiveInboxStore: DataStore<Preferences> by
|
||||
@@ -49,10 +51,10 @@ private const val MAX_ENTRIES = 100
|
||||
* newest-first, deduped by id (so a re-delivered message doesn't double up), and
|
||||
* capped at [MAX_ENTRIES]. Survives app restart.
|
||||
*
|
||||
* Demoted (2026-06-29): the agent conversation now lives as a Thread in Chat (the
|
||||
* gateway session is the durable history), so the in-app inbox view is retired.
|
||||
* This store is only fed for messages NOT shown in an open Thread; it currently
|
||||
* has no viewer and is fully retireable — see TODO.
|
||||
* Demoted (2026-06-29): once a phone gateway session exists, it is the durable
|
||||
* history. Outbound agent messages arrive before that session exists, so this
|
||||
* bounded store also backs the provisional Thread until the user's first reply
|
||||
* promotes it to a real `source=phone` session.
|
||||
*/
|
||||
class ProactiveInboxRepository(private val context: Context) {
|
||||
|
||||
|
||||
+10
-8
@@ -61,8 +61,8 @@ class ProactiveMessageHandler(
|
||||
/**
|
||||
* Show an inbound message inline in the Chat **Thread** it belongs to, when
|
||||
* that Thread is currently open. Returns true if it was shown there — in
|
||||
* which case the message is NOT also notified or added to the inbox (you're
|
||||
* already looking at the conversation). The unified-Threads counterpart of
|
||||
* which case the message is persisted but not also notified (you're already
|
||||
* looking at the conversation). The unified-Threads counterpart of
|
||||
* [toSession]; wired after construction.
|
||||
*/
|
||||
var injectIntoThread: ((ProactiveMessage) -> Boolean)? = null,
|
||||
@@ -94,13 +94,15 @@ class ProactiveMessageHandler(
|
||||
/** Route a parsed message: into the open Thread if it belongs there, else
|
||||
* the durable inbox log + the surface its hint selects. */
|
||||
private fun dispatch(msg: ProactiveMessage) {
|
||||
// Unified Threads: if this message belongs to the Thread currently open
|
||||
// in Chat, render it inline there and STOP — no notification, no inbox
|
||||
// entry (you're already looking at the conversation).
|
||||
if (injectIntoThread?.invoke(msg) == true) return
|
||||
// Otherwise the inbox is the durable log of agent-initiated messages and
|
||||
// the surfacing hint selects the additional surface.
|
||||
// Persist first even when the currently open Thread consumes the live
|
||||
// message. Agent-initiated outbound sends do not create a gateway
|
||||
// session until the phone replies, so this cache is the provisional
|
||||
// Thread transcript during that gap.
|
||||
toInbox?.invoke(msg)
|
||||
// Unified Threads: if this message belongs to the Thread currently open
|
||||
// in Chat, render it inline there and stop before raising a notification.
|
||||
if (injectIntoThread?.invoke(msg) == true) return
|
||||
// The surfacing hint selects the additional surface.
|
||||
when (msg.surfacing?.lowercase()) {
|
||||
"inbox" -> { /* inbox only — already recorded above */ }
|
||||
"session" -> {
|
||||
|
||||
+8
-4
@@ -9,6 +9,7 @@ import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.content.pm.PackageManager
|
||||
import android.os.Build
|
||||
import android.net.Uri
|
||||
import android.util.Log
|
||||
import androidx.core.app.NotificationCompat
|
||||
import androidx.core.app.NotificationManagerCompat
|
||||
@@ -47,10 +48,13 @@ object ProactiveMessageNotifier {
|
||||
/**
|
||||
* Tap route — opens Chat, where the message lives as a Thread. Must match
|
||||
* `Screen.Chat.route()` in RelayApp. Routed via the EXTRA_NAV_ROUTE deep-link
|
||||
* path (MainActivity → NavRouteRequest → RelayApp collector). Opening the
|
||||
* exact Thread by chat_id is a follow-up (see TODO).
|
||||
* path (MainActivity → NavRouteRequest → RelayApp collector), carrying the
|
||||
* `chat_id` so RelayApp opens the exact real or provisional Thread.
|
||||
*/
|
||||
private const val TAP_ROUTE = "chat"
|
||||
private fun tapRoute(chatId: String?): String =
|
||||
chatId?.takeIf { it.isNotBlank() }
|
||||
?.let { "chat?proactiveChatId=${Uri.encode(it)}" }
|
||||
?: "chat"
|
||||
|
||||
/**
|
||||
* Post (or replace) a proactive-message notification.
|
||||
@@ -80,7 +84,7 @@ object ProactiveMessageNotifier {
|
||||
|
||||
val tapIntent = Intent(context, MainActivity::class.java).apply {
|
||||
flags = Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP
|
||||
putExtra(MainActivity.EXTRA_NAV_ROUTE, TAP_ROUTE)
|
||||
putExtra(MainActivity.EXTRA_NAV_ROUTE, tapRoute(chatId))
|
||||
}
|
||||
val pendingFlags = PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE
|
||||
// Distinct requestCode per slot so each notification gets its own
|
||||
|
||||
@@ -336,17 +336,20 @@ sealed class Screen(
|
||||
// NavHost, and the NavigationBarItem click must navigate via [route]()
|
||||
// so no unresolved `{openAgentSheet}` leaks into the destination.
|
||||
data object Chat : Screen(
|
||||
"chat?openAgentSheet={openAgentSheet}&sessionId={sessionId}&profile={profile}",
|
||||
"chat?openAgentSheet={openAgentSheet}&sessionId={sessionId}&profile={profile}" +
|
||||
"&proactiveChatId={proactiveChatId}",
|
||||
"Chat",
|
||||
Icons.AutoMirrored.Filled.Chat,
|
||||
) {
|
||||
const val ARG_OPEN_AGENT_SHEET: String = "openAgentSheet"
|
||||
const val ARG_SESSION_ID: String = "sessionId"
|
||||
const val ARG_PROFILE: String = "profile"
|
||||
const val ARG_PROACTIVE_CHAT_ID: String = "proactiveChatId"
|
||||
fun route(
|
||||
openAgentSheet: Boolean = false,
|
||||
sessionId: String? = null,
|
||||
profile: String? = null,
|
||||
proactiveChatId: String? = null,
|
||||
): String {
|
||||
val params = buildList {
|
||||
if (openAgentSheet) add("$ARG_OPEN_AGENT_SHEET=true")
|
||||
@@ -356,6 +359,9 @@ sealed class Screen(
|
||||
profile?.takeIf { it.isNotBlank() }?.let {
|
||||
add("$ARG_PROFILE=${android.net.Uri.encode(it)}")
|
||||
}
|
||||
proactiveChatId?.takeIf { it.isNotBlank() }?.let {
|
||||
add("$ARG_PROACTIVE_CHAT_ID=${android.net.Uri.encode(it)}")
|
||||
}
|
||||
}
|
||||
return if (params.isEmpty()) "chat" else "chat?${params.joinToString("&")}"
|
||||
}
|
||||
@@ -1685,6 +1691,11 @@ fun RelayApp() {
|
||||
nullable = true
|
||||
defaultValue = null
|
||||
},
|
||||
navArgument(Screen.Chat.ARG_PROACTIVE_CHAT_ID) {
|
||||
type = NavType.StringType
|
||||
nullable = true
|
||||
defaultValue = null
|
||||
},
|
||||
),
|
||||
) { backStackEntry ->
|
||||
// Responsive bubble width based on screen width. The "Blend"
|
||||
@@ -1715,6 +1726,35 @@ fun RelayApp() {
|
||||
val requestedProfileRoute = backStackEntry.arguments
|
||||
?.getString(Screen.Chat.ARG_PROFILE)
|
||||
?.takeIf { it.isNotBlank() }
|
||||
val requestedProactiveChatId = backStackEntry.arguments
|
||||
?.getString(Screen.Chat.ARG_PROACTIVE_CHAT_ID)
|
||||
?.takeIf { it.isNotBlank() }
|
||||
val proactiveInboxEntries by connectionViewModel.inboxMessages.collectAsState()
|
||||
val phoneThreadChatIds by connectionViewModel.phoneThreadChatIds.collectAsState()
|
||||
LaunchedEffect(
|
||||
requestedProactiveChatId,
|
||||
proactiveInboxEntries,
|
||||
phoneThreadChatIds,
|
||||
) {
|
||||
val chatId = requestedProactiveChatId ?: return@LaunchedEffect
|
||||
val realSessionId = phoneThreadChatIds.entries
|
||||
.firstOrNull { it.value == chatId }
|
||||
?.key
|
||||
if (realSessionId != null) {
|
||||
chatViewModel.switchSession(realSessionId)
|
||||
} else {
|
||||
val entries = proactiveInboxEntries.filter {
|
||||
(it.connectionId == null || it.connectionId == activeConnectionId) &&
|
||||
(it.chatId ?: "phone") == chatId
|
||||
}
|
||||
if (entries.isEmpty()) return@LaunchedEffect
|
||||
chatViewModel.openProactiveThread(chatId, entries)
|
||||
}
|
||||
backStackEntry.arguments?.putString(
|
||||
Screen.Chat.ARG_PROACTIVE_CHAT_ID,
|
||||
null,
|
||||
)
|
||||
}
|
||||
LaunchedEffect(
|
||||
requestedSessionId,
|
||||
requestedProfileRoute,
|
||||
|
||||
+10
-1
@@ -1739,7 +1739,10 @@ fun ActiveCardRoutesSection(
|
||||
var routeEditorOriginal by remember(connection.id) {
|
||||
mutableStateOf<EndpointCandidate?>(null)
|
||||
}
|
||||
val hasTailscaleRoute = endpoints.any { it.role.equals("tailscale", ignoreCase = true) }
|
||||
val hasTailscaleRoute = hasConfiguredTailscaleRoute(
|
||||
endpoints = endpoints,
|
||||
primaryEndpointUrl = connection.primaryEndpointUrl,
|
||||
)
|
||||
val tailscalePreferred = preferredRole?.equals("tailscale", ignoreCase = true) == true
|
||||
val routeNeedsAttention = activeEndpoint == null && liveState != RelayUiState.Connected
|
||||
val showTailscaleUnavailableHint =
|
||||
@@ -2171,6 +2174,12 @@ fun ActiveCardRoutesSection(
|
||||
}
|
||||
}
|
||||
|
||||
internal fun hasConfiguredTailscaleRoute(
|
||||
endpoints: List<EndpointCandidate>,
|
||||
primaryEndpointUrl: String,
|
||||
): Boolean = endpoints.any { it.role.equals("tailscale", ignoreCase = true) } ||
|
||||
Connection.inferRouteRole(primaryEndpointUrl) == "tailscale"
|
||||
|
||||
/**
|
||||
* Numbered step row for the Manual pairing code fallback. Tightly
|
||||
* coupled to its Card 3 layout — step badge sizing + content shape —
|
||||
|
||||
@@ -104,6 +104,15 @@ data class ProfileSessionRow(
|
||||
val session: ChatSession,
|
||||
)
|
||||
|
||||
data class ProvisionalThreadRow(
|
||||
val chatId: String,
|
||||
val title: String,
|
||||
val messageCount: Int,
|
||||
val lastActivityAt: Long,
|
||||
)
|
||||
|
||||
private const val PROVISIONAL_THREAD_PREFIX = "proactive:"
|
||||
|
||||
internal fun sessionWorkLabels(session: ChatSession): List<String> = buildList {
|
||||
val repo = (session.gitRepoRoot ?: session.workingDirectory)
|
||||
?.trimEnd('/', '\\')
|
||||
@@ -169,6 +178,8 @@ fun SessionDrawerContent(
|
||||
* filter view. The first message the user types opens the conversation.
|
||||
*/
|
||||
onNewThread: ((String) -> Unit)? = null,
|
||||
provisionalThreads: List<ProvisionalThreadRow> = emptyList(),
|
||||
onSelectProvisionalThread: ((String) -> Unit)? = null,
|
||||
/** Gateway sources currently hidden from the drawer (default: cron+webhook). */
|
||||
hiddenSources: Set<String> = emptySet(),
|
||||
/** Toggle a source's visibility (persisted). Null hides the source filter. */
|
||||
@@ -191,7 +202,20 @@ fun SessionDrawerContent(
|
||||
// Threads affordance shows when the capability is active OR there's already at least one
|
||||
// agent Thread (source=phone) in the list. If the filter is on Threads but they've
|
||||
// vanished, fall back to All so the drawer never gets stuck on an empty hidden filter.
|
||||
val showThreads = threadsCapabilityActive || sessions.any { isThreadSource(it.source) }
|
||||
val provisionalSessions = provisionalThreads.map { thread ->
|
||||
ChatSession(
|
||||
sessionId = "$PROVISIONAL_THREAD_PREFIX${thread.chatId}",
|
||||
title = thread.title,
|
||||
model = null,
|
||||
messageCount = thread.messageCount,
|
||||
updatedAt = thread.lastActivityAt,
|
||||
startedAt = thread.lastActivityAt,
|
||||
lastActivityAt = thread.lastActivityAt,
|
||||
source = "phone",
|
||||
)
|
||||
}
|
||||
val allSessions = sessions + provisionalSessions
|
||||
val showThreads = threadsCapabilityActive || allSessions.any { isThreadSource(it.source) }
|
||||
val activeFilter = resolveSessionDrawerFilter(filter, showThreads, archiveSupported)
|
||||
// External gateway sources present (discord/telegram/cron/…) for the source
|
||||
// filter dropdown. Own chats (tui/api_server) + phone Threads aren't listed.
|
||||
@@ -200,7 +224,7 @@ fun SessionDrawerContent(
|
||||
.distinct()
|
||||
.filter { sourceBadge(it) != null }
|
||||
.sorted()
|
||||
val visibleSessions = sessions
|
||||
val visibleSessions = allSessions
|
||||
.asSequence()
|
||||
.filter { session ->
|
||||
when (activeFilter) {
|
||||
@@ -529,7 +553,7 @@ fun SessionDrawerContent(
|
||||
horizontalAlignment = Alignment.CenterHorizontally
|
||||
) {
|
||||
Text(
|
||||
text = if (sessions.isEmpty()) stringResource(R.string.drawer_no_sessions) else stringResource(R.string.drawer_no_matching_sessions),
|
||||
text = if (allSessions.isEmpty()) stringResource(R.string.drawer_no_sessions) else stringResource(R.string.drawer_no_matching_sessions),
|
||||
style = MaterialTheme.typography.bodyMedium,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant
|
||||
)
|
||||
@@ -553,7 +577,16 @@ fun SessionDrawerContent(
|
||||
pinned = session.pinned,
|
||||
archived = session.archived,
|
||||
archiveSupported = archiveSupported,
|
||||
onClick = { onSelectSession(session.sessionId) },
|
||||
onClick = {
|
||||
if (session.sessionId.startsWith(PROVISIONAL_THREAD_PREFIX)) {
|
||||
onSelectProvisionalThread?.invoke(
|
||||
session.sessionId.removePrefix(PROVISIONAL_THREAD_PREFIX),
|
||||
)
|
||||
} else {
|
||||
onSelectSession(session.sessionId)
|
||||
}
|
||||
},
|
||||
actionsEnabled = !session.sessionId.startsWith(PROVISIONAL_THREAD_PREFIX),
|
||||
onTogglePinned = {
|
||||
onSetSessionPinned(session.sessionId, !session.pinned)
|
||||
},
|
||||
@@ -748,6 +781,7 @@ private fun SessionItem(
|
||||
archived: Boolean,
|
||||
archiveSupported: Boolean,
|
||||
onClick: () -> Unit,
|
||||
actionsEnabled: Boolean = true,
|
||||
onTogglePinned: () -> Unit,
|
||||
onToggleArchived: () -> Unit,
|
||||
onRename: () -> Unit,
|
||||
@@ -888,7 +922,7 @@ private fun SessionItem(
|
||||
}
|
||||
}
|
||||
|
||||
Box {
|
||||
if (actionsEnabled) Box {
|
||||
IconButton(
|
||||
onClick = { menuOpen = true },
|
||||
modifier = Modifier.size(36.dp),
|
||||
|
||||
@@ -173,6 +173,7 @@ import com.hermesandroid.relay.data.HermesCardAction
|
||||
import com.hermesandroid.relay.data.MessageRole
|
||||
import com.hermesandroid.relay.data.PhysicalKeyboardEnterBehavior
|
||||
import com.hermesandroid.relay.data.ProfilePresentationPolicy
|
||||
import com.hermesandroid.relay.data.ProactiveInboxEntry
|
||||
import com.hermesandroid.relay.data.SessionActivityState
|
||||
import com.hermesandroid.relay.data.VoicePresentationMode
|
||||
import com.hermesandroid.relay.data.hermesProcessNotificationOrNull
|
||||
@@ -232,6 +233,7 @@ import com.hermesandroid.relay.ui.components.ThinkingMatrixColor
|
||||
import com.hermesandroid.relay.ui.components.ThinkingMatrixPattern
|
||||
import com.hermesandroid.relay.ui.components.SessionDrawerContent
|
||||
import com.hermesandroid.relay.ui.components.ProfileSessionRow
|
||||
import com.hermesandroid.relay.ui.components.ProvisionalThreadRow
|
||||
import com.hermesandroid.relay.ui.components.ProfileDisplayManagerDialog
|
||||
import com.hermesandroid.relay.ui.components.ProfileShelf
|
||||
import com.hermesandroid.relay.ui.components.ProfileSwitcherSheet
|
||||
@@ -2066,6 +2068,22 @@ fun ChatScreen(
|
||||
val threadsCapabilityActive = threadsProactiveEnabled &&
|
||||
threadsAuthState is com.hermesandroid.relay.auth.AuthState.Paired
|
||||
val hiddenSources by connectionViewModel.hiddenSources.collectAsState()
|
||||
val proactiveInboxEntries by connectionViewModel.inboxMessages.collectAsState()
|
||||
val phoneThreadChatIds by connectionViewModel.phoneThreadChatIds.collectAsState()
|
||||
val provisionalThreadEntries = buildProvisionalThreadRows(
|
||||
entries = proactiveInboxEntries,
|
||||
activeConnectionId = activeConnection?.id,
|
||||
realThreadChatIds = phoneThreadChatIds.values,
|
||||
)
|
||||
val provisionalThreads = provisionalThreadEntries.map { (chatId, entries) ->
|
||||
val latest = entries.maxBy { it.receivedAt }
|
||||
ProvisionalThreadRow(
|
||||
chatId = chatId,
|
||||
title = latest.title.ifBlank { "Hermes" },
|
||||
messageCount = entries.size,
|
||||
lastActivityAt = latest.receivedAt,
|
||||
)
|
||||
}
|
||||
|
||||
SessionDrawerContent(
|
||||
sessions = sessions,
|
||||
@@ -2112,6 +2130,14 @@ fun ChatScreen(
|
||||
chatViewModel.startNewThread(name)
|
||||
scope.launch { drawerState.close() }
|
||||
},
|
||||
provisionalThreads = provisionalThreads,
|
||||
onSelectProvisionalThread = { chatId ->
|
||||
chatViewModel.openProactiveThread(
|
||||
chatId,
|
||||
provisionalThreadEntries[chatId].orEmpty(),
|
||||
)
|
||||
scope.launch { drawerState.close() }
|
||||
},
|
||||
hiddenSources = hiddenSources,
|
||||
onToggleSourceHidden = { source, hidden ->
|
||||
connectionViewModel.setSourceHidden(source, hidden)
|
||||
@@ -4230,6 +4256,15 @@ fun ChatScreen(
|
||||
}
|
||||
}
|
||||
|
||||
internal fun buildProvisionalThreadRows(
|
||||
entries: List<ProactiveInboxEntry>,
|
||||
activeConnectionId: String?,
|
||||
realThreadChatIds: Collection<String>,
|
||||
): Map<String, List<ProactiveInboxEntry>> = entries
|
||||
.filter { it.connectionId == null || it.connectionId == activeConnectionId }
|
||||
.groupBy { it.chatId ?: "phone" }
|
||||
.filterKeys { it !in realThreadChatIds }
|
||||
|
||||
// --- Helper functions ---
|
||||
|
||||
@Composable
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
package com.hermesandroid.relay.util
|
||||
|
||||
import kotlinx.coroutines.flow.MutableSharedFlow
|
||||
import kotlinx.coroutines.flow.SharedFlow
|
||||
import kotlinx.coroutines.flow.asSharedFlow
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.flow.Flow
|
||||
import kotlinx.coroutines.flow.receiveAsFlow
|
||||
|
||||
/**
|
||||
* Cross-layer one-shot navigation requests.
|
||||
@@ -18,23 +18,21 @@ import kotlinx.coroutines.flow.asSharedFlow
|
||||
* `BridgeSafetySettingsScreen` instead of dropping the user on `MainActivity`'s
|
||||
* home screen.
|
||||
*
|
||||
* Buffer: `extraBufferCapacity = 4` so back-to-back tryEmit calls during
|
||||
* `onCreate → setContent` don't drop on the floor before `RelayApp`'s
|
||||
* collector subscribes. Replay 0 — late subscribers shouldn't replay stale
|
||||
* navigation intents from prior process lifetimes.
|
||||
* A buffered [Channel] is intentional here: notification taps are consumed in
|
||||
* `MainActivity.onCreate` before Compose installs RelayApp's collector. A
|
||||
* replay-0 SharedFlow drops those cold-start requests when no subscriber exists.
|
||||
* The channel retains up to four one-shot routes and hands each to the single
|
||||
* app-root collector exactly once.
|
||||
*/
|
||||
object NavRouteRequest {
|
||||
private val _requests = MutableSharedFlow<String>(
|
||||
replay = 0,
|
||||
extraBufferCapacity = 4,
|
||||
)
|
||||
private val channel = Channel<String>(capacity = 4)
|
||||
|
||||
val requests: SharedFlow<String> = _requests.asSharedFlow()
|
||||
val requests: Flow<String> = channel.receiveAsFlow()
|
||||
|
||||
/** Fire-and-forget emit. Safe to call from any thread, including the main thread. */
|
||||
fun tryRequest(route: String): Boolean = _requests.tryEmit(route)
|
||||
fun tryRequest(route: String): Boolean = channel.trySend(route).isSuccess
|
||||
|
||||
suspend fun request(route: String) {
|
||||
_requests.emit(route)
|
||||
channel.send(route)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ import com.hermesandroid.relay.data.MessageDeliveryStatus
|
||||
import com.hermesandroid.relay.data.MessageRole
|
||||
import com.hermesandroid.relay.data.parseChatQuotedPrompt
|
||||
import com.hermesandroid.relay.data.Profile
|
||||
import com.hermesandroid.relay.data.ProactiveInboxEntry
|
||||
import com.hermesandroid.relay.data.RealtimeConversationContextMessage
|
||||
import com.hermesandroid.relay.data.RealtimeTurnTrace
|
||||
import com.hermesandroid.relay.data.SessionActivityState
|
||||
@@ -2932,6 +2933,12 @@ class ChatViewModel : ViewModel() {
|
||||
fun injectThreadMessage(msg: ProactiveMessage): Boolean {
|
||||
val handler = chatHandler ?: return false
|
||||
val msgChatId = msg.chatId?.takeIf { it.isNotBlank() }
|
||||
pendingThread?.let { pending ->
|
||||
if (msgChatId == null || msgChatId == pending.chatId) {
|
||||
handler.addAgentThreadMessage(msg.text, msg.messageId, msg.title)
|
||||
return true
|
||||
}
|
||||
}
|
||||
// A freshly-created thread whose real session we're still switching to:
|
||||
// show the agent's first reply in the draft view now (the switch
|
||||
// reconciles it from history). Covers the gap before currentSessionId is
|
||||
@@ -3550,6 +3557,8 @@ class ChatViewModel : ViewModel() {
|
||||
|
||||
fun createNewChat() {
|
||||
val handler = chatHandler ?: return
|
||||
pendingThread = null
|
||||
creatingThread = null
|
||||
|
||||
// Gateway turns continue as detached siblings; SSE remains exclusive.
|
||||
releaseTurnForNavigation(handler)
|
||||
@@ -3666,6 +3675,43 @@ class ChatViewModel : ViewModel() {
|
||||
onSessionChanged?.invoke(null)
|
||||
}
|
||||
|
||||
/**
|
||||
* Open an agent-initiated Thread before the gateway has a `source=phone`
|
||||
* session for it. Outbound platform sends do not create gateway sessions;
|
||||
* the first phone reply does. Until then the durable proactive inbox is the
|
||||
* provisional transcript. The existing pending-thread send path promotes
|
||||
* this draft to the real gateway session after the user's first reply.
|
||||
*/
|
||||
fun openProactiveThread(chatId: String, entries: List<ProactiveInboxEntry>) {
|
||||
val handler = chatHandler ?: return
|
||||
val normalizedChatId = chatId.ifBlank { "phone" }
|
||||
val ordered = entries
|
||||
.filter { (it.chatId ?: "phone") == normalizedChatId }
|
||||
.sortedBy { it.receivedAt }
|
||||
if (ordered.isEmpty()) return
|
||||
|
||||
releaseTurnForNavigation(handler)
|
||||
cancelAnswerRecovery(settleUi = false)
|
||||
historyLoadGeneration.incrementAndGet()
|
||||
pendingThread = PendingThread(
|
||||
chatId = normalizedChatId,
|
||||
name = ordered.last().title.ifBlank { "Hermes" },
|
||||
)
|
||||
creatingThread = null
|
||||
gatewayClient?.clearSession()
|
||||
handler.setSessionId(null)
|
||||
selectBackgroundProcessSession(null)
|
||||
handler.clearMessages()
|
||||
ordered.forEach { entry ->
|
||||
handler.addAgentThreadMessage(entry.text, entry.id, entry.title)
|
||||
}
|
||||
_contextUsage.value = null
|
||||
_contextWindow.value = null
|
||||
dismissPendingAskNotification()
|
||||
_pendingAsk.value = null
|
||||
onSessionChanged?.invoke(null)
|
||||
}
|
||||
|
||||
/**
|
||||
* After a "+ New Thread" first send, poll the session list until the gateway
|
||||
* has created the new `source=phone` session, then switch to it (loading its
|
||||
@@ -3711,6 +3757,8 @@ class ChatViewModel : ViewModel() {
|
||||
fun switchSession(sessionId: String) {
|
||||
val handler = chatHandler ?: return
|
||||
if (streamingEndpoint != "gateway" && apiClient == null) return
|
||||
pendingThread = null
|
||||
creatingThread = null
|
||||
|
||||
// Keep a Gateway sibling alive and detach its callbacks. SSE remains a
|
||||
// single exclusive stream and is interrupted on navigation.
|
||||
|
||||
@@ -2362,6 +2362,7 @@ class ConnectionViewModel(application: Application) : AndroidViewModel(applicati
|
||||
text = msg.text,
|
||||
receivedAt = msg.sentAt ?: System.currentTimeMillis(),
|
||||
chatId = msg.chatId,
|
||||
connectionId = connectionStore.activeConnectionId.value,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
package com.hermesandroid.relay.ui.components
|
||||
|
||||
import com.hermesandroid.relay.data.ApiEndpoint
|
||||
import com.hermesandroid.relay.data.EndpointCandidate
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
class TailscaleRoutePresenceTest {
|
||||
@Test
|
||||
fun primaryTailnetIpCountsAsConfiguredRoute() {
|
||||
assertTrue(
|
||||
hasConfiguredTailscaleRoute(
|
||||
endpoints = emptyList(),
|
||||
primaryEndpointUrl = "http://100.75.1.2:9119",
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun explicitTailscaleCandidateStillCounts() {
|
||||
assertTrue(
|
||||
hasConfiguredTailscaleRoute(
|
||||
endpoints = listOf(
|
||||
EndpointCandidate(
|
||||
role = "tailscale",
|
||||
api = ApiEndpoint("server.ts.net", 8642),
|
||||
),
|
||||
),
|
||||
primaryEndpointUrl = "http://192.168.1.2:9119",
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ordinaryLanDoesNotCount() {
|
||||
assertFalse(
|
||||
hasConfiguredTailscaleRoute(
|
||||
endpoints = emptyList(),
|
||||
primaryEndpointUrl = "http://192.168.1.2:9119",
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package com.hermesandroid.relay.ui.screens
|
||||
|
||||
import com.hermesandroid.relay.data.ProactiveInboxEntry
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
class ProvisionalThreadRowsTest {
|
||||
@Test
|
||||
fun rowsAreConnectionScopedAndDisappearWhenRealThreadExists() {
|
||||
val entries = listOf(
|
||||
entry("one", connectionId = "connection-a", chatId = "phone"),
|
||||
entry("two", connectionId = "connection-b", chatId = "phone"),
|
||||
entry("three", connectionId = "connection-a", chatId = "project"),
|
||||
)
|
||||
|
||||
val rows = buildProvisionalThreadRows(
|
||||
entries = entries,
|
||||
activeConnectionId = "connection-a",
|
||||
realThreadChatIds = listOf("project"),
|
||||
)
|
||||
|
||||
assertEquals(listOf("one"), rows.getValue("phone").map { it.id })
|
||||
assertFalse("project" in rows)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun legacyUnscopedEntriesRemainVisibleOnTheActiveConnection() {
|
||||
val rows = buildProvisionalThreadRows(
|
||||
entries = listOf(entry("legacy", connectionId = null, chatId = null)),
|
||||
activeConnectionId = "connection-a",
|
||||
realThreadChatIds = emptyList(),
|
||||
)
|
||||
|
||||
assertTrue("phone" in rows)
|
||||
}
|
||||
|
||||
private fun entry(id: String, connectionId: String?, chatId: String?) =
|
||||
ProactiveInboxEntry(
|
||||
id = id,
|
||||
title = "Hermes",
|
||||
text = id,
|
||||
receivedAt = 1L,
|
||||
chatId = chatId,
|
||||
connectionId = connectionId,
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package com.hermesandroid.relay.util
|
||||
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertNull
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
import kotlinx.coroutines.flow.first
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlinx.coroutines.withTimeout
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
|
||||
class NavRouteRequestTest {
|
||||
@Test
|
||||
fun requestBeforeCollector_isRetainedAndConsumedOnce() = runBlocking {
|
||||
assertTrue(NavRouteRequest.tryRequest("chat?proactiveChatId=phone"))
|
||||
|
||||
assertEquals(
|
||||
"chat?proactiveChatId=phone",
|
||||
withTimeout(1_000) { NavRouteRequest.requests.first() },
|
||||
)
|
||||
assertNull(withTimeoutOrNull(50) { NavRouteRequest.requests.first() })
|
||||
}
|
||||
}
|
||||
+83
-4
@@ -80,11 +80,15 @@ aborts, and non-zero process exits; and keeps request context collapsed until
|
||||
explicitly expanded. Events record handler duration and request ID where
|
||||
available. The compact Overview still shows only the three newest events.
|
||||
Settings also keeps activity compact: it previews the three newest events and
|
||||
opens a dedicated Activity page. Selecting an event opens bounded request,
|
||||
stdout, stderr, result, exit, timing, and truncation evidence; sensitive request
|
||||
inputs are excluded. Handler failures and aborts are **Issues**; non-zero process exits are
|
||||
opens a dedicated Activity page. Selecting an event opens a truthful lifecycle
|
||||
stepper plus bounded request, stdout, stderr, result, exit, timing, and
|
||||
truncation evidence; sensitive request inputs are excluded. Screenshot events
|
||||
can retain an opaque local PNG outside the JSON log and open it in a larger
|
||||
borderless viewer. **Settings → Activity → Screenshot evidence** controls this
|
||||
as Off, 1 day, 7 days (default), or 30 days, shows local file count/usage, and
|
||||
caps storage at 20 files and 10 MB per image. Handler failures and aborts are **Issues**; non-zero process exits are
|
||||
shown separately because probing commands may legitimately use them. **Clear**
|
||||
removes both current and rotated local audit history after confirmation.
|
||||
removes current/rotated audit history and retained screenshot evidence after confirmation.
|
||||
|
||||
Clicking a card under **Hosts** opens that host's detail page; it does not change
|
||||
the active connection. The detail page is the per-host hub for its local display
|
||||
@@ -441,6 +445,81 @@ assist/control grant is active because approved input inherits that privilege.
|
||||
|
||||
Default computer-use policy blocks password managers, credential prompts, banking/payment/crypto surfaces, OS security/admin settings, and private-key/token material. `~/.hermes/desktop-control.json` lets operators tighten or extend that baseline.
|
||||
|
||||
#### Preferred CUA Driver engine
|
||||
|
||||
On Windows, Hermes-Relay prefers a compatible local
|
||||
[CUA Driver](https://github.com/trycua/cua) runtime for structured
|
||||
computer-control engine. It stays behind the same `desktop_computer_*` tools:
|
||||
the agent does not receive CUA's raw tool surface, configuration, updater,
|
||||
recording, replay, JavaScript, application-launch, or process-termination
|
||||
operations.
|
||||
|
||||
Windows input is the explicit compatibility backend. A backend is selected once
|
||||
when each authenticated control session starts and cannot change mid-session;
|
||||
changing the setting affects only new sessions. If preferred CUA is unavailable
|
||||
before a session starts, that session can use compatibility mode.
|
||||
Inspect the detected runtime and selected/effective engine with:
|
||||
|
||||
```powershell
|
||||
hermes-relay computer-use status --json
|
||||
hermes-relay computer-use cua status
|
||||
hermes-relay computer-use cua health # explicit accessibility recheck
|
||||
hermes-relay computer-use cua check-update
|
||||
hermes-relay computer-use cua install --yes
|
||||
hermes-relay computer-use cua update --yes
|
||||
hermes-relay computer-use engine cua # preferred; requires a ready runtime
|
||||
hermes-relay computer-use engine legacy # explicit compatibility backend
|
||||
hermes-relay computer-use cursor on
|
||||
```
|
||||
|
||||
The management UI exposes the same controls under **Settings → Computer
|
||||
control**. CUA is selected only when its canonical Windows package resolves from
|
||||
`%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe`, its supported
|
||||
version and manifest agree, its required tools are present, and its permission mode
|
||||
is not unrestricted. The live health report is an explicit diagnostic while the
|
||||
temporary Windows workaround for trycua/cua#3103 is active. Hermes ignores an unrelated
|
||||
or stale `cua-driver.exe` found earlier on `PATH`.
|
||||
|
||||
Background dispatch is mandatory. If an application cannot accept a
|
||||
background action, the action fails instead of silently stealing focus.
|
||||
**Allow foreground escalation** is reserved for a later explicitly approved
|
||||
path; this release always reports it off and dispatches CUA actions in the
|
||||
background. **Animated agent cursor** shows a virtual, session-scoped pointer
|
||||
for agent activity; it does not move the operator's physical Windows cursor and
|
||||
is not another hardware pointer. Hermes binds driver sessions and snapshot
|
||||
tokens to its own control authority, target PID/window, grant, and fresh
|
||||
snapshot; an element token cannot be reused across windows or after it is
|
||||
consumed or expires.
|
||||
|
||||
CUA Driver is not bundled with the Hermes-Relay installer. The explicit
|
||||
`computer-use cua install|update --yes` commands use the canonical upstream
|
||||
GitHub release manifest and installer. Hermes verifies the manifest's
|
||||
repository/product/version and installer SHA-256 before execution under a
|
||||
sanitized child-process environment, then checks
|
||||
the canonical binary's path, version, own manifest, tool surface, and permission
|
||||
mode. Accessibility health can be rechecked separately. This is release-metadata/checksum validation—not a Windows
|
||||
publisher signature. A native update newer than the supported `>=0.19.3,
|
||||
<0.20.0` range is displayed but refused. There is no silent install/update,
|
||||
and every child invocation forces CUA telemetry off. `hermes-relay update`
|
||||
continues to manage only the CLI and management UI.
|
||||
|
||||
Window-scoped snapshots and semantic actions use the selected control backend.
|
||||
The existing full-display screenshot remains a separate read-only
|
||||
`system_capture` path, so observation does not cause a mid-session backend
|
||||
switch. The local audit and UI Activity timeline record bounded high-level
|
||||
evidence such as backend, background dispatch, control session, target
|
||||
application/window identifiers, action, phase, and verification state.
|
||||
Accessibility text, screenshot bytes, entered values, and raw CUA responses are
|
||||
excluded from that drilldown.
|
||||
|
||||
CUA improves structured screen/input isolation, but it is not a sandbox for
|
||||
general commands. If Commands, PowerShell, or terminal execution is allowed,
|
||||
that trusted path can still use ordinary operating-system automation. Disable
|
||||
raw command access when CUA's scoped UI-control boundary is part of the security
|
||||
model. Full Access can remove ordinary task prompts, but it does not bypass
|
||||
authenticated targeting, sensitive-surface checks, snapshot freshness, UAC or
|
||||
Windows-session boundaries, audit, driver health, or emergency stop.
|
||||
|
||||
### Devices — server-side session management
|
||||
|
||||
```sh
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@hermes-relay/cli",
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@hermes-relay/cli",
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"hermes-relay": "bin/hermes-relay.js"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@hermes-relay/cli",
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"description": "Thin-client CLI for Hermes-Relay — talk to a remote Hermes agent over WSS with pairing auth, stream-renders tool calls and responses to plain stdout.",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[switch]$Run,
|
||||
[string]$CuaDriver = "$env:USERPROFILE\.cua-driver\packages\current\cua-driver.exe"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
if (-not $Run) {
|
||||
Write-Host 'Hermes CUA live acceptance is opt-in because it opens and controls two Calculator windows.'
|
||||
Write-Host 'Re-run with: powershell -ExecutionPolicy Bypass -File scripts/test-cua-windows.ps1 -Run'
|
||||
exit 0
|
||||
}
|
||||
|
||||
if (-not (Test-Path -LiteralPath $CuaDriver -PathType Leaf)) {
|
||||
throw "Canonical CUA Driver was not found at $CuaDriver"
|
||||
}
|
||||
|
||||
Add-Type -TypeDefinition @'
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
public static class HermesCuaAcceptanceNative {
|
||||
[StructLayout(LayoutKind.Sequential)] public struct POINT { public int X; public int Y; }
|
||||
[DllImport("user32.dll")] public static extern bool GetCursorPos(out POINT point);
|
||||
[DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
|
||||
}
|
||||
'@
|
||||
|
||||
function Invoke-Cua([string]$Tool, [hashtable]$Payload) {
|
||||
$json = $Payload | ConvertTo-Json -Depth 12 -Compress
|
||||
$raw = $json | & $CuaDriver call $Tool
|
||||
if ($LASTEXITCODE -ne 0) { throw "CUA $Tool failed with exit code $LASTEXITCODE" }
|
||||
$result = $raw | ConvertFrom-Json
|
||||
if ($result.isError -eq $true) { throw "CUA $Tool rejected the request: $raw" }
|
||||
return $result
|
||||
}
|
||||
|
||||
function Get-DesktopSentinel {
|
||||
$point = New-Object HermesCuaAcceptanceNative+POINT
|
||||
[void][HermesCuaAcceptanceNative]::GetCursorPos([ref]$point)
|
||||
[pscustomobject]@{
|
||||
CursorX = $point.X
|
||||
CursorY = $point.Y
|
||||
Foreground = [HermesCuaAcceptanceNative]::GetForegroundWindow().ToInt64()
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-Unchanged([object]$Before, [string]$Step) {
|
||||
$after = Get-DesktopSentinel
|
||||
if ($after.CursorX -ne $Before.CursorX -or $after.CursorY -ne $Before.CursorY) {
|
||||
throw "$Step moved the physical cursor from ($($Before.CursorX),$($Before.CursorY)) to ($($after.CursorX),$($after.CursorY))"
|
||||
}
|
||||
if ($after.Foreground -ne $Before.Foreground) {
|
||||
throw "$Step changed the foreground HWND from $($Before.Foreground) to $($after.Foreground)"
|
||||
}
|
||||
}
|
||||
|
||||
function Get-Window([object]$Launch) {
|
||||
$window = @($Launch.windows | Where-Object { $_.is_on_screen -ne $false }) | Select-Object -First 1
|
||||
if (-not $window) { throw "CUA launch did not return a usable window: $($Launch | ConvertTo-Json -Depth 8 -Compress)" }
|
||||
return $window
|
||||
}
|
||||
|
||||
$sessionA = "hermes-live-a-$([guid]::NewGuid().ToString('N'))"
|
||||
$sessionB = "hermes-live-b-$([guid]::NewGuid().ToString('N'))"
|
||||
$opened = @()
|
||||
try {
|
||||
$health = Invoke-Cua health_report @{}
|
||||
if ($health.overall -ne 'healthy') { throw "CUA health must be healthy, got $($health.overall)" }
|
||||
|
||||
[void](Invoke-Cua start_session @{ session = $sessionA; capture_scope = 'window' })
|
||||
[void](Invoke-Cua start_session @{ session = $sessionB; capture_scope = 'window' })
|
||||
|
||||
$sentinel = Get-DesktopSentinel
|
||||
$launchA = Invoke-Cua launch_app @{ aumid = 'Microsoft.WindowsCalculator_8wekyb3d8bbwe!App'; creates_new_application_instance = $true; session = $sessionA }
|
||||
$launchB = Invoke-Cua launch_app @{ aumid = 'Microsoft.WindowsCalculator_8wekyb3d8bbwe!App'; creates_new_application_instance = $true; session = $sessionB }
|
||||
$opened = @($launchA.pid, $launchB.pid)
|
||||
Start-Sleep -Milliseconds 800
|
||||
Assert-Unchanged $sentinel 'background launch'
|
||||
|
||||
$windowA = Get-Window $launchA
|
||||
$windowB = Get-Window $launchB
|
||||
if ($windowA.window_id -eq $windowB.window_id) { throw 'isolated sessions resolved to the same Calculator window' }
|
||||
|
||||
$beforeA = Invoke-Cua get_window_state @{ pid = $launchA.pid; window_id = $windowA.window_id; session = $sessionA; query = 'Seven'; include_screenshot = $false }
|
||||
$seven = @($beforeA.elements | Where-Object { $_.label -eq 'Seven' -or $_.name -eq 'Seven' }) | Select-Object -First 1
|
||||
if (-not $seven.element_token) { throw 'Calculator Seven element did not expose an opaque token' }
|
||||
|
||||
[void](Invoke-Cua click @{ pid = $launchA.pid; window_id = $windowA.window_id; element_token = $seven.element_token; session = $sessionA; scope = 'window'; delivery_mode = 'background' })
|
||||
Assert-Unchanged $sentinel 'background element action'
|
||||
$afterA = Invoke-Cua get_window_state @{ pid = $launchA.pid; window_id = $windowA.window_id; session = $sessionA; query = 'Display'; include_screenshot = $false }
|
||||
if ($afterA.snapshot_id -eq $beforeA.snapshot_id) { throw 'post-action snapshot did not advance its generation' }
|
||||
|
||||
$staleRaw = (@{ pid = $launchA.pid; window_id = $windowA.window_id; element_token = $seven.element_token; session = $sessionA; scope = 'window'; delivery_mode = 'background' } | ConvertTo-Json -Compress) | & $CuaDriver call click
|
||||
$stale = $staleRaw | ConvertFrom-Json
|
||||
if ($stale.isError -ne $true) { throw 'stale element token was accepted after a newer snapshot' }
|
||||
|
||||
$cursorA = Invoke-Cua get_agent_cursor_state @{ session = $sessionA }
|
||||
$cursorB = Invoke-Cua get_agent_cursor_state @{ session = $sessionB }
|
||||
if (($cursorA | ConvertTo-Json -Depth 8 -Compress) -eq ($cursorB | ConvertTo-Json -Depth 8 -Compress)) {
|
||||
throw 'two control sessions did not expose isolated cursor state'
|
||||
}
|
||||
|
||||
[void](Invoke-Cua end_session @{ session = $sessionA })
|
||||
$endedRaw = (@{ session = $sessionA } | ConvertTo-Json -Compress) | & $CuaDriver call get_session_state
|
||||
$ended = $endedRaw | ConvertFrom-Json
|
||||
if ($ended.code -ne 'session_not_started') { throw 'ended session remained active' }
|
||||
|
||||
Write-Host 'PASS: physical cursor and foreground stayed unchanged.'
|
||||
Write-Host 'PASS: background Calculator action was bracketed by snapshots.'
|
||||
Write-Host 'PASS: stale token rejection and two isolated cursor sessions were verified.'
|
||||
Write-Host 'PASS: ending a session immediately revoked its CUA state.'
|
||||
} finally {
|
||||
foreach ($session in @($sessionA, $sessionB)) {
|
||||
try { [void](Invoke-Cua end_session @{ session = $session }) } catch { Write-Warning $_ }
|
||||
}
|
||||
Write-Host "Calculator processes created by this acceptance run: $($opened -join ', '). Close them manually after inspection."
|
||||
}
|
||||
@@ -5,7 +5,8 @@
|
||||
// the audit flagged as the biggest desktop-tools transparency gap.
|
||||
|
||||
import type { ParsedArgs } from '../cli.js'
|
||||
import { auditLogPath, readRecentAudit } from '../lib/auditLog.js'
|
||||
import { auditLogPath, auditScreenshotEvidenceStatus, clearAuditScreenshotEvidence, pruneAuditScreenshotEvidence, readRecentAudit } from '../lib/auditLog.js'
|
||||
import { readDesktopUseSettings, setActivityScreenshotRetention } from '../lib/desktopUseSettings.js'
|
||||
import { renderTable } from '../lib/table.js'
|
||||
import { SYMBOLS, theme as makeTheme } from '../lib/theme.js'
|
||||
import { printUsage, type UsageSpec } from '../lib/usage.js'
|
||||
@@ -13,10 +14,12 @@ import { printUsage, type UsageSpec } from '../lib/usage.js'
|
||||
const AUDIT_USAGE: UsageSpec = {
|
||||
name: 'audit',
|
||||
summary: 'show recent desktop-tool activity the agent ran on this machine',
|
||||
usage: ['audit [--limit <n>] [--json]'],
|
||||
usage: ['audit [--limit <n>] [--json]', 'audit screenshots [on|off] [--days <1|7|30>] [--yes] [--json]'],
|
||||
flags: [
|
||||
{ flag: '--limit <n>', desc: 'How many recent entries to show (default 50)' },
|
||||
{ flag: '--json', desc: 'Emit raw audit entries as JSON' }
|
||||
{ flag: '--json', desc: 'Emit raw audit entries as JSON' },
|
||||
{ flag: '--days <1|7|30>', desc: 'Local screenshot retention period' },
|
||||
{ flag: '--yes', desc: 'Confirm a retention change' }
|
||||
],
|
||||
examples: ['hermes-relay audit', 'hermes-relay audit --limit 20']
|
||||
}
|
||||
@@ -36,6 +39,34 @@ export async function auditCommand(args: ParsedArgs): Promise<number> {
|
||||
return 0
|
||||
}
|
||||
|
||||
if (args.positional[0] === 'screenshots') {
|
||||
const settings = await readDesktopUseSettings()
|
||||
const mode = args.positional[1]
|
||||
if (mode === 'on' || mode === 'off') {
|
||||
if (args.flags.yes !== true) {
|
||||
process.stderr.write('audit screenshots: retention changes require --yes\n')
|
||||
return 1
|
||||
}
|
||||
const rawDays = typeof args.flags.days === 'string' ? Number(args.flags.days) : settings.activity_screenshot_retention_days
|
||||
if (rawDays !== 1 && rawDays !== 7 && rawDays !== 30) {
|
||||
process.stderr.write('audit screenshots: --days must be 1, 7, or 30\n')
|
||||
return 1
|
||||
}
|
||||
await setActivityScreenshotRetention(mode === 'on', rawDays)
|
||||
if (mode === 'off') await clearAuditScreenshotEvidence()
|
||||
else await pruneAuditScreenshotEvidence(rawDays)
|
||||
} else if (mode) {
|
||||
process.stderr.write('audit screenshots: expected on or off\n')
|
||||
return 1
|
||||
}
|
||||
const current = await readDesktopUseSettings()
|
||||
const evidence = await auditScreenshotEvidenceStatus()
|
||||
const result = { enabled: current.activity_screenshot_retention_enabled, days: current.activity_screenshot_retention_days, ...evidence }
|
||||
if (args.flags.json) process.stdout.write(JSON.stringify(result, null, 2) + '\n')
|
||||
else process.stdout.write(`Screenshot evidence: ${result.enabled ? `${result.days} days` : 'off'} · ${result.count} file${result.count === 1 ? '' : 's'}\n`)
|
||||
return 0
|
||||
}
|
||||
|
||||
const rawLimit = typeof args.flags.limit === 'string' ? parseInt(args.flags.limit, 10) : 50
|
||||
const limit = Number.isFinite(rawLimit) && rawLimit > 0 ? rawLimit : 50
|
||||
|
||||
|
||||
@@ -5,11 +5,19 @@ import { readDaemonStatus, isDaemonProcessAlive } from '../lib/daemonStatus.js'
|
||||
import {
|
||||
readDesktopUseSettings,
|
||||
requestComputerGrantCancellation,
|
||||
setComputerControlSettings,
|
||||
setDesktopUseEnabled
|
||||
} from '../lib/desktopUseSettings.js'
|
||||
import { listPendingGrantRequests } from '../lib/grantBridge.js'
|
||||
import { theme as makeTheme } from '../lib/theme.js'
|
||||
import { printUsage, type UsageSpec, unknownSubcommand } from '../lib/usage.js'
|
||||
import { CuaDriverAdapter, type CuaRuntimeStatus } from '../tools/cuaDriver.js'
|
||||
import {
|
||||
checkCuaUpdate,
|
||||
getCuaManagementStatus,
|
||||
installCuaDriver,
|
||||
updateCuaDriver
|
||||
} from '../tools/cuaManagement.js'
|
||||
|
||||
const COMPUTER_USE_USAGE: UsageSpec = {
|
||||
name: 'computer-use',
|
||||
@@ -18,21 +26,31 @@ const COMPUTER_USE_USAGE: UsageSpec = {
|
||||
'computer-use status [--json]',
|
||||
'computer-use enable [--yes]',
|
||||
'computer-use disable',
|
||||
'computer-use cancel'
|
||||
'computer-use cancel',
|
||||
'computer-use engine <legacy|cua>',
|
||||
'computer-use cursor <on|off>',
|
||||
'computer-use cua <status|health|install|check-update|update> [--json] [--yes]'
|
||||
],
|
||||
subcommands: [
|
||||
{ verb: 'status', desc: 'Show preference, daemon state, active grant, and pending requests' },
|
||||
{ verb: 'enable', desc: 'Persist desktop-use enablement after explicit confirmation' },
|
||||
{ verb: 'disable', desc: 'Disable desktop use and request cancellation of any active grant' },
|
||||
{ verb: 'cancel', desc: 'Cancel the active task-scoped desktop grant' }
|
||||
{ verb: 'cancel', desc: 'Cancel the active task-scoped desktop grant' },
|
||||
{ verb: 'engine', desc: 'Choose legacy Windows input or a ready CUA Driver backend' },
|
||||
{ verb: 'cursor', desc: 'Show or hide the CUA virtual agent cursor' },
|
||||
{ verb: 'cua', desc: 'Manage the canonical CUA Driver package and recheck accessibility health' }
|
||||
],
|
||||
flags: [
|
||||
{ flag: '--json', desc: 'Emit machine-readable status' },
|
||||
{ flag: '--yes', desc: 'Confirm enablement non-interactively' }
|
||||
{ flag: '--yes', desc: 'Confirm enablement, CUA installation, or CUA update explicitly' }
|
||||
],
|
||||
examples: [
|
||||
'hermes-relay computer-use status',
|
||||
'hermes-relay computer-use enable',
|
||||
'hermes-relay computer-use cua status',
|
||||
'hermes-relay computer-use cua health',
|
||||
'hermes-relay computer-use cua check-update',
|
||||
'hermes-relay computer-use cua install --yes',
|
||||
'hermes-relay computer-use cancel',
|
||||
'hermes-relay computer-use disable'
|
||||
]
|
||||
@@ -62,6 +80,24 @@ async function statusPayload(): Promise<Record<string, unknown>> {
|
||||
const activeGrant = daemonAlive && daemon?.computer_grant?.active === true
|
||||
? daemon.computer_grant
|
||||
: null
|
||||
let cua: CuaRuntimeStatus | null = null
|
||||
try {
|
||||
cua = await CuaDriverAdapter.status()
|
||||
} catch {
|
||||
// The optional backend must not make ordinary desktop-use status fail.
|
||||
}
|
||||
const cuaReason = cua?.reason?.toLowerCase() ?? ''
|
||||
const cuaState = !cua?.available
|
||||
? 'not_installed'
|
||||
: cua.ready
|
||||
? 'ready'
|
||||
: /(?:incompatible|unsupported|version|manifest|permission mode|missing required tools)/.test(cuaReason)
|
||||
? 'incompatible'
|
||||
: /(?:degraded|health)/.test(cuaReason)
|
||||
? 'degraded'
|
||||
: 'error'
|
||||
const cuaReady = cuaState === 'ready'
|
||||
const lifecycle = daemonAlive ? daemon?.computer_control : undefined
|
||||
return {
|
||||
enabled: settings.computer_use_enabled,
|
||||
daemon_alive: daemonAlive,
|
||||
@@ -69,7 +105,32 @@ async function statusPayload(): Promise<Record<string, unknown>> {
|
||||
daemon_computer_use_enabled: daemonAlive ? (daemon?.computer_use_enabled ?? false) : false,
|
||||
active_grant: activeGrant,
|
||||
pending_grants: pending.length,
|
||||
restart_required: daemonAlive && daemon?.computer_use_enabled !== settings.computer_use_enabled
|
||||
restart_required: daemonAlive && daemon?.computer_use_enabled !== settings.computer_use_enabled,
|
||||
computer_control_engine: {
|
||||
selected: settings.computer_control_engine,
|
||||
effective: settings.computer_control_engine === 'cua'
|
||||
? lifecycle?.active_backend === 'cua'
|
||||
? 'cua'
|
||||
: lifecycle?.active_backend === 'legacy_compat'
|
||||
? 'legacy'
|
||||
: cuaReady ? 'cua' : 'legacy'
|
||||
: 'legacy',
|
||||
available: cua?.available === true,
|
||||
state: cuaState,
|
||||
version: cua?.binaryVersion ?? null,
|
||||
health: cua?.health ?? null,
|
||||
path: cua?.binaryPath ?? null,
|
||||
cursor_enabled: settings.cua_cursor_enabled,
|
||||
active_sessions: lifecycle?.active_sessions ?? 0,
|
||||
active_backend: lifecycle?.active_backend ?? 'idle',
|
||||
last_action: lifecycle?.last_action ?? null,
|
||||
foreground_escalation_enabled: false,
|
||||
message: settings.computer_control_engine === 'cua' && !cuaReady
|
||||
? cuaState === 'degraded'
|
||||
? `CUA Driver is installed, but UI Automation is degraded; new sessions use Windows Input compatibility mode. ${cua?.reason ?? ''}`.trim()
|
||||
: `CUA Driver is unavailable before control starts; new sessions use Windows Input compatibility mode. ${cua?.reason ?? ''}`.trim()
|
||||
: cua?.reason ?? null
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,6 +142,68 @@ export async function computerUseCommand(args: ParsedArgs): Promise<number> {
|
||||
}
|
||||
|
||||
const subcommand = args.positional[0] ?? 'status'
|
||||
if (subcommand === 'cua') {
|
||||
const action = args.positional[1] ?? 'status'
|
||||
const json = args.flags.json === true
|
||||
try {
|
||||
if (action === 'health') {
|
||||
const health = await CuaDriverAdapter.healthStatus()
|
||||
if (json) process.stdout.write(JSON.stringify(health, null, 2) + '\n')
|
||||
else {
|
||||
process.stdout.write(t.bold('CUA Driver accessibility health') + `\n state: ${health.state}\n`)
|
||||
if (health.reason) process.stdout.write(t.warnLine(` ${health.reason}`) + '\n')
|
||||
process.stdout.write(t.muted(' This diagnostic does not disable the runtime while the temporary Windows compatibility policy is active.') + '\n')
|
||||
}
|
||||
// The probe result is data, not command failure. Callers (including the
|
||||
// tray) inspect state while still receiving the JSON for degradation.
|
||||
return 0
|
||||
}
|
||||
const payload = action === 'status'
|
||||
? await getCuaManagementStatus()
|
||||
: action === 'check-update'
|
||||
? await checkCuaUpdate()
|
||||
: action === 'install'
|
||||
? args.flags.yes === true
|
||||
? await installCuaDriver()
|
||||
: null
|
||||
: action === 'update'
|
||||
? args.flags.yes === true
|
||||
? await updateCuaDriver()
|
||||
: null
|
||||
: undefined
|
||||
if (payload === undefined) {
|
||||
process.stderr.write(t.err('cua action must be status, health, install, check-update, or update') + '\n')
|
||||
return 1
|
||||
}
|
||||
if (payload === null) {
|
||||
process.stderr.write(t.err(`CUA ${action} requires explicit confirmation with --yes`) + '\n')
|
||||
return 1
|
||||
}
|
||||
if (json) {
|
||||
process.stdout.write(JSON.stringify(payload, null, 2) + '\n')
|
||||
return 0
|
||||
}
|
||||
const version = payload.current_version ?? 'not installed'
|
||||
process.stdout.write(t.bold('CUA Driver') + `\n version: ${version}\n`)
|
||||
process.stdout.write(` package: ${payload.canonical_path ?? 'not installed'}\n`)
|
||||
process.stdout.write(` compatibility: ${payload.compatible ? 'supported' : payload.compatibility_reason ?? 'not ready'}\n`)
|
||||
if (payload.stale_path_shim) {
|
||||
process.stdout.write(t.warnLine(` PATH resolves a competing copy: ${payload.discovered_path}`) + '\n')
|
||||
process.stdout.write(t.muted(' Hermes uses the canonical package/current install instead.') + '\n')
|
||||
}
|
||||
if (payload.update) {
|
||||
if (payload.update.error) process.stdout.write(t.warnLine(` update check: ${payload.update.error}`) + '\n')
|
||||
else if (payload.update.update_available) {
|
||||
process.stdout.write(` update: ${payload.update.latest_version}${payload.update.compatible ? ' available' : ' available but unsupported'}\n`)
|
||||
} else process.stdout.write(' update: up to date\n')
|
||||
}
|
||||
if (payload.operation) process.stdout.write(t.okLine(`CUA ${payload.operation.kind} completed`) + '\n')
|
||||
return 0
|
||||
} catch (error) {
|
||||
process.stderr.write(t.err(error instanceof Error ? error.message : String(error)) + '\n')
|
||||
return 1
|
||||
}
|
||||
}
|
||||
if (subcommand === 'status') {
|
||||
const payload = await statusPayload()
|
||||
if (args.flags.json) {
|
||||
@@ -129,6 +252,42 @@ export async function computerUseCommand(args: ParsedArgs): Promise<number> {
|
||||
return 0
|
||||
}
|
||||
|
||||
if (subcommand === 'engine') {
|
||||
const engine = args.positional[1]
|
||||
if (engine !== 'legacy' && engine !== 'cua') {
|
||||
process.stderr.write(t.err('engine must be legacy or cua') + '\n')
|
||||
return 1
|
||||
}
|
||||
if (engine === 'cua') {
|
||||
const payload = await statusPayload()
|
||||
const status = payload.computer_control_engine as { state?: string }
|
||||
if (status.state !== 'ready') {
|
||||
process.stderr.write(t.err('CUA Driver is not ready; engine selection was not changed') + '\n')
|
||||
return 1
|
||||
}
|
||||
}
|
||||
await setComputerControlSettings({ computer_control_engine: engine })
|
||||
process.stdout.write(t.okLine(`computer control engine set to ${engine}`) + '\n')
|
||||
return 0
|
||||
}
|
||||
|
||||
if (subcommand === 'cursor') {
|
||||
const value = args.positional[1]
|
||||
if (value !== 'on' && value !== 'off') {
|
||||
process.stderr.write(t.err(`${subcommand} must be on or off`) + '\n')
|
||||
return 1
|
||||
}
|
||||
const payload = await statusPayload()
|
||||
const status = payload.computer_control_engine as { selected?: string; state?: string }
|
||||
if (status.selected !== 'cua' || status.state !== 'ready') {
|
||||
process.stderr.write(t.err(`CUA Driver must be selected and ready before changing ${subcommand}`) + '\n')
|
||||
return 1
|
||||
}
|
||||
await setComputerControlSettings({ cua_cursor_enabled: value === 'on' })
|
||||
process.stdout.write(t.okLine(`CUA ${subcommand} ${value}`) + '\n')
|
||||
return 0
|
||||
}
|
||||
|
||||
return unknownSubcommand(COMPUTER_USE_USAGE, subcommand, t)
|
||||
}
|
||||
|
||||
|
||||
@@ -50,6 +50,7 @@ import {
|
||||
type DaemonStatus
|
||||
} from '../lib/daemonStatus.js'
|
||||
import { rpcErrorMessage, asRpcResult } from '../lib/rpc.js'
|
||||
import { appendAudit } from '../lib/auditLog.js'
|
||||
import { effectiveHostAccessMode, effectiveHostCapabilityPolicies, getHostAccessMode, getHostCapabilityPolicies } from '../lib/hostAccessPolicy.js'
|
||||
import { theme as makeTheme } from '../lib/theme.js'
|
||||
import { printUsage, type UsageSpec } from '../lib/usage.js'
|
||||
@@ -62,12 +63,13 @@ import {
|
||||
shouldAdvertiseComputerUse
|
||||
} from '../tools/handlerSet.js'
|
||||
import {
|
||||
cancelComputerGrant,
|
||||
cancelAllComputerGrants,
|
||||
configureComputerUseRuntime,
|
||||
getActiveComputerGrant,
|
||||
expireComputerControlSessions,
|
||||
setComputerGrantChangeListener,
|
||||
type ComputerGrant
|
||||
} from '../tools/computerGrants.js'
|
||||
import { closeCuaControlSession, setComputerControlLifecycleListener } from '../tools/cuaDriver.js'
|
||||
import { DesktopToolRouter } from '../tools/router.js'
|
||||
import { configureCapabilityPolicies } from '../tools/capabilityRuntime.js'
|
||||
import { adbBackendAvailable } from '../tools/handlers/adb.js'
|
||||
@@ -853,17 +855,26 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
|
||||
? (info as { attempt?: number; delayMs?: number })
|
||||
: {}
|
||||
log.warn({ event: 'reconnecting', attempt: attempt ?? null, delay_ms: delayMs ?? null })
|
||||
updateStatus({ state: 'reconnecting', last_event: 'reconnecting' })
|
||||
const retryAt = nowSec() + Math.ceil((delayMs ?? 0) / 1000)
|
||||
updateStatus({ state: 'reconnecting', last_event: 'reconnecting', reconnect_attempt: attempt ?? null, retry_at: retryAt, last_error: 'Relay connection interrupted' })
|
||||
if ((attempt ?? 1) === 1) {
|
||||
void appendAudit({
|
||||
ts: Date.now(), kind: 'connection.state', tool: 'daemon.reconnecting', category: 'system', ok: false,
|
||||
host_url: configuredUrl, summary: 'Automatic reconnect started', error: 'Relay connection interrupted'
|
||||
})
|
||||
}
|
||||
})
|
||||
relay.on('reconnected', () => {
|
||||
log.info({ event: 'reconnected' })
|
||||
updateStatus({ state: 'connected', last_event: 'reconnected' })
|
||||
updateStatus({ state: 'connected', last_event: 'reconnected', reconnect_attempt: null, retry_at: null, last_error: null })
|
||||
void appendAudit({ ts: Date.now(), kind: 'connection.state', tool: 'daemon.reconnected', category: 'system', ok: true, host_url: configuredUrl, summary: 'Relay tunnel restored' })
|
||||
})
|
||||
relay.on('exit', (code: unknown) => {
|
||||
// Transport gave up (auth.fail, reconnect gate returned false, or
|
||||
// reconnect attempts exhausted). Daemon exits non-zero so the
|
||||
// service manager decides whether to restart.
|
||||
log.error({ event: 'transport_exited', code: typeof code === 'number' ? code : null })
|
||||
void appendAudit({ ts: Date.now(), kind: 'connection.state', tool: 'daemon.disconnected', category: 'system', ok: false, host_url: configuredUrl, summary: 'Relay transport stopped', error: 'Automatic reconnect stopped' })
|
||||
// Defer exit so the log line flushes before the process dies.
|
||||
setImmediate(() => process.exit(1))
|
||||
})
|
||||
@@ -873,6 +884,8 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
|
||||
const outcome = await relay.whenAuthResolved()
|
||||
if (!outcome.ok) {
|
||||
log.error({ event: 'auth_failed', reason: outcome.reason })
|
||||
updateStatus({ state: 'stopped', last_event: 'auth_failed', last_error: outcome.reason, reconnect_attempt: null, retry_at: null })
|
||||
void appendAudit({ ts: Date.now(), kind: 'connection.state', tool: 'daemon.auth_failed', category: 'system', ok: false, host_url: configuredUrl, summary: 'Relay authentication failed', error: outcome.reason })
|
||||
try {
|
||||
relay.kill()
|
||||
} catch {
|
||||
@@ -886,7 +899,7 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
|
||||
server_version: relay.serverVersion ?? null,
|
||||
transport: relay.authMeta?.transportHint ?? null
|
||||
})
|
||||
updateStatus({ state: 'connected', server_version: relay.serverVersion ?? null, last_event: 'authed' })
|
||||
updateStatus({ state: 'connected', server_version: relay.serverVersion ?? null, last_event: 'authed', reconnect_attempt: null, retry_at: null, last_error: null })
|
||||
|
||||
// Signal downstream handlers that we're running headless. The router
|
||||
// also checks this env var in its detectInteractive() fallback, so any
|
||||
@@ -925,8 +938,14 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
|
||||
expires_at: grant?.expires_at ?? null,
|
||||
reason: grant?.reason
|
||||
})
|
||||
const restoreGrantListener = setComputerGrantChangeListener(grant => {
|
||||
const restoreGrantListener = setComputerGrantChangeListener((grant, controlSessionId) => {
|
||||
updateStatus({ computer_grant: toDaemonGrantStatus(grant), last_event: 'grant_changed' })
|
||||
if (!grant && controlSessionId) {
|
||||
void closeCuaControlSession(controlSessionId, 'computer grant ended')
|
||||
}
|
||||
})
|
||||
const restoreControlLifecycleListener = setComputerControlLifecycleListener(computerControl => {
|
||||
updateStatus({ computer_control: computerControl })
|
||||
})
|
||||
|
||||
let cancellationCheckRunning = false
|
||||
@@ -936,10 +955,10 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
|
||||
try {
|
||||
const request = await consumeComputerGrantCancellation()
|
||||
if (request) {
|
||||
const result = cancelComputerGrant(request.reason)
|
||||
const result = { cancelled: cancelAllComputerGrants(request.reason) }
|
||||
log.info({ event: 'computer_grant_cancelled_locally', reason: request.reason, result })
|
||||
} else {
|
||||
getActiveComputerGrant()
|
||||
expireComputerControlSessions()
|
||||
}
|
||||
} catch (error) {
|
||||
log.warn({ event: 'computer_grant_control_failed', message: rpcErrorMessage(error) })
|
||||
@@ -1027,6 +1046,7 @@ export async function daemonCommand(args: ParsedArgs): Promise<number> {
|
||||
clearInterval(statusHeartbeat)
|
||||
clearInterval(grantControlInterval)
|
||||
restoreGrantListener()
|
||||
restoreControlLifecycleListener()
|
||||
try {
|
||||
await clearDaemonStatus()
|
||||
} catch {
|
||||
|
||||
+145
-3
@@ -9,7 +9,8 @@
|
||||
//
|
||||
// Best-effort by design: a logging failure must never break a tool dispatch.
|
||||
|
||||
import { appendFile, mkdir, readFile, rename, stat } from 'node:fs/promises'
|
||||
import { appendFile, mkdir, readFile, readdir, rename, stat, unlink, writeFile } from 'node:fs/promises'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { homedir } from 'node:os'
|
||||
import { dirname, join } from 'node:path'
|
||||
|
||||
@@ -17,7 +18,7 @@ export interface AuditEntry {
|
||||
/** Epoch milliseconds when the command completed. */
|
||||
ts: number
|
||||
/** Stable event type for consumers that do not want to infer it from fields. */
|
||||
kind?: 'tool.completed'
|
||||
kind?: 'tool.completed' | 'connection.state'
|
||||
tool: string
|
||||
category?: AuditCategory
|
||||
ok: boolean
|
||||
@@ -25,6 +26,20 @@ export interface AuditEntry {
|
||||
request_id?: string
|
||||
/** Relay identity that issued the action, when known. */
|
||||
host_url?: string
|
||||
relay_session_id?: string
|
||||
requester_device_id?: string
|
||||
run_id?: string
|
||||
target_device_id?: string
|
||||
backend?: 'cua' | 'legacy_compat' | 'system_capture'
|
||||
dispatch?: 'background' | 'foreground_compatibility'
|
||||
control_session_id?: string
|
||||
target_app?: string
|
||||
target_title?: string
|
||||
target_pid?: number
|
||||
target_window_id?: number
|
||||
action?: string
|
||||
verification?: 'snapshot_captured' | 'not_requested' | 'failed'
|
||||
phase?: 'structured_primary' | 'explicit_compatibility' | 'pre_session_safe_fallback'
|
||||
/** Handler wall time, excluding relay transport latency. */
|
||||
duration_ms?: number
|
||||
/** Process exit code when the handler returns one. Non-zero is attention-worthy. */
|
||||
@@ -45,6 +60,11 @@ export interface AuditEntry {
|
||||
/** Short success summary (path / exit code / first stdout line). */
|
||||
summary?: string
|
||||
error?: string
|
||||
/** Opaque local evidence identifier. Screenshot pixels never enter JSONL. */
|
||||
screenshot_evidence_id?: string
|
||||
screenshot_mime_type?: 'image/png'
|
||||
screenshot_width?: number
|
||||
screenshot_height?: number
|
||||
}
|
||||
|
||||
export type AuditCategory = 'command' | 'files' | 'screen' | 'input' | 'devices' | 'system' | 'other'
|
||||
@@ -52,11 +72,109 @@ export type AuditCategory = 'command' | 'files' | 'screen' | 'input' | 'devices'
|
||||
/** Rotate the log once it crosses ~1 MB, keeping a single `.1` backup. */
|
||||
const MAX_BYTES = 1_000_000
|
||||
const MAX_DETAIL_BYTES = 32_768
|
||||
const MAX_SCREENSHOT_BYTES = 10_000_000
|
||||
const MAX_SCREENSHOT_FILES = 20
|
||||
|
||||
export function auditLogPath(): string {
|
||||
return join(homedir(), '.hermes', 'desktop-audit.jsonl')
|
||||
}
|
||||
|
||||
export function auditEvidenceDirectory(): string {
|
||||
return process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR ?? join(homedir(), '.hermes', 'activity-evidence')
|
||||
}
|
||||
|
||||
type ScreenshotEvidence = Pick<AuditEntry, 'screenshot_evidence_id' | 'screenshot_mime_type' | 'screenshot_width' | 'screenshot_height'>
|
||||
|
||||
function screenshotPayload(result: unknown): { base64: string; width?: number; height?: number } | null {
|
||||
if (!result || typeof result !== 'object' || Array.isArray(result)) return null
|
||||
const record = result as Record<string, unknown>
|
||||
const nested = record.after_screenshot
|
||||
if (nested && typeof nested === 'object' && !Array.isArray(nested)) {
|
||||
const found = screenshotPayload(nested)
|
||||
if (found) return found
|
||||
}
|
||||
const base64 = typeof record.screenshot_base64 === 'string'
|
||||
? record.screenshot_base64
|
||||
: typeof record.bytes_base64 === 'string'
|
||||
? record.bytes_base64
|
||||
: null
|
||||
if (!base64) return null
|
||||
const display = record.display && typeof record.display === 'object' && !Array.isArray(record.display)
|
||||
? record.display as Record<string, unknown>
|
||||
: null
|
||||
const width = typeof record.screenshot_width === 'number' ? record.screenshot_width
|
||||
: typeof display?.width === 'number' ? display.width : undefined
|
||||
const height = typeof record.screenshot_height === 'number' ? record.screenshot_height
|
||||
: typeof display?.height === 'number' ? display.height : undefined
|
||||
return { base64, width, height }
|
||||
}
|
||||
|
||||
async function pruneScreenshotEvidence(directory: string, retentionDays: number): Promise<void> {
|
||||
const now = Date.now()
|
||||
const maxAgeMs = retentionDays * 24 * 60 * 60 * 1000
|
||||
const records = await Promise.all((await readdir(directory, { withFileTypes: true }))
|
||||
.filter(entry => entry.isFile() && /^[a-f0-9]{32}\.png$/.test(entry.name))
|
||||
.map(async entry => ({ name: entry.name, stats: await stat(join(directory, entry.name)) })))
|
||||
records.sort((left, right) => right.stats.mtimeMs - left.stats.mtimeMs)
|
||||
await Promise.all(records
|
||||
.filter((record, index) => index >= MAX_SCREENSHOT_FILES || now - record.stats.mtimeMs > maxAgeMs)
|
||||
.map(record => unlink(join(directory, record.name)).catch(() => {})))
|
||||
}
|
||||
|
||||
export async function pruneAuditScreenshotEvidence(retentionDays: 1 | 7 | 30): Promise<void> {
|
||||
try {
|
||||
await pruneScreenshotEvidence(auditEvidenceDirectory(), retentionDays)
|
||||
} catch {
|
||||
/* Missing or unreadable evidence is equivalent to an empty store. */
|
||||
}
|
||||
}
|
||||
|
||||
/** Retain a bounded, local screenshot for the activity evidence viewer. The
|
||||
* audit log stores only an opaque identifier; clearing activity removes the
|
||||
* evidence directory too. */
|
||||
export async function persistAuditScreenshot(result: unknown, requestId: string, retentionDays = 7): Promise<ScreenshotEvidence> {
|
||||
const payload = screenshotPayload(result)
|
||||
if (!payload) return {}
|
||||
let bytes: Buffer
|
||||
try {
|
||||
bytes = Buffer.from(payload.base64, 'base64')
|
||||
} catch {
|
||||
return {}
|
||||
}
|
||||
if (!bytes.length || bytes.length > MAX_SCREENSHOT_BYTES || bytes.subarray(0, 8).toString('hex') !== '89504e470d0a1a0a') return {}
|
||||
const id = createHash('sha256').update(`${requestId}:${Date.now()}`).digest('hex').slice(0, 32)
|
||||
const directory = auditEvidenceDirectory()
|
||||
try {
|
||||
await mkdir(directory, { recursive: true })
|
||||
await writeFile(join(directory, `${id}.png`), bytes, { mode: 0o600, flag: 'wx' })
|
||||
await pruneScreenshotEvidence(directory, retentionDays)
|
||||
return {
|
||||
screenshot_evidence_id: id,
|
||||
screenshot_mime_type: 'image/png',
|
||||
...(payload.width ? { screenshot_width: payload.width } : {}),
|
||||
...(payload.height ? { screenshot_height: payload.height } : {})
|
||||
}
|
||||
} catch {
|
||||
return {}
|
||||
}
|
||||
}
|
||||
|
||||
export async function clearAuditScreenshotEvidence(): Promise<void> {
|
||||
const { rm } = await import('node:fs/promises')
|
||||
await rm(auditEvidenceDirectory(), { recursive: true, force: true })
|
||||
}
|
||||
|
||||
export async function auditScreenshotEvidenceStatus(): Promise<{ count: number; bytes: number }> {
|
||||
try {
|
||||
const records = await Promise.all((await readdir(auditEvidenceDirectory(), { withFileTypes: true }))
|
||||
.filter(entry => entry.isFile() && /^[a-f0-9]{32}\.png$/.test(entry.name))
|
||||
.map(entry => stat(join(auditEvidenceDirectory(), entry.name))))
|
||||
return { count: records.length, bytes: records.reduce((total, value) => total + value.size, 0) }
|
||||
} catch {
|
||||
return { count: 0, bytes: 0 }
|
||||
}
|
||||
}
|
||||
|
||||
export async function appendAudit(entry: AuditEntry): Promise<void> {
|
||||
const path = auditLogPath()
|
||||
try {
|
||||
@@ -127,7 +245,8 @@ function boundedText(value: string): { text: string; truncated: boolean } {
|
||||
* bodies, environment values, or unbounded process output. */
|
||||
export function auditDetails(
|
||||
args: Record<string, unknown>,
|
||||
result?: unknown
|
||||
result?: unknown,
|
||||
options: { redactComputerContent?: boolean } = {}
|
||||
): Pick<AuditEntry, 'request_detail' | 'stdout' | 'stderr' | 'result_detail' | 'request_truncated' | 'stdout_truncated' | 'stderr_truncated' | 'result_truncated'> {
|
||||
const safeRequest: Record<string, unknown> = {}
|
||||
for (const key of [
|
||||
@@ -141,6 +260,29 @@ export function auditDetails(
|
||||
request_detail: request.text,
|
||||
request_truncated: request.truncated || undefined
|
||||
}
|
||||
if (options.redactComputerContent) {
|
||||
if (result && typeof result === 'object' && !Array.isArray(result)) {
|
||||
const record = result as Record<string, unknown>
|
||||
const target = record.target && typeof record.target === 'object' && !Array.isArray(record.target)
|
||||
? record.target as Record<string, unknown>
|
||||
: {}
|
||||
const safeResult = {
|
||||
backend: record.backend,
|
||||
action: record.action,
|
||||
status: record.status,
|
||||
code: record.code,
|
||||
target: {
|
||||
pid: target.pid,
|
||||
windowId: target.windowId
|
||||
},
|
||||
redacted: true
|
||||
}
|
||||
const value = boundedText(JSON.stringify(safeResult, null, 2))
|
||||
details.result_detail = value.text
|
||||
details.result_truncated = value.truncated || undefined
|
||||
}
|
||||
return details
|
||||
}
|
||||
if (!result || typeof result !== 'object') {
|
||||
if (result !== undefined) {
|
||||
const value = boundedText(String(result))
|
||||
|
||||
@@ -14,6 +14,7 @@ import { execFileSync } from 'node:child_process'
|
||||
|
||||
import type { ProcessPrivilege } from './processPrivilege.js'
|
||||
import type { HostAccessMode } from './hostAccessPolicy.js'
|
||||
import type { ComputerControlLifecycleStatus } from '../tools/cuaDriver.js'
|
||||
|
||||
export type DaemonState = 'starting' | 'connected' | 'reconnecting' | 'stopped'
|
||||
|
||||
@@ -44,12 +45,16 @@ export interface DaemonStatus {
|
||||
advertised_tools?: number
|
||||
voice_url?: string | null
|
||||
last_event?: string
|
||||
reconnect_attempt?: number | null
|
||||
retry_at?: number | null
|
||||
last_error?: string | null
|
||||
username?: string
|
||||
privilege?: ProcessPrivilege
|
||||
computer_use_enabled?: boolean
|
||||
access_mode?: HostAccessMode
|
||||
tools_enabled?: boolean
|
||||
computer_grant?: DaemonComputerGrantStatus
|
||||
computer_control?: ComputerControlLifecycleStatus
|
||||
}
|
||||
|
||||
export function daemonStatusPath(): string {
|
||||
|
||||
@@ -7,6 +7,10 @@ import { grantBridgeDir } from './grantBridge.js'
|
||||
|
||||
export interface DesktopUseSettings {
|
||||
computer_use_enabled: boolean
|
||||
computer_control_engine: 'legacy' | 'cua'
|
||||
cua_cursor_enabled: boolean
|
||||
activity_screenshot_retention_enabled: boolean
|
||||
activity_screenshot_retention_days: 1 | 7 | 30
|
||||
updated_at?: string
|
||||
}
|
||||
|
||||
@@ -27,22 +31,36 @@ export function computerGrantCancellationPath(): string {
|
||||
|
||||
function normalizeSettings(value: unknown): DesktopUseSettings {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
return { computer_use_enabled: false }
|
||||
return defaultSettings()
|
||||
}
|
||||
const raw = value as Partial<DesktopUseSettings>
|
||||
return {
|
||||
computer_use_enabled: raw.computer_use_enabled === true,
|
||||
computer_control_engine: raw.computer_control_engine === 'legacy' ? 'legacy' : 'cua',
|
||||
cua_cursor_enabled: raw.cua_cursor_enabled === true,
|
||||
activity_screenshot_retention_enabled: raw.activity_screenshot_retention_enabled !== false,
|
||||
activity_screenshot_retention_days: raw.activity_screenshot_retention_days === 1 || raw.activity_screenshot_retention_days === 30 ? raw.activity_screenshot_retention_days : 7,
|
||||
updated_at: typeof raw.updated_at === 'string' ? raw.updated_at : undefined
|
||||
}
|
||||
}
|
||||
|
||||
function defaultSettings(): DesktopUseSettings {
|
||||
return {
|
||||
computer_use_enabled: false,
|
||||
computer_control_engine: 'cua',
|
||||
cua_cursor_enabled: false,
|
||||
activity_screenshot_retention_enabled: true,
|
||||
activity_screenshot_retention_days: 7
|
||||
}
|
||||
}
|
||||
|
||||
export function readDesktopUseSettingsSync(
|
||||
filePath = desktopUseSettingsPath()
|
||||
): DesktopUseSettings {
|
||||
try {
|
||||
return normalizeSettings(JSON.parse(readFileSync(filePath, 'utf8')))
|
||||
} catch {
|
||||
return { computer_use_enabled: false }
|
||||
return defaultSettings()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -52,7 +70,7 @@ export async function readDesktopUseSettings(
|
||||
try {
|
||||
return normalizeSettings(JSON.parse(await readFile(filePath, 'utf8')))
|
||||
} catch {
|
||||
return { computer_use_enabled: false }
|
||||
return defaultSettings()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -68,6 +86,7 @@ export async function setDesktopUseEnabled(
|
||||
filePath = desktopUseSettingsPath()
|
||||
): Promise<DesktopUseSettings> {
|
||||
const settings: DesktopUseSettings = {
|
||||
...await readDesktopUseSettings(filePath),
|
||||
computer_use_enabled: enabled,
|
||||
updated_at: new Date().toISOString()
|
||||
}
|
||||
@@ -75,6 +94,35 @@ export async function setDesktopUseEnabled(
|
||||
return settings
|
||||
}
|
||||
|
||||
export async function setComputerControlSettings(
|
||||
update: Partial<Pick<DesktopUseSettings,
|
||||
'computer_control_engine' | 'cua_cursor_enabled'>>,
|
||||
filePath = desktopUseSettingsPath()
|
||||
): Promise<DesktopUseSettings> {
|
||||
const settings: DesktopUseSettings = {
|
||||
...await readDesktopUseSettings(filePath),
|
||||
...update,
|
||||
updated_at: new Date().toISOString()
|
||||
}
|
||||
await writeJsonAtomic(filePath, settings)
|
||||
return settings
|
||||
}
|
||||
|
||||
export async function setActivityScreenshotRetention(
|
||||
enabled: boolean,
|
||||
days: 1 | 7 | 30,
|
||||
filePath = desktopUseSettingsPath()
|
||||
): Promise<DesktopUseSettings> {
|
||||
const settings: DesktopUseSettings = {
|
||||
...await readDesktopUseSettings(filePath),
|
||||
activity_screenshot_retention_enabled: enabled,
|
||||
activity_screenshot_retention_days: days,
|
||||
updated_at: new Date().toISOString()
|
||||
}
|
||||
await writeJsonAtomic(filePath, settings)
|
||||
return settings
|
||||
}
|
||||
|
||||
export async function requestComputerGrantCancellation(
|
||||
reason = 'cancelled from local desktop controls',
|
||||
filePath = computerGrantCancellationPath()
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
|
||||
export const LEGACY_CONTROL_SESSION_ID = 'legacy-local-control-session'
|
||||
|
||||
export interface ComputerControlAuthority {
|
||||
/** Server-attested identity for one attached desktop router lifecycle. */
|
||||
controlSessionId: string
|
||||
/** Canonical relay URL when known. Never supplied by remote tool arguments. */
|
||||
hostUrl?: string
|
||||
/** Request currently being dispatched. Bound by the router, not the caller. */
|
||||
requestId?: string
|
||||
/** Reserved for authenticated relay identity once the wire protocol supplies it. */
|
||||
relaySessionId?: string
|
||||
requesterDeviceId?: string
|
||||
chatSessionId?: string
|
||||
runId?: string
|
||||
targetDeviceId?: string
|
||||
}
|
||||
|
||||
export interface ComputerTarget {
|
||||
pid: number
|
||||
windowId: number
|
||||
app?: string
|
||||
title?: string
|
||||
executable?: string
|
||||
display?: string
|
||||
folder?: string
|
||||
}
|
||||
|
||||
export interface SnapshotBindingInput {
|
||||
authority: ComputerControlAuthority
|
||||
grantId: string | null
|
||||
target: ComputerTarget
|
||||
snapshotGeneration: string
|
||||
driverElementToken?: string
|
||||
ttlMs?: number
|
||||
}
|
||||
|
||||
interface SnapshotBinding extends SnapshotBindingInput {
|
||||
expiresAt: number
|
||||
}
|
||||
|
||||
export type SensitiveTargetDecision =
|
||||
| { allowed: true; reason: 'not_sensitive' }
|
||||
| { allowed: false; reason: 'missing_target_identity' | 'sensitive_target'; matched?: string }
|
||||
|
||||
const DEFAULT_SENSITIVE_PATTERNS = Object.freeze([
|
||||
/\b(password|passkey|credential|authenticator|security key)\b/i,
|
||||
/\b(1password|bitwarden|keepass|lastpass|dashlane|proton pass)\b/i,
|
||||
/\b(bank|banking|payment|checkout|wallet|crypto|cryptocurrency)\b/i,
|
||||
/\b(certificate|private[ _-]?key|api[ _-]?key|access[ _-]?token|secret)\b/i,
|
||||
/\b(user account control|windows security|credential manager|security settings)\b/i
|
||||
])
|
||||
|
||||
/**
|
||||
* Hermes-owned state for a single router lifecycle. It intentionally does not
|
||||
* execute input. It binds remote requests, local grants, snapshots, and opaque
|
||||
* element handles before an execution backend (legacy or CUA) is invoked.
|
||||
*/
|
||||
export class ComputerControlSecurityState {
|
||||
private readonly seenRequestIds = new Map<string, { at: number; controlSessionId: string }>()
|
||||
private readonly snapshotBindings = new Map<string, SnapshotBinding>()
|
||||
|
||||
constructor(
|
||||
readonly authority: ComputerControlAuthority,
|
||||
private readonly maxRememberedRequests = 2_048
|
||||
) {}
|
||||
|
||||
bindRequest(
|
||||
requestId: string,
|
||||
authority: ComputerControlAuthority = this.authority
|
||||
): ComputerControlAuthority | null {
|
||||
const normalized = requestId.trim()
|
||||
if (!normalized || normalized.length > 256 || this.seenRequestIds.has(normalized)) return null
|
||||
this.seenRequestIds.set(normalized, { at: Date.now(), controlSessionId: authority.controlSessionId })
|
||||
while (this.seenRequestIds.size > this.maxRememberedRequests) {
|
||||
const oldest = this.seenRequestIds.keys().next().value as string | undefined
|
||||
if (!oldest) break
|
||||
this.seenRequestIds.delete(oldest)
|
||||
}
|
||||
return { ...authority, requestId: normalized }
|
||||
}
|
||||
|
||||
issueSnapshotToken(input: SnapshotBindingInput): string {
|
||||
if (!input.authority.controlSessionId.trim()) throw new Error('snapshot authority requires a control session id')
|
||||
const token = `hermes-snapshot-${randomUUID()}`
|
||||
const ttlMs = Math.max(1_000, Math.min(input.ttlMs ?? 60_000, 300_000))
|
||||
this.snapshotBindings.set(token, { ...input, expiresAt: Date.now() + ttlMs })
|
||||
return token
|
||||
}
|
||||
|
||||
consumeSnapshotToken(
|
||||
token: string,
|
||||
expected: Omit<SnapshotBindingInput, 'ttlMs' | 'driverElementToken' | 'snapshotGeneration'> & {
|
||||
snapshotGeneration?: string
|
||||
}
|
||||
): SnapshotBinding | null {
|
||||
const binding = this.snapshotBindings.get(token)
|
||||
this.snapshotBindings.delete(token)
|
||||
if (!binding || binding.expiresAt <= Date.now()) return null
|
||||
if (binding.authority.controlSessionId !== expected.authority.controlSessionId) return null
|
||||
if (binding.grantId !== expected.grantId) return null
|
||||
if (binding.target.pid !== expected.target.pid || binding.target.windowId !== expected.target.windowId) return null
|
||||
if (expected.snapshotGeneration && binding.snapshotGeneration !== expected.snapshotGeneration) return null
|
||||
return binding
|
||||
}
|
||||
|
||||
revoke(): void {
|
||||
this.seenRequestIds.clear()
|
||||
this.snapshotBindings.clear()
|
||||
}
|
||||
|
||||
/** Revoke only artifacts owned by one concurrent relay control session. */
|
||||
revokeAuthority(controlSessionId: string): void {
|
||||
for (const [requestId, binding] of this.seenRequestIds) {
|
||||
if (binding.controlSessionId === controlSessionId) this.seenRequestIds.delete(requestId)
|
||||
}
|
||||
for (const [token, binding] of this.snapshotBindings) {
|
||||
if (binding.authority.controlSessionId === controlSessionId) this.snapshotBindings.delete(token)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function hasAuthenticatedControlIdentity(
|
||||
authority: ComputerControlAuthority | undefined
|
||||
): authority is ComputerControlAuthority & Required<Pick<ComputerControlAuthority,
|
||||
'relaySessionId' | 'requesterDeviceId' | 'runId' | 'targetDeviceId'>> {
|
||||
return !!authority && [
|
||||
authority.controlSessionId,
|
||||
authority.relaySessionId,
|
||||
authority.requesterDeviceId,
|
||||
authority.runId,
|
||||
authority.targetDeviceId
|
||||
].every(value => typeof value === 'string' && value.trim().length > 0)
|
||||
}
|
||||
|
||||
/** Fail closed when a brokered action does not identify its app/window. */
|
||||
export function evaluateSensitiveTarget(
|
||||
target: Pick<ComputerTarget, 'app' | 'title' | 'executable'>,
|
||||
extraPatterns: readonly RegExp[] = []
|
||||
): SensitiveTargetDecision {
|
||||
const identity = [target.app, target.title, target.executable]
|
||||
.filter((value): value is string => typeof value === 'string' && value.trim().length > 0)
|
||||
.join(' | ')
|
||||
if (!identity) return { allowed: false, reason: 'missing_target_identity' }
|
||||
for (const pattern of [...DEFAULT_SENSITIVE_PATTERNS, ...extraPatterns]) {
|
||||
pattern.lastIndex = 0
|
||||
const match = pattern.exec(identity)
|
||||
if (match) return { allowed: false, reason: 'sensitive_target', matched: match[0] }
|
||||
}
|
||||
return { allowed: true, reason: 'not_sensitive' }
|
||||
}
|
||||
@@ -1,9 +1,16 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
|
||||
import {
|
||||
DEFAULT_CAPABILITY_POLICIES,
|
||||
presetCapabilityPolicies,
|
||||
type CapabilityPolicies,
|
||||
type HostAccessMode
|
||||
} from '../lib/hostAccessPolicy.js'
|
||||
import {
|
||||
LEGACY_CONTROL_SESSION_ID,
|
||||
type ComputerControlAuthority,
|
||||
type ComputerTarget
|
||||
} from './computerControlSecurity.js'
|
||||
|
||||
export type ComputerGrantMode = 'observe' | 'assist' | 'control'
|
||||
|
||||
@@ -30,14 +37,34 @@ export interface ComputerUseRuntime {
|
||||
capabilities: CapabilityPolicies
|
||||
}
|
||||
|
||||
let activeGrant: ComputerGrant | null = null
|
||||
let grantChangeListener: ((grant: ComputerGrant | null) => void) | null = null
|
||||
let runtime: ComputerUseRuntime = {
|
||||
interface ControlSessionState {
|
||||
activeGrant: ComputerGrant | null
|
||||
runtime: ComputerUseRuntime
|
||||
}
|
||||
|
||||
const defaultRuntime = (): ComputerUseRuntime => ({
|
||||
url: null,
|
||||
computerUseConsented: false,
|
||||
consentSource: 'none',
|
||||
accessMode: 'ask',
|
||||
capabilities: { ...DEFAULT_CAPABILITY_POLICIES }
|
||||
})
|
||||
|
||||
const controlSessions = new Map<string, ControlSessionState>()
|
||||
let grantChangeListener: ((grant: ComputerGrant | null, controlSessionId?: string) => void) | null = null
|
||||
|
||||
function authorityKey(authority?: ComputerControlAuthority): string {
|
||||
return authority?.controlSessionId || LEGACY_CONTROL_SESSION_ID
|
||||
}
|
||||
|
||||
function sessionState(authority?: ComputerControlAuthority): ControlSessionState {
|
||||
const key = authorityKey(authority)
|
||||
let state = controlSessions.get(key)
|
||||
if (!state) {
|
||||
state = { activeGrant: null, runtime: defaultRuntime() }
|
||||
controlSessions.set(key, state)
|
||||
}
|
||||
return state
|
||||
}
|
||||
|
||||
function nowMs(): number {
|
||||
@@ -45,7 +72,7 @@ function nowMs(): number {
|
||||
}
|
||||
|
||||
function newGrantId(): string {
|
||||
return `computer-grant-${nowMs().toString(36)}-${Math.random().toString(36).slice(2, 8)}`
|
||||
return `computer-grant-${randomUUID()}`
|
||||
}
|
||||
|
||||
function parseScope(value: unknown): ComputerGrantScope {
|
||||
@@ -82,18 +109,19 @@ export function normalizeComputerGrantDurationSeconds(value: unknown): number {
|
||||
: 900
|
||||
}
|
||||
|
||||
function expireIfNeeded(): void {
|
||||
if (!activeGrant) {
|
||||
function expireIfNeeded(authority?: ComputerControlAuthority): void {
|
||||
const state = sessionState(authority)
|
||||
if (!state.activeGrant) {
|
||||
return
|
||||
}
|
||||
if (Date.parse(activeGrant.expires_at) <= nowMs()) {
|
||||
activeGrant = null
|
||||
grantChangeListener?.(null)
|
||||
if (Date.parse(state.activeGrant.expires_at) <= nowMs()) {
|
||||
state.activeGrant = null
|
||||
grantChangeListener?.(null, authorityKey(authority))
|
||||
}
|
||||
}
|
||||
|
||||
export function setComputerGrantChangeListener(
|
||||
listener: ((grant: ComputerGrant | null) => void) | null
|
||||
listener: ((grant: ComputerGrant | null, controlSessionId?: string) => void) | null
|
||||
): () => void {
|
||||
const previous = grantChangeListener
|
||||
grantChangeListener = listener
|
||||
@@ -102,24 +130,25 @@ export function setComputerGrantChangeListener(
|
||||
}
|
||||
}
|
||||
|
||||
export function getActiveComputerGrant(): ComputerGrant | null {
|
||||
expireIfNeeded()
|
||||
return activeGrant
|
||||
export function getActiveComputerGrant(authority?: ComputerControlAuthority): ComputerGrant | null {
|
||||
expireIfNeeded(authority)
|
||||
return sessionState(authority).activeGrant
|
||||
}
|
||||
|
||||
export function getComputerGrantSummary(): Record<string, unknown> {
|
||||
if (runtime.capabilities.screen_input === 'allow') {
|
||||
export function getComputerGrantSummary(authority?: ComputerControlAuthority): Record<string, unknown> {
|
||||
const state = sessionState(authority)
|
||||
if (state.runtime.capabilities.screen_input === 'allow') {
|
||||
return {
|
||||
active: true,
|
||||
mode: runtime.accessMode === 'full_access' ? 'full_access' : 'capability_allow',
|
||||
mode: state.runtime.accessMode === 'full_access' ? 'full_access' : 'capability_allow',
|
||||
expires_at: null,
|
||||
scope: null,
|
||||
reason: runtime.accessMode === 'full_access'
|
||||
reason: state.runtime.accessMode === 'full_access'
|
||||
? 'This host has Full Access.'
|
||||
: 'Screen and input are allowed by this host policy.'
|
||||
}
|
||||
}
|
||||
const grant = getActiveComputerGrant()
|
||||
const grant = getActiveComputerGrant(authority)
|
||||
if (!grant) {
|
||||
return {
|
||||
active: false,
|
||||
@@ -138,16 +167,18 @@ export function getComputerGrantSummary(): Record<string, unknown> {
|
||||
}
|
||||
}
|
||||
|
||||
export function configureComputerUseRuntime(next: Partial<ComputerUseRuntime>): void {
|
||||
const capabilities = next.capabilities ?? (next.accessMode ? presetCapabilityPolicies(next.accessMode) : runtime.capabilities)
|
||||
runtime = {
|
||||
...runtime,
|
||||
export function configureComputerUseRuntime(next: Partial<ComputerUseRuntime>, authority?: ComputerControlAuthority): void {
|
||||
const state = sessionState(authority)
|
||||
const capabilities = next.capabilities ?? (next.accessMode ? presetCapabilityPolicies(next.accessMode) : state.runtime.capabilities)
|
||||
state.runtime = {
|
||||
...state.runtime,
|
||||
...next,
|
||||
capabilities
|
||||
}
|
||||
}
|
||||
|
||||
export function getComputerUseRuntimeSummary(): Record<string, unknown> {
|
||||
export function getComputerUseRuntimeSummary(authority?: ComputerControlAuthority): Record<string, unknown> {
|
||||
const runtime = sessionState(authority).runtime
|
||||
return {
|
||||
url: runtime.url,
|
||||
consented: runtime.computerUseConsented,
|
||||
@@ -165,12 +196,14 @@ export interface RequestComputerGrantInput {
|
||||
reason?: unknown
|
||||
}
|
||||
|
||||
export function requestComputerGrant(input: RequestComputerGrantInput): Record<string, unknown> {
|
||||
export function requestComputerGrant(input: RequestComputerGrantInput, authority?: ComputerControlAuthority): Record<string, unknown> {
|
||||
const state = sessionState(authority)
|
||||
const runtime = state.runtime
|
||||
if (runtime.capabilities.screen_input === 'allow') {
|
||||
return {
|
||||
ok: true,
|
||||
full_access: runtime.accessMode === 'full_access',
|
||||
grant: getComputerGrantSummary(),
|
||||
grant: getComputerGrantSummary(authority),
|
||||
message: 'This host already has Full Access; no task grant is required.'
|
||||
}
|
||||
}
|
||||
@@ -184,7 +217,7 @@ export function requestComputerGrant(input: RequestComputerGrantInput): Record<s
|
||||
code: 'computer_use_consent_required',
|
||||
message:
|
||||
'Assist/control grants require local desktop-tool consent for this relay URL before task-scoped input grants can be created.',
|
||||
grant: getComputerGrantSummary()
|
||||
grant: getComputerGrantSummary(authority)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -197,12 +230,12 @@ export function requestComputerGrant(input: RequestComputerGrantInput): Record<s
|
||||
created_at: createdAt.toISOString(),
|
||||
expires_at: new Date(createdAt.getTime() + durationSeconds * 1000).toISOString()
|
||||
}
|
||||
activeGrant = grant
|
||||
grantChangeListener?.(grant)
|
||||
state.activeGrant = grant
|
||||
grantChangeListener?.(grant, authorityKey(authority))
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
grant: getComputerGrantSummary(),
|
||||
grant: getComputerGrantSummary(authority),
|
||||
message:
|
||||
mode === 'observe'
|
||||
? 'Observe grant active. Screenshot/status tools may run.'
|
||||
@@ -210,29 +243,84 @@ export function requestComputerGrant(input: RequestComputerGrantInput): Record<s
|
||||
}
|
||||
}
|
||||
|
||||
export function cancelComputerGrant(reason = 'cancelled'): Record<string, unknown> {
|
||||
const previous = getActiveComputerGrant()
|
||||
activeGrant = null
|
||||
if (previous) grantChangeListener?.(null)
|
||||
export function cancelComputerGrant(reason = 'cancelled', authority?: ComputerControlAuthority): Record<string, unknown> {
|
||||
const state = sessionState(authority)
|
||||
const previous = getActiveComputerGrant(authority)
|
||||
state.activeGrant = null
|
||||
if (previous) grantChangeListener?.(null, authorityKey(authority))
|
||||
return {
|
||||
ok: true,
|
||||
cancelled: previous !== null,
|
||||
previous_grant: previous,
|
||||
reason,
|
||||
grant: getComputerGrantSummary()
|
||||
grant: getComputerGrantSummary(authority)
|
||||
}
|
||||
}
|
||||
|
||||
export function hasComputerInputGrant(): boolean {
|
||||
export function hasComputerInputGrant(authority?: ComputerControlAuthority): boolean {
|
||||
const runtime = sessionState(authority).runtime
|
||||
if (runtime.capabilities.screen_input === 'allow') return true
|
||||
const grant = getActiveComputerGrant()
|
||||
const grant = getActiveComputerGrant(authority)
|
||||
return grant?.mode === 'assist' || grant?.mode === 'control'
|
||||
}
|
||||
|
||||
export function hasComputerObserveGrant(): boolean {
|
||||
return runtime.capabilities.screen_input === 'allow' || getActiveComputerGrant() !== null
|
||||
export function hasComputerObserveGrant(authority?: ComputerControlAuthority): boolean {
|
||||
const runtime = sessionState(authority).runtime
|
||||
return runtime.capabilities.screen_input === 'allow' || getActiveComputerGrant(authority) !== null
|
||||
}
|
||||
|
||||
export function hasFullHostAccess(): boolean {
|
||||
return runtime.accessMode === 'full_access'
|
||||
/** Seed a new router lifecycle from the legacy process-level configuration. */
|
||||
export function initializeComputerControlSession(authority: ComputerControlAuthority): void {
|
||||
if (controlSessions.has(authorityKey(authority))) return
|
||||
const legacy = sessionState().runtime
|
||||
controlSessions.set(authorityKey(authority), {
|
||||
activeGrant: null,
|
||||
runtime: { ...legacy, capabilities: { ...legacy.capabilities } }
|
||||
})
|
||||
}
|
||||
|
||||
export function hasFullHostAccess(authority?: ComputerControlAuthority): boolean {
|
||||
return sessionState(authority).runtime.accessMode === 'full_access'
|
||||
}
|
||||
|
||||
export function revokeComputerControlSession(authority: ComputerControlAuthority, reason = 'control session ended'): void {
|
||||
cancelComputerGrant(reason, authority)
|
||||
controlSessions.delete(authorityKey(authority))
|
||||
}
|
||||
|
||||
export function cancelAllComputerGrants(_reason = 'cancelled locally'): number {
|
||||
let cancelled = 0
|
||||
for (const [controlSessionId, state] of controlSessions) {
|
||||
if (!state.activeGrant) continue
|
||||
state.activeGrant = null
|
||||
cancelled += 1
|
||||
grantChangeListener?.(null, controlSessionId)
|
||||
}
|
||||
return cancelled
|
||||
}
|
||||
|
||||
export function expireComputerControlSessions(): void {
|
||||
for (const controlSessionId of controlSessions.keys()) {
|
||||
expireIfNeeded({ controlSessionId })
|
||||
}
|
||||
}
|
||||
|
||||
/** Enforce the scope shown in the approval prompt before a broker acts. */
|
||||
export function computerGrantAllowsTarget(authority: ComputerControlAuthority, target: ComputerTarget): boolean {
|
||||
const runtime = sessionState(authority).runtime
|
||||
if (runtime.capabilities.screen_input === 'allow') return true
|
||||
const grant = getActiveComputerGrant(authority)
|
||||
if (!grant) return false
|
||||
if (grant.scope.display && grant.scope.display !== target.display) return false
|
||||
if (grant.scope.app) {
|
||||
const actual = [target.app, target.executable].filter(Boolean).join(' ').toLowerCase()
|
||||
if (!actual || !actual.includes(grant.scope.app.toLowerCase())) return false
|
||||
}
|
||||
if (grant.scope.folder) {
|
||||
if (!target.folder) return false
|
||||
const expected = grant.scope.folder.replaceAll('\\', '/').replace(/\/+$/, '').toLowerCase()
|
||||
const actual = target.folder.replaceAll('\\', '/').toLowerCase()
|
||||
if (actual !== expected && !actual.startsWith(`${expected}/`)) return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -0,0 +1,866 @@
|
||||
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { access, realpath } from 'node:fs/promises'
|
||||
import { homedir } from 'node:os'
|
||||
import { basename, dirname, join, relative, resolve, sep } from 'node:path'
|
||||
|
||||
import { readDesktopUseSettingsSync } from '../lib/desktopUseSettings.js'
|
||||
|
||||
const SUPPORTED_MIN_VERSION = [0, 19, 3] as const
|
||||
const SUPPORTED_MAX_VERSION = [0, 20, 0] as const
|
||||
const DEFAULT_TIMEOUT_MS = 8_000
|
||||
const MAX_OUTPUT_BYTES = 4 * 1024 * 1024
|
||||
const REQUIRED_TOOLS = Object.freeze([
|
||||
'health_report',
|
||||
'start_session',
|
||||
'end_session',
|
||||
'list_windows',
|
||||
'get_window_state',
|
||||
'click',
|
||||
'set_value',
|
||||
'press_key',
|
||||
'scroll'
|
||||
])
|
||||
const ALLOWED_TOOLS = Object.freeze([...REQUIRED_TOOLS, 'set_agent_cursor_enabled'])
|
||||
|
||||
export interface CuaProcessResult {
|
||||
stdout: string
|
||||
stderr: string
|
||||
exitCode: number
|
||||
}
|
||||
|
||||
export interface CuaProcessRunner {
|
||||
run(
|
||||
executable: string,
|
||||
args: readonly string[],
|
||||
options?: { stdin?: string; timeoutMs?: number; signal?: AbortSignal; env?: NodeJS.ProcessEnv }
|
||||
): Promise<CuaProcessResult>
|
||||
}
|
||||
|
||||
export interface CuaRuntimeStatus {
|
||||
available: boolean
|
||||
ready: boolean
|
||||
binaryPath?: string
|
||||
binaryVersion?: string
|
||||
permissionMode?: 'standard' | 'bounded'
|
||||
health?: 'ok' | 'not_checked'
|
||||
reason?: string
|
||||
}
|
||||
|
||||
export interface CuaHealthStatus {
|
||||
state: 'healthy' | 'degraded' | 'error'
|
||||
checkedAt: string
|
||||
overall?: string
|
||||
reason?: string
|
||||
temporaryWindowsCompatibility: true
|
||||
}
|
||||
|
||||
export interface CuaControlSessionIdentity {
|
||||
controlSessionId: string
|
||||
targetDeviceId: string
|
||||
hostUrl?: string
|
||||
requestId?: string
|
||||
relaySessionId?: string
|
||||
requesterDeviceId?: string
|
||||
chatSessionId?: string
|
||||
runId?: string
|
||||
}
|
||||
|
||||
export interface CuaWindowTarget {
|
||||
pid: number
|
||||
windowId: number
|
||||
}
|
||||
|
||||
export interface CuaSnapshotOptions extends CuaWindowTarget {
|
||||
query?: string
|
||||
includeScreenshot?: boolean
|
||||
maxElements?: number
|
||||
maxDepth?: number
|
||||
}
|
||||
|
||||
export interface CuaElementTarget extends CuaWindowTarget {
|
||||
elementToken: string
|
||||
}
|
||||
|
||||
export type CuaToolResult = Record<string, unknown>
|
||||
|
||||
export type ComputerControlBackend = 'cua' | 'legacy_compat'
|
||||
|
||||
export interface ComputerControlBackendSelection {
|
||||
backend: ComputerControlBackend
|
||||
reason: 'cua_ready' | 'explicit_compatibility' | 'cua_unavailable_before_session'
|
||||
selectedAt: string
|
||||
cursorEnabled: boolean
|
||||
}
|
||||
|
||||
export interface CuaActionEvent {
|
||||
action: string
|
||||
target_app?: string
|
||||
target_pid?: number
|
||||
target_window_id?: number
|
||||
verification: 'snapshot_captured' | 'not_requested' | 'failed'
|
||||
occurred_at: string
|
||||
}
|
||||
|
||||
export interface ComputerControlLifecycleStatus {
|
||||
active_sessions: number
|
||||
cursor_enabled: boolean
|
||||
active_backend: 'cua' | 'legacy_compat' | 'mixed' | 'idle'
|
||||
last_action: CuaActionEvent | null
|
||||
}
|
||||
|
||||
interface CuaManifest {
|
||||
schema_version?: unknown
|
||||
binary_version?: unknown
|
||||
binary_path?: unknown
|
||||
}
|
||||
|
||||
interface CuaHealthReport {
|
||||
schema_version?: unknown
|
||||
driver_version?: unknown
|
||||
overall?: unknown
|
||||
}
|
||||
|
||||
interface JsonRpcResponse {
|
||||
id?: number
|
||||
result?: unknown
|
||||
error?: { message?: unknown }
|
||||
}
|
||||
|
||||
/**
|
||||
* CUA is a separately maintained child process. Do not copy the relay daemon's
|
||||
* environment into it: that environment can contain pairing tokens, provider
|
||||
* credentials, and other secrets unrelated to desktop control.
|
||||
*/
|
||||
export function cuaDriverEnvironment(extra: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv {
|
||||
const allowed = [
|
||||
'SystemRoot', 'WINDIR', 'COMSPEC', 'PATHEXT', 'PATH', 'TEMP', 'TMP',
|
||||
'LOCALAPPDATA', 'APPDATA', 'USERPROFILE', 'HOMEDRIVE', 'HOMEPATH',
|
||||
'PROCESSOR_ARCHITECTURE', 'PROCESSOR_ARCHITEW6432', 'NUMBER_OF_PROCESSORS'
|
||||
]
|
||||
const env: NodeJS.ProcessEnv = {}
|
||||
for (const key of allowed) {
|
||||
const value = process.env[key]
|
||||
if (value !== undefined) env[key] = value
|
||||
}
|
||||
return { ...env, ...extra, CUA_DRIVER_RS_TELEMETRY_ENABLED: '0' }
|
||||
}
|
||||
|
||||
class CuaMcpClient {
|
||||
private readonly child: ChildProcessWithoutNullStreams
|
||||
private readonly pending = new Map<number, { resolve(value: unknown): void; reject(error: Error): void; timer: NodeJS.Timeout }>()
|
||||
private nextId = 1
|
||||
private stdout = ''
|
||||
private closed = false
|
||||
|
||||
private constructor(binaryPath: string) {
|
||||
this.child = spawn(binaryPath, ['mcp', '--socket', '\\\\.\\pipe\\cua-driver'], {
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
windowsHide: true,
|
||||
env: cuaDriverEnvironment()
|
||||
})
|
||||
this.child.stdout.on('data', (chunk: Buffer) => this.receive(chunk))
|
||||
this.child.on('error', error => this.failAll(new CuaRuntimeError(`CUA MCP transport failed: ${error.message}`, 'transport')))
|
||||
this.child.on('close', code => this.failAll(new CuaRuntimeError(`CUA MCP transport closed (${code ?? 'unknown'})`, 'transport')))
|
||||
}
|
||||
|
||||
static async connect(binaryPath: string, expectedVersion: string): Promise<CuaMcpClient> {
|
||||
const client = new CuaMcpClient(binaryPath)
|
||||
const initialized = await client.request('initialize', {
|
||||
protocolVersion: '2025-06-18',
|
||||
capabilities: {},
|
||||
clientInfo: { name: 'hermes-relay-desktop', version: '1' }
|
||||
}) as { protocolVersion?: unknown; serverInfo?: { version?: unknown } }
|
||||
if (initialized.protocolVersion !== '2025-06-18') {
|
||||
client.close()
|
||||
throw new CuaRuntimeError('CUA MCP protocol version is incompatible', 'incompatible')
|
||||
}
|
||||
if (initialized.serverInfo?.version !== expectedVersion) {
|
||||
client.close()
|
||||
throw new CuaRuntimeError(
|
||||
`CUA MCP daemon version ${String(initialized.serverInfo?.version ?? 'unknown')} does not match the verified binary ${expectedVersion}`,
|
||||
'incompatible'
|
||||
)
|
||||
}
|
||||
client.notify('notifications/initialized', {})
|
||||
return client
|
||||
}
|
||||
|
||||
async call(tool: string, args: Record<string, unknown>): Promise<CuaToolResult> {
|
||||
const result = await this.request('tools/call', { name: tool, arguments: args }) as {
|
||||
isError?: unknown
|
||||
structuredContent?: unknown
|
||||
content?: Array<{ type?: unknown; text?: unknown; data?: unknown; mimeType?: unknown }>
|
||||
}
|
||||
if (result.isError === true) {
|
||||
// Driver errors can quote UI labels or entered values. Treat the broker
|
||||
// result as sensitive and expose only the stable operation name.
|
||||
throw new CuaRuntimeError(`CUA Driver ${tool} rejected the action`, 'transport')
|
||||
}
|
||||
if (result.structuredContent && typeof result.structuredContent === 'object' && !Array.isArray(result.structuredContent)) {
|
||||
const structured = { ...(result.structuredContent as CuaToolResult) }
|
||||
const image = result.content?.find(item => item.type === 'image' && typeof item.data === 'string')
|
||||
if (image) {
|
||||
structured.screenshot_base64 = image.data
|
||||
structured.screenshot_mime_type = image.mimeType
|
||||
}
|
||||
return structured
|
||||
}
|
||||
const text = result.content?.find(item => item.type === 'text' && typeof item.text === 'string')?.text
|
||||
return text ? parseJsonObject(String(text), `CUA Driver ${tool}`) : {}
|
||||
}
|
||||
|
||||
close(): void {
|
||||
if (this.closed) return
|
||||
this.closed = true
|
||||
this.child.stdin.end()
|
||||
const timer = setTimeout(() => this.child.kill('SIGKILL'), 1_000)
|
||||
timer.unref?.()
|
||||
}
|
||||
|
||||
private request(method: string, params: Record<string, unknown>): Promise<unknown> {
|
||||
if (this.closed) return Promise.reject(new CuaRuntimeError('CUA MCP transport is closed', 'closed'))
|
||||
const id = this.nextId++
|
||||
return new Promise((resolvePromise, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
this.pending.delete(id)
|
||||
reject(new CuaRuntimeError(`CUA MCP ${method} timed out`, 'transport'))
|
||||
}, DEFAULT_TIMEOUT_MS)
|
||||
timer.unref?.()
|
||||
this.pending.set(id, { resolve: resolvePromise, reject, timer })
|
||||
this.child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', id, method, params })}\n`)
|
||||
})
|
||||
}
|
||||
|
||||
private notify(method: string, params: Record<string, unknown>): void {
|
||||
this.child.stdin.write(`${JSON.stringify({ jsonrpc: '2.0', method, params })}\n`)
|
||||
}
|
||||
|
||||
private receive(chunk: Buffer): void {
|
||||
this.stdout += chunk.toString('utf8')
|
||||
if (Buffer.byteLength(this.stdout, 'utf8') > MAX_OUTPUT_BYTES) {
|
||||
this.failAll(new CuaRuntimeError('CUA MCP response exceeded the output limit', 'transport'))
|
||||
this.child.kill('SIGKILL')
|
||||
return
|
||||
}
|
||||
let newline = this.stdout.indexOf('\n')
|
||||
while (newline >= 0) {
|
||||
const line = this.stdout.slice(0, newline).trim()
|
||||
this.stdout = this.stdout.slice(newline + 1)
|
||||
if (line) this.resolveLine(line)
|
||||
newline = this.stdout.indexOf('\n')
|
||||
}
|
||||
}
|
||||
|
||||
private resolveLine(line: string): void {
|
||||
let message: JsonRpcResponse
|
||||
try { message = JSON.parse(line) as JsonRpcResponse } catch { return }
|
||||
if (typeof message.id !== 'number') return
|
||||
const pending = this.pending.get(message.id)
|
||||
if (!pending) return
|
||||
this.pending.delete(message.id)
|
||||
clearTimeout(pending.timer)
|
||||
if (message.error) pending.reject(new CuaRuntimeError(`CUA MCP error: ${String(message.error.message ?? 'unknown')}`, 'transport'))
|
||||
else pending.resolve(message.result)
|
||||
}
|
||||
|
||||
private failAll(error: Error): void {
|
||||
if (this.closed && this.pending.size === 0) return
|
||||
this.closed = true
|
||||
for (const pending of this.pending.values()) {
|
||||
clearTimeout(pending.timer)
|
||||
pending.reject(error)
|
||||
}
|
||||
this.pending.clear()
|
||||
}
|
||||
}
|
||||
|
||||
export interface CuaRuntimeOptions {
|
||||
platform?: NodeJS.Platform
|
||||
homeDir?: string
|
||||
runner?: CuaProcessRunner
|
||||
}
|
||||
|
||||
export class CuaRuntimeError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
readonly code: 'unavailable' | 'incompatible' | 'degraded' | 'transport' | 'closed'
|
||||
) {
|
||||
super(message)
|
||||
this.name = 'CuaRuntimeError'
|
||||
}
|
||||
}
|
||||
|
||||
export class SpawnCuaProcessRunner implements CuaProcessRunner {
|
||||
run(
|
||||
executable: string,
|
||||
args: readonly string[],
|
||||
options: { stdin?: string; timeoutMs?: number; signal?: AbortSignal; env?: NodeJS.ProcessEnv } = {}
|
||||
): Promise<CuaProcessResult> {
|
||||
return new Promise((resolvePromise, reject) => {
|
||||
const child = spawn(executable, [...args], {
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
windowsHide: true,
|
||||
env: options.env ?? cuaDriverEnvironment()
|
||||
})
|
||||
let stdout = ''
|
||||
let stderr = ''
|
||||
let settled = false
|
||||
let outputBytes = 0
|
||||
const finish = (error?: Error, exitCode = -1) => {
|
||||
if (settled) return
|
||||
settled = true
|
||||
clearTimeout(timer)
|
||||
options.signal?.removeEventListener('abort', abort)
|
||||
if (error) reject(error)
|
||||
else resolvePromise({ stdout, stderr, exitCode })
|
||||
}
|
||||
const abort = () => {
|
||||
child.kill('SIGKILL')
|
||||
finish(new CuaRuntimeError('CUA Driver process was cancelled', 'transport'))
|
||||
}
|
||||
const append = (current: string, chunk: Buffer): string => {
|
||||
outputBytes += chunk.length
|
||||
if (outputBytes > MAX_OUTPUT_BYTES) {
|
||||
child.kill('SIGKILL')
|
||||
finish(new CuaRuntimeError('CUA Driver response exceeded the output limit', 'transport'))
|
||||
return current
|
||||
}
|
||||
return current + chunk.toString('utf8')
|
||||
}
|
||||
const timer = setTimeout(() => {
|
||||
child.kill('SIGKILL')
|
||||
finish(new CuaRuntimeError('CUA Driver process timed out', 'transport'))
|
||||
}, options.timeoutMs ?? DEFAULT_TIMEOUT_MS)
|
||||
timer.unref?.()
|
||||
options.signal?.addEventListener('abort', abort, { once: true })
|
||||
child.stdout.on('data', (chunk: Buffer) => { stdout = append(stdout, chunk) })
|
||||
child.stderr.on('data', (chunk: Buffer) => { stderr = append(stderr, chunk) })
|
||||
child.on('error', error => finish(new CuaRuntimeError(`CUA Driver process failed: ${error.message}`, 'transport')))
|
||||
child.on('close', code => finish(undefined, code ?? -1))
|
||||
child.stdin.end(options.stdin ?? '')
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function compareVersion(left: readonly number[], right: readonly number[]): number {
|
||||
for (let i = 0; i < 3; i += 1) {
|
||||
const delta = (left[i] ?? 0) - (right[i] ?? 0)
|
||||
if (delta !== 0) return delta
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
function parseVersion(value: string): [number, number, number] | null {
|
||||
const match = /(?:^|\s)(\d+)\.(\d+)\.(\d+)(?:\s|$)/.exec(value.trim())
|
||||
return match ? [Number(match[1]), Number(match[2]), Number(match[3])] : null
|
||||
}
|
||||
|
||||
function parseJsonObject(text: string, label: string): Record<string, unknown> {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(text)
|
||||
} catch {
|
||||
throw new CuaRuntimeError(`${label} returned invalid JSON`, 'transport')
|
||||
}
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new CuaRuntimeError(`${label} returned a non-object response`, 'transport')
|
||||
}
|
||||
return parsed as Record<string, unknown>
|
||||
}
|
||||
|
||||
function validatePositiveInteger(value: number, name: string): void {
|
||||
if (!Number.isSafeInteger(value) || value <= 0) {
|
||||
throw new CuaRuntimeError(`${name} must be a positive integer`, 'transport')
|
||||
}
|
||||
}
|
||||
|
||||
function validateElementToken(value: string): void {
|
||||
if (!/^e[0-9a-f]+$/i.test(value)) {
|
||||
throw new CuaRuntimeError('elementToken is not a valid opaque CUA element token', 'transport')
|
||||
}
|
||||
}
|
||||
|
||||
function derivedSessionId(identity: CuaControlSessionIdentity): string {
|
||||
const fields = [
|
||||
identity.controlSessionId,
|
||||
identity.targetDeviceId,
|
||||
identity.hostUrl ?? '',
|
||||
identity.relaySessionId ?? '',
|
||||
identity.requesterDeviceId ?? '',
|
||||
identity.chatSessionId ?? '',
|
||||
identity.runId ?? ''
|
||||
]
|
||||
if (!identity.controlSessionId.trim() || !identity.targetDeviceId.trim()) {
|
||||
throw new CuaRuntimeError('CUA control session identity is incomplete', 'transport')
|
||||
}
|
||||
return `hermes-${createHash('sha256').update(fields.join('\u0000')).digest('hex').slice(0, 32)}`
|
||||
}
|
||||
|
||||
function controlIdentityFingerprint(identity: CuaControlSessionIdentity): string {
|
||||
return createHash('sha256').update([
|
||||
identity.controlSessionId,
|
||||
identity.targetDeviceId,
|
||||
identity.hostUrl ?? '',
|
||||
identity.relaySessionId ?? '',
|
||||
identity.requesterDeviceId ?? '',
|
||||
identity.chatSessionId ?? '',
|
||||
identity.runId ?? ''
|
||||
].join('\u0000')).digest('hex')
|
||||
}
|
||||
|
||||
async function canonicalBinary(options: CuaRuntimeOptions): Promise<string> {
|
||||
const platform = options.platform ?? process.platform
|
||||
if (platform !== 'win32') {
|
||||
throw new CuaRuntimeError(`CUA Driver adapter is not enabled on ${platform}`, 'unavailable')
|
||||
}
|
||||
const home = resolve(options.homeDir ?? homedir())
|
||||
const packages = join(home, '.cua-driver', 'packages')
|
||||
const releases = join(packages, 'releases')
|
||||
const candidate = join(packages, 'current', 'cua-driver.exe')
|
||||
let resolvedCandidate: string
|
||||
let resolvedReleases: string
|
||||
try {
|
||||
;[resolvedCandidate, resolvedReleases] = await Promise.all([realpath(candidate), realpath(releases)])
|
||||
await access(resolvedCandidate)
|
||||
} catch {
|
||||
throw new CuaRuntimeError('Canonical CUA Driver package is not installed', 'unavailable')
|
||||
}
|
||||
const inside = relative(resolvedReleases, resolvedCandidate)
|
||||
if (!inside || inside.startsWith(`..${sep}`) || inside === '..' || resolve(resolvedCandidate) === resolve(resolvedReleases)) {
|
||||
throw new CuaRuntimeError('Canonical CUA Driver package resolved outside its release store', 'incompatible')
|
||||
}
|
||||
if (basename(resolvedCandidate).toLowerCase() !== 'cua-driver.exe' || basename(dirname(resolvedCandidate)).length === 0) {
|
||||
throw new CuaRuntimeError('Canonical CUA Driver executable path is invalid', 'incompatible')
|
||||
}
|
||||
return resolvedCandidate
|
||||
}
|
||||
|
||||
export class CuaDriverAdapter {
|
||||
readonly binaryPath: string
|
||||
readonly binaryVersion: string
|
||||
readonly permissionMode: 'standard' | 'bounded'
|
||||
private readonly sessions = new Map<string, CuaControlSession>()
|
||||
|
||||
private constructor(
|
||||
binaryPath: string,
|
||||
binaryVersion: string,
|
||||
permissionMode: 'standard' | 'bounded',
|
||||
private readonly runner: CuaProcessRunner,
|
||||
private readonly usePersistentMcp: boolean,
|
||||
private readonly supportsCursorToggle: boolean
|
||||
) {
|
||||
this.binaryPath = binaryPath
|
||||
this.binaryVersion = binaryVersion
|
||||
this.permissionMode = permissionMode
|
||||
}
|
||||
|
||||
static async connect(options: CuaRuntimeOptions = {}): Promise<CuaDriverAdapter> {
|
||||
const runner = options.runner ?? new SpawnCuaProcessRunner()
|
||||
const binaryPath = await canonicalBinary(options)
|
||||
const run = async (args: readonly string[], stdin?: string): Promise<string> => {
|
||||
const result = await runner.run(binaryPath, args, {
|
||||
stdin,
|
||||
env: cuaDriverEnvironment()
|
||||
})
|
||||
if (result.exitCode !== 0) {
|
||||
throw new CuaRuntimeError(`CUA Driver probe ${args[0] ?? 'command'} failed`, 'transport')
|
||||
}
|
||||
return result.stdout.trim()
|
||||
}
|
||||
const versionText = await run(['--version'])
|
||||
const versionTuple = parseVersion(versionText)
|
||||
if (!versionTuple || compareVersion(versionTuple, SUPPORTED_MIN_VERSION) < 0 || compareVersion(versionTuple, SUPPORTED_MAX_VERSION) >= 0) {
|
||||
throw new CuaRuntimeError(`Unsupported CUA Driver version: ${versionText || 'unknown'}`, 'incompatible')
|
||||
}
|
||||
const manifest = parseJsonObject(await run(['manifest', '--pretty']), 'CUA Driver manifest') as CuaManifest
|
||||
if (manifest.schema_version !== '1' || manifest.binary_version !== versionTuple.join('.')) {
|
||||
throw new CuaRuntimeError('CUA Driver manifest schema or version is incompatible', 'incompatible')
|
||||
}
|
||||
const manifestPath = typeof manifest.binary_path === 'string' ? await realpath(manifest.binary_path).catch(() => '') : ''
|
||||
if (resolve(manifestPath).toLowerCase() !== resolve(binaryPath).toLowerCase()) {
|
||||
throw new CuaRuntimeError('CUA Driver manifest identifies a different executable', 'incompatible')
|
||||
}
|
||||
const toolNames = new Set((await run(['list-tools'])).split(/\r?\n/).map(line => line.split(':', 1)[0]?.trim()).filter(Boolean))
|
||||
const missingTools = REQUIRED_TOOLS.filter(tool => !toolNames.has(tool))
|
||||
if (missingTools.length > 0) {
|
||||
throw new CuaRuntimeError(`CUA Driver is missing required tools: ${missingTools.join(', ')}`, 'incompatible')
|
||||
}
|
||||
const statusText = await run(['status'])
|
||||
const permissionMatch = /permission mode:\s*(standard|bounded|unrestricted)\b/i.exec(statusText)
|
||||
if (!permissionMatch || permissionMatch[1]?.toLowerCase() === 'unrestricted') {
|
||||
throw new CuaRuntimeError('CUA Driver permission mode is unavailable or unrestricted', 'incompatible')
|
||||
}
|
||||
const permissionMode = permissionMatch[1]!.toLowerCase() as 'standard' | 'bounded'
|
||||
// Temporary Windows compatibility policy for trycua/cua#3103. The global
|
||||
// health_report performs a whole-desktop UIA walk with a fixed timeout and
|
||||
// can poison the driver's busy flag after a false timeout. Runtime
|
||||
// readiness is therefore based on the canonical binary, manifest, tool
|
||||
// contract, daemon status, and safe permission mode. Individual structured
|
||||
// actions still fail closed. Keep health_report as an explicit diagnostic
|
||||
// via healthStatus(), and remove this split when upstream fixes #3103.
|
||||
return new CuaDriverAdapter(
|
||||
binaryPath,
|
||||
versionTuple.join('.'),
|
||||
permissionMode,
|
||||
runner,
|
||||
options.runner === undefined,
|
||||
toolNames.has('set_agent_cursor_enabled')
|
||||
)
|
||||
}
|
||||
|
||||
static async status(options: CuaRuntimeOptions = {}): Promise<CuaRuntimeStatus> {
|
||||
try {
|
||||
const adapter = await CuaDriverAdapter.connect(options)
|
||||
return {
|
||||
available: true,
|
||||
ready: true,
|
||||
binaryPath: adapter.binaryPath,
|
||||
binaryVersion: adapter.binaryVersion,
|
||||
permissionMode: adapter.permissionMode,
|
||||
health: 'not_checked',
|
||||
reason: 'Runtime checks passed; Windows accessibility health is checked separately.'
|
||||
}
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error)
|
||||
return { available: error instanceof CuaRuntimeError && error.code !== 'unavailable', ready: false, reason }
|
||||
}
|
||||
}
|
||||
|
||||
static async healthStatus(options: CuaRuntimeOptions = {}): Promise<CuaHealthStatus> {
|
||||
const checkedAt = new Date().toISOString()
|
||||
try {
|
||||
const adapter = await CuaDriverAdapter.connect(options)
|
||||
const runner = options.runner ?? new SpawnCuaProcessRunner()
|
||||
const result = await runner.run(adapter.binaryPath, ['call', 'health_report'], {
|
||||
stdin: '{}',
|
||||
timeoutMs: DEFAULT_TIMEOUT_MS,
|
||||
env: cuaDriverEnvironment()
|
||||
})
|
||||
if (result.exitCode !== 0) {
|
||||
return {
|
||||
state: 'error', checkedAt, temporaryWindowsCompatibility: true,
|
||||
reason: `CUA Driver health probe exited ${result.exitCode}`
|
||||
}
|
||||
}
|
||||
const health = parseJsonObject(result.stdout.trim(), 'CUA Driver health report') as CuaHealthReport
|
||||
if (health.schema_version !== '1' || health.driver_version !== adapter.binaryVersion) {
|
||||
return {
|
||||
state: 'error', checkedAt, temporaryWindowsCompatibility: true,
|
||||
reason: 'CUA Driver health report schema or version is incompatible'
|
||||
}
|
||||
}
|
||||
const overall = String(health.overall ?? 'unknown')
|
||||
return overall === 'ok'
|
||||
? { state: 'healthy', checkedAt, overall, temporaryWindowsCompatibility: true }
|
||||
: {
|
||||
state: 'degraded', checkedAt, overall, temporaryWindowsCompatibility: true,
|
||||
reason: `CUA Driver reported ${overall}; runtime remains available and actions still fail closed.`
|
||||
}
|
||||
} catch (error) {
|
||||
return {
|
||||
state: 'error', checkedAt, temporaryWindowsCompatibility: true,
|
||||
reason: error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async openSession(identity: CuaControlSessionIdentity, signal?: AbortSignal, cursorEnabled = false): Promise<CuaControlSession> {
|
||||
const id = derivedSessionId(identity)
|
||||
if (this.sessions.has(id)) {
|
||||
throw new CuaRuntimeError('CUA control session is already active', 'transport')
|
||||
}
|
||||
if (signal?.aborted) throw new CuaRuntimeError('CUA control session was cancelled', 'transport')
|
||||
const mcp = this.usePersistentMcp ? await CuaMcpClient.connect(this.binaryPath, this.binaryVersion) : null
|
||||
const invoke = async (tool: string, args: Record<string, unknown>, invokeSignal?: AbortSignal): Promise<CuaToolResult> => {
|
||||
if (invokeSignal?.aborted) throw new CuaRuntimeError('CUA action was cancelled', 'transport')
|
||||
if (!ALLOWED_TOOLS.includes(tool)) throw new CuaRuntimeError('Attempted to invoke a non-allowlisted CUA Driver tool', 'transport')
|
||||
if (mcp) return mcp.call(tool, args)
|
||||
const result = await this.runner.run(this.binaryPath, ['call', tool], {
|
||||
stdin: JSON.stringify(args),
|
||||
timeoutMs: DEFAULT_TIMEOUT_MS,
|
||||
signal: invokeSignal,
|
||||
env: cuaDriverEnvironment()
|
||||
})
|
||||
if (result.exitCode !== 0) {
|
||||
throw new CuaRuntimeError(`CUA Driver ${tool} failed`, 'transport')
|
||||
}
|
||||
const payload = parseJsonObject(result.stdout.trim(), `CUA Driver ${tool}`)
|
||||
if (payload.isError === true) throw new CuaRuntimeError(`CUA Driver ${tool} rejected the action`, 'transport')
|
||||
return payload
|
||||
}
|
||||
await invoke('start_session', { session: id, capture_scope: 'window' }, signal).catch(error => {
|
||||
mcp?.close()
|
||||
throw error
|
||||
})
|
||||
if (this.supportsCursorToggle) {
|
||||
await invoke('set_agent_cursor_enabled', { session: id, enabled: cursorEnabled }, signal).catch(error => {
|
||||
mcp?.close()
|
||||
throw error
|
||||
})
|
||||
}
|
||||
const session = new CuaControlSession(
|
||||
id,
|
||||
invoke,
|
||||
() => mcp?.close(),
|
||||
() => this.sessions.delete(id)
|
||||
)
|
||||
this.sessions.set(id, session)
|
||||
return session
|
||||
}
|
||||
|
||||
async closeAll(reason = 'Hermes control authority ended'): Promise<void> {
|
||||
const sessions = [...this.sessions.values()]
|
||||
await Promise.allSettled(sessions.map(session => session.close(reason)))
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
export class CuaControlSession {
|
||||
private closed = false
|
||||
|
||||
constructor(
|
||||
private readonly id: string,
|
||||
private readonly invoke: (tool: string, args: Record<string, unknown>, signal?: AbortSignal) => Promise<CuaToolResult>,
|
||||
private readonly closeTransport: () => void,
|
||||
private readonly onClose: () => void
|
||||
) {}
|
||||
|
||||
private ensureOpen(): void {
|
||||
if (this.closed) throw new CuaRuntimeError('CUA control session is closed', 'closed')
|
||||
}
|
||||
|
||||
async listWindows(pid?: number, signal?: AbortSignal): Promise<CuaToolResult> {
|
||||
this.ensureOpen()
|
||||
if (pid !== undefined) validatePositiveInteger(pid, 'pid')
|
||||
return this.invoke('list_windows', pid === undefined ? {} : { pid }, signal)
|
||||
}
|
||||
|
||||
async snapshot(options: CuaSnapshotOptions, signal?: AbortSignal): Promise<CuaToolResult> {
|
||||
this.ensureOpen()
|
||||
validatePositiveInteger(options.pid, 'pid')
|
||||
validatePositiveInteger(options.windowId, 'windowId')
|
||||
return this.invoke('get_window_state', {
|
||||
pid: options.pid,
|
||||
window_id: options.windowId,
|
||||
session: this.id,
|
||||
query: options.query,
|
||||
include_screenshot: options.includeScreenshot,
|
||||
max_elements: options.maxElements ?? 1_000,
|
||||
max_depth: options.maxDepth ?? 20
|
||||
}, signal)
|
||||
}
|
||||
|
||||
async clickElement(target: CuaElementTarget, signal?: AbortSignal): Promise<CuaToolResult> {
|
||||
this.ensureOpen()
|
||||
validatePositiveInteger(target.pid, 'pid')
|
||||
validatePositiveInteger(target.windowId, 'windowId')
|
||||
validateElementToken(target.elementToken)
|
||||
return this.invoke('click', {
|
||||
pid: target.pid,
|
||||
window_id: target.windowId,
|
||||
element_token: target.elementToken,
|
||||
session: this.id,
|
||||
scope: 'window',
|
||||
delivery_mode: 'background'
|
||||
}, signal)
|
||||
}
|
||||
|
||||
async setElementValue(target: CuaElementTarget, value: string, signal?: AbortSignal): Promise<CuaToolResult> {
|
||||
this.ensureOpen()
|
||||
validatePositiveInteger(target.pid, 'pid')
|
||||
validatePositiveInteger(target.windowId, 'windowId')
|
||||
validateElementToken(target.elementToken)
|
||||
return this.invoke('set_value', {
|
||||
pid: target.pid,
|
||||
window_id: target.windowId,
|
||||
element_token: target.elementToken,
|
||||
value,
|
||||
session: this.id
|
||||
}, signal)
|
||||
}
|
||||
|
||||
async pressKey(target: CuaWindowTarget, key: string, signal?: AbortSignal): Promise<CuaToolResult> {
|
||||
this.ensureOpen()
|
||||
validatePositiveInteger(target.pid, 'pid')
|
||||
validatePositiveInteger(target.windowId, 'windowId')
|
||||
if (!/^(?:return|tab|escape|up|down|left|right|space|delete|home|end|pageup|pagedown|f(?:[1-9]|1[0-2])|[a-z0-9])$/i.test(key)) {
|
||||
throw new CuaRuntimeError('key is not in the allowlisted CUA key vocabulary', 'transport')
|
||||
}
|
||||
return this.invoke('press_key', {
|
||||
pid: target.pid,
|
||||
window_id: target.windowId,
|
||||
key: key.toLowerCase(),
|
||||
session: this.id,
|
||||
scope: 'window',
|
||||
delivery_mode: 'background'
|
||||
}, signal)
|
||||
}
|
||||
|
||||
async scroll(target: CuaElementTarget, direction: 'up' | 'down' | 'left' | 'right', amount: number, signal?: AbortSignal): Promise<CuaToolResult> {
|
||||
this.ensureOpen()
|
||||
validatePositiveInteger(target.pid, 'pid')
|
||||
validatePositiveInteger(target.windowId, 'windowId')
|
||||
validateElementToken(target.elementToken)
|
||||
if (!Number.isSafeInteger(amount) || amount < 1 || amount > 20) {
|
||||
throw new CuaRuntimeError('scroll amount must be an integer from 1 to 20', 'transport')
|
||||
}
|
||||
return this.invoke('scroll', {
|
||||
pid: target.pid,
|
||||
window_id: target.windowId,
|
||||
element_token: target.elementToken,
|
||||
direction,
|
||||
amount,
|
||||
session: this.id,
|
||||
delivery_mode: 'background'
|
||||
}, signal)
|
||||
}
|
||||
|
||||
async close(_reason = 'Hermes control session ended'): Promise<void> {
|
||||
if (this.closed) return
|
||||
this.closed = true
|
||||
this.onClose()
|
||||
try {
|
||||
await this.invoke('end_session', { session: this.id })
|
||||
} finally {
|
||||
this.closeTransport()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let sharedAdapter: Promise<CuaDriverAdapter> | null = null
|
||||
const sharedSessions = new Map<string, Promise<CuaControlSession>>()
|
||||
const sharedSessionIdentities = new Map<string, string>()
|
||||
const backendSelections = new Map<string, ComputerControlBackendSelection>()
|
||||
let lastActionEvent: CuaActionEvent | null = null
|
||||
let lifecycleListener: ((status: ComputerControlLifecycleStatus) => void) | null = null
|
||||
|
||||
export function setComputerControlLifecycleListener(
|
||||
listener: ((status: ComputerControlLifecycleStatus) => void) | null
|
||||
): () => void {
|
||||
lifecycleListener = listener
|
||||
listener?.(computerControlLifecycleStatus())
|
||||
return () => { if (lifecycleListener === listener) lifecycleListener = null }
|
||||
}
|
||||
|
||||
function publishLifecycle(): void {
|
||||
lifecycleListener?.(computerControlLifecycleStatus())
|
||||
}
|
||||
let testSessionFactory: ((identity: CuaControlSessionIdentity) => Promise<CuaControlSession>) | null = null
|
||||
|
||||
/** Test-only dependency seam; production callers must never configure this. */
|
||||
export function setCuaControlSessionFactoryForTests(
|
||||
factory: ((identity: CuaControlSessionIdentity) => Promise<CuaControlSession>) | null
|
||||
): void {
|
||||
testSessionFactory = factory
|
||||
sharedSessions.clear()
|
||||
sharedSessionIdentities.clear()
|
||||
backendSelections.clear()
|
||||
sharedAdapter = null
|
||||
}
|
||||
|
||||
/** Select once per outer Hermes control session; an active session never changes backend. */
|
||||
export async function selectComputerControlBackend(
|
||||
controlSessionId: string,
|
||||
selected: 'legacy' | 'cua',
|
||||
cursorEnabled: boolean
|
||||
): Promise<ComputerControlBackendSelection> {
|
||||
const existing = backendSelections.get(controlSessionId)
|
||||
if (existing) return existing
|
||||
let selection: ComputerControlBackendSelection
|
||||
if (selected === 'legacy') {
|
||||
selection = {
|
||||
backend: 'legacy_compat',
|
||||
reason: 'explicit_compatibility',
|
||||
selectedAt: new Date().toISOString(),
|
||||
cursorEnabled: false
|
||||
}
|
||||
} else if (testSessionFactory) {
|
||||
selection = { backend: 'cua', reason: 'cua_ready', selectedAt: new Date().toISOString(), cursorEnabled }
|
||||
} else {
|
||||
try {
|
||||
sharedAdapter ??= CuaDriverAdapter.connect()
|
||||
await sharedAdapter
|
||||
selection = { backend: 'cua', reason: 'cua_ready', selectedAt: new Date().toISOString(), cursorEnabled }
|
||||
} catch {
|
||||
sharedAdapter = null
|
||||
selection = {
|
||||
backend: 'legacy_compat',
|
||||
reason: 'cua_unavailable_before_session',
|
||||
selectedAt: new Date().toISOString(),
|
||||
cursorEnabled: false
|
||||
}
|
||||
}
|
||||
}
|
||||
backendSelections.set(controlSessionId, selection)
|
||||
publishLifecycle()
|
||||
return selection
|
||||
}
|
||||
|
||||
export function recordCuaActionEvent(event: Omit<CuaActionEvent, 'occurred_at'>): void {
|
||||
lastActionEvent = { ...event, occurred_at: new Date().toISOString() }
|
||||
publishLifecycle()
|
||||
}
|
||||
|
||||
export function computerControlLifecycleStatus(): ComputerControlLifecycleStatus {
|
||||
const active = [...backendSelections.values()]
|
||||
const backends = new Set(active.map(item => item.backend))
|
||||
return {
|
||||
active_sessions: active.length,
|
||||
cursor_enabled: active.some(item => item.backend === 'cua' && item.cursorEnabled),
|
||||
active_backend: active.length === 0 ? 'idle' : backends.size > 1 ? 'mixed' : active[0]!.backend,
|
||||
last_action: lastActionEvent
|
||||
}
|
||||
}
|
||||
|
||||
/** Return the one bounded CUA session owned by an authenticated Hermes control session. */
|
||||
export async function getCuaControlSession(identity: CuaControlSessionIdentity): Promise<CuaControlSession> {
|
||||
const key = identity.controlSessionId
|
||||
const fingerprint = controlIdentityFingerprint(identity)
|
||||
const existingFingerprint = sharedSessionIdentities.get(key)
|
||||
if (existingFingerprint && existingFingerprint !== fingerprint) {
|
||||
throw new CuaRuntimeError('CUA control session identity changed after session creation', 'incompatible')
|
||||
}
|
||||
let session = sharedSessions.get(key)
|
||||
if (!session) {
|
||||
if (testSessionFactory) {
|
||||
session = testSessionFactory(identity)
|
||||
} else {
|
||||
sharedAdapter ??= CuaDriverAdapter.connect()
|
||||
const settings = readDesktopUseSettingsSync()
|
||||
const selection = backendSelections.get(key)
|
||||
session = sharedAdapter.then(adapter => adapter.openSession(
|
||||
identity,
|
||||
undefined,
|
||||
selection?.backend === 'cua' ? selection.cursorEnabled : settings.cua_cursor_enabled
|
||||
))
|
||||
}
|
||||
sharedSessions.set(key, session)
|
||||
sharedSessionIdentities.set(key, fingerprint)
|
||||
void session.catch(() => {
|
||||
sharedSessions.delete(key)
|
||||
sharedSessionIdentities.delete(key)
|
||||
})
|
||||
}
|
||||
return session
|
||||
}
|
||||
|
||||
/** Revoke a CUA cursor/session when its outer Hermes authority ends. */
|
||||
export async function closeCuaControlSession(controlSessionId: string, reason?: string): Promise<void> {
|
||||
const session = sharedSessions.get(controlSessionId)
|
||||
sharedSessions.delete(controlSessionId)
|
||||
sharedSessionIdentities.delete(controlSessionId)
|
||||
backendSelections.delete(controlSessionId)
|
||||
publishLifecycle()
|
||||
if (session) await session.then(value => value.close(reason)).catch(() => undefined)
|
||||
}
|
||||
|
||||
export async function closeAllCuaControlSessions(reason?: string): Promise<void> {
|
||||
const sessions = [...sharedSessions.values()]
|
||||
sharedSessions.clear()
|
||||
sharedSessionIdentities.clear()
|
||||
backendSelections.clear()
|
||||
publishLifecycle()
|
||||
await Promise.allSettled(sessions.map(session => session.then(value => value.close(reason))))
|
||||
sharedAdapter = null
|
||||
}
|
||||
@@ -0,0 +1,390 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { createReadStream } from 'node:fs'
|
||||
import { access, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises'
|
||||
import { homedir, tmpdir } from 'node:os'
|
||||
import { delimiter, join, relative, resolve, sep, win32 } from 'node:path'
|
||||
|
||||
import {
|
||||
CuaDriverAdapter,
|
||||
SpawnCuaProcessRunner,
|
||||
cuaDriverEnvironment,
|
||||
type CuaProcessRunner
|
||||
} from './cuaDriver.js'
|
||||
|
||||
export const CUA_SUPPORTED_MIN_VERSION = '0.19.3'
|
||||
export const CUA_SUPPORTED_MAX_EXCLUSIVE = '0.20.0'
|
||||
const TRUSTED_REPOSITORY = 'trycua/cua'
|
||||
const TRUSTED_PRODUCT = 'cua-driver-rs'
|
||||
const RELEASE_BASE = 'https://github.com/trycua/cua/releases/download'
|
||||
const MAX_INSTALLER_BYTES = 2 * 1024 * 1024
|
||||
|
||||
interface FetchResponse {
|
||||
ok: boolean
|
||||
status: number
|
||||
arrayBuffer(): Promise<ArrayBuffer>
|
||||
json(): Promise<unknown>
|
||||
}
|
||||
|
||||
export type CuaManagementFetch = (url: string) => Promise<FetchResponse>
|
||||
|
||||
export interface CuaUpdateStatus {
|
||||
checked_at?: string
|
||||
current_version?: string
|
||||
latest_version?: string
|
||||
update_available: boolean
|
||||
compatible: boolean
|
||||
error?: string
|
||||
release_notes_url?: string
|
||||
}
|
||||
|
||||
export interface CuaManagementStatus {
|
||||
installed: boolean
|
||||
canonical_path: string | null
|
||||
discovered_path: string | null
|
||||
stale_path_shim: boolean
|
||||
current_version: string | null
|
||||
compatible: boolean
|
||||
compatibility_reason: string | null
|
||||
binary_sha256: string | null
|
||||
release_source: typeof TRUSTED_REPOSITORY
|
||||
telemetry_enabled: false
|
||||
bundled: false
|
||||
supported_range: {
|
||||
minimum: typeof CUA_SUPPORTED_MIN_VERSION
|
||||
maximum_exclusive: typeof CUA_SUPPORTED_MAX_EXCLUSIVE
|
||||
}
|
||||
update?: CuaUpdateStatus
|
||||
operation?: {
|
||||
kind: 'install' | 'update'
|
||||
state: 'completed'
|
||||
version: string
|
||||
release_manifest_verified: true
|
||||
installer_checksum_verified: true
|
||||
runtime_verified: true
|
||||
}
|
||||
}
|
||||
|
||||
export interface CuaManagementOptions {
|
||||
platform?: NodeJS.Platform
|
||||
arch?: string
|
||||
homeDir?: string
|
||||
path?: string
|
||||
runner?: CuaProcessRunner
|
||||
fetch?: CuaManagementFetch
|
||||
systemRoot?: string
|
||||
}
|
||||
|
||||
interface TrustedRelease {
|
||||
version: string
|
||||
installerUrl: string
|
||||
installerSha256: string
|
||||
}
|
||||
|
||||
function versionTuple(value: string): [number, number, number] | null {
|
||||
const match = /(?:^|\s|v)(\d+)\.(\d+)\.(\d+)(?:\s|$)/.exec(value.trim())
|
||||
return match ? [Number(match[1]), Number(match[2]), Number(match[3])] : null
|
||||
}
|
||||
|
||||
function compareVersion(left: string, right: string): number | null {
|
||||
const a = versionTuple(left)
|
||||
const b = versionTuple(right)
|
||||
if (!a || !b) return null
|
||||
for (let index = 0; index < 3; index += 1) {
|
||||
const delta = a[index] - b[index]
|
||||
if (delta !== 0) return delta
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
export function isSupportedCuaVersion(value: string): boolean {
|
||||
const minimum = compareVersion(value, CUA_SUPPORTED_MIN_VERSION)
|
||||
const maximum = compareVersion(value, CUA_SUPPORTED_MAX_EXCLUSIVE)
|
||||
return minimum !== null && maximum !== null && minimum >= 0 && maximum < 0
|
||||
}
|
||||
|
||||
function canonicalPaths(homeDir: string): { executable: string; releases: string } {
|
||||
const packages = join(resolve(homeDir), '.cua-driver', 'packages')
|
||||
return {
|
||||
executable: join(packages, 'current', 'cua-driver.exe'),
|
||||
releases: join(packages, 'releases')
|
||||
}
|
||||
}
|
||||
|
||||
async function resolveCanonical(options: CuaManagementOptions): Promise<string | null> {
|
||||
if ((options.platform ?? process.platform) !== 'win32') return null
|
||||
const paths = canonicalPaths(options.homeDir ?? homedir())
|
||||
try {
|
||||
const [binary, releases] = await Promise.all([
|
||||
realpath(paths.executable),
|
||||
realpath(paths.releases)
|
||||
])
|
||||
await access(binary)
|
||||
const inside = relative(releases, binary)
|
||||
if (!inside || inside === '..' || inside.startsWith(`..${sep}`)) return null
|
||||
return binary
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
async function firstPathCandidate(options: CuaManagementOptions): Promise<string | null> {
|
||||
if ((options.platform ?? process.platform) !== 'win32') return null
|
||||
for (const entry of (options.path ?? process.env.PATH ?? '').split(delimiter)) {
|
||||
const trimmed = entry.trim().replace(/^"|"$/g, '')
|
||||
if (!trimmed) continue
|
||||
const candidate = join(trimmed, 'cua-driver.exe')
|
||||
try {
|
||||
await access(candidate)
|
||||
return await realpath(candidate)
|
||||
} catch {
|
||||
// Continue to the next PATH entry.
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
async function sha256File(path: string): Promise<string> {
|
||||
return new Promise((resolvePromise, reject) => {
|
||||
const hash = createHash('sha256')
|
||||
const stream = createReadStream(path)
|
||||
stream.on('data', chunk => hash.update(chunk))
|
||||
stream.on('error', reject)
|
||||
stream.on('end', () => resolvePromise(hash.digest('hex')))
|
||||
})
|
||||
}
|
||||
|
||||
function safeError(error: unknown): string {
|
||||
return (error instanceof Error ? error.message : String(error)).slice(0, 500)
|
||||
}
|
||||
|
||||
async function binaryVersion(binary: string, runner: CuaProcessRunner): Promise<string | null> {
|
||||
const result = await runner.run(binary, ['--version'], {
|
||||
timeoutMs: 8_000,
|
||||
env: cuaDriverEnvironment()
|
||||
})
|
||||
if (result.exitCode !== 0) return null
|
||||
const tuple = versionTuple(result.stdout)
|
||||
return tuple?.join('.') ?? null
|
||||
}
|
||||
|
||||
export async function getCuaManagementStatus(
|
||||
options: CuaManagementOptions = {}
|
||||
): Promise<CuaManagementStatus> {
|
||||
const runner = options.runner ?? new SpawnCuaProcessRunner()
|
||||
const [canonical, discovered] = await Promise.all([
|
||||
resolveCanonical(options),
|
||||
firstPathCandidate(options)
|
||||
])
|
||||
let currentVersion: string | null = null
|
||||
let binarySha256: string | null = null
|
||||
let compatibilityReason: string | null = null
|
||||
if (canonical) {
|
||||
try {
|
||||
;[currentVersion, binarySha256] = await Promise.all([
|
||||
binaryVersion(canonical, runner),
|
||||
sha256File(canonical)
|
||||
])
|
||||
} catch (error) {
|
||||
compatibilityReason = safeError(error)
|
||||
}
|
||||
}
|
||||
const compatible = !!currentVersion && isSupportedCuaVersion(currentVersion)
|
||||
if (canonical && !compatibilityReason && !compatible) {
|
||||
compatibilityReason = currentVersion
|
||||
? `CUA Driver ${currentVersion} is outside the supported range`
|
||||
: 'CUA Driver version could not be verified'
|
||||
}
|
||||
return {
|
||||
installed: canonical !== null,
|
||||
canonical_path: canonical,
|
||||
discovered_path: discovered,
|
||||
stale_path_shim: !!canonical && !!discovered && resolve(canonical).toLowerCase() !== resolve(discovered).toLowerCase(),
|
||||
current_version: currentVersion,
|
||||
compatible,
|
||||
compatibility_reason: compatibilityReason,
|
||||
binary_sha256: binarySha256,
|
||||
release_source: TRUSTED_REPOSITORY,
|
||||
telemetry_enabled: false,
|
||||
bundled: false,
|
||||
supported_range: {
|
||||
minimum: CUA_SUPPORTED_MIN_VERSION,
|
||||
maximum_exclusive: CUA_SUPPORTED_MAX_EXCLUSIVE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeUpdatePayload(payload: Record<string, unknown>): CuaUpdateStatus {
|
||||
const latest = typeof payload.latest_version === 'string' ? payload.latest_version : undefined
|
||||
const error = typeof payload.error === 'string' && payload.error.trim() ? payload.error.slice(0, 500) : undefined
|
||||
return {
|
||||
checked_at: typeof payload.checked_at === 'string' ? payload.checked_at : undefined,
|
||||
current_version: typeof payload.current_version === 'string' ? payload.current_version : undefined,
|
||||
latest_version: latest,
|
||||
update_available: payload.update_available === true,
|
||||
compatible: !!latest && isSupportedCuaVersion(latest),
|
||||
error,
|
||||
release_notes_url: typeof payload.release_notes_url === 'string' ? payload.release_notes_url : undefined
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkCuaUpdate(
|
||||
options: CuaManagementOptions = {}
|
||||
): Promise<CuaManagementStatus> {
|
||||
const runner = options.runner ?? new SpawnCuaProcessRunner()
|
||||
let status = await getCuaManagementStatus({ ...options, runner })
|
||||
for (let attempt = 0; !status.installed && attempt < 2; attempt += 1) {
|
||||
await new Promise(resolvePromise => setTimeout(resolvePromise, 50))
|
||||
status = await getCuaManagementStatus({ ...options, runner })
|
||||
}
|
||||
if (!status.canonical_path) {
|
||||
return { ...status, update: { update_available: false, compatible: false, error: 'CUA Driver is not installed' } }
|
||||
}
|
||||
const result = await runner.run(status.canonical_path, ['check-update', '--json', '--no-cache'], {
|
||||
timeoutMs: 30_000,
|
||||
env: cuaDriverEnvironment()
|
||||
})
|
||||
if (result.exitCode !== 0) {
|
||||
return {
|
||||
...status,
|
||||
update: {
|
||||
update_available: false,
|
||||
compatible: false,
|
||||
error: result.stderr.trim().slice(0, 500) || `check-update exited ${result.exitCode}`
|
||||
}
|
||||
}
|
||||
}
|
||||
try {
|
||||
const payload = JSON.parse(result.stdout) as Record<string, unknown>
|
||||
return { ...status, update: normalizeUpdatePayload(payload) }
|
||||
} catch {
|
||||
return { ...status, update: { update_available: false, compatible: false, error: 'CUA Driver returned invalid update JSON' } }
|
||||
}
|
||||
}
|
||||
|
||||
async function trustedRelease(version: string, fetchImpl: CuaManagementFetch): Promise<TrustedRelease> {
|
||||
if (!isSupportedCuaVersion(version)) {
|
||||
throw new Error(`CUA Driver ${version} is outside the Hermes-supported range`)
|
||||
}
|
||||
const tag = `cua-driver-rs-v${version}`
|
||||
const base = `${RELEASE_BASE}/${tag}`
|
||||
const manifestResponse = await fetchImpl(`${base}/release-manifest.json`)
|
||||
if (!manifestResponse.ok) throw new Error(`CUA release manifest request failed (${manifestResponse.status})`)
|
||||
const manifest = await manifestResponse.json() as Record<string, unknown>
|
||||
if (
|
||||
manifest.schemaVersion !== 1 || manifest.repository !== TRUSTED_REPOSITORY ||
|
||||
manifest.product !== TRUSTED_PRODUCT || manifest.version !== version || manifest.tag !== tag
|
||||
) {
|
||||
throw new Error('CUA release publisher or version manifest is invalid')
|
||||
}
|
||||
const assets = Array.isArray(manifest.assets) ? manifest.assets : []
|
||||
const installer = assets.find(asset =>
|
||||
!!asset && typeof asset === 'object' && (asset as Record<string, unknown>).name === 'install.ps1'
|
||||
) as Record<string, unknown> | undefined
|
||||
const checksum = typeof installer?.sha256 === 'string' ? installer.sha256.toLowerCase() : ''
|
||||
if (!/^[0-9a-f]{64}$/.test(checksum)) throw new Error('CUA release installer checksum is missing')
|
||||
return { version, installerUrl: `${base}/install.ps1`, installerSha256: checksum }
|
||||
}
|
||||
|
||||
async function applyTrustedRelease(
|
||||
release: TrustedRelease,
|
||||
runner: CuaProcessRunner,
|
||||
fetchImpl: CuaManagementFetch,
|
||||
systemRootOverride?: string
|
||||
): Promise<void> {
|
||||
const response = await fetchImpl(release.installerUrl)
|
||||
if (!response.ok) throw new Error(`CUA installer request failed (${response.status})`)
|
||||
const script = Buffer.from(await response.arrayBuffer())
|
||||
if (script.byteLength === 0 || script.byteLength > MAX_INSTALLER_BYTES) {
|
||||
throw new Error('CUA installer size is invalid')
|
||||
}
|
||||
const actual = createHash('sha256').update(script).digest('hex')
|
||||
if (actual !== release.installerSha256) throw new Error('CUA installer checksum verification failed')
|
||||
const directory = await mkdtemp(join(tmpdir(), 'hermes-cua-install-'))
|
||||
const path = join(directory, 'install.ps1')
|
||||
try {
|
||||
await writeFile(path, script, { mode: 0o600 })
|
||||
const configuredSystemRoot = systemRootOverride ?? process.env.SystemRoot ?? process.env.WINDIR
|
||||
if (!configuredSystemRoot || !configuredSystemRoot.match(/^[A-Za-z]:[\\/]/)) {
|
||||
throw new Error('Windows system PowerShell path is unavailable')
|
||||
}
|
||||
const powershell = win32.join(win32.resolve(configuredSystemRoot), 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe')
|
||||
const result = await runner.run(powershell, [
|
||||
'-NoProfile',
|
||||
'-NonInteractive',
|
||||
'-ExecutionPolicy',
|
||||
'Bypass',
|
||||
'-File',
|
||||
path
|
||||
], {
|
||||
timeoutMs: 15 * 60_000,
|
||||
env: cuaDriverEnvironment({
|
||||
CUA_DRIVER_RS_VERSION: release.version,
|
||||
CUA_DRIVER_INSTALL_CHANNEL: 'install_script'
|
||||
})
|
||||
})
|
||||
if (result.exitCode !== 0) {
|
||||
throw new Error(result.stderr.trim().slice(0, 500) || `CUA installer exited ${result.exitCode}`)
|
||||
}
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
|
||||
async function mutateCua(
|
||||
kind: 'install' | 'update',
|
||||
version: string,
|
||||
options: CuaManagementOptions
|
||||
): Promise<CuaManagementStatus> {
|
||||
if ((options.platform ?? process.platform) !== 'win32') throw new Error('CUA management is currently available on Windows only')
|
||||
const runner = options.runner ?? new SpawnCuaProcessRunner()
|
||||
const fetchImpl = options.fetch ?? (globalThis.fetch as unknown as CuaManagementFetch)
|
||||
const release = await trustedRelease(version, fetchImpl)
|
||||
await applyTrustedRelease(release, runner, fetchImpl, options.systemRoot)
|
||||
const status = await getCuaManagementStatus({ ...options, runner })
|
||||
if (!status.installed || status.current_version !== version || !status.compatible) {
|
||||
throw new Error(
|
||||
`CUA canonical package did not pass post-install version verification ` +
|
||||
`(installed=${status.installed}, version=${status.current_version ?? 'unknown'}, compatible=${status.compatible})`
|
||||
)
|
||||
}
|
||||
const runtime = await CuaDriverAdapter.status({
|
||||
platform: options.platform,
|
||||
homeDir: options.homeDir,
|
||||
runner
|
||||
})
|
||||
if (
|
||||
!runtime.ready || runtime.binaryVersion !== version ||
|
||||
!runtime.binaryPath || !status.canonical_path ||
|
||||
resolve(runtime.binaryPath).toLowerCase() !== resolve(status.canonical_path).toLowerCase()
|
||||
) {
|
||||
throw new Error(runtime.reason ?? 'CUA canonical manifest, path, or health verification failed')
|
||||
}
|
||||
return {
|
||||
...status,
|
||||
operation: {
|
||||
kind,
|
||||
state: 'completed',
|
||||
version,
|
||||
release_manifest_verified: true,
|
||||
installer_checksum_verified: true,
|
||||
runtime_verified: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function installCuaDriver(options: CuaManagementOptions = {}): Promise<CuaManagementStatus> {
|
||||
const existing = await getCuaManagementStatus(options)
|
||||
if (existing.installed) return existing
|
||||
return mutateCua('install', CUA_SUPPORTED_MIN_VERSION, options)
|
||||
}
|
||||
|
||||
export async function updateCuaDriver(options: CuaManagementOptions = {}): Promise<CuaManagementStatus> {
|
||||
const checked = await checkCuaUpdate(options)
|
||||
const update = checked.update
|
||||
if (!update || update.error) throw new Error(update?.error ?? 'CUA update check failed')
|
||||
if (!update.update_available) return checked
|
||||
if (!update.latest_version || !update.compatible) {
|
||||
throw new Error(`CUA Driver ${update.latest_version ?? 'unknown'} is available but unsupported by this Hermes Relay build`)
|
||||
}
|
||||
return mutateCua('update', update.latest_version, options)
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { spawn } from 'node:child_process'
|
||||
|
||||
import {
|
||||
cancelComputerGrant,
|
||||
computerGrantAllowsTarget,
|
||||
getActiveComputerGrant,
|
||||
getComputerGrantSummary,
|
||||
getComputerUseRuntimeSummary,
|
||||
@@ -13,12 +14,22 @@ import {
|
||||
requestComputerGrant,
|
||||
type ComputerGrantMode
|
||||
} from '../computerGrants.js'
|
||||
import { evaluateSensitiveTarget, hasAuthenticatedControlIdentity, type ComputerTarget } from '../computerControlSecurity.js'
|
||||
import {
|
||||
CuaDriverAdapter,
|
||||
closeCuaControlSession,
|
||||
computerControlLifecycleStatus,
|
||||
getCuaControlSession,
|
||||
recordCuaActionEvent,
|
||||
selectComputerControlBackend
|
||||
} from '../cuaDriver.js'
|
||||
import { approveComputerGrant } from '../computerActionApproval.js'
|
||||
import {
|
||||
runComputerInputAction,
|
||||
validateComputerAction
|
||||
} from '../computerInput.js'
|
||||
import type { ToolHandler } from '../router.js'
|
||||
import type { ToolContext, ToolHandler } from '../router.js'
|
||||
import { readDesktopUseSettingsSync } from '../../lib/desktopUseSettings.js'
|
||||
import { screenshotHandler } from './screenshot.js'
|
||||
|
||||
const STATUS_TIMEOUT_MS = 5_000
|
||||
@@ -212,13 +223,18 @@ function pngDimensions(base64: string): { width: number; height: number } | null
|
||||
}
|
||||
}
|
||||
|
||||
function failure(code: string, message: string, extra: Record<string, unknown> = {}): Record<string, unknown> {
|
||||
function failure(
|
||||
code: string,
|
||||
message: string,
|
||||
extra: Record<string, unknown> = {},
|
||||
authority?: ToolContext['controlSession']
|
||||
): Record<string, unknown> {
|
||||
return {
|
||||
ok: false,
|
||||
code,
|
||||
message,
|
||||
...EXPERIMENTAL_META,
|
||||
grant: getComputerGrantSummary(),
|
||||
grant: getComputerGrantSummary(authority),
|
||||
...extra
|
||||
}
|
||||
}
|
||||
@@ -230,17 +246,178 @@ function parseGrantMode(value: unknown): ComputerGrantMode | null {
|
||||
return null
|
||||
}
|
||||
|
||||
function positiveInteger(value: unknown): number | null {
|
||||
return typeof value === 'number' && Number.isSafeInteger(value) && value > 0 ? value : null
|
||||
}
|
||||
|
||||
function cuaTarget(args: Record<string, unknown>): Pick<ComputerTarget, 'pid' | 'windowId'> | null {
|
||||
const pid = positiveInteger(args.pid)
|
||||
const windowId = positiveInteger(args.window_id)
|
||||
if (pid === null || windowId === null) return null
|
||||
return { pid, windowId }
|
||||
}
|
||||
|
||||
async function cuaPreflight(args: Record<string, unknown>, ctx: ToolContext): Promise<
|
||||
| { ok: true; target: ComputerTarget; grantId: string | null; session: Awaited<ReturnType<typeof getCuaControlSession>> }
|
||||
| { ok: false; result: Record<string, unknown> }
|
||||
> {
|
||||
if (!hasAuthenticatedControlIdentity(ctx.controlSession) || !ctx.controlSecurity) {
|
||||
return { ok: false, result: failure('authenticated_control_session_required', 'Structured CUA control requires server-attested relay, requester, run, and target identity.', {}, ctx.controlSession) }
|
||||
}
|
||||
const requested = cuaTarget(args)
|
||||
if (!requested) return { ok: false, result: failure('invalid_target', 'Structured CUA control requires positive pid and window_id.', {}, ctx.controlSession) }
|
||||
const session = await getCuaControlSession({ ...ctx.controlSession, targetDeviceId: ctx.controlSession.targetDeviceId })
|
||||
const listed = await session.listWindows(requested.pid, ctx.abortSignal)
|
||||
const windows = Array.isArray(listed.windows) ? listed.windows : []
|
||||
const exact = windows.find(item => isObject(item) && item.window_id === requested.windowId)
|
||||
if (!isObject(exact)) {
|
||||
return { ok: false, result: failure('target_not_found', 'CUA Driver did not report the exact requested PID and window.', {}, ctx.controlSession) }
|
||||
}
|
||||
const target: ComputerTarget = {
|
||||
...requested,
|
||||
app: typeof exact.app_name === 'string' ? exact.app_name : undefined,
|
||||
title: typeof exact.title === 'string' ? exact.title : undefined,
|
||||
executable: typeof exact.executable === 'string' ? exact.executable : undefined
|
||||
}
|
||||
const sensitive = evaluateSensitiveTarget(target)
|
||||
if (!sensitive.allowed) {
|
||||
return { ok: false, result: failure('sensitive_target_blocked', 'Structured control is blocked for missing or sensitive application identity.', { reason: sensitive.reason }, ctx.controlSession) }
|
||||
}
|
||||
if (!computerGrantAllowsTarget(ctx.controlSession, target)) {
|
||||
return { ok: false, result: failure('grant_target_mismatch', 'The active computer grant does not cover this application target.', {}, ctx.controlSession) }
|
||||
}
|
||||
return { ok: true, target, grantId: getActiveComputerGrant(ctx.controlSession)?.id ?? null, session }
|
||||
}
|
||||
|
||||
async function cuaSnapshot(args: Record<string, unknown>, ctx: ToolContext): Promise<Record<string, unknown>> {
|
||||
const check = await cuaPreflight(args, ctx)
|
||||
if (!check.ok) return check.result
|
||||
const authority = ctx.controlSession!
|
||||
const raw = await check.session.snapshot({
|
||||
pid: check.target.pid,
|
||||
windowId: check.target.windowId,
|
||||
query: argString(args.query).trim() || undefined,
|
||||
includeScreenshot: args.include_screenshot !== false,
|
||||
maxElements: typeof args.max_elements === 'number' ? args.max_elements : undefined,
|
||||
maxDepth: typeof args.max_depth === 'number' ? args.max_depth : undefined
|
||||
}, ctx.abortSignal)
|
||||
const generation = typeof raw.snapshot_id === 'string' ? raw.snapshot_id : ''
|
||||
if (!generation) return failure('invalid_backend_response', 'CUA Driver snapshot did not include a generation.', {}, authority)
|
||||
const elements = Array.isArray(raw.elements) ? raw.elements : []
|
||||
const safeElements = elements.slice(0, 1_000).map(item => {
|
||||
if (!isObject(item)) return item
|
||||
const driverToken = typeof item.element_token === 'string' ? item.element_token : undefined
|
||||
const output = { ...item }
|
||||
delete output.element_token
|
||||
if (driverToken) {
|
||||
output.snapshot_token = ctx.controlSecurity!.issueSnapshotToken({
|
||||
authority,
|
||||
grantId: check.grantId,
|
||||
target: check.target,
|
||||
snapshotGeneration: generation,
|
||||
driverElementToken: driverToken
|
||||
})
|
||||
}
|
||||
return output
|
||||
})
|
||||
return {
|
||||
ok: true,
|
||||
...EXPERIMENTAL_META,
|
||||
backend: 'cua_driver',
|
||||
snapshot_generation: generation,
|
||||
target: check.target,
|
||||
elements: safeElements,
|
||||
tree_markdown: raw.tree_markdown,
|
||||
screenshot_base64: raw.screenshot_base64,
|
||||
screenshot_width: raw.screenshot_width,
|
||||
screenshot_height: raw.screenshot_height,
|
||||
truncated: elements.length > safeElements.length
|
||||
}
|
||||
}
|
||||
|
||||
async function cuaAction(args: Record<string, unknown>, ctx: ToolContext): Promise<Record<string, unknown>> {
|
||||
const check = await cuaPreflight(args, ctx)
|
||||
if (!check.ok) return check.result
|
||||
const authority = ctx.controlSession!
|
||||
const token = argString(args.snapshot_token).trim()
|
||||
const binding = token ? ctx.controlSecurity!.consumeSnapshotToken(token, {
|
||||
authority,
|
||||
grantId: check.grantId,
|
||||
target: check.target,
|
||||
...(typeof args.snapshot_generation === 'string' ? { snapshotGeneration: args.snapshot_generation } : {})
|
||||
}) : null
|
||||
if (!binding?.driverElementToken) {
|
||||
return failure('invalid_or_stale_snapshot', 'A fresh one-use snapshot_token for this exact session, grant, PID, and window is required.', {}, authority)
|
||||
}
|
||||
const element = { pid: check.target.pid, windowId: check.target.windowId, elementToken: binding.driverElementToken }
|
||||
const action = argString(args.action).trim()
|
||||
let result: Record<string, unknown>
|
||||
if (action === 'click_element') result = await check.session.clickElement(element, ctx.abortSignal)
|
||||
else if (action === 'set_value') result = await check.session.setElementValue(element, argString(args.value), ctx.abortSignal)
|
||||
else if (action === 'press_key') result = await check.session.pressKey(check.target, argString(args.key), ctx.abortSignal)
|
||||
else if (action === 'scroll_element') {
|
||||
const direction = argString(args.direction) as 'up' | 'down' | 'left' | 'right'
|
||||
if (!['up', 'down', 'left', 'right'].includes(direction)) return failure('invalid_request', 'direction must be up, down, left, or right.', {}, authority)
|
||||
result = await check.session.scroll(element, direction, positiveInteger(args.amount) ?? 1, ctx.abortSignal)
|
||||
} else return failure('invalid_request', 'Unsupported structured CUA action.', {}, authority)
|
||||
|
||||
const after = await check.session.snapshot({ pid: check.target.pid, windowId: check.target.windowId, includeScreenshot: false }, ctx.abortSignal)
|
||||
recordCuaActionEvent({
|
||||
action,
|
||||
target_app: check.target.app,
|
||||
target_pid: check.target.pid,
|
||||
target_window_id: check.target.windowId,
|
||||
verification: 'snapshot_captured'
|
||||
})
|
||||
return {
|
||||
ok: true,
|
||||
...EXPERIMENTAL_META,
|
||||
backend: 'cua',
|
||||
dispatch: 'background',
|
||||
control_session_id: authority.controlSessionId,
|
||||
target_app: check.target.app,
|
||||
target_title: check.target.title,
|
||||
target_pid: check.target.pid,
|
||||
target_window_id: check.target.windowId,
|
||||
action,
|
||||
verification: 'snapshot_captured',
|
||||
phase: 'structured_primary',
|
||||
result,
|
||||
verification_snapshot: after
|
||||
}
|
||||
}
|
||||
|
||||
export const computerStatusHandler: ToolHandler = async (_args, ctx) => {
|
||||
const grant = getActiveComputerGrant()
|
||||
const runtime = getComputerUseRuntimeSummary()
|
||||
const grant = getActiveComputerGrant(ctx.controlSession)
|
||||
const runtime = getComputerUseRuntimeSummary(ctx.controlSession)
|
||||
const inputBackendReady = runtime.consented === true && process.platform === 'win32'
|
||||
const fullAccess = runtime.full_access === true
|
||||
const settings = readDesktopUseSettingsSync()
|
||||
const cua = settings.computer_control_engine === 'cua' ? await CuaDriverAdapter.status() : null
|
||||
const lifecycle = computerControlLifecycleStatus()
|
||||
return {
|
||||
ok: true,
|
||||
...EXPERIMENTAL_META,
|
||||
platform: process.platform,
|
||||
displays: await getDisplays(),
|
||||
runtime,
|
||||
computer_control_engine: {
|
||||
...lifecycle,
|
||||
selected: settings.computer_control_engine,
|
||||
effective: lifecycle.active_backend === 'cua'
|
||||
? 'cua'
|
||||
: lifecycle.active_backend === 'legacy_compat'
|
||||
? 'legacy'
|
||||
: settings.computer_control_engine === 'cua' && cua?.ready ? 'cua' : 'legacy',
|
||||
cursor_enabled: settings.cua_cursor_enabled,
|
||||
foreground_escalation_enabled: false,
|
||||
cua,
|
||||
message: settings.computer_control_engine === 'cua' && !cua?.ready
|
||||
? cua?.available
|
||||
? `CUA Driver is installed, but not ready; new sessions use Windows Input compatibility mode. ${cua.reason ?? ''}`.trim()
|
||||
: `CUA Driver is unavailable before control starts; new sessions use Windows Input compatibility mode. ${cua?.reason ?? ''}`.trim()
|
||||
: null
|
||||
},
|
||||
permissions: {
|
||||
screenshot: fullAccess ? 'full_access' : grant ? 'granted' : 'grant_required',
|
||||
input: fullAccess || grant?.mode === 'assist' || grant?.mode === 'control'
|
||||
@@ -248,9 +425,9 @@ export const computerStatusHandler: ToolHandler = async (_args, ctx) => {
|
||||
? fullAccess ? 'full_access' : 'granted_until_expiry'
|
||||
: 'grant_active_but_input_backend_unavailable'
|
||||
: 'not_granted',
|
||||
accessibility: 'not_implemented'
|
||||
accessibility: cua?.ready ? 'available' : 'unavailable'
|
||||
},
|
||||
grant: getComputerGrantSummary(),
|
||||
grant: getComputerGrantSummary(ctx.controlSession),
|
||||
overlay: {
|
||||
visible: ctx.interactive,
|
||||
state: ctx.interactive ? 'cli_grant_prompt_available' : 'not_available',
|
||||
@@ -267,13 +444,25 @@ export const computerStatusHandler: ToolHandler = async (_args, ctx) => {
|
||||
}
|
||||
|
||||
export const computerScreenshotHandler: ToolHandler = async (args, ctx) => {
|
||||
if (!hasComputerObserveGrant()) {
|
||||
if (!hasComputerObserveGrant(ctx.controlSession)) {
|
||||
return failure(
|
||||
'grant_required',
|
||||
'Screenshot observe mode requires an active observe/assist/control grant. Call desktop_computer_grant_request first.'
|
||||
'Screenshot observe mode requires an active observe/assist/control grant. Call desktop_computer_grant_request first.',
|
||||
{},
|
||||
ctx.controlSession
|
||||
)
|
||||
}
|
||||
|
||||
const settings = readDesktopUseSettingsSync()
|
||||
const selection = ctx.controlSession
|
||||
? await selectComputerControlBackend(ctx.controlSession.controlSessionId, settings.computer_control_engine, settings.cua_cursor_enabled)
|
||||
: null
|
||||
if (selection?.backend === 'cua') {
|
||||
// Window-scoped observation belongs to CUA. A caller may still request the
|
||||
// existing read-only display capture; it is not an input-backend fallback.
|
||||
if (args.pid !== undefined || args.window_id !== undefined) return cuaSnapshot(args, ctx)
|
||||
}
|
||||
|
||||
if (args.region !== undefined && args.region !== null) {
|
||||
return failure(
|
||||
'not_implemented',
|
||||
@@ -299,6 +488,7 @@ export const computerScreenshotHandler: ToolHandler = async (args, ctx) => {
|
||||
ok: true,
|
||||
...EXPERIMENTAL_META,
|
||||
mode: 'observe',
|
||||
backend: 'system_capture',
|
||||
format: result.format,
|
||||
bytes_base64: result.bytes_base64,
|
||||
saved_path: result.saved_path,
|
||||
@@ -318,7 +508,7 @@ export const computerScreenshotHandler: ToolHandler = async (args, ctx) => {
|
||||
applied: false,
|
||||
reason: 'Sensitive-window redaction is planned but not implemented yet.'
|
||||
},
|
||||
grant: getComputerGrantSummary()
|
||||
grant: getComputerGrantSummary(ctx.controlSession)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -327,17 +517,39 @@ export const computerActionHandler: ToolHandler = async (args, ctx) => {
|
||||
if (!action) {
|
||||
return failure('invalid_request', 'desktop_computer_action requires an action name.')
|
||||
}
|
||||
const settings = readDesktopUseSettingsSync()
|
||||
const selection = ctx.controlSession
|
||||
? await selectComputerControlBackend(ctx.controlSession.controlSessionId, settings.computer_control_engine, settings.cua_cursor_enabled)
|
||||
: null
|
||||
if (selection?.backend === 'cua') {
|
||||
if (!['click_element', 'set_value', 'press_key', 'scroll_element'].includes(action)) {
|
||||
return failure(
|
||||
'cua_structured_action_required',
|
||||
'CUA is selected; legacy coordinate and foreground-routed input is disabled. Take a structured snapshot and use an element action.',
|
||||
{ action },
|
||||
ctx.controlSession
|
||||
)
|
||||
}
|
||||
if (!hasComputerInputGrant(ctx.controlSession)) {
|
||||
return failure('grant_required', 'Structured host input requires an active assist/control grant.', { action }, ctx.controlSession)
|
||||
}
|
||||
return cuaAction(args, ctx)
|
||||
}
|
||||
if (['click_element', 'set_value', 'press_key', 'scroll_element'].includes(action)) {
|
||||
return failure('cua_unavailable', 'Structured CUA actions are unavailable in the compatibility backend.', { action }, ctx.controlSession)
|
||||
}
|
||||
const displays = await getDisplays()
|
||||
const validation = validateComputerAction(args, displays)
|
||||
if (!validation.ok) {
|
||||
return failure(validation.code, validation.message, { action })
|
||||
}
|
||||
|
||||
if (!hasComputerInputGrant()) {
|
||||
if (!hasComputerInputGrant(ctx.controlSession)) {
|
||||
return failure(
|
||||
'grant_required',
|
||||
'Host input is disabled. Request and locally approve an assist/control grant first.',
|
||||
{ action }
|
||||
{ action },
|
||||
ctx.controlSession
|
||||
)
|
||||
}
|
||||
|
||||
@@ -352,12 +564,21 @@ export const computerActionHandler: ToolHandler = async (args, ctx) => {
|
||||
performed_at: new Date().toISOString(),
|
||||
duration_ms: Date.now() - started,
|
||||
input_backend: inputResult.backend,
|
||||
backend: 'legacy_compat',
|
||||
dispatch: 'foreground_compatibility',
|
||||
phase: selection?.reason === 'cua_unavailable_before_session' ? 'pre_session_safe_fallback' : 'explicit_compatibility',
|
||||
control_session_id: ctx.controlSession?.controlSessionId,
|
||||
verification: normalized.returnScreenshot ? 'snapshot_captured' : 'not_requested',
|
||||
platform: inputResult.platform,
|
||||
grant: getComputerGrantSummary()
|
||||
grant: getComputerGrantSummary(ctx.controlSession)
|
||||
}
|
||||
if (normalized.returnScreenshot) {
|
||||
response.after_screenshot = await computerScreenshotHandler({ display: args.display ?? 'primary' }, ctx)
|
||||
}
|
||||
recordCuaActionEvent({
|
||||
action: normalized.action,
|
||||
verification: normalized.returnScreenshot ? 'snapshot_captured' : 'not_requested'
|
||||
})
|
||||
return response
|
||||
}
|
||||
|
||||
@@ -366,7 +587,7 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
|
||||
if (!mode) {
|
||||
return failure('invalid_request', 'mode must be one of observe, assist, or control.')
|
||||
}
|
||||
const runtime = getComputerUseRuntimeSummary()
|
||||
const runtime = getComputerUseRuntimeSummary(ctx.controlSession)
|
||||
if (runtime.full_access === true) {
|
||||
return {
|
||||
...requestComputerGrant({
|
||||
@@ -374,7 +595,7 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
|
||||
scope: args.scope,
|
||||
duration_seconds: args.duration_seconds,
|
||||
reason: args.reason
|
||||
}),
|
||||
}, ctx.controlSession),
|
||||
...EXPERIMENTAL_META
|
||||
}
|
||||
}
|
||||
@@ -382,7 +603,9 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
|
||||
if (runtime.consented !== true) {
|
||||
return failure(
|
||||
'computer_use_consent_required',
|
||||
'Assist/control grants require local desktop-tool consent for this relay URL before task-scoped input grants can be created.'
|
||||
'Assist/control grants require local desktop-tool consent for this relay URL before task-scoped input grants can be created.',
|
||||
{},
|
||||
ctx.controlSession
|
||||
)
|
||||
}
|
||||
const approval = await approveComputerGrant({
|
||||
@@ -406,17 +629,18 @@ export const computerGrantRequestHandler: ToolHandler = async (args, ctx) => {
|
||||
scope: args.scope,
|
||||
duration_seconds: args.duration_seconds,
|
||||
reason: args.reason
|
||||
}),
|
||||
}, ctx.controlSession),
|
||||
...EXPERIMENTAL_META
|
||||
}
|
||||
}
|
||||
|
||||
export const computerCancelHandler: ToolHandler = async (args) => {
|
||||
export const computerCancelHandler: ToolHandler = async (args, ctx) => {
|
||||
const reason = typeof args.reason === 'string' && args.reason.trim()
|
||||
? args.reason.trim()
|
||||
: 'cancelled by desktop_computer_cancel'
|
||||
if (ctx.controlSession) await closeCuaControlSession(ctx.controlSession.controlSessionId, reason)
|
||||
return {
|
||||
...cancelComputerGrant(reason),
|
||||
...cancelComputerGrant(reason, ctx.controlSession),
|
||||
...EXPERIMENTAL_META
|
||||
}
|
||||
}
|
||||
|
||||
+186
-9
@@ -30,18 +30,31 @@ import {
|
||||
auditDetails,
|
||||
categorizeTool,
|
||||
previewArgs,
|
||||
persistAuditScreenshot,
|
||||
resultExitCode,
|
||||
summarizeResult
|
||||
} from '../lib/auditLog.js'
|
||||
import { VERSION } from '../version.js'
|
||||
import { desktopDeviceId } from '../deviceIdentity.js'
|
||||
import { getComputerGrantSummary, getComputerUseRuntimeSummary } from './computerGrants.js'
|
||||
import { readDesktopUseSettingsSync } from '../lib/desktopUseSettings.js'
|
||||
import {
|
||||
getComputerGrantSummary,
|
||||
getComputerUseRuntimeSummary,
|
||||
initializeComputerControlSession,
|
||||
revokeComputerControlSession
|
||||
} from './computerGrants.js'
|
||||
import {
|
||||
ComputerControlSecurityState,
|
||||
type ComputerControlAuthority
|
||||
} from './computerControlSecurity.js'
|
||||
import { closeAllCuaControlSessions, closeCuaControlSession } from './cuaDriver.js'
|
||||
|
||||
/** The payload shape server → client for a single tool invocation. */
|
||||
export interface ToolCallPayload {
|
||||
request_id: string
|
||||
tool: string
|
||||
args: Record<string, unknown>
|
||||
control_session?: unknown
|
||||
}
|
||||
|
||||
/** Either a success with a free-form result, or a failure with an error
|
||||
@@ -67,6 +80,10 @@ export interface ToolContext {
|
||||
cwd: string
|
||||
abortSignal: AbortSignal
|
||||
interactive: boolean
|
||||
/** Server-attested control identity. Optional only for legacy handlers. */
|
||||
controlSession?: ComputerControlAuthority
|
||||
/** Hermes-owned snapshot/token binding state for this router lifecycle. */
|
||||
controlSecurity?: ComputerControlSecurityState
|
||||
}
|
||||
|
||||
/** A tool handler. Throws → router responds with `{ok:false, error}`. */
|
||||
@@ -122,6 +139,98 @@ function isToolCallPayload(x: unknown): x is ToolCallPayload {
|
||||
)
|
||||
}
|
||||
|
||||
interface ControlSessionEndPayload {
|
||||
version: 1
|
||||
id: string
|
||||
target_device_id: string
|
||||
reason?: string
|
||||
}
|
||||
|
||||
function parseControlSessionEnd(value: unknown): ControlSessionEndPayload | null {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) return null
|
||||
const raw = value as Record<string, unknown>
|
||||
const field = (key: string): string | null => {
|
||||
const value = raw[key]
|
||||
if (typeof value !== 'string') return null
|
||||
const normalized = value.trim()
|
||||
return normalized && normalized.length <= 256 && !/[\u0000-\u001f\u007f]/u.test(normalized) ? normalized : null
|
||||
}
|
||||
if (raw.version !== 1) return null
|
||||
const id = field('id')
|
||||
const targetDeviceId = field('target_device_id')
|
||||
if (!id || !targetDeviceId) return null
|
||||
const reason = raw.reason === undefined ? undefined : field('reason') ?? undefined
|
||||
return { version: 1, id, target_device_id: targetDeviceId, ...(reason ? { reason } : {}) }
|
||||
}
|
||||
|
||||
function parseControlAuthority(
|
||||
value: unknown,
|
||||
requestId: string,
|
||||
hostUrl: string | undefined
|
||||
): ComputerControlAuthority | null {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) return null
|
||||
const raw = value as Record<string, unknown>
|
||||
const required = (field: string): string | null => {
|
||||
const current = raw[field]
|
||||
if (typeof current !== 'string') return null
|
||||
const normalized = current.trim()
|
||||
if (!normalized || normalized.length > 256 || /[\u0000-\u001f\u007f]/u.test(normalized)) return null
|
||||
return normalized
|
||||
}
|
||||
if (raw.version !== 1) return null
|
||||
const id = required('id')
|
||||
const boundRequestId = required('request_id')
|
||||
const requesterDeviceId = required('requester_device_id')
|
||||
const targetDeviceId = required('target_device_id')
|
||||
const runId = required('run_id')
|
||||
if (
|
||||
!id || !boundRequestId || boundRequestId !== requestId || !requesterDeviceId ||
|
||||
!targetDeviceId || targetDeviceId !== desktopDeviceId() || !runId
|
||||
) return null
|
||||
const optional = (field: string): string | undefined => {
|
||||
const current = raw[field]
|
||||
if (typeof current !== 'string') return undefined
|
||||
const normalized = current.trim()
|
||||
return normalized && normalized.length <= 256 && !/[\u0000-\u001f\u007f]/u.test(normalized)
|
||||
? normalized
|
||||
: undefined
|
||||
}
|
||||
return {
|
||||
controlSessionId: id,
|
||||
// The relay-owned control session is the authenticated execution-session
|
||||
// identity currently available on the wire. Keep the adapter contract
|
||||
// explicit while avoiding any identity supplied through tool args.
|
||||
relaySessionId: id,
|
||||
requesterDeviceId,
|
||||
targetDeviceId,
|
||||
requestId,
|
||||
chatSessionId: optional('chat_session_id'),
|
||||
runId,
|
||||
...(hostUrl ? { hostUrl } : {})
|
||||
}
|
||||
}
|
||||
|
||||
function computerAuditMetadata(result: unknown): Record<string, unknown> {
|
||||
if (!result || typeof result !== 'object' || Array.isArray(result)) return {}
|
||||
const source = result as Record<string, unknown>
|
||||
const metadata: Record<string, unknown> = {}
|
||||
// Window titles frequently contain document names, account names, or page
|
||||
// content. Keep the app and numeric target for drilldown, but never persist
|
||||
// the title in the local activity log.
|
||||
for (const key of ['backend', 'dispatch', 'control_session_id', 'target_app', 'action', 'verification', 'phase']) {
|
||||
if (typeof source[key] === 'string') metadata[key] = String(source[key]).slice(0, 256)
|
||||
}
|
||||
for (const key of ['target_pid', 'target_window_id']) {
|
||||
if (typeof source[key] === 'number' && Number.isSafeInteger(source[key])) metadata[key] = source[key]
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
export function toolResultSucceeded(tool: string, result: unknown): boolean {
|
||||
if (!tool.startsWith('desktop_computer_')) return true
|
||||
return !result || typeof result !== 'object' || Array.isArray(result) || (result as Record<string, unknown>).ok !== false
|
||||
}
|
||||
|
||||
/** Default interactive detection — true iff stdin is a TTY AND we're not
|
||||
* flagged as the daemon subcommand. The daemon command sets
|
||||
* HERMES_RELAY_DAEMON=1 in its own process.env before constructing the
|
||||
@@ -151,6 +260,9 @@ export class DesktopToolRouter {
|
||||
* timeout, abort). Surfaced in the heartbeat so an agent or the dashboard
|
||||
* can ask "is this client healthy?" without parsing transcript history. */
|
||||
private lastError: { message: string; tool: string; ts: number } | null = null
|
||||
private readonly controlAuthority: ComputerControlAuthority
|
||||
private readonly controlSecurity: ComputerControlSecurityState
|
||||
private readonly activeControlAuthorities = new Map<string, ComputerControlAuthority>()
|
||||
|
||||
constructor(opts: DesktopToolRouterOpts) {
|
||||
this.handlers = opts.handlers
|
||||
@@ -161,6 +273,13 @@ export class DesktopToolRouter {
|
||||
// capture it once at construct time so a short shell session that
|
||||
// happens to get its stdin redirected mid-flight doesn't flip mode.
|
||||
this.interactive = opts.interactive ?? detectInteractive()
|
||||
this.controlAuthority = {
|
||||
controlSessionId: `router-${desktopDeviceId()}-${this.startedAtMs}`,
|
||||
...(this.hostUrl ? { hostUrl: this.hostUrl } : {})
|
||||
}
|
||||
this.controlSecurity = new ComputerControlSecurityState(this.controlAuthority)
|
||||
initializeComputerControlSession(this.controlAuthority)
|
||||
this.activeControlAuthorities.set(this.controlAuthority.controlSessionId, this.controlAuthority)
|
||||
}
|
||||
|
||||
/** Install the `onChannel('desktop')` listener and start heartbeats.
|
||||
@@ -188,6 +307,17 @@ export class DesktopToolRouter {
|
||||
void this.dispatch(payload)
|
||||
return
|
||||
}
|
||||
if (type === 'desktop.control_session_end') {
|
||||
const ended = parseControlSessionEnd(payload)
|
||||
if (!ended || ended.target_device_id !== desktopDeviceId()) return
|
||||
const authority = this.activeControlAuthorities.get(ended.id)
|
||||
if (!authority || authority.targetDeviceId !== ended.target_device_id) return
|
||||
this.activeControlAuthorities.delete(ended.id)
|
||||
this.controlSecurity.revokeAuthority(ended.id)
|
||||
revokeComputerControlSession(authority, ended.reason ?? 'relay control session ended')
|
||||
void closeCuaControlSession(ended.id, ended.reason ?? 'relay control session ended')
|
||||
return
|
||||
}
|
||||
// Unknown desktop.* types — ignore; server may extend later.
|
||||
})
|
||||
|
||||
@@ -202,6 +332,7 @@ export class DesktopToolRouter {
|
||||
/** Remove the channel listener and stop heartbeats. Idempotent. */
|
||||
detach(): void {
|
||||
if (!this.attached) {
|
||||
this.revokeControlAuthority()
|
||||
return
|
||||
}
|
||||
this.attached = false
|
||||
@@ -215,6 +346,16 @@ export class DesktopToolRouter {
|
||||
/* ignore */
|
||||
}
|
||||
this.relay = null
|
||||
this.revokeControlAuthority()
|
||||
}
|
||||
|
||||
private revokeControlAuthority(): void {
|
||||
this.controlSecurity.revoke()
|
||||
for (const authority of this.activeControlAuthorities.values()) {
|
||||
revokeComputerControlSession(authority, 'desktop router detached')
|
||||
}
|
||||
this.activeControlAuthorities.clear()
|
||||
void closeAllCuaControlSessions('desktop router detached')
|
||||
}
|
||||
|
||||
/** Broadcast the advertised-tools heartbeat. Safe to call when detached
|
||||
@@ -241,8 +382,8 @@ export class DesktopToolRouter {
|
||||
device_name: os.hostname()
|
||||
}
|
||||
if (this.advertisedTools.some(name => name.startsWith('desktop_computer_'))) {
|
||||
const runtime = getComputerUseRuntimeSummary()
|
||||
const grant = getComputerGrantSummary()
|
||||
const runtime = getComputerUseRuntimeSummary(this.controlAuthority)
|
||||
const grant = getComputerGrantSummary(this.controlAuthority)
|
||||
const inputGrantActive =
|
||||
grant.active === true && (grant.mode === 'assist' || grant.mode === 'control')
|
||||
payload.computer_use = {
|
||||
@@ -289,6 +430,22 @@ export class DesktopToolRouter {
|
||||
return
|
||||
}
|
||||
|
||||
const relayAuthority = parseControlAuthority(cmd.control_session, request_id, this.hostUrl)
|
||||
if (tool.startsWith('desktop_computer_') && cmd.control_session !== undefined && !relayAuthority) {
|
||||
this.sendResponse({ request_id, ok: false, error: 'invalid server control_session binding' })
|
||||
return
|
||||
}
|
||||
const requestAuthority = relayAuthority ?? this.controlAuthority
|
||||
if (relayAuthority) {
|
||||
initializeComputerControlSession(relayAuthority)
|
||||
this.activeControlAuthorities.set(relayAuthority.controlSessionId, relayAuthority)
|
||||
}
|
||||
const controlSession = this.controlSecurity.bindRequest(request_id, requestAuthority)
|
||||
if (!controlSession) {
|
||||
this.sendResponse({ request_id, ok: false, error: 'duplicate or invalid desktop request_id' })
|
||||
return
|
||||
}
|
||||
|
||||
const controller = new AbortController()
|
||||
const timeoutMs = tool.startsWith('desktop_computer_') || tool.startsWith('desktop_adb_') || tool.startsWith('desktop_usb_')
|
||||
? COMPUTER_USE_HANDLER_TIMEOUT_MS
|
||||
@@ -303,7 +460,9 @@ export class DesktopToolRouter {
|
||||
const ctx: ToolContext = {
|
||||
cwd: process.cwd(),
|
||||
abortSignal: controller.signal,
|
||||
interactive: this.interactive
|
||||
interactive: this.interactive,
|
||||
controlSession,
|
||||
controlSecurity: this.controlSecurity
|
||||
}
|
||||
|
||||
try {
|
||||
@@ -314,20 +473,34 @@ export class DesktopToolRouter {
|
||||
// work was completable.
|
||||
this.sendResponse({ request_id, ok: true, result })
|
||||
// Local audit trail — fire-and-forget so logging never delays the reply.
|
||||
void appendAudit({
|
||||
const retention = readDesktopUseSettingsSync()
|
||||
const canRetainScreenshot = tool.includes('screenshot') || tool === 'desktop_computer_action'
|
||||
const screenshotEvidence = retention.activity_screenshot_retention_enabled && canRetainScreenshot
|
||||
? persistAuditScreenshot(result, request_id, retention.activity_screenshot_retention_days)
|
||||
: Promise.resolve({})
|
||||
void screenshotEvidence.then(screenshotEvidence => appendAudit({
|
||||
ts: Date.now(),
|
||||
kind: 'tool.completed',
|
||||
tool,
|
||||
category: categorizeTool(tool),
|
||||
ok: true,
|
||||
ok: toolResultSucceeded(tool, result),
|
||||
request_id,
|
||||
host_url: this.hostUrl,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
exit_code: resultExitCode(result),
|
||||
args_preview: previewArgs(args),
|
||||
summary: summarizeResult(result),
|
||||
...auditDetails(args, result)
|
||||
})
|
||||
...(controlSession.relaySessionId ? { relay_session_id: controlSession.relaySessionId } : {}),
|
||||
...(controlSession.requesterDeviceId ? { requester_device_id: controlSession.requesterDeviceId } : {}),
|
||||
...(controlSession.runId ? { run_id: controlSession.runId } : {}),
|
||||
...(controlSession.targetDeviceId ? { target_device_id: controlSession.targetDeviceId } : {}),
|
||||
...computerAuditMetadata(result),
|
||||
...screenshotEvidence,
|
||||
...(!toolResultSucceeded(tool, result) && result && typeof result === 'object' && !Array.isArray(result) && typeof (result as Record<string, unknown>).code === 'string'
|
||||
? { error: String((result as Record<string, unknown>).code).slice(0, 128) }
|
||||
: {}),
|
||||
...auditDetails(args, result, { redactComputerContent: tool.startsWith('desktop_computer_') })
|
||||
}))
|
||||
} catch (e) {
|
||||
clearTimeout(timeoutTimer)
|
||||
// Distinguish aborts (timeout or transport teardown) from genuine
|
||||
@@ -355,7 +528,11 @@ export class DesktopToolRouter {
|
||||
host_url: this.hostUrl,
|
||||
duration_ms: Date.now() - startedAt,
|
||||
args_preview: previewArgs(args),
|
||||
...auditDetails(args),
|
||||
...(controlSession.relaySessionId ? { relay_session_id: controlSession.relaySessionId } : {}),
|
||||
...(controlSession.requesterDeviceId ? { requester_device_id: controlSession.requesterDeviceId } : {}),
|
||||
...(controlSession.runId ? { run_id: controlSession.runId } : {}),
|
||||
...(controlSession.targetDeviceId ? { target_device_id: controlSession.targetDeviceId } : {}),
|
||||
...auditDetails(args, undefined, { redactComputerContent: tool.startsWith('desktop_computer_') }),
|
||||
error: message
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
// Regenerated from package.json by gen:version script. Do not edit by hand.
|
||||
export const VERSION = "0.4.0-alpha.8" as const
|
||||
export const VERSION = "0.4.0-beta.2" as const
|
||||
|
||||
@@ -13,6 +13,7 @@ export interface WindowsInstallerLaunchOptions {
|
||||
cliPath?: string
|
||||
trayPath?: string
|
||||
delayMs?: number
|
||||
callerPid?: number
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -30,9 +31,12 @@ export function windowsInstallerLaunchPlan(
|
||||
installDir = '',
|
||||
cliPath = '',
|
||||
trayPath = '',
|
||||
delayMs = 1200
|
||||
delayMs = 1200,
|
||||
callerPid = process.pid
|
||||
} = options
|
||||
const script = [
|
||||
'$callerPid = [int]$env:HERMES_RELAY_SETUP_CALLER_PID',
|
||||
'if ($callerPid -gt 0) { Wait-Process -Id $callerPid -ErrorAction SilentlyContinue }',
|
||||
`Start-Sleep -Milliseconds ${Math.max(0, Math.floor(delayMs))}`,
|
||||
'$installer = $env:HERMES_RELAY_SETUP_PATH',
|
||||
"$installerArgs = if ($env:HERMES_RELAY_SETUP_SILENT -eq '1') { @('/S') } else { @() }",
|
||||
@@ -64,6 +68,7 @@ export function windowsInstallerLaunchPlan(
|
||||
env: {
|
||||
...process.env,
|
||||
HERMES_RELAY_SETUP_PATH: installerPath,
|
||||
HERMES_RELAY_SETUP_CALLER_PID: String(callerPid),
|
||||
HERMES_RELAY_SETUP_SILENT: silent ? '1' : '0',
|
||||
HERMES_RELAY_SETUP_RESTART_DAEMON: restartDaemon ? '1' : '0',
|
||||
HERMES_RELAY_SETUP_INSTALL_DIR: installDir,
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
|
||||
import { auditDetails, categorizeTool, resultExitCode, summarizeResult } from '../src/lib/auditLog.js'
|
||||
import { auditDetails, categorizeTool, persistAuditScreenshot, resultExitCode, summarizeResult } from '../src/lib/auditLog.js'
|
||||
import { toolResultSucceeded } from '../src/tools/router.js'
|
||||
|
||||
test('audit events classify the activity surfaces used by the tray', () => {
|
||||
assert.equal(categorizeTool('desktop_powershell'), 'command')
|
||||
@@ -31,3 +35,45 @@ test('audit events preserve process exit outcome separately from dispatch succes
|
||||
assert.equal(summarizeResult(result), 'exit 17')
|
||||
assert.equal(resultExitCode({ path: 'C:\\temp\\file.txt' }), undefined)
|
||||
})
|
||||
|
||||
test('computer-use audit redacts screenshots, UI trees, labels, and entered values', () => {
|
||||
const details = auditDetails(
|
||||
{ pid: 42, window_id: 7, value: 'top-secret-password' },
|
||||
{
|
||||
backend: 'cua_driver',
|
||||
target: { pid: 42, windowId: 7, title: 'Password Manager' },
|
||||
tree_markdown: 'Password: hunter2',
|
||||
screenshot_base64: 'sensitive-pixels',
|
||||
elements: [{ label: 'API token sk-live-secret' }]
|
||||
},
|
||||
{ redactComputerContent: true }
|
||||
)
|
||||
const serialized = JSON.stringify(details)
|
||||
assert.doesNotMatch(serialized, /hunter2|sensitive-pixels|sk-live-secret|top-secret-password|Password Manager/)
|
||||
assert.match(details.result_detail ?? '', /"redacted": true/)
|
||||
assert.match(details.result_detail ?? '', /"pid": 42/)
|
||||
})
|
||||
|
||||
test('semantic computer-control rejection is audited as failed without changing transport semantics', () => {
|
||||
assert.equal(toolResultSucceeded('desktop_computer_action', { ok: false, code: 'sensitive_target_blocked' }), false)
|
||||
assert.equal(toolResultSucceeded('desktop_computer_action', { ok: true }), true)
|
||||
assert.equal(toolResultSucceeded('desktop_read_file', { ok: false }), true)
|
||||
})
|
||||
|
||||
test('screenshot evidence is retained as a private opaque PNG instead of JSON content', async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), 'hermes-activity-evidence-'))
|
||||
const previous = process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR
|
||||
process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR = directory
|
||||
try {
|
||||
const png = Buffer.from('89504e470d0a1a0a00000000', 'hex').toString('base64')
|
||||
const evidence = await persistAuditScreenshot({ screenshot_base64: png, screenshot_width: 640, screenshot_height: 480 }, 'request-1', 7)
|
||||
assert.match(evidence.screenshot_evidence_id ?? '', /^[a-f0-9]{32}$/)
|
||||
assert.equal(evidence.screenshot_width, 640)
|
||||
assert.equal((await readFile(join(directory, `${evidence.screenshot_evidence_id}.png`))).subarray(0, 8).toString('hex'), '89504e470d0a1a0a')
|
||||
assert.equal(JSON.stringify(evidence).includes(png), false)
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR
|
||||
else process.env.HERMES_RELAY_ACTIVITY_EVIDENCE_DIR = previous
|
||||
await rm(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
|
||||
import {
|
||||
computerGrantAllowsTarget,
|
||||
configureComputerUseRuntime,
|
||||
getComputerGrantSummary,
|
||||
initializeComputerControlSession,
|
||||
cancelAllComputerGrants,
|
||||
requestComputerGrant,
|
||||
revokeComputerControlSession
|
||||
} from '../src/tools/computerGrants.js'
|
||||
import {
|
||||
ComputerControlSecurityState,
|
||||
evaluateSensitiveTarget,
|
||||
hasAuthenticatedControlIdentity,
|
||||
type ComputerControlAuthority
|
||||
} from '../src/tools/computerControlSecurity.js'
|
||||
|
||||
const authority = (id: string): ComputerControlAuthority => ({
|
||||
controlSessionId: id,
|
||||
hostUrl: 'wss://relay.example'
|
||||
})
|
||||
|
||||
test('control request ids are accepted once per router lifecycle', () => {
|
||||
const state = new ComputerControlSecurityState(authority('router-a'))
|
||||
assert.equal(state.bindRequest('request-1')?.requestId, 'request-1')
|
||||
assert.equal(state.bindRequest('request-1'), null)
|
||||
assert.equal(state.bindRequest(' '), null)
|
||||
})
|
||||
|
||||
test('snapshot tokens are opaque, one-use, and bound to grant and target', () => {
|
||||
const auth = authority('router-snapshot')
|
||||
const state = new ComputerControlSecurityState(auth)
|
||||
const token = state.issueSnapshotToken({
|
||||
authority: auth,
|
||||
grantId: 'grant-a',
|
||||
target: { pid: 42, windowId: 7, app: 'Notepad' },
|
||||
snapshotGeneration: 'generation-1',
|
||||
driverElementToken: 'driver-private-token'
|
||||
})
|
||||
assert.match(token, /^hermes-snapshot-/)
|
||||
assert.equal(state.consumeSnapshotToken(token, {
|
||||
authority: auth,
|
||||
grantId: 'grant-b',
|
||||
target: { pid: 42, windowId: 7 }
|
||||
}), null)
|
||||
assert.equal(state.consumeSnapshotToken(token, {
|
||||
authority: auth,
|
||||
grantId: 'grant-a',
|
||||
target: { pid: 42, windowId: 7 }
|
||||
}), null)
|
||||
})
|
||||
|
||||
test('snapshot tokens return their backend binding only to the exact target', () => {
|
||||
const auth = authority('router-target')
|
||||
const state = new ComputerControlSecurityState(auth)
|
||||
const token = state.issueSnapshotToken({
|
||||
authority: auth,
|
||||
grantId: null,
|
||||
target: { pid: 9, windowId: 11 },
|
||||
snapshotGeneration: 'g1',
|
||||
driverElementToken: 'opaque-driver-token'
|
||||
})
|
||||
const binding = state.consumeSnapshotToken(token, {
|
||||
authority: auth,
|
||||
grantId: null,
|
||||
target: { pid: 9, windowId: 11 },
|
||||
snapshotGeneration: 'g1'
|
||||
})
|
||||
assert.equal(binding?.driverElementToken, 'opaque-driver-token')
|
||||
assert.equal(state.consumeSnapshotToken(token, {
|
||||
authority: auth,
|
||||
grantId: null,
|
||||
target: { pid: 9, windowId: 11 }
|
||||
}), null)
|
||||
})
|
||||
|
||||
test('authority revocation removes only that concurrent session artifacts', () => {
|
||||
const first = authority('router-first')
|
||||
const second = authority('router-second')
|
||||
const state = new ComputerControlSecurityState(first)
|
||||
assert.ok(state.bindRequest('request-first', first))
|
||||
assert.ok(state.bindRequest('request-second', second))
|
||||
const firstToken = state.issueSnapshotToken({
|
||||
authority: first, grantId: null, target: { pid: 1, windowId: 1 }, snapshotGeneration: 'g1'
|
||||
})
|
||||
const secondToken = state.issueSnapshotToken({
|
||||
authority: second, grantId: null, target: { pid: 2, windowId: 2 }, snapshotGeneration: 'g2'
|
||||
})
|
||||
state.revokeAuthority(first.controlSessionId)
|
||||
assert.ok(state.bindRequest('request-first', first))
|
||||
assert.equal(state.bindRequest('request-second', second), null)
|
||||
assert.equal(state.consumeSnapshotToken(firstToken, { authority: first, grantId: null, target: { pid: 1, windowId: 1 } }), null)
|
||||
assert.ok(state.consumeSnapshotToken(secondToken, { authority: second, grantId: null, target: { pid: 2, windowId: 2 } }))
|
||||
})
|
||||
|
||||
test('sensitive target policy fails closed without identity and blocks baseline surfaces', () => {
|
||||
assert.deepEqual(evaluateSensitiveTarget({}), { allowed: false, reason: 'missing_target_identity' })
|
||||
assert.equal(evaluateSensitiveTarget({ app: 'Bitwarden' }).allowed, false)
|
||||
assert.equal(evaluateSensitiveTarget({ title: 'Windows Security settings' }).allowed, false)
|
||||
assert.deepEqual(evaluateSensitiveTarget({ app: 'Notepad', title: 'notes.txt' }), {
|
||||
allowed: true,
|
||||
reason: 'not_sensitive'
|
||||
})
|
||||
})
|
||||
|
||||
test('authenticated control identity requires relay-attested requester/run/target fields', () => {
|
||||
assert.equal(hasAuthenticatedControlIdentity(authority('local-only')), false)
|
||||
assert.equal(hasAuthenticatedControlIdentity({
|
||||
controlSessionId: 'control-1',
|
||||
relaySessionId: 'relay-1',
|
||||
requesterDeviceId: 'requester-1',
|
||||
runId: 'run-1',
|
||||
targetDeviceId: 'desktop-1'
|
||||
}), true)
|
||||
})
|
||||
|
||||
test('router control sessions inherit host policy but keep grants isolated', () => {
|
||||
configureComputerUseRuntime({
|
||||
url: 'wss://relay.example',
|
||||
computerUseConsented: true,
|
||||
consentSource: 'stored',
|
||||
accessMode: 'ask',
|
||||
capabilities: {
|
||||
commands: 'ask', files: 'ask', screen_input: 'ask', usb: 'ask', microphone: 'disabled', camera: 'disabled'
|
||||
}
|
||||
})
|
||||
const first = authority('router-first')
|
||||
const second = authority('router-second')
|
||||
initializeComputerControlSession(first)
|
||||
initializeComputerControlSession(second)
|
||||
requestComputerGrant({
|
||||
mode: 'control',
|
||||
scope: { app: 'Notepad', display: 'primary', folder: 'C:\\work' },
|
||||
duration_seconds: 60,
|
||||
reason: 'edit notes'
|
||||
}, first)
|
||||
assert.equal(getComputerGrantSummary(first).active, true)
|
||||
assert.equal(getComputerGrantSummary(second).active, false)
|
||||
assert.equal(computerGrantAllowsTarget(first, {
|
||||
pid: 1,
|
||||
windowId: 2,
|
||||
app: 'Windows Notepad',
|
||||
display: 'primary',
|
||||
folder: 'C:\\work\\notes'
|
||||
}), true)
|
||||
assert.equal(computerGrantAllowsTarget(first, {
|
||||
pid: 1,
|
||||
windowId: 2,
|
||||
app: 'Browser',
|
||||
display: 'primary',
|
||||
folder: 'C:\\work\\notes'
|
||||
}), false)
|
||||
revokeComputerControlSession(first)
|
||||
revokeComputerControlSession(second)
|
||||
})
|
||||
|
||||
test('Full Access configured before router creation is preserved without sharing grants', () => {
|
||||
configureComputerUseRuntime({
|
||||
computerUseConsented: true,
|
||||
accessMode: 'full_access'
|
||||
})
|
||||
const full = authority('router-full')
|
||||
initializeComputerControlSession(full)
|
||||
assert.equal(getComputerGrantSummary(full).mode, 'full_access')
|
||||
revokeComputerControlSession(full)
|
||||
configureComputerUseRuntime({ accessMode: 'ask' })
|
||||
})
|
||||
|
||||
test('local emergency cancellation revokes every active control session', () => {
|
||||
configureComputerUseRuntime({
|
||||
computerUseConsented: true,
|
||||
accessMode: 'ask',
|
||||
capabilities: {
|
||||
commands: 'ask', files: 'ask', screen_input: 'ask', usb: 'ask', microphone: 'disabled', camera: 'disabled'
|
||||
}
|
||||
})
|
||||
const first = authority('emergency-first')
|
||||
const second = authority('emergency-second')
|
||||
initializeComputerControlSession(first)
|
||||
initializeComputerControlSession(second)
|
||||
requestComputerGrant({ mode: 'control', reason: 'first' }, first)
|
||||
requestComputerGrant({ mode: 'observe', reason: 'second' }, second)
|
||||
assert.equal(cancelAllComputerGrants('emergency stop'), 2)
|
||||
assert.equal(getComputerGrantSummary(first).active, false)
|
||||
assert.equal(getComputerGrantSummary(second).active, false)
|
||||
revokeComputerControlSession(first)
|
||||
revokeComputerControlSession(second)
|
||||
})
|
||||
@@ -0,0 +1,260 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import test from 'node:test'
|
||||
|
||||
import { ComputerControlSecurityState, type ComputerControlAuthority } from '../src/tools/computerControlSecurity.js'
|
||||
import {
|
||||
cancelComputerGrant,
|
||||
configureComputerUseRuntime,
|
||||
requestComputerGrant
|
||||
} from '../src/tools/computerGrants.js'
|
||||
import {
|
||||
setCuaControlSessionFactoryForTests,
|
||||
type CuaControlSession,
|
||||
type CuaControlSessionIdentity
|
||||
} from '../src/tools/cuaDriver.js'
|
||||
import {
|
||||
computerActionHandler,
|
||||
computerCancelHandler,
|
||||
computerScreenshotHandler
|
||||
} from '../src/tools/handlers/computer.js'
|
||||
import type { ToolContext } from '../src/tools/router.js'
|
||||
|
||||
interface FakeCall {
|
||||
name: string
|
||||
args?: unknown
|
||||
}
|
||||
|
||||
class FakeCuaSession {
|
||||
readonly calls: FakeCall[] = []
|
||||
closed = false
|
||||
snapshotNumber = 0
|
||||
appName = 'Calculator'
|
||||
|
||||
async listWindows(pid?: number): Promise<Record<string, unknown>> {
|
||||
this.calls.push({ name: 'listWindows', args: pid })
|
||||
return {
|
||||
windows: [{
|
||||
window_id: 200,
|
||||
app_name: this.appName,
|
||||
title: this.appName,
|
||||
executable: 'C:\\WindowsApps\\CalculatorApp.exe'
|
||||
}]
|
||||
}
|
||||
}
|
||||
|
||||
async snapshot(args: unknown): Promise<Record<string, unknown>> {
|
||||
this.snapshotNumber += 1
|
||||
this.calls.push({ name: 'snapshot', args })
|
||||
return {
|
||||
snapshot_id: `s0000000${this.snapshotNumber}`,
|
||||
elements: [{ element_index: 7, element_token: 'e1234abcd', role: 'button', label: 'Seven' }],
|
||||
tree_markdown: '[7] button Seven'
|
||||
}
|
||||
}
|
||||
|
||||
async clickElement(args: unknown): Promise<Record<string, unknown>> {
|
||||
this.calls.push({ name: 'clickElement', args })
|
||||
return { clicked: true }
|
||||
}
|
||||
|
||||
async setElementValue(args: unknown, value: string): Promise<Record<string, unknown>> {
|
||||
this.calls.push({ name: 'setElementValue', args: { args, value } })
|
||||
return { changed: true }
|
||||
}
|
||||
|
||||
async pressKey(args: unknown, key: string): Promise<Record<string, unknown>> {
|
||||
this.calls.push({ name: 'pressKey', args: { args, key } })
|
||||
return { pressed: true }
|
||||
}
|
||||
|
||||
async scroll(args: unknown, direction: string, amount: number): Promise<Record<string, unknown>> {
|
||||
this.calls.push({ name: 'scroll', args: { args, direction, amount } })
|
||||
return { scrolled: true }
|
||||
}
|
||||
|
||||
async close(reason?: string): Promise<void> {
|
||||
this.closed = true
|
||||
this.calls.push({ name: 'close', args: reason })
|
||||
}
|
||||
}
|
||||
|
||||
async function fixture(): Promise<{
|
||||
authority: ComputerControlAuthority
|
||||
ctx: ToolContext
|
||||
session: FakeCuaSession
|
||||
cleanup(): Promise<void>
|
||||
}> {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'hermes-cua-handlers-'))
|
||||
const settingsPath = join(dir, 'desktop-settings.json')
|
||||
await writeFile(settingsPath, JSON.stringify({
|
||||
computer_use_enabled: true,
|
||||
computer_control_engine: 'cua',
|
||||
cua_cursor_enabled: true,
|
||||
cua_foreground_escalation_enabled: false
|
||||
}))
|
||||
process.env.HERMES_RELAY_DESKTOP_SETTINGS_PATH = settingsPath
|
||||
const authority: ComputerControlAuthority = {
|
||||
controlSessionId: 'control-handler-test',
|
||||
requestId: 'request-handler-test',
|
||||
relaySessionId: 'relay-session',
|
||||
requesterDeviceId: 'requester-device',
|
||||
runId: 'run-1',
|
||||
targetDeviceId: 'desktop-target'
|
||||
}
|
||||
const session = new FakeCuaSession()
|
||||
setCuaControlSessionFactoryForTests(async (_identity: CuaControlSessionIdentity) => session as unknown as CuaControlSession)
|
||||
configureComputerUseRuntime({
|
||||
url: 'wss://relay.example.test',
|
||||
computerUseConsented: true,
|
||||
consentSource: 'stored',
|
||||
accessMode: 'ask'
|
||||
}, authority)
|
||||
requestComputerGrant({ mode: 'control', scope: { app: 'Calculator' }, duration_seconds: 60 }, authority)
|
||||
const ctx: ToolContext = {
|
||||
cwd: dir,
|
||||
abortSignal: new AbortController().signal,
|
||||
interactive: false,
|
||||
controlSession: authority,
|
||||
controlSecurity: new ComputerControlSecurityState(authority)
|
||||
}
|
||||
return {
|
||||
authority,
|
||||
ctx,
|
||||
session,
|
||||
cleanup: async () => {
|
||||
cancelComputerGrant('test cleanup', authority)
|
||||
setCuaControlSessionFactoryForTests(null)
|
||||
delete process.env.HERMES_RELAY_DESKTOP_SETTINGS_PATH
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
test('CUA handlers issue a Hermes token, execute once, and verify with a fresh snapshot', async () => {
|
||||
const item = await fixture()
|
||||
try {
|
||||
const observed = await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx) as {
|
||||
ok: boolean
|
||||
backend: string
|
||||
elements: Array<{ snapshot_token: string; element_token?: string }>
|
||||
}
|
||||
assert.equal(observed.ok, true)
|
||||
assert.equal(observed.backend, 'cua_driver')
|
||||
assert.equal(observed.elements[0]!.element_token, undefined)
|
||||
assert.match(observed.elements[0]!.snapshot_token, /^hermes-snapshot-/)
|
||||
|
||||
const acted = await computerActionHandler({
|
||||
action: 'click_element',
|
||||
pid: 100,
|
||||
window_id: 200,
|
||||
snapshot_token: observed.elements[0]!.snapshot_token,
|
||||
snapshot_generation: 's00000001'
|
||||
}, item.ctx) as { ok: boolean; backend: string; verification_snapshot: { snapshot_id: string } }
|
||||
assert.equal(acted.ok, true)
|
||||
assert.equal(acted.backend, 'cua')
|
||||
assert.equal(acted.verification_snapshot.snapshot_id, 's00000002')
|
||||
assert.deepEqual(item.session.calls.map(call => call.name), [
|
||||
'listWindows', 'snapshot', 'listWindows', 'clickElement', 'snapshot'
|
||||
])
|
||||
|
||||
const replayed = await computerActionHandler({
|
||||
action: 'click_element',
|
||||
pid: 100,
|
||||
window_id: 200,
|
||||
snapshot_token: observed.elements[0]!.snapshot_token
|
||||
}, item.ctx) as { ok: boolean; code: string }
|
||||
assert.equal(replayed.ok, false)
|
||||
assert.equal(replayed.code, 'invalid_or_stale_snapshot')
|
||||
assert.equal(item.session.calls.filter(call => call.name === 'clickElement').length, 1)
|
||||
} finally {
|
||||
await item.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('CUA handlers reject unauthenticated and mismatched targets before input', async () => {
|
||||
const item = await fixture()
|
||||
try {
|
||||
const unauthenticatedAuthority = { controlSessionId: 'legacy-only' }
|
||||
requestComputerGrant({ mode: 'observe', duration_seconds: 60 }, unauthenticatedAuthority)
|
||||
const unauthenticated = await computerScreenshotHandler(
|
||||
{ pid: 100, window_id: 200 },
|
||||
{ ...item.ctx, controlSession: unauthenticatedAuthority }
|
||||
) as { ok: boolean; code: string }
|
||||
assert.equal(unauthenticated.ok, false)
|
||||
assert.equal(unauthenticated.code, 'authenticated_control_session_required')
|
||||
cancelComputerGrant('test cleanup', unauthenticatedAuthority)
|
||||
|
||||
const missing = await computerScreenshotHandler({ pid: 100, window_id: 201 }, item.ctx) as {
|
||||
ok: boolean
|
||||
code: string
|
||||
}
|
||||
assert.equal(missing.ok, false)
|
||||
assert.equal(missing.code, 'target_not_found')
|
||||
|
||||
requestComputerGrant({ mode: 'control', scope: { app: 'Notepad' }, duration_seconds: 60 }, item.authority)
|
||||
const wrongGrant = await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx) as {
|
||||
ok: boolean
|
||||
code: string
|
||||
}
|
||||
assert.equal(wrongGrant.ok, false)
|
||||
assert.equal(wrongGrant.code, 'grant_target_mismatch')
|
||||
|
||||
item.session.appName = 'Windows Security'
|
||||
configureComputerUseRuntime({
|
||||
url: 'wss://relay.example.test',
|
||||
computerUseConsented: true,
|
||||
consentSource: 'stored',
|
||||
accessMode: 'full_access'
|
||||
}, item.authority)
|
||||
const sensitive = await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx) as {
|
||||
ok: boolean
|
||||
code: string
|
||||
}
|
||||
assert.equal(sensitive.ok, false)
|
||||
assert.equal(sensitive.code, 'sensitive_target_blocked')
|
||||
assert.equal(item.session.calls.some(call => call.name === 'clickElement'), false)
|
||||
} finally {
|
||||
await item.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('desktop computer cancel closes the matching CUA session', async () => {
|
||||
const item = await fixture()
|
||||
try {
|
||||
await computerScreenshotHandler({ pid: 100, window_id: 200 }, item.ctx)
|
||||
const result = await computerCancelHandler({ reason: 'operator stop' }, item.ctx) as { ok: boolean }
|
||||
assert.equal(result.ok, true)
|
||||
assert.equal(item.session.closed, true)
|
||||
assert.deepEqual(item.session.calls.at(-1), { name: 'close', args: 'operator stop' })
|
||||
} finally {
|
||||
await item.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('selected CUA retains system display observation but never falls back to legacy input', async () => {
|
||||
const item = await fixture()
|
||||
try {
|
||||
if (process.platform === 'win32') {
|
||||
const screenshot = await computerScreenshotHandler({ display: 'primary' }, item.ctx) as {
|
||||
ok: boolean
|
||||
backend: string
|
||||
}
|
||||
assert.equal(screenshot.ok, true)
|
||||
assert.equal(screenshot.backend, 'system_capture')
|
||||
}
|
||||
|
||||
const coordinate = await computerActionHandler({
|
||||
action: 'left_click',
|
||||
x: 100,
|
||||
y: 200
|
||||
}, item.ctx) as { ok: boolean; code: string }
|
||||
assert.equal(coordinate.ok, false)
|
||||
assert.equal(coordinate.code, 'cua_structured_action_required')
|
||||
assert.equal(item.session.calls.some(call => call.name === 'clickElement'), false)
|
||||
} finally {
|
||||
await item.cleanup()
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,167 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
|
||||
import { desktopDeviceId } from '../src/deviceIdentity.js'
|
||||
import type { RelayTransport } from '../src/transport/RelayTransport.js'
|
||||
import { configureComputerUseRuntime, getComputerGrantSummary, requestComputerGrant } from '../src/tools/computerGrants.js'
|
||||
import {
|
||||
DesktopToolRouter,
|
||||
type ToolContext
|
||||
} from '../src/tools/router.js'
|
||||
|
||||
type ChannelListener = (type: string, payload: Record<string, unknown>) => void
|
||||
|
||||
function harness(handler: (ctx: ToolContext) => void) {
|
||||
let listener: ChannelListener | null = null
|
||||
const sent: Array<{ channel: string; type: string; payload: Record<string, unknown> }> = []
|
||||
const relay = {
|
||||
onChannel(channel: string, next: ChannelListener | null) {
|
||||
assert.equal(channel, 'desktop')
|
||||
listener = next
|
||||
},
|
||||
sendChannel(channel: string, type: string, payload: Record<string, unknown>) {
|
||||
sent.push({ channel, type, payload })
|
||||
}
|
||||
} as unknown as RelayTransport
|
||||
const router = new DesktopToolRouter({
|
||||
consentGranted: true,
|
||||
interactive: false,
|
||||
hostUrl: 'wss://relay.example',
|
||||
handlers: {
|
||||
desktop_computer_snapshot: async (_args, ctx) => {
|
||||
handler(ctx)
|
||||
return { ok: true }
|
||||
}
|
||||
}
|
||||
})
|
||||
router.attach(relay)
|
||||
return {
|
||||
router,
|
||||
sent,
|
||||
emit(type: string, payload: Record<string, unknown>) {
|
||||
assert.ok(listener)
|
||||
listener(type, payload)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function nextTurn(): Promise<void> {
|
||||
return new Promise(resolve => setImmediate(resolve))
|
||||
}
|
||||
|
||||
test('router passes a strictly validated server control session to handlers', async () => {
|
||||
let received: ToolContext | undefined
|
||||
const testHarness = harness(ctx => { received = ctx })
|
||||
const requestId = 'request-1'
|
||||
testHarness.emit('desktop.command', {
|
||||
request_id: requestId,
|
||||
tool: 'desktop_computer_snapshot',
|
||||
args: {},
|
||||
control_session: {
|
||||
version: 1,
|
||||
id: 'control-1',
|
||||
request_id: requestId,
|
||||
requester_device_id: 'paired-agent-1',
|
||||
target_device_id: desktopDeviceId(),
|
||||
chat_session_id: 'chat-1',
|
||||
run_id: 'run-1'
|
||||
}
|
||||
})
|
||||
await nextTurn()
|
||||
|
||||
assert.equal(received?.controlSession?.controlSessionId, 'control-1')
|
||||
assert.equal(received?.controlSession?.relaySessionId, 'control-1')
|
||||
assert.equal(received?.controlSession?.requestId, requestId)
|
||||
assert.equal(received?.controlSession?.requesterDeviceId, 'paired-agent-1')
|
||||
assert.equal(received?.controlSession?.targetDeviceId, desktopDeviceId())
|
||||
assert.equal(received?.controlSession?.runId, 'run-1')
|
||||
assert.equal(received?.controlSession?.chatSessionId, 'chat-1')
|
||||
assert.ok(testHarness.sent.some(item =>
|
||||
item.type === 'desktop.response' && item.payload.request_id === requestId && item.payload.ok === true
|
||||
))
|
||||
testHarness.router.detach()
|
||||
})
|
||||
|
||||
test('router rejects a supplied control session whose request binding is invalid', async () => {
|
||||
let invoked = false
|
||||
const testHarness = harness(() => { invoked = true })
|
||||
testHarness.emit('desktop.command', {
|
||||
request_id: 'request-outer',
|
||||
tool: 'desktop_computer_snapshot',
|
||||
args: {},
|
||||
control_session: {
|
||||
version: 1,
|
||||
id: 'control-1',
|
||||
request_id: 'request-other',
|
||||
requester_device_id: 'paired-agent-1',
|
||||
target_device_id: desktopDeviceId(),
|
||||
run_id: 'run-1'
|
||||
}
|
||||
})
|
||||
await nextTurn()
|
||||
|
||||
assert.equal(invoked, false)
|
||||
const response = testHarness.sent.find(item =>
|
||||
item.type === 'desktop.response' && item.payload.request_id === 'request-outer'
|
||||
)
|
||||
assert.equal(response?.payload.ok, false)
|
||||
assert.equal(response?.payload.error, 'invalid server control_session binding')
|
||||
testHarness.router.detach()
|
||||
})
|
||||
|
||||
test('router preserves legacy computer commands when the server omits identity', async () => {
|
||||
let received: ToolContext | undefined
|
||||
const testHarness = harness(ctx => { received = ctx })
|
||||
testHarness.emit('desktop.command', {
|
||||
request_id: 'legacy-request',
|
||||
tool: 'desktop_computer_snapshot',
|
||||
args: {}
|
||||
})
|
||||
await nextTurn()
|
||||
|
||||
assert.match(received?.controlSession?.controlSessionId ?? '', /^router-/)
|
||||
assert.equal(received?.controlSession?.relaySessionId, undefined)
|
||||
testHarness.router.detach()
|
||||
})
|
||||
|
||||
test('relay control-session end revokes only the exact local target authority', async () => {
|
||||
let received: ToolContext | undefined
|
||||
const testHarness = harness(ctx => {
|
||||
received = ctx
|
||||
configureComputerUseRuntime({ computerUseConsented: true, accessMode: 'ask' }, ctx.controlSession)
|
||||
requestComputerGrant({ mode: 'control', duration_seconds: 60 }, ctx.controlSession)
|
||||
})
|
||||
const requestId = 'request-ending'
|
||||
testHarness.emit('desktop.command', {
|
||||
request_id: requestId,
|
||||
tool: 'desktop_computer_snapshot',
|
||||
args: {},
|
||||
control_session: {
|
||||
version: 1,
|
||||
id: 'control-ending',
|
||||
request_id: requestId,
|
||||
requester_device_id: 'paired-agent-1',
|
||||
target_device_id: desktopDeviceId(),
|
||||
run_id: 'run-ending'
|
||||
}
|
||||
})
|
||||
await nextTurn()
|
||||
assert.equal(getComputerGrantSummary(received?.controlSession).active, true)
|
||||
|
||||
testHarness.emit('desktop.control_session_end', {
|
||||
version: 1,
|
||||
id: 'control-ending',
|
||||
target_device_id: 'another-desktop',
|
||||
reason: 'wrong target'
|
||||
})
|
||||
assert.equal(getComputerGrantSummary(received?.controlSession).active, true)
|
||||
|
||||
testHarness.emit('desktop.control_session_end', {
|
||||
version: 1,
|
||||
id: 'control-ending',
|
||||
target_device_id: desktopDeviceId(),
|
||||
reason: 'run ended'
|
||||
})
|
||||
assert.equal(getComputerGrantSummary(received?.controlSession).active, false)
|
||||
testHarness.router.detach()
|
||||
})
|
||||
@@ -0,0 +1,201 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdtemp, mkdir, rm, symlink, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import test from 'node:test'
|
||||
|
||||
import {
|
||||
CuaDriverAdapter,
|
||||
CuaRuntimeError,
|
||||
closeAllCuaControlSessions,
|
||||
getCuaControlSession,
|
||||
selectComputerControlBackend,
|
||||
setCuaControlSessionFactoryForTests,
|
||||
type CuaControlSession,
|
||||
type CuaProcessResult,
|
||||
type CuaProcessRunner
|
||||
} from '../src/tools/cuaDriver.js'
|
||||
|
||||
const REQUIRED_TOOL_LINES = [
|
||||
'health_report', 'start_session', 'end_session', 'list_windows', 'get_window_state',
|
||||
'click', 'set_value', 'press_key', 'scroll'
|
||||
].map(name => `${name}: test tool`).join('\n')
|
||||
|
||||
class FakeRunner implements CuaProcessRunner {
|
||||
readonly calls: Array<{ executable: string; args: readonly string[]; stdin?: string }> = []
|
||||
|
||||
constructor(
|
||||
private readonly binaryPath: string,
|
||||
private readonly health = 'ok',
|
||||
private readonly version = '0.19.3',
|
||||
private readonly permissionMode = 'standard'
|
||||
) {}
|
||||
|
||||
async run(executable: string, args: readonly string[], options = {}): Promise<CuaProcessResult> {
|
||||
this.calls.push({ executable, args, stdin: options.stdin })
|
||||
const command = args.join(' ')
|
||||
if (command === '--version') return ok(`cua-driver ${this.version}`)
|
||||
if (command === 'manifest --pretty') {
|
||||
return ok(JSON.stringify({ schema_version: '1', binary_version: this.version, binary_path: this.binaryPath }))
|
||||
}
|
||||
if (command === 'list-tools') return ok(REQUIRED_TOOL_LINES)
|
||||
if (command === 'status') return ok(`Cua Driver daemon is running\n permission mode: ${this.permissionMode} (test)`)
|
||||
if (command === 'call health_report') {
|
||||
return ok(JSON.stringify({ schema_version: '1', driver_version: this.version, overall: this.health }))
|
||||
}
|
||||
return ok(JSON.stringify({ ok: true, tool: args[1] }))
|
||||
}
|
||||
}
|
||||
|
||||
function ok(stdout: string): CuaProcessResult {
|
||||
return { stdout, stderr: '', exitCode: 0 }
|
||||
}
|
||||
|
||||
async function fakeInstall(): Promise<{ home: string; binary: string; cleanup(): Promise<void> }> {
|
||||
const home = await mkdtemp(join(tmpdir(), 'hermes-cua-'))
|
||||
const releases = join(home, '.cua-driver', 'packages', 'releases')
|
||||
const release = join(releases, '0.19.3-x86_64-pc-windows-msvc')
|
||||
const current = join(home, '.cua-driver', 'packages', 'current')
|
||||
await mkdir(release, { recursive: true })
|
||||
const binary = join(release, 'cua-driver.exe')
|
||||
await writeFile(binary, '')
|
||||
await symlink(release, current, 'junction')
|
||||
return { home, binary, cleanup: () => rm(home, { recursive: true, force: true }) }
|
||||
}
|
||||
|
||||
test('discovers only the canonical package/current executable and negotiates readiness', async () => {
|
||||
const install = await fakeInstall()
|
||||
try {
|
||||
const runner = new FakeRunner(install.binary)
|
||||
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner })
|
||||
assert.equal(adapter.binaryPath, install.binary)
|
||||
assert.equal(adapter.binaryVersion, '0.19.3')
|
||||
assert.equal(adapter.permissionMode, 'standard')
|
||||
assert.equal(runner.calls[0]?.executable, install.binary)
|
||||
assert.deepEqual(runner.calls.map(call => call.args.join(' ')).slice(0, 4), [
|
||||
'--version', 'manifest --pretty', 'list-tools', 'status'
|
||||
])
|
||||
assert.equal(runner.calls.some(call => call.args.join(' ') === 'call health_report'), false)
|
||||
} finally {
|
||||
await install.cleanup()
|
||||
}
|
||||
})
|
||||
test('keeps runtime ready when global health is degraded but still rejects unrestricted permission mode', async () => {
|
||||
const install = await fakeInstall()
|
||||
try {
|
||||
const adapter = await CuaDriverAdapter.connect({
|
||||
platform: 'win32', homeDir: install.home, runner: new FakeRunner(install.binary, 'degraded')
|
||||
})
|
||||
assert.equal(adapter.binaryVersion, '0.19.3')
|
||||
await assert.rejects(
|
||||
CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner: new FakeRunner(install.binary, 'ok', '0.19.3', 'unrestricted') }),
|
||||
(error: unknown) => error instanceof CuaRuntimeError && error.code === 'incompatible'
|
||||
)
|
||||
} finally {
|
||||
await install.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('explicit health recheck reports degradation without changing runtime readiness', async () => {
|
||||
const install = await fakeInstall()
|
||||
try {
|
||||
const runner = new FakeRunner(install.binary, 'degraded')
|
||||
const health = await CuaDriverAdapter.healthStatus({ platform: 'win32', homeDir: install.home, runner })
|
||||
assert.equal(health.state, 'degraded')
|
||||
assert.equal(health.overall, 'degraded')
|
||||
assert.equal(health.temporaryWindowsCompatibility, true)
|
||||
assert.equal(runner.calls.filter(call => call.args.join(' ') === 'call health_report').length, 1)
|
||||
const runtime = await CuaDriverAdapter.status({ platform: 'win32', homeDir: install.home, runner })
|
||||
assert.equal(runtime.ready, true)
|
||||
assert.equal(runtime.health, 'not_checked')
|
||||
} finally {
|
||||
await install.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('uses a locally derived session and exposes only typed background actions', async () => {
|
||||
const install = await fakeInstall()
|
||||
try {
|
||||
const runner = new FakeRunner(install.binary)
|
||||
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner })
|
||||
const session = await adapter.openSession({ controlSessionId: 'control-1', targetDeviceId: 'desktop-1', runId: 'run-1' })
|
||||
await session.snapshot({ pid: 123, windowId: 456, includeScreenshot: false })
|
||||
await session.clickElement({ pid: 123, windowId: 456, elementToken: 'e123abc' })
|
||||
await session.pressKey({ pid: 123, windowId: 456 }, 'escape')
|
||||
await session.close()
|
||||
|
||||
const invocations = runner.calls.filter(call => call.args[0] === 'call' && call.args[1] !== 'health_report')
|
||||
assert.deepEqual(invocations.map(call => call.args[1]), [
|
||||
'start_session', 'get_window_state', 'click', 'press_key', 'end_session'
|
||||
])
|
||||
const start = JSON.parse(invocations[0]!.stdin!) as { session: string; capture_scope: string }
|
||||
assert.match(start.session, /^hermes-[0-9a-f]{32}$/)
|
||||
assert.equal(start.capture_scope, 'window')
|
||||
const click = JSON.parse(invocations[2]!.stdin!) as { session: string; delivery_mode: string; scope: string }
|
||||
assert.equal(click.session, start.session)
|
||||
assert.equal(click.delivery_mode, 'background')
|
||||
assert.equal(click.scope, 'window')
|
||||
await assert.rejects(session.listWindows(), /closed/)
|
||||
} finally {
|
||||
await install.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('rejects invalid tokens and keys before invoking the driver', async () => {
|
||||
const install = await fakeInstall()
|
||||
try {
|
||||
const runner = new FakeRunner(install.binary)
|
||||
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: install.home, runner })
|
||||
const session = await adapter.openSession({ controlSessionId: 'control-2', targetDeviceId: 'desktop-1' })
|
||||
const before = runner.calls.length
|
||||
await assert.rejects(session.clickElement({ pid: 1, windowId: 2, elementToken: '../bad' }), /elementToken/)
|
||||
await assert.rejects(session.pressKey({ pid: 1, windowId: 2 }, 'win+r'), /allowlisted/)
|
||||
assert.equal(runner.calls.length, before)
|
||||
await session.close()
|
||||
} finally {
|
||||
await install.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('shared control sessions reject identity changes under the same authority id', async () => {
|
||||
const fake = { close: async () => undefined } as unknown as CuaControlSession
|
||||
setCuaControlSessionFactoryForTests(async () => fake)
|
||||
try {
|
||||
await getCuaControlSession({
|
||||
controlSessionId: 'control-bound',
|
||||
targetDeviceId: 'desktop-1',
|
||||
relaySessionId: 'relay-1',
|
||||
requesterDeviceId: 'agent-1',
|
||||
runId: 'run-1'
|
||||
})
|
||||
await assert.rejects(getCuaControlSession({
|
||||
controlSessionId: 'control-bound',
|
||||
targetDeviceId: 'desktop-1',
|
||||
relaySessionId: 'relay-1',
|
||||
requesterDeviceId: 'agent-2',
|
||||
runId: 'run-1'
|
||||
}), /identity changed/)
|
||||
} finally {
|
||||
await closeAllCuaControlSessions('test cleanup')
|
||||
setCuaControlSessionFactoryForTests(null)
|
||||
}
|
||||
})
|
||||
|
||||
test('control backend selection is immutable for the active session', async () => {
|
||||
const fake = { close: async () => undefined } as unknown as CuaControlSession
|
||||
setCuaControlSessionFactoryForTests(async () => fake)
|
||||
try {
|
||||
const primary = await selectComputerControlBackend('backend-cua', 'cua', true)
|
||||
const attemptedDowngrade = await selectComputerControlBackend('backend-cua', 'legacy', false)
|
||||
assert.equal(primary.backend, 'cua')
|
||||
assert.deepEqual(attemptedDowngrade, primary)
|
||||
|
||||
const compatibility = await selectComputerControlBackend('backend-legacy', 'legacy', false)
|
||||
const attemptedUpgrade = await selectComputerControlBackend('backend-legacy', 'cua', true)
|
||||
assert.equal(compatibility.backend, 'legacy_compat')
|
||||
assert.deepEqual(attemptedUpgrade, compatibility)
|
||||
} finally {
|
||||
await closeAllCuaControlSessions('test cleanup')
|
||||
setCuaControlSessionFactoryForTests(null)
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,149 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdir, mkdtemp, rm, symlink, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import test from 'node:test'
|
||||
|
||||
import {
|
||||
CuaDriverAdapter,
|
||||
CuaRuntimeError,
|
||||
type CuaProcessResult,
|
||||
type CuaProcessRunner
|
||||
} from '../src/tools/cuaDriver.js'
|
||||
|
||||
const tools = [
|
||||
'health_report', 'start_session', 'end_session', 'list_windows', 'get_window_state',
|
||||
'click', 'set_value', 'press_key', 'scroll'
|
||||
]
|
||||
|
||||
class StatefulFakeCua implements CuaProcessRunner {
|
||||
readonly calls: Array<{ args: readonly string[]; payload: Record<string, unknown> | null; signal?: AbortSignal }> = []
|
||||
rejectClicksAsStale = false
|
||||
|
||||
constructor(private readonly binary: string) {}
|
||||
|
||||
async run(
|
||||
_executable: string,
|
||||
args: readonly string[],
|
||||
options: { stdin?: string; signal?: AbortSignal } = {}
|
||||
): Promise<CuaProcessResult> {
|
||||
const payload = options.stdin ? JSON.parse(options.stdin) as Record<string, unknown> : null
|
||||
this.calls.push({ args: [...args], payload, signal: options.signal })
|
||||
if (options.signal?.aborted) throw new CuaRuntimeError('fake action cancelled', 'transport')
|
||||
const command = args.join(' ')
|
||||
if (command === '--version') return ok('cua-driver 0.19.3')
|
||||
if (command === 'manifest --pretty') {
|
||||
return ok(JSON.stringify({ schema_version: '1', binary_version: '0.19.3', binary_path: this.binary }))
|
||||
}
|
||||
if (command === 'list-tools') return ok(tools.map(tool => `${tool}: fake`).join('\n'))
|
||||
if (command === 'status') return ok('permission mode: bounded')
|
||||
if (command === 'call health_report') {
|
||||
return ok(JSON.stringify({ schema_version: '1', driver_version: '0.19.3', overall: 'ok' }))
|
||||
}
|
||||
if (command === 'call get_window_state') {
|
||||
return ok(JSON.stringify({
|
||||
snapshot_id: 's1234abcd',
|
||||
elements: [{ element_index: 7, element_token: 'e1234abcd', role: 'button', label: 'Seven' }]
|
||||
}))
|
||||
}
|
||||
if (command === 'call click' && this.rejectClicksAsStale) {
|
||||
return ok(JSON.stringify({ isError: true, code: 'stale_element_token' }))
|
||||
}
|
||||
return ok(JSON.stringify({ ok: true }))
|
||||
}
|
||||
}
|
||||
|
||||
function ok(stdout: string): CuaProcessResult {
|
||||
return { stdout, stderr: '', exitCode: 0 }
|
||||
}
|
||||
|
||||
async function harness(): Promise<{
|
||||
adapter: CuaDriverAdapter
|
||||
runner: StatefulFakeCua
|
||||
cleanup(): Promise<void>
|
||||
}> {
|
||||
const home = await mkdtemp(join(tmpdir(), 'hermes-cua-integration-'))
|
||||
const release = join(home, '.cua-driver', 'packages', 'releases', '0.19.3-test')
|
||||
await mkdir(release, { recursive: true })
|
||||
const binary = join(release, 'cua-driver.exe')
|
||||
await writeFile(binary, '')
|
||||
await symlink(release, join(home, '.cua-driver', 'packages', 'current'), 'junction')
|
||||
const runner = new StatefulFakeCua(binary)
|
||||
const adapter = await CuaDriverAdapter.connect({ platform: 'win32', homeDir: home, runner })
|
||||
return { adapter, runner, cleanup: () => rm(home, { recursive: true, force: true }) }
|
||||
}
|
||||
|
||||
function identity(id: string) {
|
||||
return { controlSessionId: id, targetDeviceId: 'desktop-target', runId: 'run-1' }
|
||||
}
|
||||
|
||||
test('semantic CUA action can be bracketed by fresh snapshots without foreground dispatch', async () => {
|
||||
const { adapter, runner, cleanup } = await harness()
|
||||
try {
|
||||
const session = await adapter.openSession(identity('control-snapshot'))
|
||||
const before = await session.snapshot({ pid: 100, windowId: 200, includeScreenshot: true })
|
||||
const token = (before.elements as Array<{ element_token: string }>)[0]!.element_token
|
||||
await session.clickElement({ pid: 100, windowId: 200, elementToken: token })
|
||||
await session.snapshot({ pid: 100, windowId: 200, includeScreenshot: false })
|
||||
await session.close()
|
||||
const calls = runner.calls.filter(call => call.args[0] === 'call' && call.args[1] !== 'health_report')
|
||||
assert.deepEqual(calls.map(call => call.args[1]), [
|
||||
'start_session', 'get_window_state', 'click', 'get_window_state', 'end_session'
|
||||
])
|
||||
assert.deepEqual(calls[2]!.payload, {
|
||||
pid: 100,
|
||||
window_id: 200,
|
||||
element_token: 'e1234abcd',
|
||||
session: calls[0]!.payload!.session,
|
||||
scope: 'window',
|
||||
delivery_mode: 'background'
|
||||
})
|
||||
} finally {
|
||||
await cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('stale driver token errors and cancellation both fail closed', async () => {
|
||||
const { adapter, runner, cleanup } = await harness()
|
||||
try {
|
||||
const session = await adapter.openSession(identity('control-failure'))
|
||||
runner.rejectClicksAsStale = true
|
||||
await assert.rejects(
|
||||
session.clickElement({ pid: 100, windowId: 200, elementToken: 'edeadbeef' }),
|
||||
/rejected the action/
|
||||
)
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
await assert.rejects(
|
||||
session.pressKey({ pid: 100, windowId: 200 }, 'escape', controller.signal),
|
||||
/cancelled/
|
||||
)
|
||||
const key = runner.calls.find(call => call.args[1] === 'press_key')
|
||||
assert.equal(key, undefined)
|
||||
await session.close()
|
||||
} finally {
|
||||
await cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
test('two control authorities get isolated virtual cursor sessions and independent teardown', async () => {
|
||||
const { adapter, runner, cleanup } = await harness()
|
||||
try {
|
||||
const first = await adapter.openSession(identity('control-one'))
|
||||
const second = await adapter.openSession(identity('control-two'))
|
||||
const starts = runner.calls.filter(call => call.args[1] === 'start_session')
|
||||
assert.equal(starts.length, 2)
|
||||
assert.notEqual(starts[0]!.payload!.session, starts[1]!.payload!.session)
|
||||
assert.match(String(starts[0]!.payload!.session), /^hermes-[0-9a-f]{32}$/)
|
||||
assert.equal(starts[0]!.payload!.capture_scope, 'window')
|
||||
assert.equal(starts[1]!.payload!.capture_scope, 'window')
|
||||
await first.close()
|
||||
await assert.rejects(first.snapshot({ pid: 100, windowId: 200 }), /closed/)
|
||||
await second.snapshot({ pid: 300, windowId: 400 })
|
||||
await second.close()
|
||||
const ends = runner.calls.filter(call => call.args[1] === 'end_session').map(call => call.payload!.session)
|
||||
assert.deepEqual(new Set(ends), new Set(starts.map(call => call.payload!.session)))
|
||||
} finally {
|
||||
await cleanup()
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,326 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { access, mkdir, mkdtemp, rm, symlink, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import test from 'node:test'
|
||||
|
||||
import {
|
||||
checkCuaUpdate,
|
||||
getCuaManagementStatus,
|
||||
installCuaDriver,
|
||||
isSupportedCuaVersion,
|
||||
updateCuaDriver,
|
||||
type CuaManagementFetch
|
||||
} from '../src/tools/cuaManagement.js'
|
||||
import { CuaDriverAdapter, type CuaProcessResult, type CuaProcessRunner } from '../src/tools/cuaDriver.js'
|
||||
import { computerUseCommand } from '../src/commands/computerUse.js'
|
||||
|
||||
const ok = (stdout = ''): CuaProcessResult => ({ stdout, stderr: '', exitCode: 0 })
|
||||
|
||||
async function packageHome(version = '0.19.3'): Promise<{ root: string; binary: string }> {
|
||||
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-management-'))
|
||||
const release = join(root, '.cua-driver', 'packages', 'releases', `${version}-x86_64-pc-windows-msvc`)
|
||||
const current = join(root, '.cua-driver', 'packages', 'current')
|
||||
await mkdir(release, { recursive: true })
|
||||
const binary = join(release, 'cua-driver.exe')
|
||||
await writeFile(binary, `fake-cua-${version}`)
|
||||
await symlink(release, current, 'junction')
|
||||
return { root, binary }
|
||||
}
|
||||
|
||||
class ProbeRunner implements CuaProcessRunner {
|
||||
constructor(
|
||||
readonly version = '0.19.3',
|
||||
readonly latest = version,
|
||||
readonly onRun?: (executable: string, args: readonly string[], env?: NodeJS.ProcessEnv) => Promise<void> | void,
|
||||
readonly health: string = 'ok'
|
||||
) {}
|
||||
|
||||
async run(executable: string, args: readonly string[], options: { env?: NodeJS.ProcessEnv } = {}): Promise<CuaProcessResult> {
|
||||
await this.onRun?.(executable, args, options.env)
|
||||
if (args[0] === '--version') return ok(`cua-driver ${this.version}`)
|
||||
if (args[0] === 'check-update') return ok(JSON.stringify({
|
||||
checked_at: '2026-08-13T00:00:00Z',
|
||||
current_version: this.version,
|
||||
latest_version: this.latest,
|
||||
update_available: this.latest !== this.version,
|
||||
error: null
|
||||
}))
|
||||
if (args[0] === 'manifest') return ok(JSON.stringify({
|
||||
schema_version: '1',
|
||||
binary_version: this.version,
|
||||
binary_path: executable
|
||||
}))
|
||||
if (args[0] === 'list-tools') return ok([
|
||||
'health_report:', 'start_session:', 'end_session:', 'list_windows:',
|
||||
'get_window_state:', 'click:', 'set_value:', 'press_key:', 'scroll:'
|
||||
].join('\n'))
|
||||
if (args[0] === 'status') return ok('permission mode: standard')
|
||||
if (args[0] === 'call' && args[1] === 'health_report') return ok(JSON.stringify({
|
||||
schema_version: '1', driver_version: this.version, overall: this.health
|
||||
}))
|
||||
return ok()
|
||||
}
|
||||
}
|
||||
|
||||
test('supported CUA range is bounded to the adapter contract', () => {
|
||||
assert.equal(isSupportedCuaVersion('0.19.2'), false)
|
||||
assert.equal(isSupportedCuaVersion('0.19.3'), true)
|
||||
assert.equal(isSupportedCuaVersion('0.19.99'), true)
|
||||
assert.equal(isSupportedCuaVersion('0.20.0'), false)
|
||||
})
|
||||
|
||||
test('status prefers canonical package/current and reports a competing PATH shim', async () => {
|
||||
const home = await packageHome()
|
||||
const stale = await mkdtemp(join(tmpdir(), 'hermes-cua-stale-'))
|
||||
await writeFile(join(stale, 'cua-driver.exe'), 'stale')
|
||||
try {
|
||||
const status = await getCuaManagementStatus({
|
||||
platform: 'win32', homeDir: home.root, path: stale, runner: new ProbeRunner()
|
||||
})
|
||||
assert.equal(status.installed, true)
|
||||
assert.equal(status.current_version, '0.19.3')
|
||||
assert.equal(status.compatible, true)
|
||||
assert.equal(status.stale_path_shim, true)
|
||||
assert.equal(status.canonical_path, home.binary)
|
||||
assert.equal(status.release_source, 'trycua/cua')
|
||||
assert.equal(status.telemetry_enabled, false)
|
||||
} finally {
|
||||
await rm(home.root, { recursive: true, force: true })
|
||||
await rm(stale, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('check-update exposes a newer incompatible release without applying it', async () => {
|
||||
const home = await packageHome()
|
||||
try {
|
||||
const status = await checkCuaUpdate({
|
||||
platform: 'win32', homeDir: home.root, path: '', runner: new ProbeRunner('0.19.3', '0.20.0')
|
||||
})
|
||||
assert.equal(status.update?.update_available, true)
|
||||
assert.equal(status.update?.latest_version, '0.20.0')
|
||||
assert.equal(status.update?.compatible, false)
|
||||
} finally {
|
||||
await rm(home.root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('update refuses an unsupported latest release before any download or apply', async () => {
|
||||
const home = await packageHome()
|
||||
let fetches = 0
|
||||
let powershellRuns = 0
|
||||
try {
|
||||
await assert.rejects(updateCuaDriver({
|
||||
platform: 'win32',
|
||||
homeDir: home.root,
|
||||
path: '',
|
||||
runner: new ProbeRunner('0.19.3', '0.20.0', executable => {
|
||||
if (executable.toLowerCase() === 'powershell.exe') powershellRuns += 1
|
||||
}),
|
||||
fetch: async () => {
|
||||
fetches += 1
|
||||
throw new Error('must not fetch')
|
||||
}
|
||||
}), /available but unsupported/)
|
||||
assert.equal(fetches, 0)
|
||||
assert.equal(powershellRuns, 0)
|
||||
} finally {
|
||||
await rm(home.root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('install verifies trusted release metadata/checksum and sanitizes installer environment', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-install-test-'))
|
||||
const script = Buffer.from('Write-Output canonical-installer')
|
||||
const scriptSha = createHash('sha256').update(script).digest('hex')
|
||||
const fetchImpl: CuaManagementFetch = async url => ({
|
||||
ok: true,
|
||||
status: 200,
|
||||
async arrayBuffer() { return script.buffer.slice(script.byteOffset, script.byteOffset + script.byteLength) },
|
||||
async json() {
|
||||
assert.match(url, /release-manifest\.json$/)
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
repository: 'trycua/cua',
|
||||
product: 'cua-driver-rs',
|
||||
version: '0.19.3',
|
||||
tag: 'cua-driver-rs-v0.19.3',
|
||||
assets: [{ name: 'install.ps1', sha256: scriptSha }]
|
||||
}
|
||||
}
|
||||
})
|
||||
let installerEnvironment: NodeJS.ProcessEnv | undefined
|
||||
let installerPath: string | undefined
|
||||
const runner = new ProbeRunner('0.19.3', '0.19.3', async (executable, args, env) => {
|
||||
if (!executable.toLowerCase().endsWith('\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe')) return
|
||||
installerEnvironment = env
|
||||
installerPath = args[args.indexOf('-File') + 1]
|
||||
const release = join(root, '.cua-driver', 'packages', 'releases', '0.19.3-x86_64-pc-windows-msvc')
|
||||
await mkdir(release, { recursive: true })
|
||||
await writeFile(join(release, 'cua-driver.exe'), 'installed')
|
||||
await symlink(release, join(root, '.cua-driver', 'packages', 'current'), 'junction')
|
||||
})
|
||||
const priorSecret = process.env.OPENAI_API_KEY
|
||||
process.env.OPENAI_API_KEY = 'must-not-leak'
|
||||
try {
|
||||
const status = await installCuaDriver({
|
||||
platform: 'win32', homeDir: root, path: '', runner, fetch: fetchImpl, systemRoot: 'C:\\Windows'
|
||||
})
|
||||
assert.equal(status.operation?.release_manifest_verified, true)
|
||||
assert.equal(status.operation?.installer_checksum_verified, true)
|
||||
assert.equal(status.operation?.runtime_verified, true)
|
||||
assert.equal(installerEnvironment?.OPENAI_API_KEY, undefined)
|
||||
assert.equal(installerEnvironment?.CUA_DRIVER_RS_TELEMETRY_ENABLED, '0')
|
||||
assert.equal(installerEnvironment?.CUA_DRIVER_RS_VERSION, '0.19.3')
|
||||
assert.ok(installerPath)
|
||||
await assert.rejects(access(installerPath!))
|
||||
} finally {
|
||||
if (priorSecret === undefined) delete process.env.OPENAI_API_KEY
|
||||
else process.env.OPENAI_API_KEY = priorSecret
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('install rejects invalid release identity metadata before downloading the installer', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-invalid-manifest-'))
|
||||
let requests = 0
|
||||
try {
|
||||
await assert.rejects(installCuaDriver({
|
||||
platform: 'win32', homeDir: root, path: '', runner: new ProbeRunner(),
|
||||
fetch: async url => {
|
||||
requests += 1
|
||||
assert.match(url, /release-manifest\.json$/)
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
async arrayBuffer() { return new ArrayBuffer(0) },
|
||||
async json() {
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
repository: 'attacker/fork',
|
||||
product: 'cua-driver-rs',
|
||||
version: '0.19.3',
|
||||
tag: 'cua-driver-rs-v0.19.3',
|
||||
assets: [{ name: 'install.ps1', sha256: 'a'.repeat(64) }]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}), /publisher or version manifest is invalid/)
|
||||
assert.equal(requests, 1)
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('install rejects an installer whose bytes do not match release metadata', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-bad-checksum-'))
|
||||
let powershellRuns = 0
|
||||
try {
|
||||
await assert.rejects(installCuaDriver({
|
||||
platform: 'win32', homeDir: root, path: '',
|
||||
runner: new ProbeRunner('0.19.3', '0.19.3', executable => {
|
||||
if (executable.toLowerCase() === 'powershell.exe') powershellRuns += 1
|
||||
}),
|
||||
fetch: async url => ({
|
||||
ok: true,
|
||||
status: 200,
|
||||
async arrayBuffer() { return Buffer.from('tampered').buffer },
|
||||
async json() {
|
||||
assert.match(url, /release-manifest\.json$/)
|
||||
return {
|
||||
schemaVersion: 1, repository: 'trycua/cua', product: 'cua-driver-rs',
|
||||
version: '0.19.3', tag: 'cua-driver-rs-v0.19.3',
|
||||
assets: [{ name: 'install.ps1', sha256: 'a'.repeat(64) }]
|
||||
}
|
||||
}
|
||||
})
|
||||
}), /checksum verification failed/)
|
||||
assert.equal(powershellRuns, 0)
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('post-install runtime verification succeeds while explicit health remains degraded', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-degraded-'))
|
||||
const script = Buffer.from('installer')
|
||||
const checksum = createHash('sha256').update(script).digest('hex')
|
||||
const runner = new ProbeRunner('0.19.3', '0.19.3', async executable => {
|
||||
if (!executable.toLowerCase().endsWith('\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe')) return
|
||||
const release = join(root, '.cua-driver', 'packages', 'releases', '0.19.3-x86_64-pc-windows-msvc')
|
||||
await mkdir(release, { recursive: true })
|
||||
await writeFile(join(release, 'cua-driver.exe'), 'installed')
|
||||
await symlink(release, join(root, '.cua-driver', 'packages', 'current'), 'junction')
|
||||
}, 'degraded')
|
||||
try {
|
||||
const installed = await installCuaDriver({
|
||||
platform: 'win32', homeDir: root, path: '', runner, systemRoot: 'C:\\Windows',
|
||||
fetch: async url => ({
|
||||
ok: true,
|
||||
status: 200,
|
||||
async arrayBuffer() { return script.buffer.slice(script.byteOffset, script.byteOffset + script.byteLength) },
|
||||
async json() {
|
||||
assert.match(url, /release-manifest\.json$/)
|
||||
return {
|
||||
schemaVersion: 1, repository: 'trycua/cua', product: 'cua-driver-rs',
|
||||
version: '0.19.3', tag: 'cua-driver-rs-v0.19.3',
|
||||
assets: [{ name: 'install.ps1', sha256: checksum }]
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
assert.equal(installed.operation?.runtime_verified, true)
|
||||
const health = await CuaDriverAdapter.healthStatus({ platform: 'win32', homeDir: root, runner })
|
||||
assert.equal(health.state, 'degraded')
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('CLI install and update require explicit confirmation', async () => {
|
||||
let stderr = ''
|
||||
const original = process.stderr.write
|
||||
process.stderr.write = ((chunk: string | Uint8Array) => {
|
||||
stderr += chunk.toString()
|
||||
return true
|
||||
}) as typeof process.stderr.write
|
||||
try {
|
||||
assert.equal(await computerUseCommand({ command: 'computer-use', flags: {}, positional: ['cua', 'install'] }), 1)
|
||||
assert.match(stderr, /requires explicit confirmation with --yes/)
|
||||
} finally {
|
||||
process.stderr.write = original
|
||||
}
|
||||
})
|
||||
|
||||
test('install rejects an unverified installer before process execution', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'hermes-cua-bad-installer-'))
|
||||
let processStarted = false
|
||||
const fetchImpl: CuaManagementFetch = async url => ({
|
||||
ok: true,
|
||||
status: 200,
|
||||
async arrayBuffer() { return Buffer.from('tampered').buffer },
|
||||
async json() {
|
||||
assert.match(url, /release-manifest\.json$/)
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
repository: 'trycua/cua',
|
||||
product: 'cua-driver-rs',
|
||||
version: '0.19.3',
|
||||
tag: 'cua-driver-rs-v0.19.3',
|
||||
assets: [{ name: 'install.ps1', sha256: '0'.repeat(64) }]
|
||||
}
|
||||
}
|
||||
})
|
||||
const runner = new ProbeRunner('0.19.3', '0.19.3', () => { processStarted = true })
|
||||
try {
|
||||
await assert.rejects(
|
||||
installCuaDriver({ platform: 'win32', homeDir: root, path: '', runner, fetch: fetchImpl }),
|
||||
/checksum verification failed/
|
||||
)
|
||||
assert.equal(processStarted, false)
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
@@ -9,6 +9,8 @@ import {
|
||||
readDesktopUseSettings,
|
||||
readDesktopUseSettingsSync,
|
||||
requestComputerGrantCancellation,
|
||||
setActivityScreenshotRetention,
|
||||
setComputerControlSettings,
|
||||
setDesktopUseEnabled
|
||||
} from '../src/lib/desktopUseSettings.js'
|
||||
import { shouldAdvertiseComputerUse } from '../src/tools/handlerSet.js'
|
||||
@@ -34,6 +36,45 @@ test('desktop-use preference defaults off and persists explicit changes', async
|
||||
}
|
||||
})
|
||||
|
||||
test('computer control settings default fail-closed and survive desktop-use changes', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'hermes-computer-control-'))
|
||||
const settingsPath = join(dir, 'desktop-settings.json')
|
||||
try {
|
||||
assert.deepEqual(await readDesktopUseSettings(settingsPath), {
|
||||
computer_use_enabled: false,
|
||||
computer_control_engine: 'cua',
|
||||
cua_cursor_enabled: false,
|
||||
activity_screenshot_retention_enabled: true,
|
||||
activity_screenshot_retention_days: 7
|
||||
})
|
||||
await setComputerControlSettings({
|
||||
computer_control_engine: 'cua',
|
||||
cua_cursor_enabled: true
|
||||
}, settingsPath)
|
||||
await setDesktopUseEnabled(true, settingsPath)
|
||||
const settings = await readDesktopUseSettings(settingsPath)
|
||||
assert.equal(settings.computer_use_enabled, true)
|
||||
assert.equal(settings.computer_control_engine, 'cua')
|
||||
assert.equal(settings.cua_cursor_enabled, true)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('screenshot evidence retention is explicit and survives other desktop setting changes', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'hermes-screenshot-retention-'))
|
||||
const settingsPath = join(dir, 'desktop-settings.json')
|
||||
try {
|
||||
await setActivityScreenshotRetention(false, 30, settingsPath)
|
||||
await setDesktopUseEnabled(true, settingsPath)
|
||||
const settings = await readDesktopUseSettings(settingsPath)
|
||||
assert.equal(settings.activity_screenshot_retention_enabled, false)
|
||||
assert.equal(settings.activity_screenshot_retention_days, 30)
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
test('grant cancellation bridge is consumed exactly once', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'hermes-desktop-cancel-'))
|
||||
const cancelPath = join(dir, 'cancel-active.json')
|
||||
|
||||
@@ -31,6 +31,8 @@ test('installer launch is delayed until the running CLI can exit', () => {
|
||||
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_PATH, 'C:\\Temp\\hermes setup.exe')
|
||||
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_SILENT, '1')
|
||||
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_INSTALL_DIR, 'C:\\Hermes custom')
|
||||
assert.equal(plan.options.env?.HERMES_RELAY_SETUP_CALLER_PID, String(process.pid))
|
||||
assert.match(plan.args.join(' '), /Wait-Process -Id \$callerPid/)
|
||||
assert.match(plan.args.join(' '), /Start-Sleep/)
|
||||
assert.match(plan.args.join(' '), /\/D=/)
|
||||
assert.match(plan.args.join(' '), /Remove-Item -LiteralPath \$installer/)
|
||||
@@ -44,7 +46,7 @@ test('PowerShell launches an installer whose path contains spaces via environmen
|
||||
const fakeInstaller = join(scratch, 'fake setup.exe')
|
||||
await copyFile(join(process.env.WINDIR ?? 'C:\\Windows', 'System32', 'whoami.exe'), fakeInstaller)
|
||||
try {
|
||||
const plan = windowsInstallerLaunchPlan(fakeInstaller, { silent: false, delayMs: 0 })
|
||||
const plan = windowsInstallerLaunchPlan(fakeInstaller, { silent: false, delayMs: 0, callerPid: 0 })
|
||||
const result = spawnSync(plan.program, plan.args, { ...plan.options, detached: false, stdio: 'pipe' })
|
||||
assert.equal(result.status, 0, result.stderr?.toString())
|
||||
} finally {
|
||||
@@ -76,6 +78,25 @@ test('NSIS bundle cleans cooperative-update and local-development backups', asyn
|
||||
}
|
||||
})
|
||||
|
||||
test('NSIS bundle quiesces tray children and retries failed payload extraction once', async () => {
|
||||
const script = await readFile(new URL('../tray/installer/hermes-relay.nsi', import.meta.url), 'utf8')
|
||||
const core = script.slice(script.indexOf('Section "Hermes-Relay CLI and management UI"'), script.indexOf('Section "Start tray when I sign in"'))
|
||||
const trayStop = core.indexOf('/IM hermes-relay-tray.exe /T /F')
|
||||
const cliDrain = core.indexOf('/IM hermes-relay.exe /T /F')
|
||||
const firstInstall = core.indexOf('File /oname=hermes-relay.exe "${CLI_EXE}"')
|
||||
const retry = core.indexOf('install_attempt_failed:')
|
||||
|
||||
assert.ok(trayStop >= 0, 'installer should terminate the tray process tree')
|
||||
assert.ok(cliDrain < firstInstall, 'installed processes must be drained before replacement')
|
||||
assert.ok(retry > firstInstall, 'installer should retry the first failed extraction attempt')
|
||||
assert.match(core, /StrCpy \$InstallAttempt "1"[\s\S]*Goto install_attempt/)
|
||||
assert.match(core, /ClearErrors[\s\S]*File \/oname=hermes-relay\.exe[\s\S]*File \/oname=hermes-relay-tray\.exe[\s\S]*IfErrors install_attempt_failed/)
|
||||
assert.match(core, /SetErrorLevel 1\s+Quit/)
|
||||
assert.match(core, /MessageBox MB_ICONSTOP .* \/SD IDOK/)
|
||||
assert.match(core, /taskkill\.exe" \/IM hermes-relay-tray\.exe \/T \/F/)
|
||||
assert.match(core, /taskkill\.exe" \/IM hermes-relay\.exe \/T \/F/)
|
||||
})
|
||||
|
||||
test('local tray installation embeds production assets instead of loading devUrl', async () => {
|
||||
const script = await readFile(new URL('../scripts/dev-install-tray.mjs', import.meta.url), 'utf8')
|
||||
assert.match(script, /'--features', 'custom-protocol'/)
|
||||
@@ -87,6 +108,14 @@ test('CLI opens the GUI process without forcing a hidden Windows startup state',
|
||||
assert.doesNotMatch(source, /windowsHide:\s*true/)
|
||||
})
|
||||
|
||||
test('tray update helper preserves the current install directory and cleans its installer', async () => {
|
||||
const source = await readFile(new URL('../tray/src/main.rs', import.meta.url), 'utf8')
|
||||
assert.match(source, /\.env\("HERMES_UPDATE_INSTALL_DIR", install_dir\)/)
|
||||
assert.match(source, /\$installerArgs=@\('\/S',\('\/D=' \+ \$env:HERMES_UPDATE_INSTALL_DIR\)\)/)
|
||||
assert.match(source, /Remove-Item -LiteralPath \$env:HERMES_UPDATE_INSTALLER -Force/)
|
||||
assert.match(source, /exit \$exitCode/)
|
||||
})
|
||||
|
||||
test('POSIX installer only advertises artifacts produced by the release workflow', async () => {
|
||||
const script = await readFile(new URL('../scripts/install.sh', import.meta.url), 'utf8')
|
||||
assert.doesNotMatch(script, /hermes-relay-linux-arm64/)
|
||||
|
||||
Generated
+2
-1
@@ -1232,8 +1232,9 @@ checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
|
||||
|
||||
[[package]]
|
||||
name = "hermes-relay-tray"
|
||||
version = "0.4.0-alpha.8"
|
||||
version = "0.4.0-beta.2"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tauri",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "hermes-relay-tray"
|
||||
version = "0.4.0-alpha.8"
|
||||
version = "0.4.0-beta.2"
|
||||
description = "Compact Windows management UI for Hermes-Relay CLI"
|
||||
authors = ["Axiom Labs"]
|
||||
edition = "2021"
|
||||
@@ -14,6 +14,7 @@ path = "src/main.rs"
|
||||
[dependencies]
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
base64 = "0.22"
|
||||
tauri = { version = "2", features = ["tray-icon", "image-png"] }
|
||||
|
||||
[features]
|
||||
|
||||
@@ -19,6 +19,7 @@ VIAddVersionKey "FileVersion" "${VERSION}"
|
||||
VIAddVersionKey "LegalCopyright" "MIT License"
|
||||
|
||||
Var ExistingStartup
|
||||
Var InstallAttempt
|
||||
|
||||
Function .onInit
|
||||
ReadRegStr $ExistingStartup HKCU "Software\Microsoft\Windows\CurrentVersion\Run" "HermesRelayTray"
|
||||
@@ -42,20 +43,47 @@ FunctionEnd
|
||||
|
||||
Section "Hermes-Relay CLI and management UI" SEC_CORE
|
||||
SectionIn RO
|
||||
IfFileExists "$INSTDIR\hermes-relay.exe" 0 tray_stop
|
||||
; Stop the tray first so its snapshot polling cannot launch another CLI
|
||||
; process while setup is trying to replace hermes-relay.exe. The following
|
||||
; /T also terminates a poll already in flight before payload extraction.
|
||||
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay-tray.exe /T /F'
|
||||
IfFileExists "$INSTDIR\hermes-relay.exe" 0 processes_quiesced
|
||||
nsExec::ExecToLog '"$INSTDIR\hermes-relay.exe" daemon stop'
|
||||
tray_stop:
|
||||
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay-tray.exe /F'
|
||||
Sleep 250
|
||||
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay.exe /T /F'
|
||||
processes_quiesced:
|
||||
; Drain the daemon and any short-lived tray/management CLI children that
|
||||
; raced with shutdown. An explicit bundle update is the lifecycle boundary
|
||||
; for installed Hermes-Relay processes.
|
||||
Sleep 350
|
||||
Delete "$INSTDIR\hermes-relay.new.exe"
|
||||
Delete "$INSTDIR\hermes-relay.old.exe"
|
||||
Delete "$INSTDIR\hermes-relay.exe.bak"
|
||||
Delete "$INSTDIR\hermes-relay-tray.exe.bak"
|
||||
StrCpy $InstallAttempt "0"
|
||||
install_attempt:
|
||||
SetOutPath "$INSTDIR"
|
||||
ClearErrors
|
||||
File /oname=hermes-relay.exe "${CLI_EXE}"
|
||||
File /oname=hermes-relay-tray.exe "${TRAY_EXE}"
|
||||
File /oname=hermes-relay-path.ps1 "${PATH_HELPER}"
|
||||
File /oname=hermes-relay-ui.cmd "${UI_SHIM}"
|
||||
IfErrors install_attempt_failed
|
||||
|
||||
; ClearErrors/IfErrors covers both CLI and tray writes, so setup cannot
|
||||
; publish registry metadata after a locked or partial replacement.
|
||||
Goto cli_verified
|
||||
install_attempt_failed:
|
||||
StrCmp $InstallAttempt "0" 0 cli_verification_failed
|
||||
StrCpy $InstallAttempt "1"
|
||||
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay-tray.exe /T /F'
|
||||
nsExec::ExecToLog '"$SYSDIR\taskkill.exe" /IM hermes-relay.exe /T /F'
|
||||
Sleep 350
|
||||
Goto install_attempt
|
||||
cli_verification_failed:
|
||||
MessageBox MB_ICONSTOP "Hermes-Relay CLI UI ${VERSION} could not replace the running installation. Close Hermes-Relay processes and run setup again." /SD IDOK
|
||||
SetErrorLevel 1
|
||||
Quit
|
||||
cli_verified:
|
||||
WriteUninstaller "$INSTDIR\uninstall-hermes-relay.exe"
|
||||
|
||||
WriteRegStr HKCU "Software\HermesRelay" "InstallDir" "$INSTDIR"
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@hermes-relay/tray-ui",
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@hermes-relay/tray-ui",
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"dependencies": {
|
||||
"@tauri-apps/api": "^2.8.0",
|
||||
"lucide-react": "^0.468.0",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "@hermes-relay/tray-ui",
|
||||
"private": true,
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite --host 127.0.0.1",
|
||||
|
||||
+523
-20
@@ -5,6 +5,7 @@ compile_error!("hermes-relay-tray is a Windows-only optional systray");
|
||||
|
||||
#[cfg(windows)]
|
||||
mod app {
|
||||
use base64::{engine::general_purpose::STANDARD as BASE64_STANDARD, Engine as _};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
use std::{
|
||||
@@ -15,7 +16,7 @@ mod app {
|
||||
os::windows::process::CommandExt,
|
||||
path::{Path, PathBuf},
|
||||
process::{Command, Output, Stdio},
|
||||
sync::{mpsc, Arc, Mutex},
|
||||
sync::{mpsc, Arc, Mutex, OnceLock},
|
||||
thread,
|
||||
time::{Duration, SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
@@ -225,6 +226,10 @@ mod app {
|
||||
privilege: Option<String>,
|
||||
username: Option<String>,
|
||||
updated_at: Option<u64>,
|
||||
last_event: Option<String>,
|
||||
reconnect_attempt: Option<u64>,
|
||||
retry_at: Option<u64>,
|
||||
last_error: Option<String>,
|
||||
}
|
||||
|
||||
fn stopped() -> String {
|
||||
@@ -269,6 +274,26 @@ mod app {
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
host_url: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
backend: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
dispatch: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
control_session_id: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
target_app: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
target_title: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
target_pid: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
target_window_id: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
action: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
verification: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
phase: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
duration_ms: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
exit_code: Option<i64>,
|
||||
@@ -294,6 +319,14 @@ mod app {
|
||||
result_truncated: Option<bool>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
error: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
screenshot_evidence_id: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
screenshot_mime_type: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
screenshot_width: Option<u64>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
screenshot_height: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
@@ -312,15 +345,100 @@ mod app {
|
||||
active_url: Option<String>,
|
||||
daemon: DaemonStatus,
|
||||
activity: Vec<Activity>,
|
||||
activity_screenshot_retention: ActivityScreenshotRetention,
|
||||
pending_grants: Vec<PendingGrantRequest>,
|
||||
startup_enabled: bool,
|
||||
daemon_autostart_enabled: bool,
|
||||
hardware_availability: HardwareAvailability,
|
||||
computer_control_engine: Option<ComputerControlEngine>,
|
||||
ui_version: &'static str,
|
||||
cli_version: Option<String>,
|
||||
cli_path: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
struct ActivityScreenshotRetention {
|
||||
#[serde(default)]
|
||||
enabled: bool,
|
||||
#[serde(default = "default_retention_days")]
|
||||
days: u64,
|
||||
#[serde(default)]
|
||||
count: u64,
|
||||
#[serde(default)]
|
||||
bytes: u64,
|
||||
}
|
||||
|
||||
fn default_retention_days() -> u64 {
|
||||
7
|
||||
}
|
||||
|
||||
impl Default for ActivityScreenshotRetention {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
enabled: true,
|
||||
days: 7,
|
||||
count: 0,
|
||||
bytes: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
struct ComputerControlEngine {
|
||||
selected: String,
|
||||
effective: String,
|
||||
available: bool,
|
||||
state: String,
|
||||
version: Option<String>,
|
||||
health: Option<String>,
|
||||
path: Option<String>,
|
||||
#[serde(default)]
|
||||
cursor_enabled: bool,
|
||||
#[serde(default)]
|
||||
foreground_escalation_enabled: bool,
|
||||
#[serde(default)]
|
||||
active_sessions: Option<u64>,
|
||||
active_backend: Option<String>,
|
||||
last_action: Option<Value>,
|
||||
message: Option<String>,
|
||||
}
|
||||
|
||||
type ComputerControlEngineCache = Option<(SystemTime, Option<ComputerControlEngine>)>;
|
||||
|
||||
static COMPUTER_CONTROL_ENGINE_CACHE: OnceLock<Mutex<ComputerControlEngineCache>> =
|
||||
OnceLock::new();
|
||||
|
||||
fn probe_computer_control_engine() -> Option<ComputerControlEngine> {
|
||||
run_json(&["computer-use", "status", "--json"])
|
||||
.ok()
|
||||
.and_then(|value| value.get("computer_control_engine").cloned())
|
||||
.and_then(|value| serde_json::from_value::<ComputerControlEngine>(value).ok())
|
||||
}
|
||||
|
||||
fn cached_computer_control_engine() -> Option<ComputerControlEngine> {
|
||||
let cache = COMPUTER_CONTROL_ENGINE_CACHE.get_or_init(|| Mutex::new(None));
|
||||
if let Ok(guard) = cache.lock() {
|
||||
if let Some((checked_at, status)) = guard.as_ref() {
|
||||
if checked_at.elapsed().unwrap_or_default() < Duration::from_secs(30) {
|
||||
return status.clone();
|
||||
}
|
||||
}
|
||||
}
|
||||
let status = probe_computer_control_engine();
|
||||
if let Ok(mut guard) = cache.lock() {
|
||||
*guard = Some((SystemTime::now(), status.clone()));
|
||||
}
|
||||
status
|
||||
}
|
||||
|
||||
fn clear_computer_control_engine_cache() {
|
||||
if let Some(cache) = COMPUTER_CONTROL_ENGINE_CACHE.get() {
|
||||
if let Ok(mut guard) = cache.lock() {
|
||||
*guard = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
struct HardwareAvailability {
|
||||
usb: bool,
|
||||
@@ -521,17 +639,36 @@ mod app {
|
||||
let Ok(path) = home_dir().map(|h| h.join(".hermes").join("desktop-audit.jsonl")) else {
|
||||
return Vec::new();
|
||||
};
|
||||
let Ok(text) = fs::read_to_string(path) else {
|
||||
return Vec::new();
|
||||
};
|
||||
text.lines()
|
||||
.rev()
|
||||
.take(30)
|
||||
.filter_map(|line| serde_json::from_str(line).ok())
|
||||
.collect::<Vec<_>>()
|
||||
let text = [path.with_extension("jsonl.1"), path]
|
||||
.into_iter()
|
||||
.rev()
|
||||
.collect()
|
||||
.filter_map(|file| fs::read_to_string(file).ok())
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
let mut activity = text
|
||||
.lines()
|
||||
.filter_map(|line| serde_json::from_str(line).ok())
|
||||
.collect::<Vec<Activity>>();
|
||||
let evidence_directory = home_dir()
|
||||
.ok()
|
||||
.map(|home| home.join(".hermes").join("activity-evidence"));
|
||||
for entry in &mut activity {
|
||||
let available = entry
|
||||
.screenshot_evidence_id
|
||||
.as_ref()
|
||||
.zip(evidence_directory.as_ref())
|
||||
.is_some_and(|(id, directory)| directory.join(format!("{id}.png")).is_file());
|
||||
if !available {
|
||||
entry.screenshot_evidence_id = None;
|
||||
entry.screenshot_mime_type = None;
|
||||
entry.screenshot_width = None;
|
||||
entry.screenshot_height = None;
|
||||
}
|
||||
}
|
||||
activity.sort_by_key(|entry| entry.ts);
|
||||
if activity.len() > 200 {
|
||||
activity.drain(..activity.len() - 200);
|
||||
}
|
||||
activity
|
||||
}
|
||||
|
||||
fn append_management_event(
|
||||
@@ -579,6 +716,90 @@ mod app {
|
||||
fs::remove_file(&backup)
|
||||
.map_err(|error| format!("cannot remove rotated activity: {error}"))?;
|
||||
}
|
||||
let evidence = directory.join("activity-evidence");
|
||||
if evidence.exists() {
|
||||
fs::remove_dir_all(&evidence)
|
||||
.map_err(|error| format!("cannot clear screenshot evidence: {error}"))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn append_management_error(tool: &str, summary: &str, error: &str) {
|
||||
let Ok(directory) = home_dir().map(|home| home.join(".hermes")) else {
|
||||
return;
|
||||
};
|
||||
if fs::create_dir_all(&directory).is_err() {
|
||||
return;
|
||||
}
|
||||
let timestamp = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_millis() as u64;
|
||||
let event = serde_json::json!({
|
||||
"ts": timestamp,
|
||||
"kind": "management.completed",
|
||||
"tool": tool,
|
||||
"category": "system",
|
||||
"ok": false,
|
||||
"summary": summary,
|
||||
"error": error.chars().take(512).collect::<String>(),
|
||||
});
|
||||
if let Ok(mut file) = OpenOptions::new()
|
||||
.create(true)
|
||||
.append(true)
|
||||
.open(directory.join("desktop-audit.jsonl"))
|
||||
{
|
||||
let _ = writeln!(file, "{event}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn get_activity_screenshot(evidence_id: String) -> Result<String, String> {
|
||||
if evidence_id.len() != 32 || !evidence_id.bytes().all(|byte| byte.is_ascii_hexdigit()) {
|
||||
return Err("invalid screenshot evidence identifier".to_string());
|
||||
}
|
||||
let path = home_dir()?
|
||||
.join(".hermes")
|
||||
.join("activity-evidence")
|
||||
.join(format!("{}.png", evidence_id.to_ascii_lowercase()));
|
||||
let bytes =
|
||||
fs::read(path).map_err(|_| "screenshot evidence is no longer available".to_string())?;
|
||||
if bytes.is_empty()
|
||||
|| bytes.len() > 10_000_000
|
||||
|| !bytes.starts_with(&[0x89, b'P', b'N', b'G', 0x0d, 0x0a, 0x1a, 0x0a])
|
||||
{
|
||||
return Err("screenshot evidence is invalid".to_string());
|
||||
}
|
||||
Ok(format!(
|
||||
"data:image/png;base64,{}",
|
||||
BASE64_STANDARD.encode(bytes)
|
||||
))
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn set_activity_screenshot_retention(enabled: bool, days: u64) -> Result<(), String> {
|
||||
if !matches!(days, 1 | 7 | 30) {
|
||||
return Err("screenshot retention must be 1, 7, or 30 days".to_string());
|
||||
}
|
||||
let days = days.to_string();
|
||||
run_cli_checked(&[
|
||||
"audit",
|
||||
"screenshots",
|
||||
if enabled { "on" } else { "off" },
|
||||
"--days",
|
||||
&days,
|
||||
"--yes",
|
||||
])?;
|
||||
append_management_event(
|
||||
"activity.retention",
|
||||
if enabled {
|
||||
"Screenshot evidence retention changed"
|
||||
} else {
|
||||
"Screenshot evidence retention disabled"
|
||||
},
|
||||
None,
|
||||
None,
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -643,21 +864,122 @@ mod app {
|
||||
.and_then(|value| serde_json::from_value::<Vec<PendingGrantRequest>>(value).ok())
|
||||
.unwrap_or_default();
|
||||
let (cli_version, cli_path) = cli_details();
|
||||
let computer_control_engine = cached_computer_control_engine();
|
||||
let activity_screenshot_retention = run_json(&["audit", "screenshots", "--json"])
|
||||
.ok()
|
||||
.and_then(|value| serde_json::from_value(value).ok())
|
||||
.unwrap_or_default();
|
||||
Ok(Snapshot {
|
||||
hosts,
|
||||
active_url: selected,
|
||||
daemon,
|
||||
activity: read_activity(),
|
||||
activity_screenshot_retention,
|
||||
pending_grants,
|
||||
startup_enabled: startup_enabled(),
|
||||
daemon_autostart_enabled: daemon_autostart_enabled(),
|
||||
hardware_availability: hardware_availability(),
|
||||
computer_control_engine,
|
||||
ui_version: env!("CARGO_PKG_VERSION"),
|
||||
cli_version,
|
||||
cli_path,
|
||||
})
|
||||
}
|
||||
|
||||
fn computer_control_engine_status() -> Result<ComputerControlEngine, String> {
|
||||
run_json(&["computer-use", "status", "--json"])?
|
||||
.get("computer_control_engine")
|
||||
.cloned()
|
||||
.ok_or_else(|| {
|
||||
"the installed CLI does not report a computer control engine".to_string()
|
||||
})
|
||||
.and_then(|value| {
|
||||
serde_json::from_value(value).map_err(|_| {
|
||||
"the installed CLI returned an invalid computer control status".to_string()
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn computer_cua_status() -> Result<Value, String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
run_json(&["computer-use", "cua", "status", "--json"])
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("CUA status task failed: {error}"))?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn computer_cua_health() -> Result<Value, String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
run_json(&["computer-use", "cua", "health", "--json"])
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("CUA health task failed: {error}"))?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn computer_cua_install() -> Result<Value, String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
let result = run_json(&["computer-use", "cua", "install", "--yes", "--json"])?;
|
||||
clear_computer_control_engine_cache();
|
||||
Ok(result)
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("CUA install task failed: {error}"))?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn computer_cua_check_update() -> Result<Value, String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
run_json(&["computer-use", "cua", "check-update", "--json"])
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("CUA update check task failed: {error}"))?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn computer_cua_update() -> Result<Value, String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
let result = run_json(&["computer-use", "cua", "update", "--yes", "--json"])?;
|
||||
clear_computer_control_engine_cache();
|
||||
Ok(result)
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("CUA update task failed: {error}"))?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn set_computer_control_engine(engine: String) -> Result<(), String> {
|
||||
if engine != "legacy" && engine != "cua" {
|
||||
return Err("invalid computer control engine".to_string());
|
||||
}
|
||||
if engine == "cua" {
|
||||
let status = computer_control_engine_status()?;
|
||||
if !status.available || status.state != "ready" {
|
||||
return Err("CUA Driver is not ready; engine selection was not changed".to_string());
|
||||
}
|
||||
}
|
||||
let was_running = daemon_is_running();
|
||||
run_cli_checked(&["computer-use", "engine", engine.as_str()])?;
|
||||
clear_computer_control_engine_cache();
|
||||
restart_daemon_if_running(was_running)
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn set_cua_cursor_enabled(enabled: bool) -> Result<(), String> {
|
||||
let status = computer_control_engine_status()?;
|
||||
if status.selected != "cua" || status.state != "ready" {
|
||||
return Err(
|
||||
"CUA Driver must be selected and ready before changing its cursor".to_string(),
|
||||
);
|
||||
}
|
||||
let was_running = daemon_is_running();
|
||||
run_cli_checked(&["computer-use", "cursor", if enabled { "on" } else { "off" }])?;
|
||||
clear_computer_control_engine_cache();
|
||||
restart_daemon_if_running(was_running)
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
async fn get_snapshot() -> Result<Snapshot, String> {
|
||||
tauri::async_runtime::spawn_blocking(build_snapshot)
|
||||
@@ -706,14 +1028,21 @@ mod app {
|
||||
.parent()
|
||||
.map(|directory| directory.join("hermes-relay.exe"))
|
||||
.ok_or_else(|| "cannot resolve the installed CLI path".to_string())?;
|
||||
let install_dir = tray_exe
|
||||
.parent()
|
||||
.ok_or_else(|| "cannot resolve the desktop install directory".to_string())?;
|
||||
let helper_script = "$ErrorActionPreference='Stop'; "
|
||||
.to_string()
|
||||
+ "$targetPid=[int]$env:HERMES_UPDATE_PID; "
|
||||
+ "Wait-Process -Id $targetPid -ErrorAction SilentlyContinue; "
|
||||
+ "$result=Start-Process -FilePath $env:HERMES_UPDATE_INSTALLER -ArgumentList '/S' -Wait -PassThru; "
|
||||
+ "if ($result.ExitCode -eq 0) { "
|
||||
+ "$exitCode=1; try { "
|
||||
+ "$installerArgs=@('/S',('/D=' + $env:HERMES_UPDATE_INSTALL_DIR)); "
|
||||
+ "$result=Start-Process -FilePath $env:HERMES_UPDATE_INSTALLER -ArgumentList $installerArgs -Wait -PassThru; "
|
||||
+ "$exitCode=$result.ExitCode; if ($exitCode -eq 0) { "
|
||||
+ "if ($env:HERMES_UPDATE_RESTART_DAEMON -eq '1') { & $env:HERMES_UPDATE_CLI daemon start | Out-Null }; "
|
||||
+ "Start-Process -FilePath $env:HERMES_UPDATE_TRAY }";
|
||||
+ "Start-Process -FilePath $env:HERMES_UPDATE_TRAY } "
|
||||
+ "} finally { Remove-Item -LiteralPath $env:HERMES_UPDATE_INSTALLER -Force -ErrorAction SilentlyContinue }; "
|
||||
+ "exit $exitCode";
|
||||
Command::new("powershell.exe")
|
||||
.args([
|
||||
"-NoProfile",
|
||||
@@ -725,6 +1054,7 @@ mod app {
|
||||
])
|
||||
.env("HERMES_UPDATE_PID", std::process::id().to_string())
|
||||
.env("HERMES_UPDATE_INSTALLER", installer)
|
||||
.env("HERMES_UPDATE_INSTALL_DIR", install_dir)
|
||||
.env("HERMES_UPDATE_TRAY", tray_exe)
|
||||
.env("HERMES_UPDATE_CLI", cli_exe)
|
||||
.env(
|
||||
@@ -1126,7 +1456,10 @@ mod app {
|
||||
#[tauri::command]
|
||||
async fn connect_daemon() -> Result<(), String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
run_cli_checked(&["daemon", "start"])?;
|
||||
if let Err(error) = run_cli_checked(&["daemon", "start"]) {
|
||||
append_management_error("daemon.start", "Relay daemon failed to connect", &error);
|
||||
return Err(error);
|
||||
}
|
||||
append_management_event("daemon.start", "Relay daemon connected", None, None);
|
||||
Ok(())
|
||||
})
|
||||
@@ -1136,7 +1469,10 @@ mod app {
|
||||
#[tauri::command]
|
||||
async fn disconnect_daemon() -> Result<(), String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
run_cli_checked(&["daemon", "stop"])?;
|
||||
if let Err(error) = run_cli_checked(&["daemon", "stop"]) {
|
||||
append_management_error("daemon.stop", "Relay daemon failed to disconnect", &error);
|
||||
return Err(error);
|
||||
}
|
||||
append_management_event("daemon.stop", "Relay daemon disconnected", None, None);
|
||||
Ok(())
|
||||
})
|
||||
@@ -1144,10 +1480,17 @@ mod app {
|
||||
.map_err(|error| format!("disconnect daemon task failed: {error}"))?
|
||||
}
|
||||
#[tauri::command]
|
||||
fn restart_daemon() -> Result<(), String> {
|
||||
run_cli_checked(&["daemon", "restart"])?;
|
||||
append_management_event("daemon.restart", "Relay daemon restarted", None, None);
|
||||
Ok(())
|
||||
async fn restart_daemon() -> Result<(), String> {
|
||||
tauri::async_runtime::spawn_blocking(|| {
|
||||
if let Err(error) = run_cli_checked(&["daemon", "restart"]) {
|
||||
append_management_error("daemon.restart", "Relay daemon failed to restart", &error);
|
||||
return Err(error);
|
||||
}
|
||||
append_management_event("daemon.restart", "Relay daemon restarted", None, None);
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.map_err(|error| format!("restart daemon task failed: {error}"))?
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
@@ -1484,6 +1827,154 @@ mod app {
|
||||
present_grant_window_inner(&app, expanded, &tray_position)
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ConnectionNotice<'a> {
|
||||
tone: &'a str,
|
||||
title: &'a str,
|
||||
detail: String,
|
||||
}
|
||||
|
||||
fn daemon_status_from_file() -> Option<DaemonStatus> {
|
||||
let path = home_dir().ok()?.join(".hermes").join("daemon-status.json");
|
||||
serde_json::from_slice(&fs::read(path).ok()?).ok()
|
||||
}
|
||||
|
||||
fn present_connection_notice(app: &AppHandle, notice: ConnectionNotice<'_>) {
|
||||
if app
|
||||
.get_webview_window("main")
|
||||
.is_some_and(|window| window.is_visible().unwrap_or(false))
|
||||
{
|
||||
if let Some(window) = app.get_webview_window("notice") {
|
||||
let _ = window.hide();
|
||||
}
|
||||
return;
|
||||
}
|
||||
let Some(window) = app.get_webview_window("notice") else {
|
||||
return;
|
||||
};
|
||||
let Some(monitor) = window
|
||||
.current_monitor()
|
||||
.ok()
|
||||
.flatten()
|
||||
.or_else(|| window.primary_monitor().ok().flatten())
|
||||
else {
|
||||
return;
|
||||
};
|
||||
let scale = monitor.scale_factor();
|
||||
let size = PhysicalSize::new(
|
||||
(360.0 * scale).round() as u32,
|
||||
(126.0 * scale).round() as u32,
|
||||
);
|
||||
let _ = window.set_size(Size::Physical(size));
|
||||
let _ = window.set_position(bottom_right_position(monitor.work_area(), size, scale));
|
||||
let Ok(payload) = serde_json::to_string(¬ice) else {
|
||||
return;
|
||||
};
|
||||
let _ = window.eval(format!("window.dispatchEvent(new CustomEvent('hermes-connection-notice', {{ detail: {payload} }}))"));
|
||||
let _ = window.show();
|
||||
}
|
||||
|
||||
fn start_connection_watcher(app: AppHandle) {
|
||||
thread::spawn(move || {
|
||||
let mut previous = None::<String>;
|
||||
loop {
|
||||
let status = daemon_status_from_file();
|
||||
let state = status
|
||||
.as_ref()
|
||||
.map(|value| value.state.clone())
|
||||
.unwrap_or_else(|| "stopped".to_string());
|
||||
if let Some(before) = previous.as_deref() {
|
||||
if before != state {
|
||||
let notice = match state.as_str() {
|
||||
"connected" => Some(ConnectionNotice {
|
||||
tone: "connected",
|
||||
title: if before == "reconnecting" {
|
||||
"Tunnel restored"
|
||||
} else {
|
||||
"Tunnel connected"
|
||||
},
|
||||
detail: status
|
||||
.as_ref()
|
||||
.and_then(|value| value.url.clone())
|
||||
.unwrap_or_else(|| "Remote access is ready".to_string()),
|
||||
}),
|
||||
"reconnecting" => Some(ConnectionNotice {
|
||||
tone: "warning",
|
||||
title: "Connection interrupted",
|
||||
detail: format!(
|
||||
"Retrying automatically · attempt {}",
|
||||
status
|
||||
.as_ref()
|
||||
.and_then(|value| value.reconnect_attempt)
|
||||
.unwrap_or(1)
|
||||
),
|
||||
}),
|
||||
"stopped"
|
||||
if before == "connected"
|
||||
|| before == "reconnecting"
|
||||
|| before == "starting" =>
|
||||
{
|
||||
Some(ConnectionNotice {
|
||||
tone: "offline",
|
||||
title: "Tunnel disconnected",
|
||||
detail: status
|
||||
.as_ref()
|
||||
.and_then(|value| value.last_error.clone())
|
||||
.unwrap_or_else(|| "Remote access is offline".to_string()),
|
||||
})
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
if let Some(notice) = notice {
|
||||
let handle = app.clone();
|
||||
let _ = app.run_on_main_thread(move || {
|
||||
present_connection_notice(&handle, notice)
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
previous = Some(state);
|
||||
thread::sleep(Duration::from_millis(500));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn open_management_from_notice(app: AppHandle, tray_position: State<'_, TrayPositionState>) {
|
||||
if let Some(window) = app.get_webview_window("notice") {
|
||||
let _ = window.hide();
|
||||
}
|
||||
let anchor = tray_position.0.lock().ok().and_then(|value| *value);
|
||||
reveal_main_window(&app, anchor);
|
||||
}
|
||||
|
||||
#[tauri::command]
|
||||
fn present_activity_screenshot(app: AppHandle, evidence_id: String) -> Result<(), String> {
|
||||
let _ = get_activity_screenshot(evidence_id.clone())?;
|
||||
let window = app
|
||||
.get_webview_window("evidence")
|
||||
.ok_or_else(|| "screenshot viewer is unavailable".to_string())?;
|
||||
let payload = serde_json::to_string(&serde_json::json!({ "evidenceId": evidence_id }))
|
||||
.map_err(|error| error.to_string())?;
|
||||
window.eval(format!("window.dispatchEvent(new CustomEvent('hermes-screenshot-evidence', {{ detail: {payload} }}))")).map_err(|error| error.to_string())?;
|
||||
let monitor = app
|
||||
.get_webview_window("main")
|
||||
.and_then(|main| main.current_monitor().ok().flatten())
|
||||
.or_else(|| window.current_monitor().ok().flatten())
|
||||
.or_else(|| window.primary_monitor().ok().flatten());
|
||||
if let (Some(monitor), Ok(size)) = (monitor, window.outer_size()) {
|
||||
let work = monitor.work_area();
|
||||
let x =
|
||||
work.position.x + ((work.size.width as i64 - size.width as i64).max(0) / 2) as i32;
|
||||
let y = work.position.y
|
||||
+ ((work.size.height as i64 - size.height as i64).max(0) / 2) as i32;
|
||||
let _ = window.set_position(PhysicalPosition::new(x, y));
|
||||
}
|
||||
window.show().map_err(|error| error.to_string())?;
|
||||
let _ = window.set_focus();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn start_grant_watcher(app: AppHandle, tray_position: TrayPositionState) {
|
||||
thread::spawn(move || {
|
||||
let mut active_id = None::<String>;
|
||||
@@ -1549,6 +2040,7 @@ mod app {
|
||||
pair_host,
|
||||
test_host_route,
|
||||
open_management_from_grant,
|
||||
open_management_from_notice,
|
||||
forget_host,
|
||||
connect_daemon,
|
||||
disconnect_daemon,
|
||||
@@ -1562,7 +2054,17 @@ mod app {
|
||||
open_external_url,
|
||||
set_startup,
|
||||
set_daemon_autostart,
|
||||
set_computer_control_engine,
|
||||
set_cua_cursor_enabled,
|
||||
computer_cua_status,
|
||||
computer_cua_health,
|
||||
computer_cua_install,
|
||||
computer_cua_check_update,
|
||||
computer_cua_update,
|
||||
clear_activity,
|
||||
get_activity_screenshot,
|
||||
set_activity_screenshot_retention,
|
||||
present_activity_screenshot,
|
||||
present_grant_window
|
||||
])
|
||||
.setup(|app| {
|
||||
@@ -1571,6 +2073,7 @@ mod app {
|
||||
let tray_anchor = anchor.clone();
|
||||
let menu_anchor = anchor.clone();
|
||||
start_grant_watcher(app.handle().clone(), anchor);
|
||||
start_connection_watcher(app.handle().clone());
|
||||
start_activation_watcher(app.handle().clone());
|
||||
let tray_actions = start_tray_action_worker(app.handle().clone());
|
||||
let click_actions = tray_actions.clone();
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"$schema": "https://schema.tauri.app/config/2",
|
||||
"productName": "Hermes-Relay CLI UI",
|
||||
"version": "0.4.0-alpha.8",
|
||||
"version": "0.4.0-beta.2",
|
||||
"identifier": "com.axiomlabs.hermes-relay-tray",
|
||||
"build": {
|
||||
"beforeDevCommand": "npm run dev",
|
||||
@@ -47,6 +47,50 @@
|
||||
"alwaysOnTop": true,
|
||||
"skipTaskbar": true,
|
||||
"center": false
|
||||
},
|
||||
{
|
||||
"label": "notice",
|
||||
"title": "Hermes-Relay connection status",
|
||||
"width": 360,
|
||||
"height": 126,
|
||||
"resizable": false,
|
||||
"fullscreen": false,
|
||||
"decorations": false,
|
||||
"transparent": true,
|
||||
"backgroundColor": [
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"shadow": false,
|
||||
"visible": false,
|
||||
"alwaysOnTop": true,
|
||||
"skipTaskbar": true,
|
||||
"center": false
|
||||
},
|
||||
{
|
||||
"label": "evidence",
|
||||
"title": "Hermes-Relay screenshot evidence",
|
||||
"width": 900,
|
||||
"height": 650,
|
||||
"minWidth": 560,
|
||||
"minHeight": 420,
|
||||
"resizable": true,
|
||||
"fullscreen": false,
|
||||
"decorations": false,
|
||||
"transparent": true,
|
||||
"backgroundColor": [
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"shadow": true,
|
||||
"visible": false,
|
||||
"alwaysOnTop": true,
|
||||
"skipTaskbar": true,
|
||||
"center": true
|
||||
}
|
||||
],
|
||||
"security": {
|
||||
|
||||
@@ -102,6 +102,30 @@ fn management_window_owns_the_expected_narrow_surfaces() {
|
||||
"Experimental",
|
||||
"open_management_from_grant",
|
||||
"Open in UI",
|
||||
"computer_control_engine",
|
||||
"Computer control",
|
||||
"CUA Driver",
|
||||
"Not installed",
|
||||
"Incompatible",
|
||||
"Degraded",
|
||||
"Ready",
|
||||
"Animated agent cursor",
|
||||
"Background only",
|
||||
"Preferred structured engine",
|
||||
"Compatibility",
|
||||
"Active session",
|
||||
"Event timeline",
|
||||
"Post-action snapshot captured",
|
||||
"Authenticated control session",
|
||||
"computer_cua_status",
|
||||
"computer_cua_health",
|
||||
"computer_cua_install",
|
||||
"computer_cua_check_update",
|
||||
"computer_cua_update",
|
||||
"verified compatible driver",
|
||||
"set_computer_control_engine",
|
||||
"set_cua_cursor_enabled",
|
||||
"visual overlays—not additional Windows hardware pointers",
|
||||
] {
|
||||
assert!(
|
||||
source.contains(required) || security.contains(required),
|
||||
@@ -128,7 +152,9 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
|
||||
.and_then(|windows| windows.iter().find(|window| window["label"] == "grant"))
|
||||
.expect("grant window configuration");
|
||||
|
||||
assert!(ui.contains("isGrantWindow ? <GrantWindow /> : <ManagementApp />"));
|
||||
assert!(ui.contains("isGrantWindow ? <GrantWindow />"));
|
||||
assert!(ui.contains("isNoticeWindow ? <ConnectionNoticeWindow />"));
|
||||
assert!(ui.contains("isEvidenceWindow ? <EvidenceWindow />"));
|
||||
assert!(ui.contains("present_grant_window"));
|
||||
assert!(ui.contains("Remote access request"));
|
||||
assert!(native.contains("start_grant_watcher"));
|
||||
@@ -140,7 +166,8 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
|
||||
serde_json::json!([0, 0, 0, 0])
|
||||
);
|
||||
assert_eq!(grant_window["shadow"], false);
|
||||
assert!(ui.contains("(snapshot.daemon.configured_url ?? snapshot.daemon.url) === host.url"));
|
||||
assert!(ui.contains("const daemonTargetsHost"));
|
||||
assert!(ui.contains("(snapshot?.daemon.configured_url ?? snapshot?.daemon.url) === host.url"));
|
||||
assert!(ui.contains("formatGrantScope(grant.scope)"));
|
||||
assert!(ui.contains("grantAction(grant.scope)"));
|
||||
assert!(ui.contains("Requested action"));
|
||||
@@ -149,6 +176,10 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
|
||||
assert!(native.contains("management.completed"));
|
||||
assert!(native.contains("append_management_event"));
|
||||
assert!(native.contains("fn clear_activity"));
|
||||
assert!(native.contains("fn get_activity_screenshot"));
|
||||
assert!(native.contains("fn set_activity_screenshot_retention"));
|
||||
assert!(native.contains("fn start_connection_watcher"));
|
||||
assert!(native.contains("fn present_connection_notice"));
|
||||
assert!(native.contains("skip_serializing_if = \"Option::is_none\""));
|
||||
assert!(native.contains("popup_position"));
|
||||
assert!(native.contains("window.outer_size()"));
|
||||
@@ -160,6 +191,19 @@ fn grants_use_the_dedicated_card_and_host_changes_reconcile_daemon_truth() {
|
||||
assert!(native.contains("spawn_blocking(build_snapshot)"));
|
||||
assert!(native.contains("async fn check_desktop_update"));
|
||||
assert!(native.contains("async fn install_desktop_update"));
|
||||
for command in [
|
||||
"async fn computer_cua_status",
|
||||
"async fn computer_cua_health",
|
||||
"async fn computer_cua_install",
|
||||
"async fn computer_cua_check_update",
|
||||
"async fn computer_cua_update",
|
||||
] {
|
||||
assert!(
|
||||
native.contains(command),
|
||||
"blocking CUA management command: {command}"
|
||||
);
|
||||
}
|
||||
assert!(native.matches("spawn_blocking(||").count() >= 6);
|
||||
assert!(native.contains("fn set_host_capability"));
|
||||
assert!(native.contains("fn hardware_availability"));
|
||||
assert!(native.contains("HERMES_RELAY_CODE"));
|
||||
@@ -214,7 +258,7 @@ fn management_window_keeps_the_reviewed_compact_geometry() {
|
||||
assert!(config.contains("\"minWidth\": 340"));
|
||||
assert!(config.contains("\"minHeight\": 460"));
|
||||
assert!(config.contains("\"resizable\": false"));
|
||||
assert_eq!(config.matches("\"alwaysOnTop\": true").count(), 2);
|
||||
assert_eq!(config.matches("\"alwaysOnTop\": true").count(), 4);
|
||||
assert!(!ui.contains("toggleMaximize"));
|
||||
assert!(!ui.contains("data-tauri-drag-region"));
|
||||
assert!(!capability.contains("allow-start-dragging"));
|
||||
@@ -227,6 +271,10 @@ fn management_window_keeps_the_reviewed_compact_geometry() {
|
||||
assert!(ui.contains("packet packet-outbound"));
|
||||
assert!(ui.contains("packet packet-inbound"));
|
||||
assert!(ui.contains("connectionTransition"));
|
||||
assert!(ui.contains("daemon.reconnect_attempt"));
|
||||
assert!(ui.contains("Retry now"));
|
||||
assert!(ui.contains("Screenshot evidence"));
|
||||
assert!(ui.contains("View larger"));
|
||||
assert!(ui.contains("refreshInFlight"));
|
||||
assert!(ui.contains("Starting daemon and opening relay tunnel"));
|
||||
assert!(ui.contains("Stopping the local daemon"));
|
||||
|
||||
+207
-23
@@ -6,10 +6,10 @@ import {
|
||||
CircleHelp, Clock3, Download, ExternalLink, Eye, FileText, FolderOpen, Home, Info, Laptop, Link2,
|
||||
Copy, LoaderCircle, LogOut, Monitor, MousePointer2, Power, Radio, RefreshCw, Server,
|
||||
Settings, ShieldCheck, TerminalSquare, Trash2, Unplug, UserRoundX, X, Usb,
|
||||
LockKeyhole, SlidersHorizontal, Mic, Video
|
||||
LockKeyhole, SlidersHorizontal, Mic, Video, MousePointerClick, Maximize2, RotateCcw
|
||||
} from 'lucide-react'
|
||||
import logo from '../icons/icon-256.png'
|
||||
import type { AccessMode, Activity, AuthorizedClient, Capability, CapabilityMode, Host, PendingGrantRequest, Snapshot, UpdateReport } from './types'
|
||||
import type { AccessMode, Activity, AuthorizedClient, Capability, CapabilityMode, CuaHealthStatus, CuaManagementStatus, Host, PendingGrantRequest, Snapshot, UpdateReport } from './types'
|
||||
import { describeTransportSecurity } from '../../src/transportSecurity'
|
||||
import { displayLabel as displayRouteLabel, inferEndpointRole } from '../../src/endpoint'
|
||||
|
||||
@@ -18,7 +18,10 @@ type PendingAction = { type: 'access'; mode: AccessMode } | { type: 'capability'
|
||||
type RouteTestResult = { label: string; url: string; reachable: boolean; elapsed_ms: number; encrypted: boolean; security: string; error?: string | null }
|
||||
type RouteTestReport = { best?: RouteTestResult | null; routes?: RouteTestResult[] }
|
||||
|
||||
const isGrantWindow = '__TAURI_INTERNALS__' in window && getCurrentWindow().label === 'grant'
|
||||
const windowLabel = '__TAURI_INTERNALS__' in window ? getCurrentWindow().label : 'main'
|
||||
const isGrantWindow = windowLabel === 'grant'
|
||||
const isNoticeWindow = windowLabel === 'notice'
|
||||
const isEvidenceWindow = windowLabel === 'evidence'
|
||||
|
||||
const demo: Snapshot = {
|
||||
hosts: [{ url: 'wss://home-hermes.local:8767', name: 'Docker-Server', server_version: '1.6.3', endpoint_role: 'tailscale', paired_at: 1786458000, is_active: true, access_mode: 'full-access', capabilities: { commands: 'allow', files: 'allow', screen_input: 'allow', usb: 'allow', microphone: 'allow', camera: 'allow' } }],
|
||||
@@ -30,6 +33,11 @@ const demo: Snapshot = {
|
||||
cli_version: '0.4.0-alpha.4',
|
||||
cli_path: 'C:\\Program Files\\Hermes-Relay CLI\\hermes-relay.exe',
|
||||
hardware_availability: { usb: true, adb: true, microphone: false, camera: false },
|
||||
activity_screenshot_retention: { enabled: true, days: 7, count: 2, bytes: 842_000 },
|
||||
computer_control_engine: {
|
||||
selected: 'legacy', effective: 'legacy', available: false, state: 'not_installed',
|
||||
foreground_escalation_enabled: false, message: 'CUA Driver is not installed. Legacy input remains active.'
|
||||
},
|
||||
pending_grants: [],
|
||||
activity: [
|
||||
{ ts: Date.now() - 110_000, tool: 'desktop_powershell', ok: true, summary: 'exit 0', request_detail: '{\n "script": "Get-Process | Select-Object -First 5"\n}', stdout: 'Handles NPM(K) PM(K) WS(K) CPU(s) Id ProcessName\n------- ------ ----- ----- ------ -- -----------\n 412 31 74248 98312 4.18 812 powershell' },
|
||||
@@ -48,6 +56,8 @@ async function call<T>(command: string, args?: Record<string, unknown>): Promise
|
||||
if (command === 'check_desktop_update') return { current: '0.4.0-alpha.3', up_to_date: true, ahead_of_latest: true, latest_version: '0.4.0-alpha.2', installed: false, needs_restart: false } as T
|
||||
if (command === 'install_desktop_update') return { current: '0.4.0-alpha.3', up_to_date: true, ahead_of_latest: false, installed: true, needs_restart: true } as T
|
||||
if (command === 'test_host_route') return { best: { label: 'LAN', url: 'ws://172.16.24.250:8767', reachable: true, elapsed_ms: 36, encrypted: false, security: 'Unencrypted relay connection' }, routes: [] } as T
|
||||
if (command === 'computer_cua_status') return { installed: false, stale_path_shim: false, compatible: false, compatibility_reason: 'CUA Driver is not installed', supported_range: { minimum: '0.19.3', maximum_exclusive: '0.20.0' } } as T
|
||||
if (command === 'computer_cua_health') return { state: 'degraded', checkedAt: new Date().toISOString(), overall: 'degraded', reason: 'UI Automation desktop enumeration exceeded 2000ms.', temporaryWindowsCompatibility: true } as T
|
||||
return undefined as T
|
||||
}
|
||||
return invoke<T>(command, args)
|
||||
@@ -116,11 +126,56 @@ function activityName(tool: string): string {
|
||||
'host.access': 'Changed host access', 'host.pair': 'Pair host',
|
||||
'client.revoke': 'Deauthorized client', 'grant.resolve': 'Resolved access request',
|
||||
'daemon.start': 'Connected daemon', 'daemon.stop': 'Disconnected daemon',
|
||||
'daemon.reconnecting': 'Connection interrupted', 'daemon.reconnected': 'Tunnel restored',
|
||||
'daemon.disconnected': 'Tunnel disconnected', 'daemon.auth_failed': 'Connection failed',
|
||||
'daemon.restart': 'Restarted daemon', 'startup.change': 'Changed startup setting'
|
||||
}
|
||||
return names[tool] ?? tool.replace(/^desktop[._]/, '').replaceAll('_', ' ').replaceAll('.', ' ').replace(/\b\w/g, value => value.toUpperCase())
|
||||
}
|
||||
|
||||
function isComputerControl(entry: Activity): boolean {
|
||||
return entry.backend === 'cua' || entry.backend === 'legacy_compat' || entry.tool.startsWith('desktop_computer_')
|
||||
}
|
||||
|
||||
function controlActionLabel(entry: Activity): string {
|
||||
return (entry.action ?? activityName(entry.tool)).replaceAll('_', ' ').replace(/\b\w/g, value => value.toUpperCase())
|
||||
}
|
||||
|
||||
function controlVerificationLabel(value?: string): string {
|
||||
return value === 'snapshot_captured' ? 'Post-action snapshot captured'
|
||||
: value === 'failed' ? 'Verification failed'
|
||||
: value ? value.replaceAll('_', ' ') : 'Not reported'
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`
|
||||
if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB`
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`
|
||||
}
|
||||
|
||||
type ActivityStep = { title: string; detail: string; state: 'done' | 'failed' | 'pending' }
|
||||
|
||||
function activitySteps(entry: Activity): ActivityStep[] {
|
||||
if (isComputerControl(entry)) return [
|
||||
{ title: 'Authorized session', detail: entry.control_session_id ? 'Authenticated control session' : 'Authenticated by Hermes', state: 'done' },
|
||||
{ title: controlActionLabel(entry), detail: `${entry.backend === 'cua' ? 'CUA structured engine' : 'Windows input · Compatibility'} · ${entry.dispatch ?? 'background'}`, state: entry.ok ? 'done' : 'failed' },
|
||||
{ title: 'Verification', detail: controlVerificationLabel(entry.verification), state: entry.verification === 'failed' ? 'failed' : entry.verification ? 'done' : 'pending' }
|
||||
]
|
||||
const category = activityCategory(entry)
|
||||
if (category === 'system') {
|
||||
const reconnecting = entry.tool === 'daemon.reconnecting'
|
||||
return [
|
||||
{ title: 'Connection state changed', detail: entry.summary ?? activityName(entry.tool), state: entry.ok ? 'done' : 'failed' },
|
||||
...(reconnecting ? [{ title: 'Automatic retry', detail: 'Relay transport is retrying with backoff', state: 'pending' as const }] : [])
|
||||
]
|
||||
}
|
||||
return [
|
||||
{ title: 'Request received', detail: entry.args_preview ?? 'Validated local request', state: 'done' },
|
||||
{ title: activityName(entry.tool), detail: entry.aborted ? 'Stopped before completion' : entry.error ?? entry.summary ?? 'Local execution', state: entry.ok ? 'done' : 'failed' },
|
||||
{ title: 'Result recorded', detail: entry.ok ? (isNonZeroExit(entry) ? `Process exited ${activityExitCode(entry)}` : 'Evidence saved to local activity') : 'Failure details recorded', state: entry.ok ? 'done' : 'failed' }
|
||||
]
|
||||
}
|
||||
|
||||
function age(ts?: number): string {
|
||||
if (!ts) return 'Not seen yet'
|
||||
const seconds = Math.max(0, Math.floor(Date.now() / 1000) - ts)
|
||||
@@ -196,7 +251,54 @@ function hostAccessLabel(host: Host): string {
|
||||
}
|
||||
|
||||
export default function App() {
|
||||
return isGrantWindow ? <GrantWindow /> : <ManagementApp />
|
||||
return isGrantWindow ? <GrantWindow /> : isNoticeWindow ? <ConnectionNoticeWindow /> : isEvidenceWindow ? <EvidenceWindow /> : <ManagementApp />
|
||||
}
|
||||
|
||||
type ConnectionNotice = { tone: 'connected' | 'warning' | 'offline'; title: string; detail: string }
|
||||
|
||||
function ConnectionNoticeWindow() {
|
||||
const [notice, setNotice] = useState<ConnectionNotice | null>(null)
|
||||
const hideTimer = useRef<number | null>(null)
|
||||
useEffect(() => {
|
||||
const receive = (event: Event) => {
|
||||
const detail = (event as CustomEvent<ConnectionNotice>).detail
|
||||
setNotice(detail)
|
||||
if (hideTimer.current) window.clearTimeout(hideTimer.current)
|
||||
hideTimer.current = window.setTimeout(() => void getCurrentWindow().hide(), detail.tone === 'warning' ? 6500 : 4200)
|
||||
}
|
||||
window.addEventListener('hermes-connection-notice', receive)
|
||||
return () => { window.removeEventListener('hermes-connection-notice', receive); if (hideTimer.current) window.clearTimeout(hideTimer.current) }
|
||||
}, [])
|
||||
if (!notice) return null
|
||||
return <div className={`connection-notice-shell ${notice.tone}`}>
|
||||
<section className="connection-notice-card" role="status" aria-live="polite">
|
||||
<span className="connection-notice-icon">{notice.tone === 'connected' ? <Check /> : notice.tone === 'warning' ? <RotateCcw /> : <Unplug />}</span>
|
||||
<span><small>Hermes-Relay tunnel</small><strong>{notice.title}</strong><p>{notice.detail}</p></span>
|
||||
<button className="connection-notice-open" onClick={() => call('open_management_from_notice')}>Open</button>
|
||||
<button className="connection-notice-close" aria-label="Dismiss" onClick={() => getCurrentWindow().hide()}><X /></button>
|
||||
</section>
|
||||
</div>
|
||||
}
|
||||
|
||||
function EvidenceWindow() {
|
||||
const [evidenceId, setEvidenceId] = useState<string | null>(null)
|
||||
const [source, setSource] = useState<string | null>(null)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
useEffect(() => {
|
||||
const receive = (event: Event) => {
|
||||
const id = (event as CustomEvent<{ evidenceId: string }>).detail.evidenceId
|
||||
setEvidenceId(id); setSource(null); setError(null)
|
||||
void call<string>('get_activity_screenshot', { evidenceId: id }).then(setSource).catch(value => setError(String(value)))
|
||||
}
|
||||
const close = (event: KeyboardEvent) => { if (event.key === 'Escape') void getCurrentWindow().hide() }
|
||||
window.addEventListener('hermes-screenshot-evidence', receive)
|
||||
window.addEventListener('keydown', close)
|
||||
return () => { window.removeEventListener('hermes-screenshot-evidence', receive); window.removeEventListener('keydown', close) }
|
||||
}, [])
|
||||
return <div className="evidence-shell">
|
||||
<header><span><Eye /><strong>Screenshot evidence</strong><small>Stored locally with this activity event</small></span><button aria-label="Close screenshot" onClick={() => getCurrentWindow().hide()}><X /></button></header>
|
||||
<main>{source ? <img src={source} alt="Retained desktop screenshot" /> : error ? <div className="evidence-error"><AlertTriangle /><strong>Screenshot unavailable</strong><small>{error}</small></div> : <div className="evidence-loading"><LoaderCircle className="spin" /><span>{evidenceId ? 'Loading screenshot…' : 'Preparing viewer…'}</span></div>}</main>
|
||||
</div>
|
||||
}
|
||||
|
||||
function ManagementApp() {
|
||||
@@ -251,11 +353,13 @@ function ManagementApp() {
|
||||
finally { refreshInFlight.current = false }
|
||||
}, [])
|
||||
|
||||
const daemonRetrying = Boolean(snapshot?.daemon.running && snapshot.daemon.state === 'reconnecting')
|
||||
|
||||
useEffect(() => {
|
||||
refresh()
|
||||
const timer = window.setInterval(refresh, connectionTransition ? 350 : 5000)
|
||||
const timer = window.setInterval(refresh, connectionTransition ? 350 : daemonRetrying ? 1000 : 5000)
|
||||
return () => window.clearInterval(timer)
|
||||
}, [refresh, connectionTransition])
|
||||
}, [refresh, connectionTransition, daemonRetrying])
|
||||
|
||||
useEffect(() => {
|
||||
const close = (event: MouseEvent) => {
|
||||
@@ -266,7 +370,11 @@ function ManagementApp() {
|
||||
}, [])
|
||||
|
||||
const host = useMemo(() => snapshot?.hosts.find(item => item.url === selectedUrl) ?? null, [snapshot, selectedUrl])
|
||||
const connected = Boolean(snapshot?.daemon.running && snapshot.daemon.state === 'connected' && host && (snapshot.daemon.configured_url ?? snapshot.daemon.url) === host.url)
|
||||
const daemonTargetsHost = Boolean(host && (snapshot?.daemon.configured_url ?? snapshot?.daemon.url) === host.url)
|
||||
const daemonActive = Boolean(snapshot?.daemon.running && daemonTargetsHost)
|
||||
const connected = Boolean(daemonActive && snapshot?.daemon.state === 'connected')
|
||||
const reconnecting = Boolean(daemonActive && snapshot?.daemon.state === 'reconnecting')
|
||||
const retrySeconds = reconnecting && snapshot?.daemon.retry_at ? Math.max(0, snapshot.daemon.retry_at - Math.floor(Date.now() / 1000)) : null
|
||||
|
||||
useEffect(() => {
|
||||
if (page !== 'host-detail' || !detailUrl) return
|
||||
@@ -287,8 +395,8 @@ function ManagementApp() {
|
||||
|
||||
async function changeConnection() {
|
||||
if (busy) return
|
||||
const command = connected ? 'disconnect_daemon' : 'connect_daemon'
|
||||
const transition = connected ? 'disconnecting' : 'connecting'
|
||||
const command = daemonActive ? 'disconnect_daemon' : 'connect_daemon'
|
||||
const transition = daemonActive ? 'disconnecting' : 'connecting'
|
||||
setBusy(command)
|
||||
setConnectionTransition(transition)
|
||||
setError(null)
|
||||
@@ -303,6 +411,16 @@ function ManagementApp() {
|
||||
}
|
||||
}
|
||||
|
||||
async function retryConnection() {
|
||||
if (busy) return
|
||||
setBusy('restart_daemon')
|
||||
setConnectionTransition('connecting')
|
||||
setError(null)
|
||||
try { await call('restart_daemon'); await refresh() }
|
||||
catch (e) { setError(String(e)) }
|
||||
finally { setConnectionTransition(null); setBusy(null) }
|
||||
}
|
||||
|
||||
async function testRoute(remote: string) {
|
||||
setBusy('test_host_route')
|
||||
setRouteTest(null)
|
||||
@@ -418,7 +536,7 @@ function ManagementApp() {
|
||||
|
||||
<main className="content" ref={contentRef}>
|
||||
{page === 'overview' && <>
|
||||
<section className={`connection-route ${connected ? 'online' : 'offline'} ${connectionTransition ?? ''} ${snapshot.daemon.active_route === 'plugin_proxy' ? 'secure-link' : ''}`} aria-busy={connectionTransition !== null}>
|
||||
<section className={`connection-route ${connected ? 'online' : reconnecting ? 'retrying' : 'offline'} ${connectionTransition ?? ''} ${snapshot.daemon.active_route === 'plugin_proxy' ? 'secure-link' : ''}`} aria-busy={connectionTransition !== null || reconnecting}>
|
||||
{snapshot.hosts.length === 0 ?
|
||||
<button className="empty-pair" onClick={() => openPair()}><Link2 /><span><strong>Pair host</strong><small>Connect this PC to a Hermes instance</small></span><ChevronRight /></button> :
|
||||
<div className="route-grid" ref={selectorRef}>
|
||||
@@ -445,10 +563,11 @@ function ManagementApp() {
|
||||
const activeRole = snapshot.daemon.active_route ?? host?.endpoint_role ?? inferEndpointRole(snapshot.daemon.url ?? host?.url ?? '')
|
||||
const security = describeTransportSecurity(snapshot.daemon.url ?? host?.url ?? '', activeRole)
|
||||
return <div className={`route-status ${connected && !security.encrypted ? 'insecure' : ''}`} aria-live="polite" aria-atomic="true">
|
||||
<strong>{connectionTransition === 'connecting' ? 'Connecting' : connectionTransition === 'disconnecting' ? 'Disconnecting' : connected ? 'Connected' : 'Disconnected'}</strong>
|
||||
<strong>{connectionTransition === 'connecting' ? 'Connecting' : connectionTransition === 'disconnecting' ? 'Disconnecting' : reconnecting ? 'Reconnecting' : connected ? 'Connected' : 'Disconnected'}</strong>
|
||||
{connected && <button className={`route-badge ${security.kind}`} aria-expanded={routeDetailsOpen} onClick={() => setRouteDetailsOpen(open => !open)}><ShieldCheck />{displayRouteLabel(activeRole ?? 'custom')}<ChevronDown /></button>}
|
||||
{connectionTransition && <small>{connectionTransition === 'connecting' ? 'Starting daemon and opening relay tunnel' : 'Closing relay tunnel'}</small>}
|
||||
{!connected && !connectionTransition && <small>Relay connection offline</small>}
|
||||
{reconnecting && !connectionTransition && <><small>Attempt {snapshot.daemon.reconnect_attempt ?? 1}{retrySeconds !== null ? ` · retry in ${retrySeconds}s` : ' · retry scheduled'}</small><button className="retry-now" disabled={busy !== null} onClick={() => void retryConnection()}><RefreshCw /> Retry now</button></>}
|
||||
{!connected && !reconnecting && !connectionTransition && <small>{snapshot.daemon.last_error ?? 'Relay connection offline'}</small>}
|
||||
{connected && routeDetailsOpen && <aside className="route-detail-card"><div><ShieldCheck /><span><strong>{displayRouteLabel(activeRole ?? 'custom')}</strong><small>{security.detail}</small></span></div><dl><div><dt>Security</dt><dd>{security.label}</dd></div><div><dt>Endpoint</dt><dd title={snapshot.daemon.url ?? undefined}>{snapshot.daemon.url ?? 'Not reported'}</dd></div></dl><button className="route-test-button" disabled={busy === 'test_host_route'} onClick={() => host && testRoute(host.url)}>{busy === 'test_host_route' ? <LoaderCircle className="spin" /> : routeTest ? <RefreshCw /> : <ActivityIcon />}<span>{busy === 'test_host_route' ? <><strong>Testing connection…</strong><small>Checking every saved route</small></> : <><strong>{routeTest ? 'Test again' : 'Test connection'}</strong><small>Measure reachability and latency</small></>}</span></button>{routeTest && <div className={`route-test-result ${routeTest.best ? 'reachable' : 'unreachable'}`} aria-live="polite">{routeTest.best ? <><div className="route-test-summary"><span><Check /></span><strong>{routeTest.best.label} reachable</strong><em>{routeTest.best.elapsed_ms} ms</em></div><dl><div><dt>Protection</dt><dd className={routeTest.best.encrypted ? 'secure' : 'warning'}>{routeTest.best.security}</dd></div><div><dt>Tested endpoint</dt><dd title={routeTest.best.url}>{routeTest.best.url}</dd></div></dl><small>{Math.max(1, routeTest.routes?.length ?? 0)} saved route{(routeTest.routes?.length ?? 0) === 1 ? '' : 's'} checked</small></> : <div className="route-test-summary"><span><X /></span><strong>No route reachable</strong><em>Check host</em></div>}</div>}</aside>}
|
||||
</div>
|
||||
})()}
|
||||
@@ -470,9 +589,9 @@ function ManagementApp() {
|
||||
</button>
|
||||
</section>}
|
||||
|
||||
<button className={`tunnel-button ${connectionTransition ? 'pending' : ''}`} disabled={busy !== null} aria-busy={connectionTransition !== null} onClick={() => void changeConnection()}>
|
||||
{connectionTransition ? <LoaderCircle className="spin" /> : <Power />}
|
||||
<span><strong>{connectionTransition === 'connecting' ? 'Connecting…' : connectionTransition === 'disconnecting' ? 'Disconnecting…' : connected ? 'Disconnect Tunnel' : 'Connect Tunnel'}</strong>{connectionTransition && <small>{connectionTransition === 'connecting' ? 'Waiting for the relay' : 'Stopping the local daemon'}</small>}</span>
|
||||
<button className={`tunnel-button ${connectionTransition || reconnecting ? 'pending' : ''}`} disabled={busy !== null} aria-busy={connectionTransition !== null} onClick={() => void changeConnection()}>
|
||||
{connectionTransition ? <LoaderCircle className="spin" /> : reconnecting ? <Unplug /> : <Power />}
|
||||
<span><strong>{connectionTransition === 'connecting' ? 'Connecting…' : connectionTransition === 'disconnecting' ? 'Disconnecting…' : reconnecting ? 'Disconnect Tunnel' : connected ? 'Disconnect Tunnel' : 'Connect Tunnel'}</strong>{(connectionTransition || reconnecting) && <small>{connectionTransition === 'connecting' ? 'Waiting for the relay' : connectionTransition === 'disconnecting' ? 'Stopping the local daemon' : 'Automatic retry remains active'}</small>}</span>
|
||||
</button>
|
||||
|
||||
<section className="activity-section">
|
||||
@@ -489,7 +608,7 @@ function ManagementApp() {
|
||||
{page === 'hosts' && <HostsPage hosts={snapshot.hosts} selected={host} onOpen={url => { setDetailUrl(url); setSelectedUrl(url); setPage('host-detail') }} onPair={() => openPair()} />}
|
||||
{page === 'pair-host' && <PairHostPage initialUrl={pairInitialUrl} busy={busy === 'pair_host'} onBack={() => setPage('hosts')} onPair={pairHost} />}
|
||||
{page === 'host-detail' && <HostDetailPage host={snapshot.hosts.find(item => item.url === detailUrl) ?? null} clients={clients} busy={busy !== null} onBack={() => setPage('hosts')} onConnect={connectHost} onRename={(remote, name) => action('rename_host', { remote, name })} onAccess={() => { setPolicyBack('host-detail'); setPage('access') }} onCapabilities={() => { setPolicyBack('host-detail'); setPage('capabilities') }} onRevoke={(remote, client) => setPending({ type: 'revoke', client, remote })} onRepair={host => setPending({ type: 'repair', host })} onForget={host => setPending({ type: 'forget', host })} />}
|
||||
{page === 'settings' && <SettingsPage daemon={snapshot.daemon} startup={snapshot.startup_enabled} daemonAutostart={snapshot.daemon_autostart_enabled ?? false} activity={snapshot.activity} onAction={action} onStartup={value => action('set_startup', { enabled: value })} onDaemonAutostart={value => action('set_daemon_autostart', { enabled: value })} onHelp={() => setPage('help')} onViewActivity={() => { setActivityBack('settings'); setPage('activity') }} onOpenActivity={entry => { setSelectedActivity(entry); setActivityDetailBack('settings'); setPage('activity-detail') }} />}
|
||||
{page === 'settings' && <SettingsPage daemon={snapshot.daemon} computerControl={snapshot.computer_control_engine ?? null} startup={snapshot.startup_enabled} daemonAutostart={snapshot.daemon_autostart_enabled ?? false} activity={snapshot.activity} screenshotRetention={snapshot.activity_screenshot_retention} onAction={action} onStartup={value => action('set_startup', { enabled: value })} onDaemonAutostart={value => action('set_daemon_autostart', { enabled: value })} onHelp={() => setPage('help')} onViewActivity={() => { setActivityBack('settings'); setPage('activity') }} onOpenActivity={entry => { setSelectedActivity(entry); setActivityDetailBack('settings'); setPage('activity-detail') }} />}
|
||||
{page === 'help' && <HelpPage snapshot={snapshot} host={host} onBack={() => { setSelectedUrl(snapshot.active_url ?? null); setPage('settings') }} onAction={action} />}
|
||||
{page === 'activity' && <ActivityPage entries={snapshot.activity} host={host} onBack={() => setPage(activityBack)} onClear={() => setPending({ type: 'clear-activity' })} onOpen={entry => { setSelectedActivity(entry); setActivityDetailBack('activity'); setPage('activity-detail') }} />}
|
||||
{page === 'activity-detail' && <ActivityDetailPage entry={selectedActivity} host={host} onBack={() => setPage(activityDetailBack)} />}
|
||||
@@ -623,12 +742,14 @@ function ActivityList({ entries, host, onOpen }: { entries: Activity[]; host: Ho
|
||||
const warning = isNonZeroExit(entry)
|
||||
const key = entry.request_id ?? `${entry.ts}-${i}`
|
||||
const Icon = category === 'command' ? TerminalSquare : category === 'files' ? FileText : category === 'screen' ? Eye : category === 'input' ? MousePointer2 : category === 'devices' ? Usb : category === 'system' ? LogOut : ActivityIcon
|
||||
const detail = entry.error ?? entry.summary ?? (entry.aborted ? 'Request aborted' : 'Completed')
|
||||
const computerControl = isComputerControl(entry)
|
||||
const target = entry.target_app ?? entry.target_title
|
||||
const detail = entry.error ?? (computerControl ? `${entry.backend === 'cua' ? 'CUA' : 'Compatibility'} · ${entry.dispatch ?? 'background'}${target ? ` · ${target}` : ''}` : entry.summary) ?? (entry.aborted ? 'Request aborted' : 'Completed')
|
||||
const eventHost = entry.host_url ? displayHost(entry.host_url) : host?.name ?? 'Local daemon'
|
||||
return <article className="activity-item" key={key}>
|
||||
<button className="activity-row" disabled={!onOpen} onClick={() => onOpen?.(entry)}>
|
||||
<span className={`activity-icon ${attention || warning ? 'amber' : category === 'system' ? 'green' : 'violet'}`}><Icon /></span>
|
||||
<span className="activity-copy"><strong>{activityName(entry.tool)}</strong><small className={attention ? 'attention' : warning ? 'warning' : ''}>{detail} · {eventHost}</small></span>
|
||||
<span className="activity-copy"><strong>{computerControl ? controlActionLabel(entry) : activityName(entry.tool)}</strong><small className={attention ? 'attention' : warning ? 'warning' : ''}>{detail} · {eventHost}</small></span>
|
||||
<span className="activity-tail"><time>{formatTime(entry.ts)}</time>{onOpen && <ChevronRight />}</span>
|
||||
</button>
|
||||
</article>
|
||||
@@ -663,22 +784,29 @@ function ActivityPage({ entries, host, onBack, onClear, onOpen }: { entries: Act
|
||||
}
|
||||
|
||||
function ActivityDetailPage({ entry, host, onBack }: { entry: Activity | null; host: Host | null; onBack: () => void }) {
|
||||
const [evidenceError, setEvidenceError] = useState<string | null>(null)
|
||||
if (!entry) return <section className="page-panel"><button className="back-button" onClick={onBack}><ArrowLeft /> Back to Activity</button><div className="large-empty"><ActivityIcon /><h2>Event unavailable</h2></div></section>
|
||||
const attention = needsAttention(entry)
|
||||
const warning = isNonZeroExit(entry)
|
||||
const status = entry.aborted ? 'Aborted' : !entry.ok ? 'Failed' : warning ? `Exit ${activityExitCode(entry)}` : 'Completed'
|
||||
const eventHost = entry.host_url ? displayHost(entry.host_url) : host?.name ?? 'Local daemon'
|
||||
const computerControl = isComputerControl(entry)
|
||||
const steps = activitySteps(entry)
|
||||
const blocks = [
|
||||
['Request', entry.request_detail ?? entry.args_preview, entry.request_truncated],
|
||||
['Standard output', entry.stdout, entry.stdout_truncated],
|
||||
['Standard error', entry.stderr, entry.stderr_truncated],
|
||||
['Result', entry.result_detail, entry.result_truncated],
|
||||
['Error', entry.error, false]
|
||||
['Result', entry.result_detail, entry.result_truncated]
|
||||
] as const
|
||||
return <section className="page-panel activity-detail-page">
|
||||
<button className="back-button" onClick={onBack}><ArrowLeft /> Back to Activity</button>
|
||||
<div className="activity-detail-title"><span className={`activity-icon ${attention || warning ? 'amber' : 'violet'}`}><TerminalSquare /></span><span><p>{activityCategory(entry)}</p><h1>{activityName(entry.tool)}</h1><small>{eventHost}</small></span></div>
|
||||
<div className="activity-detail-title"><span className={`activity-icon ${attention || warning ? 'amber' : 'violet'}`}>{computerControl ? <MousePointerClick /> : <TerminalSquare />}</span><span><p>{computerControl ? 'Computer control' : activityCategory(entry)}</p><h1>{computerControl ? controlActionLabel(entry) : activityName(entry.tool)}</h1><small>{eventHost}</small></span></div>
|
||||
<dl className="activity-detail-meta"><div><dt>Status</dt><dd className={attention ? 'attention' : warning ? 'warning' : 'success'}>{status}</dd></div><div><dt>When</dt><dd>{formatDateTime(entry.ts)}</dd></div><div><dt>Duration</dt><dd>{formatDuration(entry.duration_ms)}</dd></div></dl>
|
||||
<section className="control-timeline" aria-label="Event timeline">{steps.map((step, index) => <div className={step.state} key={`${step.title}-${index}`}><i /><span><strong>{step.title}</strong><small>{step.detail}</small></span></div>)}</section>
|
||||
{computerControl && (entry.target_app || entry.target_title || entry.target_pid || entry.target_window_id) && <dl className="control-target"><div><dt>Application</dt><dd>{entry.target_app ?? 'Not reported'}</dd></div><div><dt>Window</dt><dd title={entry.target_title}>{entry.target_title ?? 'Not reported'}</dd></div><div><dt>Target</dt><dd>{entry.target_pid ? `PID ${entry.target_pid}` : 'PID —'} · {entry.target_window_id ? `Window ${entry.target_window_id}` : 'Window —'}</dd></div></dl>}
|
||||
{entry.error && <aside className="activity-error-callout" role="alert"><AlertTriangle /><span><strong>{entry.aborted ? 'Action stopped' : 'Action failed'}</strong><small>{entry.error}</small></span></aside>}
|
||||
{entry.screenshot_evidence_id && <button className="screenshot-evidence-card" onClick={() => { setEvidenceError(null); void call('present_activity_screenshot', { evidenceId: entry.screenshot_evidence_id }).catch(error => setEvidenceError(String(error))) }}><span><Eye /><strong>Screenshot captured</strong><small>{entry.screenshot_width && entry.screenshot_height ? `${entry.screenshot_width} × ${entry.screenshot_height} PNG` : 'Retained local evidence'}</small></span><em><Maximize2 /> View larger</em></button>}
|
||||
{evidenceError && <aside className="activity-error-callout"><AlertTriangle /><span><strong>Screenshot unavailable</strong><small>{evidenceError}</small></span></aside>}
|
||||
<div className="activity-output-list">{blocks.filter(([, value]) => value).map(([label, value, truncated]) => <section key={label}><header><strong>{label}</strong>{truncated && <em>Truncated</em>}</header><pre>{value}</pre></section>)}</div>
|
||||
{entry.request_id && <div className="activity-request-id">Request ID {entry.request_id}</div>}
|
||||
</section>
|
||||
@@ -783,10 +911,32 @@ function HostDetailPage({ host, clients, busy, onBack, onConnect, onRename, onAc
|
||||
</section>
|
||||
}
|
||||
|
||||
function SettingsPage({ daemon, startup, daemonAutostart, activity, onAction, onStartup, onDaemonAutostart, onHelp, onViewActivity, onOpenActivity }: { daemon: Snapshot['daemon']; startup: boolean; daemonAutostart: boolean; activity: Activity[]; onAction: (name: string, args?: Record<string, unknown>) => Promise<unknown>; onStartup: (value: boolean) => void; onDaemonAutostart: (value: boolean) => void; onHelp: () => void; onViewActivity: () => void; onOpenActivity: (entry: Activity) => void }) {
|
||||
function SettingsPage({ daemon, computerControl, startup, daemonAutostart, activity, screenshotRetention, onAction, onStartup, onDaemonAutostart, onHelp, onViewActivity, onOpenActivity }: { daemon: Snapshot['daemon']; computerControl: Snapshot['computer_control_engine']; startup: boolean; daemonAutostart: boolean; activity: Activity[]; screenshotRetention: Snapshot['activity_screenshot_retention']; onAction: (name: string, args?: Record<string, unknown>) => Promise<unknown>; onStartup: (value: boolean) => void; onDaemonAutostart: (value: boolean) => void; onHelp: () => void; onViewActivity: () => void; onOpenActivity: (entry: Activity) => void }) {
|
||||
const [update, setUpdate] = useState<UpdateReport | null>(null)
|
||||
const [updateBusy, setUpdateBusy] = useState<'check' | 'install' | null>(null)
|
||||
const [updateError, setUpdateError] = useState<string | null>(null)
|
||||
const [cuaManagement, setCuaManagement] = useState<CuaManagementStatus | null>(null)
|
||||
const [cuaHealth, setCuaHealth] = useState<CuaHealthStatus | null>(null)
|
||||
const [cuaBusy, setCuaBusy] = useState<'status' | 'health' | 'install' | 'check' | 'update' | null>(null)
|
||||
const [cuaError, setCuaError] = useState<string | null>(null)
|
||||
|
||||
const cuaOperation = useCallback(async (operation: 'status' | 'install' | 'check' | 'update') => {
|
||||
setCuaBusy(operation)
|
||||
try {
|
||||
setCuaManagement(await call<CuaManagementStatus>(operation === 'status' ? 'computer_cua_status' : operation === 'install' ? 'computer_cua_install' : operation === 'check' ? 'computer_cua_check_update' : 'computer_cua_update'))
|
||||
setCuaError(null)
|
||||
} catch (error) { setCuaError(String(error).replace(/^Error:\s*/i, '')) }
|
||||
finally { setCuaBusy(null) }
|
||||
}, [])
|
||||
|
||||
const recheckCuaHealth = useCallback(async () => {
|
||||
setCuaBusy('health')
|
||||
try {
|
||||
setCuaHealth(await call<CuaHealthStatus>('computer_cua_health'))
|
||||
setCuaError(null)
|
||||
} catch (error) { setCuaError(String(error).replace(/^Error:\s*/i, '')) }
|
||||
finally { setCuaBusy(null) }
|
||||
}, [])
|
||||
|
||||
const checkUpdate = useCallback(async () => {
|
||||
setUpdateBusy('check')
|
||||
@@ -803,6 +953,7 @@ function SettingsPage({ daemon, startup, daemonAutostart, activity, onAction, on
|
||||
}, [])
|
||||
|
||||
useEffect(() => { void checkUpdate() }, [checkUpdate])
|
||||
useEffect(() => { void cuaOperation('status') }, [cuaOperation])
|
||||
|
||||
const updateSummary = updateError
|
||||
? updateError
|
||||
@@ -816,12 +967,44 @@ function SettingsPage({ daemon, startup, daemonAutostart, activity, onAction, on
|
||||
? `${update.current} → ${update.latest_version} available`
|
||||
: 'Check the desktop release channel.'
|
||||
|
||||
const cuaReady = computerControl?.available === true && computerControl.state === 'ready'
|
||||
const engineState = computerControl?.state ?? 'not_installed'
|
||||
const engineLabel = engineState === 'not_installed' ? 'Not installed' : engineState === 'incompatible' ? 'Incompatible' : engineState === 'degraded' ? 'Degraded' : engineState === 'ready' ? 'Ready' : 'Unavailable'
|
||||
const engineDetail = computerControl?.message
|
||||
?? (engineState === 'not_installed' ? 'CUA Driver is not installed. Windows input remains available for compatibility.'
|
||||
: engineState === 'incompatible' ? 'The installed CUA Driver version is not compatible with this CLI.'
|
||||
: engineState === 'degraded' ? 'CUA Driver reported a health problem. Windows compatibility input remains active.'
|
||||
: engineState === 'ready' ? `CUA Driver ${computerControl?.version ?? ''} is compatible and healthy.`.trim()
|
||||
: 'CUA Driver status could not be verified. Hermes uses the safe fallback.')
|
||||
const effectiveEngine = computerControl?.effective === 'cua' ? 'CUA Driver' : 'Windows input'
|
||||
const engineRole = computerControl?.effective === 'cua' ? 'Preferred structured engine' : 'Compatibility'
|
||||
const activeSessions = computerControl?.active_sessions ?? 0
|
||||
const activeBackend = computerControl?.active_backend === 'cua' ? 'CUA active'
|
||||
: computerControl?.active_backend === 'legacy_compat' ? 'Compatibility active'
|
||||
: computerControl?.active_backend === 'mixed' ? 'Mixed backends' : 'Idle'
|
||||
const healthLabel = cuaHealth?.state === 'healthy' ? 'Healthy' : cuaHealth?.state === 'degraded' ? 'Degraded' : cuaHealth?.state === 'error' ? 'Check failed' : 'Not checked'
|
||||
const healthDetail = cuaHealth?.reason
|
||||
?? (cuaHealth?.state === 'healthy'
|
||||
? 'The latest explicit accessibility check passed.'
|
||||
: 'Optional diagnostic; it does not disable the runtime while the temporary Windows workaround is active.')
|
||||
|
||||
return <section className="page-panel settings-page"><div className="page-title"><div><p>Local management</p><h1>Settings</h1></div></div>
|
||||
<div className="settings-group"><h2>Relay daemon</h2><div className="settings-card"><div className="setting-row"><span><strong>Daemon status</strong><small>{daemon.running ? `${daemon.state} · ${daemon.privilege ?? 'user'}` : 'Stopped'}</small></span><button className="compact-button" onClick={() => onAction('restart_daemon')}><RefreshCw /> Restart</button></div><div className="setting-row"><span><strong>{daemon.privilege === 'administrator' ? 'Administrator mode' : 'User mode'}</strong><small>{daemon.privilege === 'administrator' ? 'Remote actions inherit elevated rights.' : 'Recommended for normal operation.'}</small></span><button className={`compact-button privilege-action ${daemon.privilege === 'administrator' ? '' : 'admin-action'}`} onClick={() => onAction(daemon.privilege === 'administrator' ? 'restart_daemon_as_user' : 'restart_daemon_as_administrator')}>{daemon.privilege === 'administrator' ? 'Return to user mode' : 'Restart as Administrator…'}</button></div><label className="setting-row toggle-row"><span><strong>Start UI at sign-in</strong><small>Launch the tray after you sign in.</small></span><input type="checkbox" checked={startup} onChange={e => onStartup(e.target.checked)} /><i /></label><label className="setting-row toggle-row"><span><strong>Start daemon with UI</strong><small>Connect remote access when the tray starts.</small></span><input type="checkbox" checked={daemonAutostart} onChange={e => onDaemonAutostart(e.target.checked)} /><i /></label></div></div>
|
||||
<div className="settings-group"><h2>Computer control</h2><div className={`settings-card engine-card engine-${engineState}`}>
|
||||
<div className="engine-status"><span className="setting-icon"><MousePointerClick /></span><span><strong>{effectiveEngine}</strong><small>{engineRole} · {engineDetail}</small></span><em>{engineLabel}</em></div>
|
||||
{cuaReady && <div className="engine-options">
|
||||
<div className="engine-choice"><span><strong>Control engine</strong><small>CUA is preferred; Windows input is the compatibility fallback.</small></span><div role="radiogroup" aria-label="Computer control engine"><button role="radio" aria-checked={computerControl?.selected === 'cua'} className={computerControl?.selected === 'cua' ? 'active' : ''} onClick={() => onAction('set_computer_control_engine', { engine: 'cua' })}>CUA</button><button role="radio" aria-checked={computerControl?.selected !== 'cua'} className={computerControl?.selected !== 'cua' ? 'active' : ''} onClick={() => onAction('set_computer_control_engine', { engine: 'legacy' })}>Compatibility</button></div></div>
|
||||
<div className="engine-live"><span><strong>{activeSessions}</strong><small>Active session{activeSessions === 1 ? '' : 's'}</small></span><em className={activeSessions ? 'active' : ''}><i /> {activeBackend}</em></div>
|
||||
<label className="setting-row toggle-row"><span><strong>Animated agent cursor</strong><small>Labeled · smooth glide · click pulse. It does not move your physical mouse.</small></span><input type="checkbox" disabled={computerControl?.selected !== 'cua'} checked={computerControl?.selected === 'cua' && computerControl.cursor_enabled === true} onChange={e => onAction('set_cua_cursor_enabled', { enabled: e.target.checked })} /><i /></label>
|
||||
<div className="setting-row background-only"><span><strong>Window interaction</strong><small>CUA actions stay in the background and never bring an app forward.</small></span><em>Background only</em></div>
|
||||
<div className="setting-row cua-health"><span><strong>Accessibility health</strong><small>{healthDetail}</small></span><div><em className={`health-${cuaHealth?.state ?? 'unchecked'}`}>{healthLabel}</em><button className="compact-button" disabled={cuaBusy !== null} onClick={() => void recheckCuaHealth()}>{cuaBusy === 'health' ? <LoaderCircle className="spin" /> : <RefreshCw />} Recheck</button></div></div>
|
||||
</div>}
|
||||
<div className="cua-maintenance"><span><strong>{cuaManagement?.installed ? `CUA Driver ${cuaManagement.current_version ?? ''}`.trim() : 'CUA Driver'}</strong><small>{cuaError ?? cuaManagement?.update?.error ?? cuaManagement?.compatibility_reason ?? (cuaManagement?.update?.update_available ? `${cuaManagement.update.latest_version} available` : cuaManagement?.installed ? 'Installed from the verified upstream release.' : 'Install the verified compatible driver explicitly.')}</small></span><div>{!cuaManagement?.installed ? <button disabled={cuaBusy !== null} onClick={() => void cuaOperation('install')}>{cuaBusy === 'install' ? <LoaderCircle className="spin" /> : <Download />} Install</button> : cuaManagement.update?.update_available && cuaManagement.update.compatible ? <button disabled={cuaBusy !== null} onClick={() => void cuaOperation('update')}>{cuaBusy === 'update' ? <LoaderCircle className="spin" /> : <Download />} Update</button> : <button disabled={cuaBusy !== null} onClick={() => void cuaOperation('check')}>{cuaBusy === 'check' || cuaBusy === 'status' ? <LoaderCircle className="spin" /> : <RefreshCw />} Check</button>}</div></div>
|
||||
</div><p className="group-help engine-help"><ShieldCheck /> CUA is the preferred structured engine. Hermes permissions, grants, audit, and emergency stop remain in control.</p></div>
|
||||
<div className="settings-group"><h2>CLI & diagnostics</h2><div className="settings-card quick-action-grid"><button onClick={() => onAction('open_terminal')}><TerminalSquare /><span>Open terminal</span></button><button onClick={() => onAction('open_cli_terminal')}><Bot /><span>Open Hermes CLI</span></button><button onClick={() => onAction('open_logs')}><FolderOpen /><span>View daemon log</span></button><button onClick={() => onAction('run_diagnostics')}><ActivityIcon /><span>Run diagnostics</span></button></div></div>
|
||||
<div className="settings-group"><h2>Updates</h2><div className={`settings-card update-card ${updateError ? 'error' : update?.ahead_of_latest ? 'ahead' : update?.up_to_date ? 'current' : ''}`}><div className="setting-row update-row"><span><strong>Hermes-Relay CLI UI</strong><small>{updateSummary}</small></span>{update && !update.up_to_date && !update.ahead_of_latest && !update.installed ? <button className="compact-button update-button" disabled={updateBusy !== null} onClick={installUpdate}>{updateBusy === 'install' ? <LoaderCircle className="spin" /> : <Download />} Install</button> : <button className="compact-button" disabled={updateBusy !== null} onClick={checkUpdate}>{updateBusy === 'check' ? <LoaderCircle className="spin" /> : <RefreshCw />} Check</button>}</div></div><p className="group-help update-help">Updates the management UI and CLI together, then restarts the tray automatically.</p></div>
|
||||
<button className="about-link" onClick={onHelp}><span className="setting-icon"><Info /></span><span><strong>Help & About</strong><small>Versions, documentation and troubleshooting.</small></span><ChevronRight /></button>
|
||||
<div className="settings-group"><div className="settings-group-heading"><h2>Activity</h2><button onClick={onViewActivity}>View all <ChevronRight /></button></div><p className="group-help">Recent remote actions recorded on this PC.</p><div className="settings-card padded"><ActivityList entries={activity.slice(-3).reverse()} host={null} onOpen={onOpenActivity} /></div></div>
|
||||
<div className="settings-group"><div className="settings-group-heading"><h2>Activity</h2><button onClick={onViewActivity}>View all <ChevronRight /></button></div><p className="group-help">Recent remote actions recorded on this PC.</p><div className="settings-card activity-retention-card"><div className="setting-row"><span><strong>Screenshot evidence</strong><small>{screenshotRetention.count} retained · {formatBytes(screenshotRetention.bytes)} · stored only on this PC</small></span><div className="retention-options" role="radiogroup" aria-label="Screenshot evidence retention">{([{ label: 'Off', enabled: false, days: 7 }, { label: '1d', enabled: true, days: 1 }, { label: '7d', enabled: true, days: 7 }, { label: '30d', enabled: true, days: 30 }] as const).map(option => <button key={option.label} role="radio" aria-checked={screenshotRetention.enabled === option.enabled && (!option.enabled || screenshotRetention.days === option.days)} className={screenshotRetention.enabled === option.enabled && (!option.enabled || screenshotRetention.days === option.days) ? 'active' : ''} onClick={() => onAction('set_activity_screenshot_retention', { enabled: option.enabled, days: option.days })}>{option.label}</button>)}</div></div></div><div className="settings-card padded"><ActivityList entries={activity.slice(-3).reverse()} host={null} onOpen={onOpenActivity} /></div></div>
|
||||
</section>
|
||||
}
|
||||
|
||||
@@ -834,6 +1017,7 @@ function HelpPage({ snapshot, host, onBack, onAction }: { snapshot: Snapshot; ho
|
||||
return <section className="page-panel help-page"><button className="back-button" onClick={onBack}><ArrowLeft /> Back to Settings</button><div className="about-hero"><img src={logo} alt="" /><span><p>Desktop companion</p><h1>Hermes-Relay CLI UI</h1><small>Compact control for this PC</small></span></div>
|
||||
<div className="settings-group"><h2>About</h2><div className="settings-card about-facts"><dl><div><dt>UI version</dt><dd>{snapshot.ui_version ?? 'Unknown'}</dd></div><div><dt>CLI version</dt><dd>{snapshot.cli_version ?? 'Unknown'}</dd></div><div><dt>CLI path</dt><dd title={snapshot.cli_path ?? undefined}>{snapshot.cli_path ?? 'Not reported'}</dd></div><div><dt>Relay server</dt><dd>{host?.server_version ?? 'Not connected'}</dd></div></dl></div></div>
|
||||
<div className="settings-group"><h2>Get help</h2><div className="settings-card management-links">{links.map(([label, url]) => <button key={url} onClick={() => onAction('open_external_url', { url })}><span><strong>{label}</strong></span><ExternalLink /></button>)}</div></div>
|
||||
<div className="settings-group"><h2>Computer control</h2><div className="settings-card help-note"><ShieldCheck /><span><strong>CUA Driver is an optional control engine</strong><small>When compatible and healthy, it can use app-aware background control and separate animated agent cursors. These are visual overlays—not additional Windows hardware pointers. Foreground switching is not available; control remains background only. Full Access never bypasses targeting, sensitive-surface rules, audit, or emergency stop.</small></span></div></div>
|
||||
<button className="diagnostic-action" onClick={() => onAction('run_diagnostics')}><ActivityIcon /><span><strong>Run diagnostics</strong><small>Check the daemon, installation and active relay.</small></span><ChevronRight /></button>
|
||||
</section>
|
||||
}
|
||||
|
||||
@@ -68,6 +68,8 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
|
||||
.route-endpoint small { font-size: 10px; font-weight: 550; white-space: nowrap; }
|
||||
.route-link { position: relative; z-index: 0; display: block; height: 1px; border-top: 1.5px dashed #4fdf72; opacity: .9; }
|
||||
.connection-route.offline .route-link { border-color: #66717d; opacity: .55; }
|
||||
.connection-route.retrying .route-link { border-color: var(--amber); opacity: .7; animation: retry-link 1.4s ease-in-out infinite; }
|
||||
@keyframes retry-link { 50% { opacity: .28; } }
|
||||
.route-traffic { position: absolute; z-index: 1; pointer-events: none; left: 49px; right: 49px; top: 29px; height: 1px; overflow: hidden; }
|
||||
.packet { position: absolute; top: -2px; left: 0; width: 9px; height: 5px; border-radius: 2px; background: #a3f9b2; box-shadow: 0 0 4px #72ed89, 0 0 10px #52df70; opacity: 0; }
|
||||
.packet::after { content: ''; position: absolute; inset: 1px 2px; border-radius: 1px; background: #effff2; }
|
||||
@@ -107,6 +109,10 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
|
||||
.connection-route.offline .route-status strong { color: #8d98a3; }
|
||||
.connection-route.connecting .route-status strong { color: var(--violet); }
|
||||
.connection-route.disconnecting .route-status strong { color: var(--amber); }
|
||||
.connection-route.retrying .route-status > strong { color: var(--amber); }
|
||||
.retry-now { border: 0; background: transparent; color: #d3a84f; padding: 2px 5px; display: inline-flex; align-items: center; gap: 4px; font-size: 9.5px; cursor: pointer; }
|
||||
.retry-now:hover { color: #ffd77b; }
|
||||
.retry-now svg { width: 11px; height: 11px; }
|
||||
.connection-route.connecting .route-link { opacity: .9; animation-duration: .65s; }
|
||||
.connection-route.disconnecting .route-link, .connection-route.disconnecting .route-traffic { opacity: .38; transition: opacity .18s ease; }
|
||||
.route-status span, .route-status small { color: var(--muted); font-size: 11px; }
|
||||
@@ -283,6 +289,34 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
|
||||
.activity-output-list header strong { font-size: 10px; font-weight: 550; }
|
||||
.activity-output-list header em { color: var(--amber); font-size: 8px; font-style: normal; text-transform: uppercase; letter-spacing: .5px; }
|
||||
.activity-output-list pre { max-height: 155px; margin: 0; padding: 9px; overflow: auto; color: #cdd2d7; font: 10px/1.45 'Cascadia Mono', Consolas, monospace; white-space: pre-wrap; overflow-wrap: anywhere; user-select: text; }
|
||||
.control-timeline { margin: 0 0 10px; border: 1px solid var(--line); border-radius: 7px; background: #0d151c; padding: 8px 10px; display: grid; }
|
||||
.control-timeline > div { min-height: 39px; position: relative; display: grid; grid-template-columns: 14px 1fr; gap: 8px; }
|
||||
.control-timeline > div:not(:last-child)::after { content: ''; position: absolute; left: 5px; top: 15px; bottom: -2px; border-left: 1px dashed #4b5864; }
|
||||
.control-timeline i { width: 11px; height: 11px; margin-top: 3px; border: 2px solid #53606c; border-radius: 50%; background: #0d151c; z-index: 1; }
|
||||
.control-timeline .done i { border-color: var(--green); box-shadow: 0 0 6px #50e57755; }
|
||||
.control-timeline .failed i { border-color: var(--amber); }
|
||||
.control-timeline span { display: grid; align-content: start; gap: 1px; }
|
||||
.control-timeline strong { font-size: 10.5px; font-weight: 560; }
|
||||
.control-timeline small { color: var(--muted); font-size: 9.5px; overflow-wrap: anywhere; }
|
||||
.control-timeline .pending i { border-color: #6d7782; border-style: dashed; }
|
||||
.activity-error-callout { margin: 0 0 10px; padding: 9px 10px; border: 1px solid #77502c; border-radius: 7px; background: #2b1b0f; color: #f4c878; display: flex; gap: 8px; align-items: flex-start; }
|
||||
.activity-error-callout > svg { width: 17px; flex: 0 0 17px; margin-top: 1px; }
|
||||
.activity-error-callout span { min-width: 0; display: grid; gap: 2px; }
|
||||
.activity-error-callout strong { font-size: 11px; }
|
||||
.activity-error-callout small { color: #ddb986; font-size: 10px; line-height: 1.35; user-select: text; overflow-wrap: anywhere; }
|
||||
.screenshot-evidence-card { width: 100%; min-height: 53px; margin: 0 0 10px; border: 1px solid #4c3d69; border-radius: 8px; background: linear-gradient(110deg, #171329, #101922); padding: 8px 10px; display: flex; justify-content: space-between; align-items: center; text-align: left; cursor: pointer; }
|
||||
.screenshot-evidence-card:hover { border-color: #8e62d5; background: linear-gradient(110deg, #201638, #111b24); }
|
||||
.screenshot-evidence-card > span { min-width: 0; display: grid; grid-template-columns: 25px 1fr; grid-template-rows: auto auto; column-gap: 7px; }
|
||||
.screenshot-evidence-card > span svg { grid-row: 1 / 3; width: 20px; color: var(--violet); align-self: center; }
|
||||
.screenshot-evidence-card strong { font-size: 11.5px; }
|
||||
.screenshot-evidence-card small { color: var(--muted); font-size: 9.5px; }
|
||||
.screenshot-evidence-card em { color: #c39aff; font-size: 10px; font-style: normal; display: flex; align-items: center; gap: 4px; white-space: nowrap; }
|
||||
.screenshot-evidence-card em svg { width: 13px; }
|
||||
.control-target { margin: 0 0 10px; display: grid; grid-template-columns: .8fr 1.2fr 1fr; border: 1px solid var(--line); border-radius: 7px; overflow: hidden; }
|
||||
.control-target div { min-width: 0; padding: 7px 8px; border-right: 1px solid var(--line); display: grid; gap: 2px; }
|
||||
.control-target div:last-child { border: 0; }
|
||||
.control-target dt { color: var(--faint); font-size: 8px; text-transform: uppercase; letter-spacing: .55px; }
|
||||
.control-target dd { margin: 0; color: #c8cdd2; font-size: 9.5px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
||||
.admin-warning { min-height: 44px; border: 1px solid #5b4615; border-radius: 6px; background: #3c2d0d3d; margin-top: 11px; padding: 0 12px; display: flex; align-items: center; gap: 11px; font-size: 12px; }
|
||||
.admin-warning svg { color: var(--amber); width: 22px; }
|
||||
.admin-warning span { flex: 1; }
|
||||
@@ -445,6 +479,60 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
|
||||
.settings-group h2 span { color: var(--faint); font-weight: 400; margin-left: 4px; }
|
||||
.group-help { margin: -2px 0 8px; }
|
||||
.settings-card { border: 1px solid var(--line); border-radius: 8px; background: #111922a8; overflow: hidden; }
|
||||
.engine-card { position: relative; }
|
||||
.engine-card::before { content: ''; position: absolute; inset: 0 auto 0 0; width: 2px; background: var(--faint); }
|
||||
.engine-card.engine-ready::before { background: var(--green); box-shadow: 0 0 10px #50e57788; }
|
||||
.engine-card.engine-degraded::before, .engine-card.engine-incompatible::before { background: var(--amber); }
|
||||
.engine-status { min-height: 67px; padding: 10px 11px 10px 13px; display: flex; align-items: center; gap: 9px; }
|
||||
.engine-status > span:nth-child(2) { min-width: 0; flex: 1; display: grid; gap: 3px; }
|
||||
.engine-status strong { font-size: 13px; font-weight: 570; }
|
||||
.engine-status small { color: var(--muted); font-size: 10px; line-height: 1.35; }
|
||||
.engine-status em { flex: 0 0 auto; align-self: flex-start; margin-top: 2px; border: 1px solid #46515d; border-radius: 10px; color: var(--muted); padding: 2px 6px; font-size: 8.5px; font-style: normal; text-transform: uppercase; letter-spacing: .5px; }
|
||||
.engine-ready .engine-status em { border-color: #346443; color: var(--green); background: #163c2524; }
|
||||
.engine-degraded .engine-status em, .engine-incompatible .engine-status em { border-color: #705528; color: var(--amber); background: #251e12; }
|
||||
.engine-options { border-top: 1px solid var(--line); }
|
||||
.engine-choice { min-height: 68px; padding: 9px 11px 9px 13px; display: flex; align-items: center; gap: 10px; border-bottom: 1px solid var(--line); }
|
||||
.engine-choice > span { min-width: 0; flex: 1; display: grid; gap: 3px; }
|
||||
.engine-choice strong { font-size: 12.5px; font-weight: 550; }
|
||||
.engine-choice small { color: var(--muted); font-size: 9.5px; line-height: 1.25; }
|
||||
.engine-choice > div { flex: 0 0 auto; display: grid; grid-template-columns: 1fr 1fr; border: 1px solid #46515d; border-radius: 6px; overflow: hidden; }
|
||||
.engine-choice button { min-width: 54px; height: 28px; border: 0; border-right: 1px solid #46515d; background: #121b24; color: var(--muted); font-size: 10px; cursor: pointer; }
|
||||
.engine-choice button:last-child { border-right: 0; }
|
||||
.engine-choice button.active { background: #6137bd; color: #fff; }
|
||||
.engine-live { min-height: 38px; padding: 6px 11px 6px 13px; border-bottom: 1px solid var(--line); display: flex; align-items: center; justify-content: space-between; }
|
||||
.engine-live > span { display: flex; align-items: baseline; gap: 5px; }
|
||||
.engine-live strong { font-size: 15px; color: #dfe4e9; }
|
||||
.engine-live small { color: var(--muted); font-size: 9.5px; }
|
||||
.engine-live em { color: var(--faint); font-size: 8.5px; font-style: normal; text-transform: uppercase; letter-spacing: .45px; }
|
||||
.engine-live em i { display: inline-block; width: 5px; height: 5px; border-radius: 50%; background: var(--faint); margin-right: 3px; }
|
||||
.engine-live em.active { color: var(--green); }
|
||||
.engine-live em.active i { background: var(--green); box-shadow: 0 0 6px var(--green); }
|
||||
.engine-options .setting-row { min-height: 64px; padding-left: 13px; }
|
||||
.engine-options .setting-row strong { font-size: 12.5px; }
|
||||
.engine-options .setting-row small { max-width: 250px; font-size: 9.5px; line-height: 1.3; }
|
||||
.background-only { background: #5c41150f; }
|
||||
.background-only em { flex: 0 0 auto; color: #b5bdc6; font-size: 9px; font-style: normal; text-transform: uppercase; letter-spacing: .45px; }
|
||||
.cua-health { background: #101820; }
|
||||
.cua-health > div { flex: 0 0 auto; display: flex; align-items: center; gap: 7px; }
|
||||
.cua-health em { font-size: 8.5px; font-style: normal; text-transform: uppercase; letter-spacing: .4px; color: var(--faint); }
|
||||
.cua-health em.health-healthy { color: var(--green); }
|
||||
.cua-health em.health-degraded, .cua-health em.health-error { color: var(--amber); }
|
||||
.cua-health .compact-button { min-width: 68px; height: 27px; padding: 4px 7px; font-size: 9px; }
|
||||
.cua-health .compact-button svg { width: 11px; height: 11px; }
|
||||
.cua-maintenance { min-height: 49px; padding: 7px 10px 7px 13px; border-top: 1px solid var(--line); display: flex; align-items: center; gap: 8px; }
|
||||
.cua-maintenance > span { min-width: 0; flex: 1; display: grid; gap: 2px; }
|
||||
.cua-maintenance strong { font-size: 10.5px; font-weight: 560; }
|
||||
.cua-maintenance small { color: var(--muted); font-size: 9px; line-height: 1.25; }
|
||||
.cua-maintenance button { min-width: 60px; height: 27px; border: 1px solid #4c5762; border-radius: 5px; background: #17212a; color: #dce1e6; display: flex; align-items: center; justify-content: center; gap: 4px; font-size: 9.5px; cursor: pointer; }
|
||||
.cua-maintenance button:disabled { opacity: .55; cursor: wait; }
|
||||
.cua-maintenance button svg { width: 12px; height: 12px; }
|
||||
.engine-help { display: flex; align-items: flex-start; gap: 5px; margin: 6px 2px 0; color: var(--faint); font-size: 9.5px; line-height: 1.35; }
|
||||
.engine-help svg { width: 13px; height: 13px; flex: 0 0 13px; color: var(--violet); }
|
||||
.help-note { min-height: 92px; padding: 11px; display: flex; align-items: flex-start; gap: 9px; }
|
||||
.help-note > svg { width: 18px; flex: 0 0 18px; color: var(--violet); }
|
||||
.help-note span { display: grid; gap: 4px; }
|
||||
.help-note strong { font-size: 12px; }
|
||||
.help-note small { color: var(--muted); font-size: 10px; line-height: 1.45; }
|
||||
.settings-card.padded { padding: 0 11px; }
|
||||
.activity-panel { display: grid; gap: 8px; }
|
||||
.activity-summary { display: grid; grid-template-columns: 1fr 1fr auto; gap: 7px; align-items: stretch; }
|
||||
@@ -466,6 +554,11 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
|
||||
.clear-activity:disabled { opacity: .45; cursor: default; }
|
||||
.activity-card { padding: 0; }
|
||||
.activity-card .empty-state { min-height: 90px; }
|
||||
.activity-retention-card { margin-bottom: 8px; }
|
||||
.activity-retention-card .setting-row { align-items: center; }
|
||||
.retention-options { flex: 0 0 auto; display: grid; grid-template-columns: repeat(4, auto); gap: 2px; padding: 2px; border: 1px solid #35414c; border-radius: 7px; background: #0b1218; }
|
||||
.retention-options button { min-width: 29px; height: 24px; padding: 0 5px; border: 0; border-radius: 5px; background: transparent; color: var(--muted); font-size: 9.5px; cursor: pointer; }
|
||||
.retention-options button.active { background: #7042d6; color: white; box-shadow: inset 0 1px #c9a5ff33; }
|
||||
.settings-group-heading { display: flex; align-items: center; justify-content: space-between; }
|
||||
.settings-group-heading h2 { margin-bottom: 7px; }
|
||||
.settings-group-heading button { border: 0; background: transparent; color: var(--violet); display: flex; align-items: center; gap: 2px; padding: 0 0 7px; font-size: 11.5px; cursor: pointer; }
|
||||
@@ -617,6 +710,40 @@ button:focus-visible, input:focus-visible { outline: 2px solid var(--violet); ou
|
||||
.grant-actions button:hover:not(:disabled) { filter: brightness(1.1); }
|
||||
.grant-actions button:disabled { opacity: .6; cursor: wait; }
|
||||
|
||||
.connection-notice-shell { position: fixed; inset: 0; padding: 7px; display: grid; align-items: stretch; animation: notice-in .2s cubic-bezier(.2,.8,.2,1); }
|
||||
@keyframes notice-in { from { opacity: 0; transform: translateY(12px) scale(.98); } }
|
||||
.connection-notice-card { position: relative; border: 1px solid #3c4854; border-radius: 12px; background: #101922f5; box-shadow: 0 15px 38px #000b, inset 0 1px #ffffff08; padding: 13px 56px 12px 13px; display: grid; grid-template-columns: 38px minmax(0,1fr); gap: 10px; align-items: center; overflow: hidden; }
|
||||
.connection-notice-card::before { content: ''; position: absolute; inset: 0 auto 0 0; width: 3px; background: var(--green); box-shadow: 0 0 13px var(--green); }
|
||||
.connection-notice-shell.warning .connection-notice-card::before { background: var(--amber); box-shadow: 0 0 13px var(--amber); }
|
||||
.connection-notice-shell.offline .connection-notice-card::before { background: #8c96a0; box-shadow: none; }
|
||||
.connection-notice-icon { width: 36px; height: 36px; border-radius: 10px; background: #21432b; color: var(--green); display: grid; place-items: center; }
|
||||
.connection-notice-shell.warning .connection-notice-icon { background: #493817; color: var(--amber); }
|
||||
.connection-notice-shell.offline .connection-notice-icon { background: #29323b; color: #aeb7c0; }
|
||||
.connection-notice-icon svg { width: 21px; }
|
||||
.connection-notice-card > span:nth-child(2) { min-width: 0; display: grid; gap: 1px; }
|
||||
.connection-notice-card small { color: var(--violet); font-size: 8.5px; text-transform: uppercase; letter-spacing: .8px; }
|
||||
.connection-notice-card strong { font-size: 13px; }
|
||||
.connection-notice-card p { margin: 1px 0 0; color: var(--muted); font-size: 10px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
||||
.connection-notice-open { position: absolute; right: 12px; bottom: 12px; border: 0; background: transparent; color: #c59aff; font-size: 9.5px; cursor: pointer; }
|
||||
.connection-notice-close { position: absolute; right: 8px; top: 8px; width: 25px; height: 25px; border: 0; border-radius: 6px; background: transparent; color: #8f99a4; display: grid; place-items: center; cursor: pointer; }
|
||||
.connection-notice-close:hover { color: white; background: #ffffff0c; }
|
||||
.connection-notice-close svg { width: 14px; }
|
||||
|
||||
.evidence-shell { position: fixed; inset: 0; border: 1px solid #3a4652; border-radius: 11px; background: #0b1218; box-shadow: 0 24px 70px #000d; display: grid; grid-template-rows: 52px minmax(0,1fr); overflow: hidden; }
|
||||
.evidence-shell > header { border-bottom: 1px solid var(--line); background: #111a23; padding: 0 13px 0 16px; display: flex; align-items: center; justify-content: space-between; }
|
||||
.evidence-shell > header > span { display: grid; grid-template-columns: 28px auto; grid-template-rows: auto auto; column-gap: 8px; }
|
||||
.evidence-shell > header svg { grid-row: 1 / 3; align-self: center; width: 20px; color: var(--violet); }
|
||||
.evidence-shell > header strong { font-size: 13px; }
|
||||
.evidence-shell > header small { color: var(--muted); font-size: 9.5px; }
|
||||
.evidence-shell > header button { width: 34px; height: 34px; border: 0; border-radius: 7px; background: transparent; color: #aab2bb; display: grid; place-items: center; cursor: pointer; }
|
||||
.evidence-shell > header button:hover { color: white; background: #c42b1c; }
|
||||
.evidence-shell > main { min-width: 0; min-height: 0; padding: 14px; display: grid; place-items: center; background: radial-gradient(circle at center, #18232c, #080e13 68%); }
|
||||
.evidence-shell img { max-width: 100%; max-height: 100%; object-fit: contain; border-radius: 5px; box-shadow: 0 8px 30px #000a; user-select: none; }
|
||||
.evidence-loading, .evidence-error { color: var(--muted); display: grid; place-items: center; gap: 8px; text-align: center; }
|
||||
.evidence-error svg { color: var(--amber); width: 30px; }
|
||||
.evidence-error strong { color: var(--ink); font-size: 14px; }
|
||||
.evidence-error small { max-width: 420px; font-size: 11px; }
|
||||
|
||||
@media (max-width: 410px) {
|
||||
.content { padding-left: 20px; padding-right: 20px; }
|
||||
.access-control button { gap: 5px; }
|
||||
|
||||
@@ -23,17 +23,76 @@ export interface DaemonStatus {
|
||||
privilege?: string | null
|
||||
username?: string | null
|
||||
updated_at?: number | null
|
||||
last_event?: string | null
|
||||
reconnect_attempt?: number | null
|
||||
retry_at?: number | null
|
||||
last_error?: string | null
|
||||
}
|
||||
|
||||
export interface ComputerControlEngine {
|
||||
selected: 'legacy' | 'cua'
|
||||
effective: 'legacy' | 'cua' | 'unavailable'
|
||||
available: boolean
|
||||
state: 'not_installed' | 'incompatible' | 'degraded' | 'ready' | 'error'
|
||||
version?: string | null
|
||||
health?: string | null
|
||||
path?: string | null
|
||||
cursor_enabled?: boolean
|
||||
foreground_escalation_enabled: boolean
|
||||
active_sessions?: number
|
||||
active_backend?: 'cua' | 'legacy_compat' | 'mixed' | 'idle' | null
|
||||
last_action?: {
|
||||
action?: string | null
|
||||
target_app?: string | null
|
||||
target_title?: string | null
|
||||
target_pid?: number | null
|
||||
target_window_id?: number | null
|
||||
verification?: string | null
|
||||
occurred_at?: number | string | null
|
||||
} | null
|
||||
message?: string | null
|
||||
}
|
||||
|
||||
export interface CuaManagementStatus {
|
||||
installed: boolean
|
||||
canonical_path?: string | null
|
||||
discovered_path?: string | null
|
||||
stale_path_shim: boolean
|
||||
current_version?: string | null
|
||||
compatible: boolean
|
||||
compatibility_reason?: string | null
|
||||
supported_range: { minimum: string; maximum_exclusive: string }
|
||||
update?: { latest_version?: string; update_available: boolean; compatible: boolean; error?: string; release_notes_url?: string }
|
||||
operation?: { kind: 'install' | 'update'; state: 'completed'; version: string }
|
||||
}
|
||||
|
||||
export interface CuaHealthStatus {
|
||||
state: 'healthy' | 'degraded' | 'error'
|
||||
checkedAt: string
|
||||
overall?: string
|
||||
reason?: string
|
||||
temporaryWindowsCompatibility: true
|
||||
}
|
||||
|
||||
export interface Activity {
|
||||
ts: number
|
||||
kind?: 'tool.completed' | 'management.completed'
|
||||
kind?: 'tool.completed' | 'management.completed' | 'connection.state'
|
||||
tool: string
|
||||
category?: 'command' | 'files' | 'screen' | 'input' | 'devices' | 'system' | 'other'
|
||||
ok: boolean
|
||||
aborted?: boolean
|
||||
request_id?: string
|
||||
host_url?: string
|
||||
backend?: string
|
||||
dispatch?: string
|
||||
control_session_id?: string
|
||||
target_app?: string
|
||||
target_title?: string
|
||||
target_pid?: number
|
||||
target_window_id?: number
|
||||
action?: string
|
||||
verification?: string
|
||||
phase?: string
|
||||
duration_ms?: number
|
||||
exit_code?: number
|
||||
summary?: string
|
||||
@@ -47,6 +106,10 @@ export interface Activity {
|
||||
stderr_truncated?: boolean
|
||||
result_truncated?: boolean
|
||||
error?: string
|
||||
screenshot_evidence_id?: string
|
||||
screenshot_mime_type?: 'image/png'
|
||||
screenshot_width?: number
|
||||
screenshot_height?: number
|
||||
}
|
||||
|
||||
export interface Snapshot {
|
||||
@@ -54,6 +117,7 @@ export interface Snapshot {
|
||||
active_url?: string | null
|
||||
daemon: DaemonStatus
|
||||
activity: Activity[]
|
||||
activity_screenshot_retention: { enabled: boolean; days: number; count: number; bytes: number }
|
||||
pending_grants: PendingGrantRequest[]
|
||||
startup_enabled: boolean
|
||||
daemon_autostart_enabled?: boolean
|
||||
@@ -61,6 +125,7 @@ export interface Snapshot {
|
||||
cli_version?: string | null
|
||||
cli_path?: string | null
|
||||
hardware_availability: { usb: boolean; adb: boolean; microphone: boolean; camera: boolean }
|
||||
computer_control_engine?: ComputerControlEngine | null
|
||||
}
|
||||
|
||||
export interface PendingGrantRequest {
|
||||
|
||||
@@ -270,6 +270,8 @@ not require an API endpoint or API bearer when dashboard chat is ready.
|
||||
>
|
||||
> **Decision (2026-06-29) — unified-session "Threads", not a separate surface:** the proactive agent↔phone conversation is **not** a separate app lane/tab/segment. It is a **source-tagged session inside the one Chat surface** — a **Thread** (`source=phone`). The three things distinguishing a Thread from a normal gateway chat are *session properties*, not a separate UI: (a) the agent can initiate a turn, (b) it rides the relay `proactive` transport and is relay-gated, (c) it's a standing/named DM. **Scrollback = the gateway session store** (same read path Chat uses); **live receive = the relay `proactive` push** (→ notification); **send = `proactive.reply`**. The local `ProactiveInboxStore` is demoted to a live-push cache + outbox (no parallel history). The Thread capability is surfaced in the **connection best-path/capability UI** (a relay-tier capability, like terminal/bridge/voice) and as a clean **Threads** entry (thread-spool icon, NOT a phone glyph) pinned atop the session drawer when active — never a connection-wizard step. Degrades cleanly: no relay plugin → no `source=phone` sessions → Chat is unchanged (standard-path-safe). This **supersedes the earlier "separate Agent lane / 4th nav segment" sketch** and folds in the "show chat source/platform attribution in Chat" goal in one stroke.
|
||||
>
|
||||
> **Outbound-first refinement (2026-08-14):** an outbound platform send does not itself create a gateway session; the `source=phone` session is created only when the phone first replies. Android groups the bounded proactive cache by connection + `chat_id` and renders a provisional Thread during that gap. A notification tap opens that exact provisional Thread, and the first reply uses the existing phone-platform path before promoting the view to the real gateway session. Once `/phone/threads` reports the mapping, the provisional row disappears. This cache is a bootstrap transcript, not a second long-term history store.
|
||||
>
|
||||
> **Why unified, not separate:** a separate "agent chat" tab is redundant — a Thread is just a chat session the agent can also start. One Chat surface (sessions tagged by source) matches the Discord/messaging-app model the product targets. **Two distinct senses of "gateway" to keep straight:** the *messaging gateway / platform layer* (`gateway/platforms/*` — phone/Discord/Slack as platforms; this is the Thread's `source`) vs. the *dashboard gateway transport* (`/api/ws` tui_gateway — how live chat bytes flow). A Thread is defined by its **platform/source**, not its transport; the two are orthogonal.
|
||||
|
||||
**Original research (2026-04-07, retained as lineage — the `mobile:<device_id>` syntax and the ~16-file upstream-fork registration below are SUPERSEDED by the no-fork plugin path):**
|
||||
@@ -3041,3 +3043,152 @@ Hermes authorization remains local and the vanilla upstream path remains
|
||||
independent. Implementing the client requires a TLS-over-broker byte-stream
|
||||
adapter; advertising the candidate is forbidden until that adapter and the
|
||||
cross-platform acceptance gates are complete.
|
||||
|
||||
---
|
||||
|
||||
## ADR 56 — Structured desktop control may use CUA Driver behind Hermes policy
|
||||
|
||||
**Status:** Accepted for phased implementation (2026-08-13).
|
||||
|
||||
**Implementation note (2026-08-13).** The first Windows integration keeps the
|
||||
public `desktop_computer_*` contract and adds canonical runtime discovery,
|
||||
manifest/version/tool/permission/health checks, an allowlisted CUA adapter,
|
||||
server-owned control-session envelopes, per-session grants, sensitive-target
|
||||
preflight, single-use snapshot tokens, pre/action/post snapshot flow, and local
|
||||
CLI/UI engine status. CUA actions are background-only in this phase; foreground
|
||||
escalation remains reserved and reports disabled. The optional driver is not
|
||||
bundled or updated by Hermes-Relay, and Hermes forces driver telemetry off for
|
||||
every child process it starts. The legacy engine remains the default and the
|
||||
fail-closed fallback while the live Windows acceptance and remaining scope,
|
||||
redaction, and grant-bridge hardening gates tracked in `TODO.md` stay open.
|
||||
|
||||
**Second-phase refinement (2026-08-13).** CUA is the preferred/default setting
|
||||
for new structured-control sessions; the original Windows input path is named
|
||||
`legacy_compat` and remains available only as an explicit compatibility choice
|
||||
or a fail-closed pre-session fallback when CUA is unavailable. The backend is
|
||||
selected once per authenticated control session and cannot change mid-session.
|
||||
Window-scoped snapshots/actions use CUA, while the existing read-only full-
|
||||
display screenshot remains a separate `system_capture` path. Local audit and
|
||||
the UI activity timeline expose bounded high-level fields—backend, dispatch,
|
||||
control session, target app/window identifiers, action, phase, and verification
|
||||
state—while omitting accessibility text, screenshots, entered values, and raw
|
||||
driver responses.
|
||||
|
||||
Hermes now owns an explicit Windows lifecycle surface without bundling the
|
||||
driver: `computer-use cua status|health|install|check-update|update`, with `--yes`
|
||||
required for install/update. Mutations accept only supported upstream releases
|
||||
(`>=0.19.3 <0.20.0`), verify the `trycua/cua` product/version manifest and
|
||||
installer SHA-256 before running a temporary installer under a sanitized
|
||||
environment, then verify the canonical `packages/current` binary, manifest,
|
||||
version, path, and permission mode. Accessibility health is an explicit
|
||||
diagnostic. There is no automatic install or
|
||||
update.
|
||||
|
||||
**Context.** The first Windows input backend uses PowerShell, `SetCursorPos`,
|
||||
`mouse_event`, and `SendKeys`. It can move the operator's physical pointer and
|
||||
depends on foreground focus. CUA Driver provides target-process/window UI
|
||||
Automation, accessibility snapshots, background input, and session-scoped
|
||||
animated agent cursors without moving the physical pointer. Its full local tool
|
||||
surface also includes foreground activation, arbitrary application control,
|
||||
recording, configuration, and update operations that exceed Hermes-Relay's
|
||||
screen/input capability.
|
||||
|
||||
**Decision.** Adopt CUA Driver only as an optional, version-pinned structured
|
||||
computer-control backend behind the existing `desktop_computer_*` contract.
|
||||
Hermes remains the outer authority for Relay authentication, target-device
|
||||
routing, per-host capability policy, local grants, emergency stop, and audit.
|
||||
The daemon translates an allowlisted Hermes schema to CUA operations; it never
|
||||
forwards arbitrary CUA tool names or JSON.
|
||||
|
||||
Background dispatch is the default. A `background_unavailable` result returns
|
||||
to the caller instead of silently foregrounding another application. Foreground
|
||||
dispatch, application launch/termination, JavaScript execution, recording,
|
||||
replay, configuration, and driver updates require separate, explicit local
|
||||
authority. Full Access may bypass ordinary task prompts, but never authenticated
|
||||
targeting, sensitive-surface blocks, UAC/session boundaries, audit, emergency
|
||||
stop, runtime validation, or per-action failures.
|
||||
|
||||
Every semantic element action requires a fresh pre-action window snapshot, an
|
||||
opaque element token bound to its snapshot generation, authenticated principal,
|
||||
grant, PID, and window, followed by a post-action snapshot that records whether
|
||||
the expected state changed. Grant scope and sensitive-surface policy apply to
|
||||
both accessibility text and pixels.
|
||||
|
||||
Multiple animated pointers are virtual agent overlays, not additional Windows
|
||||
hardware cursors. Relay must first attach a server-owned authenticated control
|
||||
session identity—at minimum the Relay session, requester device, chat/run,
|
||||
target device, and request—to each command. The desktop derives cursor identity
|
||||
locally and owns one bounded CUA session per active control session. Model
|
||||
arguments cannot choose or share cursor IDs. Grant expiry, cancellation,
|
||||
disconnect, re-pair, policy downgrade, emergency stop, desktop lock/session
|
||||
change, or daemon shutdown ends the corresponding driver session immediately.
|
||||
|
||||
CUA runs in the interactive user's logon session, never Windows Session 0.
|
||||
Initial packaging remains optional and resolves the canonical installed package
|
||||
rather than an untrusted PATH entry. Readiness uses the driver's live manifest,
|
||||
schema, tool surface, daemon status, and permission mode and fails closed on an
|
||||
absent or incompatible backend. Telemetry and driver updates remain explicit
|
||||
operator choices.
|
||||
|
||||
**Temporary Windows implementation note (2026-08-14).** Until
|
||||
`trycua/cua#3103` is fixed in the supported driver range, the whole-desktop
|
||||
`health_report` is not a session-start gate: its fixed UIA timeout can report a
|
||||
false degradation and leave the driver temporarily busy. Operators can re-run
|
||||
that diagnostic from the CLI or UI. Structured actions retain their existing
|
||||
target, grant, snapshot, timeout, and fail-closed checks. Remove this exception
|
||||
when the upstream probe is bounded and cannot poison later actions.
|
||||
|
||||
**Consequences.** Hermes can gain background, element-aware control and clean
|
||||
per-agent animated cursors without replacing its Relay protocol or permission
|
||||
model. The existing PowerShell/User32 backend remains a compatibility fallback
|
||||
until authenticated control-session identity, scoped grants, sensitive-surface
|
||||
enforcement, and lifecycle tests ship. Raw command execution remains an honest
|
||||
escape hatch: strong computer-control isolation is meaningful only when command
|
||||
execution is disabled or the host is already fully trusted.
|
||||
|
||||
---
|
||||
|
||||
## ADR 57 — Official Desktop Relay UI is a lazy, unified-package runtime plugin
|
||||
|
||||
**Status:** Accepted (2026-08-14).
|
||||
|
||||
**Context.** Official Hermes Desktop now discovers a regular agent plugin's
|
||||
`desktop/plugin.js` and loads it through `@hermes/plugin-sdk`. The SDK provides
|
||||
profile-aware `ctx.rest()` access to the same `plugin_api.py` namespace used by
|
||||
the web Dashboard, plus native pane, sidebar, status-bar, palette, close,
|
||||
reveal, drag, dock, i18n, query, and unload lifecycles. Registering a pane also
|
||||
adopts it into the live layout, which would violate Hermes-Relay's requirement
|
||||
that plugin load and application lifecycle events never open or focus Relay.
|
||||
|
||||
**Decision.** Ship `plugin/desktop/plugin.js` beside `plugin.yaml` and the
|
||||
existing Dashboard half. Keep the Desktop half opt-in and use only public SDK
|
||||
imports. At registration time contribute three labeled open actions, but no
|
||||
pane and no network work. The first explicit action registers one dismissible
|
||||
management pane and calls the plugin-scoped `ctx.panes.reveal()` method.
|
||||
Subsequent actions reveal the same pane. The host owns close, focus, drag,
|
||||
docking, enable state, hot reload, and disposer execution.
|
||||
|
||||
The pane reads and mutates the existing Relay Dashboard backend through
|
||||
`ctx.rest()` only. It maintains no server cache, starts no timer/socket/polling
|
||||
loop, sends no notification, and makes no request until the user opens a view.
|
||||
Every query key contains the active Desktop profile, while the SDK binds the
|
||||
request to that profile's authenticated backend namespace. Pairing, revocation,
|
||||
and remote-access actions stay user initiated; destructive or host-changing
|
||||
actions require an additional in-pane confirmation.
|
||||
|
||||
**Security boundary.** Runtime Desktop plugins are trusted local ESM with full
|
||||
renderer authority; upstream provides error isolation, not a sandbox. Relay
|
||||
therefore ships fixed reviewed UI code only. It does not load generated ESM,
|
||||
Kotlin, Python, remote modules, or arbitrary action schemas, and it never stores
|
||||
tokens, keys, QR secrets, media paths, or duplicated Relay state. The existing
|
||||
Relay-gated declarative Android Plugin Studio remains a separate host-rendered,
|
||||
digest-approved capability model.
|
||||
|
||||
**Consequences.** One plugin install now exposes the Relay-specific management
|
||||
role in both the web Dashboard and official Desktop without patching Hermes or
|
||||
replacing the standalone Relay CLI/tray. The SDK has no public programmatic
|
||||
pane-coordinate API and its agent `focus_pane` surface excludes contributed
|
||||
IDs, so movement remains native drag/dock and agent-driven reveal remains
|
||||
unsupported. Physical Desktop certification remains required for multi-window,
|
||||
named-profile, remote/SSH-mapped profile, close/reopen, drag/dock, hot-reload,
|
||||
and renderer-log behavior.
|
||||
|
||||
+29
-1
@@ -387,7 +387,8 @@ Sources: `plugin/relay/channels/notifications.py`, `app/src/main/kotlin/.../noti
|
||||
|
||||
| Type | Direction | Payload |
|
||||
|------|-----------|---------|
|
||||
| `desktop.command` | Server → Client | `{request_id, tool, args}` |
|
||||
| `desktop.command` | Server → Client | `{request_id, tool, args, control_session?}` |
|
||||
| `desktop.control_session_end` | Server → Client | `{version: 1, id, target_device_id, reason?}` |
|
||||
| `desktop.response` | Client → Server | `{request_id, ok: true, result}` or `{request_id, ok: false, error}` |
|
||||
| `desktop.status` | Client → Server | `{advertised_tools, device_id, host, platform, version, computer_use?}` |
|
||||
| `desktop.workspace` | Client → Server | Workspace context snapshot |
|
||||
@@ -403,6 +404,33 @@ pending request to the selected WebSocket, ignores responses from other PCs,
|
||||
and includes resolved target metadata in the response. `/desktop/health` lists
|
||||
the connected targets.
|
||||
|
||||
For `desktop_computer_*`, current relays can attach a server-owned
|
||||
`control_session` object with `version`, opaque `id`, `request_id`,
|
||||
`requester_device_id`, `target_device_id`, and `run_id`, plus optional
|
||||
`chat_session_id` and `profile`. The desktop client validates the version and
|
||||
all required fields, requires the embedded request and target to match the
|
||||
outer command and local installation, and never reads identity from tool
|
||||
arguments. A malformed supplied binding is rejected. Omission remains
|
||||
backward-compatible for legacy computer handlers, but identity-sensitive CUA
|
||||
operations fail closed until the relay supplies an authenticated binding.
|
||||
|
||||
The plugin's standard tool route is loopback-only. Executor context is passed
|
||||
in internal headers and is trusted only inside the existing same-user host
|
||||
boundary; any local process running as that user can reach the same boundary.
|
||||
When a paired bearer is present, the relay derives `requester_device_id` from
|
||||
the authenticated plugin session instead. The model-facing tool wrapper drops
|
||||
reserved identity arguments, and the relay—not the caller—mints the opaque
|
||||
control-session ID and binds each selected target and request.
|
||||
|
||||
The relay ends a control authority when the same authenticated requester/chat/
|
||||
profile starts a different run for that target, after 15 minutes of inactivity,
|
||||
on bounded capacity eviction, or when the target desktop disconnects. For the
|
||||
first three cases it emits `desktop.control_session_end`; the desktop accepts
|
||||
that event only from its attached relay channel, requires the target to match
|
||||
its stable local device ID and the opaque ID to match an active authority, then
|
||||
revokes that authority's grants and closes its exact CUA cursor/MCP session.
|
||||
Desktop disconnect and local cancellation independently revoke local state.
|
||||
|
||||
Per-host Structured access withholds `desktop_terminal`,
|
||||
`desktop_powershell`, `desktop_spawn_detached`, and `desktop_job_start` from
|
||||
`advertised_tools`. Brokered USB tools use the same request/response envelope:
|
||||
|
||||
@@ -1046,6 +1046,7 @@ Current Android dependency versions. Source of truth is `gradle/libs.versions.to
|
||||
| **Plugin system** | `register_tool()` via `ctx` for `android_*` and `desktop_*` tools |
|
||||
| **Relay plugin** | `hermes pair`, `hermes relay start`, `hermes relay doctor`, `hermes relay compat`, dashboard `/relay` plugin tab |
|
||||
| **Dashboard plugin** | Lives at `plugin/dashboard/`; see §10.1 below |
|
||||
| **Official Desktop plugin** | Unified-package `plugin/desktop/plugin.js`; official `@hermes/plugin-sdk` only; see §10.2 below |
|
||||
|
||||
### 10.1 Dashboard plugin
|
||||
|
||||
@@ -1073,6 +1074,34 @@ Hermes-Relay ships a hermes-agent Dashboard Plugin that surfaces relay-specific
|
||||
|
||||
**Frontend.** Source under `plugin/dashboard/src/` (JSX + esbuild), committed pre-built IIFE at `plugin/dashboard/dist/index.js` (~16 KB minified). Uses the dashboard's `window.__HERMES_PLUGIN_SDK__` global for React + shadcn primitives + `fetchJSON()` — no external HTTP library, no bundled React. See ADR 19 in [`docs/decisions.md`](decisions.md) for the architectural rationale.
|
||||
|
||||
### 10.2 Official Desktop plugin
|
||||
|
||||
The same installable `hermes-relay` plugin folder includes
|
||||
`plugin/desktop/plugin.js`, discovered through the upstream unified-package path
|
||||
`$HERMES_HOME/plugins/hermes-relay/desktop/plugin.js`. It imports only
|
||||
`@hermes/plugin-sdk`, React, and the React JSX runtime. Backend access uses the
|
||||
SDK's profile-aware `ctx.rest()` door, so Desktop and the web Dashboard share
|
||||
the existing `/api/plugins/hermes-relay/*` backend without copying state or
|
||||
introducing a second control plane.
|
||||
|
||||
The Desktop half is opt-in. Enabling or loading it registers only labeled
|
||||
sidebar, status-bar, and command-palette entry points. The management pane is
|
||||
registered lazily after one of those explicit actions, then restored and
|
||||
focused with `ctx.panes.reveal()`. Startup, reconnect, profile change, hot
|
||||
reload, update, navigation restoration, and background events never reveal or
|
||||
focus it. Closing uses the official dismissible-pane lifecycle; moving and
|
||||
docking use the host's native drag targets. The current SDK does not expose
|
||||
programmatic move coordinates or agent-driven focus for contributed pane IDs,
|
||||
so Hermes-Relay does not emulate either with private layout or Electron hooks.
|
||||
|
||||
The pane provides four manually refreshed views: Relay management and pairing,
|
||||
bridge activity, sanitized media metadata, and remote access. Mutations require
|
||||
explicit labeled actions; session revocation and remote-access changes add an
|
||||
in-pane confirmation. The plugin has no timers, notifications, background
|
||||
polling, arbitrary renderer code generation, telemetry, or direct external
|
||||
networking. Query keys include the active profile and `ctx.rest()` supplies the
|
||||
matching authenticated backend scope.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"label": "Relay",
|
||||
"description": "Paired devices, bridge activity, media inspection, and remote access for hermes-relay",
|
||||
"icon": "Activity",
|
||||
"version": "1.7.0",
|
||||
"version": "1.8.0",
|
||||
"tab": {
|
||||
"path": "/relay",
|
||||
"position": "after:skills"
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "hermes-relay-dashboard",
|
||||
"version": "1.7.0",
|
||||
"version": "1.8.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "hermes-relay-dashboard",
|
||||
"version": "1.7.0",
|
||||
"version": "1.8.0",
|
||||
"devDependencies": {
|
||||
"esbuild": "^0.25.12",
|
||||
"qrcode": "^1.5.4"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "hermes-relay-dashboard",
|
||||
"version": "1.7.0",
|
||||
"version": "1.8.0",
|
||||
"private": true,
|
||||
"description": "Hermes-Relay dashboard plugin frontend (IIFE bundle). Loaded verbatim by the hermes-agent dashboard via the Plugin SDK global.",
|
||||
"scripts": {
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Hermes-Relay plugin for official Hermes Desktop
|
||||
|
||||
This is the Desktop half of the unified `hermes-relay` plugin package. Official
|
||||
Hermes Desktop discovers it at `plugins/hermes-relay/desktop/plugin.js`, beside
|
||||
the existing Python and web Dashboard halves.
|
||||
|
||||
The plugin is opt-in and never opens itself. Enabling or loading it only adds
|
||||
three labeled native entry points: **Relay** in the sidebar and status bar, and
|
||||
**Hermes Relay: Open** in the command palette. One of those explicit actions
|
||||
registers, restores, and focuses the management pane. The pane uses the host's
|
||||
native tab close and drag/dock affordances.
|
||||
|
||||
All backend calls use the official profile-aware `ctx.rest()` namespace and the
|
||||
existing `dashboard/plugin_api.py` routes. The pane keeps no server state, does
|
||||
not poll, does not notify, and does not perform network work while closed.
|
||||
|
||||
## SDK baseline
|
||||
|
||||
Implemented against upstream Hermes Desktop source
|
||||
`0296c7d8aa1d1cdb7bec8522031ca6b14eadc604` (2026-08-14), using only:
|
||||
|
||||
- `PluginContext.register`, `registerMany`, `onDispose`, `rest`, `os`, and
|
||||
`panes.reveal`
|
||||
- pane, sidebar, status-bar, and palette contribution areas
|
||||
- the exported React/query/i18n/UI-kit surface
|
||||
|
||||
The SDK currently has no public API for programmatic pane move coordinates or
|
||||
for agent-driven focus of contributed pane IDs. Users move/dock the pane with
|
||||
native drag targets; the plugin does not use private layout or Electron hooks.
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
cd plugin/desktop
|
||||
npm test
|
||||
```
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"name": "hermes-relay-desktop-plugin",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "node --test --experimental-loader ./test/loader.mjs test/*.test.mjs"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,576 @@
|
||||
import {
|
||||
Badge,
|
||||
Button,
|
||||
Codicon,
|
||||
EmptyState,
|
||||
ErrorState,
|
||||
Input,
|
||||
PANES_AREA,
|
||||
PALETTE_AREA,
|
||||
SIDEBAR_NAV_AREA,
|
||||
STATUSBAR_AREAS,
|
||||
StatusDot,
|
||||
Tip,
|
||||
host,
|
||||
useMutation,
|
||||
usePluginI18n,
|
||||
useQuery,
|
||||
useQueryClient,
|
||||
useValue
|
||||
} from '@hermes/plugin-sdk'
|
||||
import { useState } from 'react'
|
||||
import { Fragment, jsx, jsxs } from 'react/jsx-runtime'
|
||||
|
||||
const PLUGIN_ID = 'hermes-relay'
|
||||
|
||||
const messages = {
|
||||
en: {
|
||||
'action.cancel': 'Cancel',
|
||||
'action.confirm': 'Confirm',
|
||||
'action.copy': 'Copy invite',
|
||||
'action.open': 'Open Hermes Relay',
|
||||
'action.pair': 'Pair new device',
|
||||
'action.refresh': 'Refresh',
|
||||
'activity.empty': 'No recent bridge activity.',
|
||||
'activity.title': 'Bridge activity',
|
||||
'management.empty': 'No paired devices.',
|
||||
'management.title': 'Relay management',
|
||||
'media.empty': 'No active media deliveries.',
|
||||
'media.title': 'Media',
|
||||
'nav.label': 'Relay',
|
||||
'pairing.copyFailed': 'The Desktop clipboard bridge is unavailable. Select and copy the invite manually.',
|
||||
'pairing.expires': 'One-time invite. Keep it private and use it before it expires.',
|
||||
'pairing.title': 'Pairing invite',
|
||||
'pane.description': 'Profile-scoped views of the existing Hermes-Relay service.',
|
||||
'pane.title': 'Hermes Relay',
|
||||
'remote.disable': 'Disable Tailscale serving',
|
||||
'remote.enable': 'Enable Tailscale serving',
|
||||
'remote.probe': 'Probe URL',
|
||||
'remote.save': 'Save public URL',
|
||||
'remote.title': 'Remote access',
|
||||
'session.revoke': 'Revoke',
|
||||
'session.revokeConfirm': 'Revoke this device session? The device must pair again.',
|
||||
'status.detail': 'Open the profile-scoped Relay management pane',
|
||||
'status.label': 'Relay'
|
||||
}
|
||||
}
|
||||
|
||||
export function profileQueryKey(profile, resource) {
|
||||
return [PLUGIN_ID, profile || 'default', resource]
|
||||
}
|
||||
|
||||
function text(value, fallback = '—') {
|
||||
if (value === null || value === undefined || value === '') return fallback
|
||||
if (typeof value === 'object') return JSON.stringify(value)
|
||||
return String(value)
|
||||
}
|
||||
|
||||
function list(value, key) {
|
||||
if (Array.isArray(value)) return value
|
||||
if (value && Array.isArray(value[key])) return value[key]
|
||||
return []
|
||||
}
|
||||
|
||||
function friendlyError(error) {
|
||||
if (!error) return 'Unknown error'
|
||||
if (error instanceof Error) return error.message
|
||||
return String(error)
|
||||
}
|
||||
|
||||
function formatTime(value) {
|
||||
if (!value) return '—'
|
||||
const numeric = Number(value)
|
||||
const millis = Number.isFinite(numeric) && numeric < 10_000_000_000 ? numeric * 1000 : numeric
|
||||
const date = new Date(Number.isFinite(millis) ? millis : value)
|
||||
return Number.isNaN(date.getTime()) ? text(value) : date.toLocaleString()
|
||||
}
|
||||
|
||||
function Field({ label, value }) {
|
||||
return jsxs('div', {
|
||||
className: 'min-w-0 rounded-md border border-(--ui-stroke-tertiary) px-3 py-2',
|
||||
children: [
|
||||
jsx('div', {
|
||||
className: 'text-[0.625rem] font-medium uppercase tracking-wide text-(--ui-text-quaternary)',
|
||||
children: label
|
||||
}),
|
||||
jsx('div', {
|
||||
className: 'mt-1 break-words text-xs text-(--ui-text-secondary)',
|
||||
children: text(value)
|
||||
})
|
||||
]
|
||||
})
|
||||
}
|
||||
|
||||
function SectionHeader({ action, description, title }) {
|
||||
return jsxs('header', {
|
||||
className: 'mb-4 flex flex-wrap items-start justify-between gap-3',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
children: [
|
||||
jsx('h2', { className: 'text-sm font-semibold text-(--ui-text-primary)', children: title }),
|
||||
description
|
||||
? jsx('p', { className: 'mt-1 text-xs leading-5 text-(--ui-text-tertiary)', children: description })
|
||||
: null
|
||||
]
|
||||
}),
|
||||
action
|
||||
]
|
||||
})
|
||||
}
|
||||
|
||||
function QueryState({ empty, query, render }) {
|
||||
if (query.isPending) {
|
||||
return jsx('p', { className: 'py-8 text-xs text-(--ui-text-tertiary)', role: 'status', children: 'Loading…' })
|
||||
}
|
||||
if (query.error) {
|
||||
return jsx(ErrorState, { description: friendlyError(query.error), title: 'Relay unavailable' })
|
||||
}
|
||||
if (empty(query.data)) {
|
||||
return jsx(EmptyState, { description: '', title: 'Nothing to show' })
|
||||
}
|
||||
return render(query.data)
|
||||
}
|
||||
|
||||
function createManagement(ctx) {
|
||||
return function Management() {
|
||||
const t = usePluginI18n(PLUGIN_ID)
|
||||
const profile = useValue(host.state.profile)
|
||||
const client = useQueryClient()
|
||||
const [confirmPrefix, setConfirmPrefix] = useState(null)
|
||||
const [pairing, setPairing] = useState(null)
|
||||
const [copyError, setCopyError] = useState(null)
|
||||
const overviewKey = profileQueryKey(profile, 'overview')
|
||||
const sessionsKey = profileQueryKey(profile, 'sessions')
|
||||
const overview = useQuery({ queryKey: overviewKey, queryFn: () => ctx.rest('/overview'), retry: false })
|
||||
const sessions = useQuery({ queryKey: sessionsKey, queryFn: () => ctx.rest('/sessions'), retry: false })
|
||||
const refresh = () => {
|
||||
void client.invalidateQueries({ queryKey: overviewKey })
|
||||
void client.invalidateQueries({ queryKey: sessionsKey })
|
||||
}
|
||||
const pair = useMutation({
|
||||
mutationFn: () => ctx.rest('/pairing', { method: 'POST', body: { mode: 'auto' } }),
|
||||
onSuccess: result => {
|
||||
setCopyError(null)
|
||||
setPairing(result)
|
||||
}
|
||||
})
|
||||
const revoke = useMutation({
|
||||
mutationFn: prefix => ctx.rest(`/sessions/${encodeURIComponent(prefix)}`, { method: 'DELETE' }),
|
||||
onSuccess: () => {
|
||||
setConfirmPrefix(null)
|
||||
void client.invalidateQueries({ queryKey: sessionsKey })
|
||||
}
|
||||
})
|
||||
const rows = list(sessions.data, 'sessions')
|
||||
|
||||
const copyInvite = async () => {
|
||||
const invite = pairing?.pairing_url || pairing?.qr_payload
|
||||
if (!invite) return
|
||||
if (!(await ctx.os.writeClipboard(invite))) setCopyError(t('pairing.copyFailed'))
|
||||
}
|
||||
|
||||
return jsxs('section', {
|
||||
children: [
|
||||
jsx(SectionHeader, {
|
||||
title: t('management.title'),
|
||||
description: `Profile: ${profile || 'default'}`,
|
||||
action: jsxs('div', {
|
||||
className: 'flex gap-2',
|
||||
children: [
|
||||
jsx(Button, { size: 'sm', variant: 'outline', onClick: refresh, children: t('action.refresh') }),
|
||||
jsx(Button, {
|
||||
size: 'sm',
|
||||
disabled: pair.isPending,
|
||||
onClick: () => pair.mutate(),
|
||||
children: pair.isPending ? 'Creating…' : t('action.pair')
|
||||
})
|
||||
]
|
||||
})
|
||||
}),
|
||||
overview.error
|
||||
? jsx(ErrorState, { title: 'Relay unavailable', description: friendlyError(overview.error) })
|
||||
: overview.data
|
||||
? jsxs('div', {
|
||||
className: 'mb-5 grid gap-2 sm:grid-cols-2 lg:grid-cols-4',
|
||||
children: [
|
||||
jsx(Field, { label: 'Health', value: overview.data.health || overview.data.status }),
|
||||
jsx(Field, { label: 'Version', value: overview.data.version }),
|
||||
jsx(Field, { label: 'Uptime (seconds)', value: overview.data.uptime_seconds }),
|
||||
jsx(Field, { label: 'Connected', value: overview.data.connected_clients ?? overview.data.active_sessions })
|
||||
]
|
||||
})
|
||||
: null,
|
||||
pair.error ? jsx(ErrorState, { title: 'Pairing failed', description: friendlyError(pair.error) }) : null,
|
||||
pairing
|
||||
? jsxs('div', {
|
||||
className: 'mb-5 rounded-md border border-(--ui-accent) p-3',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
className: 'flex flex-wrap items-center justify-between gap-2',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
children: [
|
||||
jsx('h3', { className: 'text-xs font-semibold text-(--ui-text-primary)', children: t('pairing.title') }),
|
||||
jsx('p', { className: 'mt-1 text-xs text-(--ui-text-tertiary)', children: t('pairing.expires') })
|
||||
]
|
||||
}),
|
||||
jsx(Button, { size: 'sm', variant: 'outline', onClick: copyInvite, children: t('action.copy') })
|
||||
]
|
||||
}),
|
||||
pairing.code
|
||||
? jsx('div', { className: 'mt-3 font-mono text-xl tracking-widest', children: pairing.code })
|
||||
: null,
|
||||
jsx('div', {
|
||||
className: 'mt-2 select-all break-all font-mono text-[0.6875rem] text-(--ui-text-tertiary)',
|
||||
children: pairing.pairing_url || pairing.qr_payload || 'Invite created'
|
||||
}),
|
||||
copyError ? jsx('p', { className: 'mt-2 text-xs text-destructive', children: copyError }) : null
|
||||
]
|
||||
})
|
||||
: null,
|
||||
revoke.error ? jsx(ErrorState, { title: 'Revoke failed', description: friendlyError(revoke.error) }) : null,
|
||||
jsx(QueryState, {
|
||||
query: sessions,
|
||||
empty: data => list(data, 'sessions').length === 0,
|
||||
render: () => jsxs('div', {
|
||||
className: 'overflow-hidden rounded-md border border-(--ui-stroke-tertiary)',
|
||||
children: rows.map((session, index) => {
|
||||
const prefix = session.token_prefix || session.prefix || session.id || `session-${index}`
|
||||
const confirming = confirmPrefix === prefix
|
||||
return jsxs('div', {
|
||||
className: 'flex flex-wrap items-center justify-between gap-3 border-b border-(--ui-stroke-tertiary) px-3 py-3 last:border-b-0',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
className: 'min-w-0',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
className: 'flex flex-wrap items-center gap-2',
|
||||
children: [
|
||||
jsx('span', { className: 'font-medium text-xs text-(--ui-text-primary)', children: session.device_name || session.name || prefix }),
|
||||
jsx(Badge, { children: session.device_type || session.client_type || 'device' })
|
||||
]
|
||||
}),
|
||||
jsx('p', {
|
||||
className: 'mt-1 text-[0.6875rem] text-(--ui-text-tertiary)',
|
||||
children: `${prefix} · paired ${formatTime(session.paired_at)}`
|
||||
})
|
||||
]
|
||||
}),
|
||||
confirming
|
||||
? jsxs('div', {
|
||||
className: 'flex items-center gap-2',
|
||||
children: [
|
||||
jsx('span', { className: 'max-w-52 text-right text-[0.6875rem] text-(--ui-text-tertiary)', children: t('session.revokeConfirm') }),
|
||||
jsx(Button, { size: 'sm', variant: 'ghost', onClick: () => setConfirmPrefix(null), children: t('action.cancel') }),
|
||||
jsx(Button, { size: 'sm', disabled: revoke.isPending, onClick: () => revoke.mutate(prefix), children: t('action.confirm') })
|
||||
]
|
||||
})
|
||||
: jsx(Button, { size: 'sm', variant: 'outline', onClick: () => setConfirmPrefix(prefix), children: t('session.revoke') })
|
||||
]
|
||||
}, prefix)
|
||||
})
|
||||
})
|
||||
})
|
||||
]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function createActivity(ctx) {
|
||||
return function Activity() {
|
||||
const t = usePluginI18n(PLUGIN_ID)
|
||||
const profile = useValue(host.state.profile)
|
||||
const query = useQuery({
|
||||
queryKey: profileQueryKey(profile, 'bridge-activity'),
|
||||
queryFn: () => ctx.rest('/bridge-activity?limit=100'),
|
||||
retry: false
|
||||
})
|
||||
return jsxs('section', {
|
||||
children: [
|
||||
jsx(SectionHeader, {
|
||||
title: t('activity.title'),
|
||||
description: 'Recent Relay-owned bridge commands. No live subscription or background polling.'
|
||||
}),
|
||||
jsx(QueryState, {
|
||||
query,
|
||||
empty: data => list(data, 'activity').length === 0,
|
||||
render: data => jsx('div', {
|
||||
className: 'space-y-2',
|
||||
children: list(data, 'activity').map((item, index) => jsxs('div', {
|
||||
className: 'rounded-md border border-(--ui-stroke-tertiary) px-3 py-2',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
className: 'flex flex-wrap items-center gap-2',
|
||||
children: [
|
||||
jsx(StatusDot, { tone: item.ok === false || item.error ? 'bad' : 'good' }),
|
||||
jsx('span', { className: 'text-xs font-medium text-(--ui-text-primary)', children: item.command || item.type || item.action || 'Bridge command' }),
|
||||
jsx('span', { className: 'ml-auto text-[0.6875rem] text-(--ui-text-quaternary)', children: formatTime(item.timestamp || item.created_at || item.at) })
|
||||
]
|
||||
}),
|
||||
jsx('p', { className: 'mt-1 break-words text-[0.6875rem] text-(--ui-text-tertiary)', children: text(item.summary || item.result || item.error || item.status) })
|
||||
]
|
||||
}, item.id || `${index}`))
|
||||
})
|
||||
})
|
||||
]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function createMedia(ctx) {
|
||||
return function Media() {
|
||||
const t = usePluginI18n(PLUGIN_ID)
|
||||
const profile = useValue(host.state.profile)
|
||||
const query = useQuery({
|
||||
queryKey: profileQueryKey(profile, 'media'),
|
||||
queryFn: () => ctx.rest('/media'),
|
||||
retry: false
|
||||
})
|
||||
return jsxs('section', {
|
||||
children: [
|
||||
jsx(SectionHeader, {
|
||||
title: t('media.title'),
|
||||
description: 'Sanitized Relay media metadata. Filesystem paths and bearer tokens are not requested.'
|
||||
}),
|
||||
jsx(QueryState, {
|
||||
query,
|
||||
empty: data => list(data, 'media').length === 0,
|
||||
render: data => jsx('div', {
|
||||
className: 'grid gap-2 sm:grid-cols-2',
|
||||
children: list(data, 'media').map((item, index) => jsxs('div', {
|
||||
className: 'rounded-md border border-(--ui-stroke-tertiary) px-3 py-3',
|
||||
children: [
|
||||
jsx('div', { className: 'truncate text-xs font-medium text-(--ui-text-primary)', children: item.filename || item.name || `Media ${index + 1}` }),
|
||||
jsx('p', { className: 'mt-1 text-[0.6875rem] text-(--ui-text-tertiary)', children: `${text(item.content_type || item.mime_type)} · ${text(item.size_bytes || item.size, 'unknown size')}` }),
|
||||
jsx('p', { className: 'mt-1 text-[0.6875rem] text-(--ui-text-quaternary)', children: `expires ${formatTime(item.expires_at)}` })
|
||||
]
|
||||
}, item.id || item.token_prefix || `${index}`))
|
||||
})
|
||||
})
|
||||
]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function createRemoteAccess(ctx) {
|
||||
return function RemoteAccess() {
|
||||
const t = usePluginI18n(PLUGIN_ID)
|
||||
const profile = useValue(host.state.profile)
|
||||
const client = useQueryClient()
|
||||
const key = profileQueryKey(profile, 'remote-access')
|
||||
const status = useQuery({ queryKey: key, queryFn: () => ctx.rest('/remote-access/status'), retry: false })
|
||||
const [url, setUrl] = useState('')
|
||||
const [confirmAction, setConfirmAction] = useState(null)
|
||||
const [result, setResult] = useState(null)
|
||||
const refresh = () => void client.invalidateQueries({ queryKey: key })
|
||||
const mutate = useMutation({
|
||||
mutationFn: async action => {
|
||||
if (action === 'enable') return ctx.rest('/remote-access/tailscale/enable', { method: 'POST', body: { stack: true } })
|
||||
if (action === 'disable') return ctx.rest('/remote-access/tailscale/disable', { method: 'POST', body: { stack: true } })
|
||||
if (action === 'save') return ctx.rest('/remote-access/public-url', { method: 'PUT', body: { url: url.trim() || null } })
|
||||
if (action === 'probe') return ctx.rest('/remote-access/probe', { method: 'POST', body: { candidates: [url.trim()] } })
|
||||
throw new Error('Unknown remote access action')
|
||||
},
|
||||
onSuccess: value => {
|
||||
setResult(value)
|
||||
setConfirmAction(null)
|
||||
refresh()
|
||||
}
|
||||
})
|
||||
const actionButton = (action, label, destructive = false) => confirmAction === action
|
||||
? jsxs('span', {
|
||||
className: 'inline-flex items-center gap-1',
|
||||
children: [
|
||||
jsx(Button, { size: 'sm', variant: 'ghost', onClick: () => setConfirmAction(null), children: t('action.cancel') }),
|
||||
jsx(Button, { size: 'sm', disabled: mutate.isPending, onClick: () => mutate.mutate(action), children: t('action.confirm') })
|
||||
]
|
||||
})
|
||||
: jsx(Button, { size: 'sm', variant: destructive ? 'outline' : 'outline', onClick: () => setConfirmAction(action), children: label })
|
||||
|
||||
return jsxs('section', {
|
||||
children: [
|
||||
jsx(SectionHeader, {
|
||||
title: t('remote.title'),
|
||||
description: 'All changes require a labeled action and a second confirmation.'
|
||||
}),
|
||||
jsx(QueryState, {
|
||||
query: status,
|
||||
empty: data => !data,
|
||||
render: data => jsxs(Fragment, {
|
||||
children: [
|
||||
jsxs('div', {
|
||||
className: 'grid gap-2 sm:grid-cols-2',
|
||||
children: [
|
||||
jsx(Field, { label: 'Tailscale', value: data.tailscale?.state || (data.tailscale?.enabled ? 'enabled' : 'disabled') }),
|
||||
jsx(Field, { label: 'Public URL', value: data.public?.url || data.public_url }),
|
||||
jsx(Field, { label: 'Secure Link', value: data.secure_link?.state || (data.secure_link?.enabled ? 'enabled' : 'disabled') }),
|
||||
jsx(Field, { label: 'Upstream helper', value: data.upstream_canonical ? 'available' : 'not detected' })
|
||||
]
|
||||
}),
|
||||
jsxs('div', {
|
||||
className: 'mt-4 flex flex-wrap gap-2',
|
||||
children: [
|
||||
actionButton('enable', t('remote.enable')),
|
||||
actionButton('disable', t('remote.disable'), true),
|
||||
jsx(Button, { size: 'sm', variant: 'ghost', onClick: refresh, children: t('action.refresh') })
|
||||
]
|
||||
})
|
||||
]
|
||||
})
|
||||
}),
|
||||
jsxs('div', {
|
||||
className: 'mt-5 rounded-md border border-(--ui-stroke-tertiary) p-3',
|
||||
children: [
|
||||
jsx('label', { className: 'text-xs font-medium text-(--ui-text-secondary)', htmlFor: 'hermes-relay-public-url', children: 'Public relay URL' }),
|
||||
jsx(Input, {
|
||||
id: 'hermes-relay-public-url',
|
||||
className: 'mt-2',
|
||||
placeholder: 'https://relay.example.com',
|
||||
value: url,
|
||||
onChange: event => setUrl(event.target.value)
|
||||
}),
|
||||
jsxs('div', {
|
||||
className: 'mt-2 flex flex-wrap gap-2',
|
||||
children: [
|
||||
actionButton('save', t('remote.save')),
|
||||
actionButton('probe', t('remote.probe'))
|
||||
]
|
||||
})
|
||||
]
|
||||
}),
|
||||
mutate.error ? jsx(ErrorState, { title: 'Remote access action failed', description: friendlyError(mutate.error) }) : null,
|
||||
result ? jsx('pre', { className: 'mt-3 max-h-48 overflow-auto whitespace-pre-wrap rounded-md border border-(--ui-stroke-tertiary) p-3 text-[0.6875rem] text-(--ui-text-tertiary)', children: JSON.stringify(result, null, 2) }) : null
|
||||
]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function createRelayPane(ctx) {
|
||||
const Management = createManagement(ctx)
|
||||
const Activity = createActivity(ctx)
|
||||
const Media = createMedia(ctx)
|
||||
const RemoteAccess = createRemoteAccess(ctx)
|
||||
|
||||
return function RelayPane() {
|
||||
const t = usePluginI18n(PLUGIN_ID)
|
||||
const [tab, setTab] = useState('management')
|
||||
const tabs = [
|
||||
['management', 'server', t('management.title')],
|
||||
['activity', 'pulse', t('activity.title')],
|
||||
['media', 'file-media', t('media.title')],
|
||||
['remote', 'remote-explorer', t('remote.title')]
|
||||
]
|
||||
const content = tab === 'activity'
|
||||
? jsx(Activity, {})
|
||||
: tab === 'media'
|
||||
? jsx(Media, {})
|
||||
: tab === 'remote'
|
||||
? jsx(RemoteAccess, {})
|
||||
: jsx(Management, {})
|
||||
|
||||
return jsxs('div', {
|
||||
className: 'flex h-full min-h-0 flex-col',
|
||||
children: [
|
||||
jsxs('header', {
|
||||
className: 'border-b border-(--ui-stroke-tertiary) px-4 py-3',
|
||||
children: [
|
||||
jsxs('div', {
|
||||
className: 'flex items-center gap-2',
|
||||
children: [
|
||||
jsx(Codicon, { name: 'radio-tower', size: '1rem' }),
|
||||
jsx('h1', { className: 'text-sm font-semibold text-(--ui-text-primary)', children: t('pane.title') })
|
||||
]
|
||||
}),
|
||||
jsx('p', { className: 'mt-1 text-xs text-(--ui-text-tertiary)', children: t('pane.description') }),
|
||||
jsx('nav', {
|
||||
'aria-label': 'Hermes Relay sections',
|
||||
className: 'mt-3 flex flex-wrap gap-1',
|
||||
children: tabs.map(([id, icon, label]) => jsx(Button, {
|
||||
size: 'sm',
|
||||
variant: tab === id ? 'secondary' : 'ghost',
|
||||
onClick: () => setTab(id),
|
||||
children: jsxs(Fragment, { children: [jsx(Codicon, { name: icon, size: '0.75rem' }), label] })
|
||||
}, id))
|
||||
})
|
||||
]
|
||||
}),
|
||||
jsx('main', { className: 'min-h-0 flex-1 overflow-y-auto p-4', children: content })
|
||||
]
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
function RelayStatus({ open }) {
|
||||
const t = usePluginI18n(PLUGIN_ID)
|
||||
return jsx(Tip, {
|
||||
label: t('status.detail'),
|
||||
children: jsxs('button', {
|
||||
type: 'button',
|
||||
className: 'inline-flex h-full items-center gap-1.5 px-1.5 text-[0.6875rem] text-(--ui-text-tertiary) hover:bg-(--chrome-action-hover) hover:text-(--ui-text-primary)',
|
||||
onClick: open,
|
||||
children: [jsx(StatusDot, { tone: 'muted' }), jsx('span', { children: t('status.label') })]
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
const plugin = {
|
||||
id: PLUGIN_ID,
|
||||
name: 'Hermes Relay',
|
||||
description: 'Profile-scoped Hermes-Relay management through the official Desktop Plugin SDK.',
|
||||
defaultEnabled: false,
|
||||
register(ctx) {
|
||||
ctx.i18n.register(messages)
|
||||
const RelayPane = createRelayPane(ctx)
|
||||
let paneRegistered = false
|
||||
|
||||
const open = () => {
|
||||
if (!paneRegistered) {
|
||||
ctx.register({
|
||||
id: 'management',
|
||||
area: PANES_AREA,
|
||||
title: ctx.i18n.t('pane.title'),
|
||||
data: {
|
||||
closeBehavior: 'dismiss',
|
||||
placement: 'right',
|
||||
dock: { pane: 'workspace', pos: 'right' },
|
||||
width: 'min(420px, 42vw)'
|
||||
},
|
||||
render: () => jsx(RelayPane, {})
|
||||
})
|
||||
paneRegistered = true
|
||||
}
|
||||
ctx.panes.reveal('management')
|
||||
}
|
||||
|
||||
ctx.onDispose(() => {
|
||||
paneRegistered = false
|
||||
})
|
||||
|
||||
ctx.registerMany([
|
||||
{
|
||||
id: 'nav',
|
||||
area: SIDEBAR_NAV_AREA,
|
||||
order: 75,
|
||||
data: { codicon: 'radio-tower', label: ctx.i18n.t('nav.label'), onSelect: open }
|
||||
},
|
||||
{
|
||||
id: 'status',
|
||||
area: STATUSBAR_AREAS.right,
|
||||
order: 110,
|
||||
render: () => jsx(RelayStatus, { open })
|
||||
},
|
||||
{
|
||||
id: 'open',
|
||||
area: PALETTE_AREA,
|
||||
data: {
|
||||
id: 'hermes-relay.open',
|
||||
label: ctx.i18n.t('action.open'),
|
||||
keywords: ['relay', 'phone', 'paired devices', 'bridge', 'media', 'remote access'],
|
||||
run: open
|
||||
}
|
||||
}
|
||||
])
|
||||
}
|
||||
}
|
||||
|
||||
export default plugin
|
||||
@@ -0,0 +1,40 @@
|
||||
const sdk = `
|
||||
export const PANES_AREA = 'panes'
|
||||
export const SIDEBAR_NAV_AREA = 'sidebar.nav'
|
||||
export const STATUSBAR_AREAS = { right: 'statusBar.right' }
|
||||
export const PALETTE_AREA = 'palette'
|
||||
export const Badge = 'Badge'
|
||||
export const Button = 'Button'
|
||||
export const Codicon = 'Codicon'
|
||||
export const EmptyState = 'EmptyState'
|
||||
export const ErrorState = 'ErrorState'
|
||||
export const Input = 'Input'
|
||||
export const StatusDot = 'StatusDot'
|
||||
export const Tip = 'Tip'
|
||||
export const host = { state: { profile: { get: () => 'default' } }, notify: () => {} }
|
||||
export const useMutation = () => ({ mutate: () => {}, isPending: false })
|
||||
export const usePluginI18n = () => key => key
|
||||
export const useQuery = () => ({ data: null, error: null, isPending: false, refetch: () => {} })
|
||||
export const useQueryClient = () => ({ invalidateQueries: () => {} })
|
||||
export const useValue = atom => atom.get()
|
||||
`
|
||||
|
||||
const react = `export const useState = initial => [initial, () => {}]`
|
||||
const jsx = `
|
||||
export const Fragment = Symbol.for('fixture.fragment')
|
||||
export const jsx = (type, props) => ({ type, props: props || {} })
|
||||
export const jsxs = jsx
|
||||
`
|
||||
|
||||
export async function resolve(specifier, context, nextResolve) {
|
||||
if (specifier === '@hermes/plugin-sdk') {
|
||||
return { shortCircuit: true, url: `data:text/javascript,${encodeURIComponent(sdk)}` }
|
||||
}
|
||||
if (specifier === 'react') {
|
||||
return { shortCircuit: true, url: `data:text/javascript,${encodeURIComponent(react)}` }
|
||||
}
|
||||
if (specifier === 'react/jsx-runtime') {
|
||||
return { shortCircuit: true, url: `data:text/javascript,${encodeURIComponent(jsx)}` }
|
||||
}
|
||||
return nextResolve(specifier, context)
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import test from 'node:test'
|
||||
|
||||
import plugin, { profileQueryKey } from '../plugin.js'
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url))
|
||||
|
||||
function harness() {
|
||||
const contributions = []
|
||||
const disposed = []
|
||||
const revealed = []
|
||||
const cleanups = []
|
||||
const ctx = {
|
||||
i18n: {
|
||||
register() {
|
||||
const dispose = () => disposed.push('i18n')
|
||||
cleanups.push(dispose)
|
||||
return dispose
|
||||
},
|
||||
t: key => key
|
||||
},
|
||||
onDispose(fn) {
|
||||
cleanups.push(fn)
|
||||
},
|
||||
os: { writeClipboard: async () => true },
|
||||
panes: { reveal: id => revealed.push(id) },
|
||||
register(contribution) {
|
||||
contributions.push(contribution)
|
||||
const dispose = () => disposed.push(contribution.id)
|
||||
cleanups.push(dispose)
|
||||
return dispose
|
||||
},
|
||||
registerMany(batch) {
|
||||
contributions.push(...batch)
|
||||
const dispose = () => disposed.push(...batch.map(item => item.id))
|
||||
cleanups.push(dispose)
|
||||
return dispose
|
||||
},
|
||||
rest: async () => ({})
|
||||
}
|
||||
return { cleanups, contributions, ctx, disposed, revealed }
|
||||
}
|
||||
|
||||
test('ships opt-in and registers no pane or network work at load', () => {
|
||||
const state = harness()
|
||||
let restCalls = 0
|
||||
state.ctx.rest = async () => {
|
||||
restCalls += 1
|
||||
return {}
|
||||
}
|
||||
|
||||
plugin.register(state.ctx)
|
||||
|
||||
assert.equal(plugin.defaultEnabled, false)
|
||||
assert.deepEqual(state.contributions.map(item => item.id), ['nav', 'status', 'open'])
|
||||
assert.equal(state.contributions.some(item => item.area === 'panes'), false)
|
||||
assert.equal(state.revealed.length, 0)
|
||||
assert.equal(restCalls, 0)
|
||||
})
|
||||
|
||||
test('startup, reconnect, profile-change, and reload registration never reveal the pane', () => {
|
||||
for (const lifecycle of ['startup', 'reconnect', 'profile-change', 'reload']) {
|
||||
const state = harness()
|
||||
plugin.register(state.ctx)
|
||||
assert.equal(state.contributions.some(item => item.area === 'panes'), false, lifecycle)
|
||||
assert.deepEqual(state.revealed, [], lifecycle)
|
||||
}
|
||||
})
|
||||
|
||||
test('every labeled entry point opens lazily and repeated opens reuse the pane', () => {
|
||||
const state = harness()
|
||||
plugin.register(state.ctx)
|
||||
const nav = state.contributions.find(item => item.id === 'nav')
|
||||
const status = state.contributions.find(item => item.id === 'status')
|
||||
const command = state.contributions.find(item => item.id === 'open')
|
||||
|
||||
nav.data.onSelect()
|
||||
const statusElement = status.render()
|
||||
const renderedStatus = statusElement.type(statusElement.props)
|
||||
renderedStatus.props.children.props.onClick()
|
||||
command.data.run()
|
||||
|
||||
const panes = state.contributions.filter(item => item.area === 'panes')
|
||||
assert.equal(panes.length, 1)
|
||||
assert.equal(panes[0].data.closeBehavior, 'dismiss')
|
||||
assert.equal(panes[0].data.placement, 'right')
|
||||
assert.deepEqual(state.revealed, ['management', 'management', 'management'])
|
||||
})
|
||||
|
||||
test('unload disposes entry points, lazy pane, locale bundle, and module state', () => {
|
||||
const state = harness()
|
||||
plugin.register(state.ctx)
|
||||
state.contributions.find(item => item.id === 'nav').data.onSelect()
|
||||
|
||||
for (const cleanup of [...state.cleanups].reverse()) cleanup()
|
||||
|
||||
assert.ok(state.disposed.includes('management'))
|
||||
assert.ok(state.disposed.includes('nav'))
|
||||
assert.ok(state.disposed.includes('i18n'))
|
||||
})
|
||||
|
||||
test('query keys isolate cached backend state by active profile', () => {
|
||||
assert.deepEqual(profileQueryKey('default', 'overview'), ['hermes-relay', 'default', 'overview'])
|
||||
assert.notDeepEqual(profileQueryKey('work', 'sessions'), profileQueryKey('personal', 'sessions'))
|
||||
})
|
||||
|
||||
test('unified package uses only runtime-plugin imports and contains no auto-open primitive', async () => {
|
||||
const source = await readFile(resolve(here, '..', 'plugin.js'), 'utf8')
|
||||
const imports = [...source.matchAll(/from\s+['\"]([^'\"]+)['\"]/g)].map(match => match[1])
|
||||
assert.deepEqual([...new Set(imports)].sort(), ['@hermes/plugin-sdk', 'react', 'react/jsx-runtime'])
|
||||
assert.equal(/setInterval|setTimeout|host\.navigate|window\.location|focus\s*\(/.test(source), false)
|
||||
assert.match(source, /const open = \(\) => \{/)
|
||||
assert.match(source, /paneRegistered = true/)
|
||||
})
|
||||
|
||||
test('desktop half is bundled beside the existing dashboard half', async () => {
|
||||
const pluginRoot = resolve(here, '..', '..')
|
||||
const manifest = await readFile(resolve(pluginRoot, 'plugin.yaml'), 'utf8')
|
||||
const dashboard = await readFile(resolve(pluginRoot, 'dashboard', 'manifest.json'), 'utf8')
|
||||
const desktop = await readFile(resolve(pluginRoot, 'desktop', 'plugin.js'), 'utf8')
|
||||
assert.match(manifest, /^name:\s+hermes-relay/m)
|
||||
assert.equal(JSON.parse(dashboard).name, 'hermes-relay')
|
||||
assert.match(desktop, /id:\s*PLUGIN_ID/)
|
||||
})
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
name: hermes-relay
|
||||
manifest_version: 1
|
||||
version: 1.7.0
|
||||
version: 1.8.0
|
||||
description: "Hermes-Relay plugin for QR pairing, relay sessions, dashboard management, remote desktop/phone tooling, and optional legacy compatibility diagnostics. Standard chat, Manage, and dashboard voice remain vanilla upstream Hermes surfaces."
|
||||
author: Axiom Labs
|
||||
# All three are OPTIONAL — only needed if you use the relay's extra /
|
||||
|
||||
@@ -19,7 +19,7 @@ See ``plugin/relay/server.py`` for the aiohttp server,
|
||||
# Desktop releases use desktop/package.json and desktop-v* tags. The /health endpoint
|
||||
# reports this plugin version, and stale values make live diagnosis harder than
|
||||
# it should be.
|
||||
__version__ = "1.7.0"
|
||||
__version__ = "1.8.0"
|
||||
|
||||
from .server import create_app, main # noqa: E402 — must come after __version__
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ Two jobs over the same WSS ``desktop`` envelope stream:
|
||||
client auth re-advertises if the relay restarts.
|
||||
|
||||
Wire envelopes (frozen — do not rename fields):
|
||||
* ``desktop.command`` — server → client: ``{request_id, tool, args}``
|
||||
* ``desktop.command`` — server → client: ``{request_id, tool, args, control_session?}``
|
||||
* ``desktop.response`` — client → server: ``{request_id, status, result}``
|
||||
* ``desktop.status`` — client → server: ``{advertised_tools, host?, platform?, cwd?, ...}``
|
||||
* ``desktop.workspace`` — client → server: opaque dict (``cwd`` / ``git_root`` / ``git_branch`` / ...)
|
||||
@@ -58,12 +58,33 @@ _REDACT_KEYS = frozenset({"password", "token", "secret", "otp", "bearer", "api_k
|
||||
|
||||
# Cap for the recent-commands ring buffer.
|
||||
RECENT_COMMANDS_MAX = 100
|
||||
CONTROL_SESSIONS_MAX = 256
|
||||
CONTROL_SESSION_IDLE_SECONDS = 900.0
|
||||
|
||||
|
||||
class DesktopError(Exception):
|
||||
"""Raised when a desktop command cannot be dispatched or times out."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DesktopRequesterContext:
|
||||
"""Trusted caller context supplied by the loopback HTTP boundary."""
|
||||
|
||||
requester_device_id: str | None = None
|
||||
chat_session_id: str | None = None
|
||||
run_id: str | None = None
|
||||
profile: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class _ControlSessionRecord:
|
||||
session_id: str
|
||||
touched_at: float
|
||||
target_ws: web.WebSocketResponse
|
||||
target_device_id: str
|
||||
expiry_task: asyncio.Task[None]
|
||||
|
||||
|
||||
def _redact_args(value: Any) -> Any:
|
||||
"""Return a copy of ``value`` with sensitive-key values replaced by
|
||||
``"[redacted]"``. Recurses into nested dicts and lists.
|
||||
@@ -190,6 +211,108 @@ class DesktopHandler:
|
||||
# registry. Cleared per-ws in :meth:`detach_ws` so disconnected
|
||||
# clients don't leak session structs.
|
||||
self._sessions: dict[web.WebSocketResponse, DesktopSession] = {}
|
||||
# Stable opaque ids for active computer-control runs. Callers provide
|
||||
# only trusted context fields; this handler always owns the id and
|
||||
# binds every emitted identity to the selected target + request id.
|
||||
self._control_sessions: dict[tuple[str, str, str, str, str], _ControlSessionRecord] = {}
|
||||
|
||||
async def _send_control_session_end(
|
||||
self,
|
||||
record: _ControlSessionRecord,
|
||||
reason: str,
|
||||
) -> None:
|
||||
if record.target_ws.closed:
|
||||
return
|
||||
await record.target_ws.send_str(json.dumps({
|
||||
"channel": "desktop",
|
||||
"type": "desktop.control_session_end",
|
||||
"payload": {
|
||||
"version": 1,
|
||||
"id": record.session_id,
|
||||
"target_device_id": record.target_device_id,
|
||||
"reason": reason[:256],
|
||||
},
|
||||
}))
|
||||
|
||||
async def _expire_control_session(
|
||||
self,
|
||||
key: tuple[str, str, str, str, str],
|
||||
session_id: str,
|
||||
) -> None:
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(CONTROL_SESSION_IDLE_SECONDS)
|
||||
record = self._control_sessions.get(key)
|
||||
if record is None or record.session_id != session_id:
|
||||
return
|
||||
if time.monotonic() - record.touched_at < CONTROL_SESSION_IDLE_SECONDS:
|
||||
continue
|
||||
self._control_sessions.pop(key, None)
|
||||
await self._send_control_session_end(record, "control session idle timeout")
|
||||
return
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
|
||||
async def _control_session(
|
||||
self,
|
||||
*,
|
||||
request_id: str,
|
||||
target_device_id: str,
|
||||
target_ws: web.WebSocketResponse,
|
||||
requester: DesktopRequesterContext | None,
|
||||
) -> dict[str, Any] | None:
|
||||
context = requester or DesktopRequesterContext()
|
||||
# A server-owned UUID is not useful authority by itself. Require an
|
||||
# executor-authenticated requester and run before advertising the
|
||||
# identity; older/unauthenticated callers stay in compatibility mode.
|
||||
if not context.requester_device_id or not context.run_id:
|
||||
return None
|
||||
now = time.monotonic()
|
||||
key = (
|
||||
context.requester_device_id,
|
||||
context.chat_session_id or "",
|
||||
context.run_id or "",
|
||||
context.profile or "",
|
||||
target_device_id or "legacy-desktop",
|
||||
)
|
||||
# One requester/chat/profile/target may own only one active run. End
|
||||
# the prior authority before minting a replacement for a new run.
|
||||
transitions = [
|
||||
(other_key, record)
|
||||
for other_key, record in self._control_sessions.items()
|
||||
if other_key[0] == key[0] and other_key[1] == key[1]
|
||||
and other_key[3] == key[3] and other_key[4] == key[4]
|
||||
and other_key[2] != key[2]
|
||||
]
|
||||
for other_key, record in transitions:
|
||||
self._control_sessions.pop(other_key, None)
|
||||
record.expiry_task.cancel()
|
||||
await self._send_control_session_end(record, "requester run changed")
|
||||
current = self._control_sessions.get(key)
|
||||
if current is None:
|
||||
if len(self._control_sessions) >= CONTROL_SESSIONS_MAX:
|
||||
oldest = min(self._control_sessions, key=lambda item: self._control_sessions[item].touched_at)
|
||||
evicted = self._control_sessions.pop(oldest)
|
||||
evicted.expiry_task.cancel()
|
||||
await self._send_control_session_end(evicted, "control session capacity eviction")
|
||||
session_id = f"control-{uuid.uuid4()}"
|
||||
task = asyncio.create_task(self._expire_control_session(key, session_id))
|
||||
self._control_sessions[key] = _ControlSessionRecord(
|
||||
session_id, now, target_ws, key[4], task
|
||||
)
|
||||
else:
|
||||
session_id = current.session_id
|
||||
current.touched_at = now
|
||||
return {
|
||||
"version": 1,
|
||||
"id": session_id,
|
||||
"request_id": request_id,
|
||||
"requester_device_id": key[0],
|
||||
"target_device_id": key[4],
|
||||
**({"chat_session_id": context.chat_session_id} if context.chat_session_id else {}),
|
||||
**({"run_id": context.run_id} if context.run_id else {}),
|
||||
**({"profile": context.profile} if context.profile else {}),
|
||||
}
|
||||
|
||||
# ── Envelope dispatch ────────────────────────────────────────────────
|
||||
|
||||
@@ -389,6 +512,7 @@ class DesktopHandler:
|
||||
method: str,
|
||||
args: dict[str, Any] | None = None,
|
||||
device: str | None = None,
|
||||
requester: DesktopRequesterContext | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Dispatch a ``desktop_*`` tool call to the connected client.
|
||||
|
||||
@@ -433,6 +557,15 @@ class DesktopHandler:
|
||||
else None
|
||||
),
|
||||
}
|
||||
if method.startswith("desktop_computer_"):
|
||||
control_session = await self._control_session(
|
||||
request_id=request_id,
|
||||
target_device_id=str(command_payload["target_device_id"] or "legacy-desktop"),
|
||||
target_ws=ws,
|
||||
requester=requester,
|
||||
)
|
||||
if control_session is not None:
|
||||
command_payload["control_session"] = control_session
|
||||
logger.info(
|
||||
"desktop >>> %s args=%s",
|
||||
method,
|
||||
@@ -736,6 +869,18 @@ class DesktopHandler:
|
||||
"desktop: session detached (had workspace from %s)",
|
||||
session.workspace_context.get("hostname", "?"),
|
||||
)
|
||||
if session is not None:
|
||||
target_device_id = (
|
||||
session.device_id
|
||||
or str(session.client_status.get("device_id", "") or "").strip()
|
||||
)
|
||||
if target_device_id:
|
||||
removed = [
|
||||
key for key in self._control_sessions if key[4] == target_device_id
|
||||
]
|
||||
for key in removed:
|
||||
record = self._control_sessions.pop(key)
|
||||
record.expiry_task.cancel()
|
||||
|
||||
# Tool-routing cleanup — only if this ws was the latched client.
|
||||
if self.client_ws is ws:
|
||||
@@ -774,6 +919,9 @@ class DesktopHandler:
|
||||
self.client_ws = None
|
||||
self.advertised_tools = set()
|
||||
self._sessions.clear()
|
||||
for record in self._control_sessions.values():
|
||||
record.expiry_task.cancel()
|
||||
self._control_sessions.clear()
|
||||
await self._fail_pending("Relay server shutting down")
|
||||
|
||||
|
||||
|
||||
+33
-1
@@ -1273,8 +1273,40 @@ async def handle_desktop_dispatch(request: web.Request) -> web.Response:
|
||||
device = args.pop("device", None) or args.pop("device_id", None)
|
||||
if not isinstance(device, str):
|
||||
device = None
|
||||
# These headers are emitted by the in-process Hermes tool wrapper and are
|
||||
# not part of the model-facing schema. The route is loopback-only; the
|
||||
# Relay still mints the opaque id and binds target/request fields itself.
|
||||
from .channels.desktop import DesktopRequesterContext
|
||||
|
||||
def _context_header(name: str) -> str | None:
|
||||
value = request.headers.get(name, "").strip()
|
||||
if not value:
|
||||
return None
|
||||
return "".join(ch for ch in value if ch.isprintable())[:256] or None
|
||||
|
||||
chat_session_id = _context_header("X-Hermes-Relay-Chat-Session")
|
||||
requester_device_id: str | None = None
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
if auth_header.startswith("Bearer "):
|
||||
# A supplied bearer must validate; never downgrade a bad credential to
|
||||
# loopback trust. This yields the real paired requester device.
|
||||
_server, authenticated_session = _require_bearer_session(request)
|
||||
requester_device_id = authenticated_session.device_id or None
|
||||
elif chat_session_id:
|
||||
# Standard plugin calls are loopback-only. Bind their principal to the
|
||||
# executor-owned Hermes session rather than inventing a host identity.
|
||||
# Same-user localhost processes are inside this existing trust boundary.
|
||||
requester_device_id = f"hermes-agent:{chat_session_id}"
|
||||
requester = DesktopRequesterContext(
|
||||
requester_device_id=requester_device_id,
|
||||
chat_session_id=chat_session_id,
|
||||
run_id=_context_header("X-Hermes-Relay-Run-Id"),
|
||||
profile=_context_header("X-Hermes-Relay-Profile"),
|
||||
)
|
||||
try:
|
||||
result = await server.desktop.handle_command(tool_name, args, device=device)
|
||||
result = await server.desktop.handle_command(
|
||||
tool_name, args, device=device, requester=requester
|
||||
)
|
||||
return web.json_response(result)
|
||||
except Exception as exc: # DesktopError or asyncio.TimeoutError
|
||||
msg = str(exc) or exc.__class__.__name__
|
||||
|
||||
@@ -9,7 +9,11 @@ from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, Mock, patch
|
||||
|
||||
from plugin.relay.channels.desktop import DesktopError, DesktopHandler
|
||||
from plugin.relay.channels.desktop import (
|
||||
DesktopError,
|
||||
DesktopHandler,
|
||||
DesktopRequesterContext,
|
||||
)
|
||||
from plugin.relay.server import handle_desktop_dispatch
|
||||
from plugin.tools import desktop_tool
|
||||
|
||||
@@ -35,7 +39,11 @@ def _status(host: str) -> dict[str, Any]:
|
||||
"type": "desktop.status",
|
||||
"payload": {
|
||||
"host": host,
|
||||
"advertised_tools": ["desktop_powershell", "desktop_read_file"],
|
||||
"advertised_tools": [
|
||||
"desktop_powershell",
|
||||
"desktop_read_file",
|
||||
"desktop_computer_snapshot",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
@@ -119,6 +127,30 @@ class DesktopMultiDeviceTests(unittest.IsolatedAsyncioTestCase):
|
||||
self.assertEqual(post.call_args.kwargs["json"]["device"], "desktop-1")
|
||||
self.assertEqual(post.call_args.kwargs["json"]["script"], "'ok'")
|
||||
|
||||
async def test_tool_dispatch_uses_executor_context_and_drops_model_identity(self) -> None:
|
||||
response = Mock(status_code=200)
|
||||
response.json.return_value = {"ok": True, "result": {}}
|
||||
with patch.object(desktop_tool.requests, "post", return_value=response) as post:
|
||||
desktop_tool._HANDLERS["desktop_computer_action"](
|
||||
{
|
||||
"action": "click",
|
||||
"x": 10,
|
||||
"y": 20,
|
||||
"control_session": {"id": "model-forged"},
|
||||
"control_context": {"run_id": "model-forged"},
|
||||
},
|
||||
session_id="chat-authenticated",
|
||||
task_id="run-authenticated",
|
||||
profile="default",
|
||||
)
|
||||
body = post.call_args.kwargs["json"]
|
||||
headers = post.call_args.kwargs["headers"]
|
||||
self.assertNotIn("control_session", body)
|
||||
self.assertNotIn("control_context", body)
|
||||
self.assertEqual(headers["X-Hermes-Relay-Chat-Session"], "chat-authenticated")
|
||||
self.assertEqual(headers["X-Hermes-Relay-Run-Id"], "run-authenticated")
|
||||
self.assertEqual(headers["X-Hermes-Relay-Profile"], "default")
|
||||
|
||||
async def test_http_dispatch_strips_selector_before_client_forwarding(self) -> None:
|
||||
desktop = Mock()
|
||||
desktop.handle_command = AsyncMock(return_value={"ok": True, "result": {}})
|
||||
@@ -126,6 +158,7 @@ class DesktopMultiDeviceTests(unittest.IsolatedAsyncioTestCase):
|
||||
remote="127.0.0.1",
|
||||
app={"server": Mock(desktop=desktop)},
|
||||
match_info={"tool_name": "desktop_powershell"},
|
||||
headers={},
|
||||
)
|
||||
request.json = AsyncMock(
|
||||
return_value={"script": "'ok'", "device": "desktop-1"}
|
||||
@@ -138,8 +171,233 @@ class DesktopMultiDeviceTests(unittest.IsolatedAsyncioTestCase):
|
||||
"desktop_powershell",
|
||||
{"script": "'ok'"},
|
||||
device="desktop-1",
|
||||
requester=DesktopRequesterContext(),
|
||||
)
|
||||
|
||||
async def test_http_dispatch_binds_loopback_executor_context(self) -> None:
|
||||
desktop = Mock()
|
||||
desktop.handle_command = AsyncMock(return_value={"ok": True, "result": {}})
|
||||
request = Mock(
|
||||
remote="127.0.0.1",
|
||||
app={"server": Mock(desktop=desktop)},
|
||||
match_info={"tool_name": "desktop_computer_snapshot"},
|
||||
headers={
|
||||
"X-Hermes-Relay-Chat-Session": "chat-1",
|
||||
"X-Hermes-Relay-Run-Id": "run-1",
|
||||
"X-Hermes-Relay-Profile": "default",
|
||||
},
|
||||
)
|
||||
request.json = AsyncMock(return_value={"device": "desktop-1"})
|
||||
|
||||
response = await handle_desktop_dispatch(request)
|
||||
|
||||
self.assertEqual(response.status, 200)
|
||||
desktop.handle_command.assert_awaited_once_with(
|
||||
"desktop_computer_snapshot",
|
||||
{},
|
||||
device="desktop-1",
|
||||
requester=DesktopRequesterContext(
|
||||
requester_device_id="hermes-agent:chat-1",
|
||||
chat_session_id="chat-1",
|
||||
run_id="run-1",
|
||||
profile="default",
|
||||
),
|
||||
)
|
||||
|
||||
async def test_http_dispatch_prefers_authenticated_paired_requester(self) -> None:
|
||||
desktop = Mock()
|
||||
desktop.handle_command = AsyncMock(return_value={"ok": True, "result": {}})
|
||||
server = Mock(desktop=desktop)
|
||||
request = Mock(
|
||||
remote="127.0.0.1",
|
||||
app={"server": server},
|
||||
match_info={"tool_name": "desktop_computer_snapshot"},
|
||||
headers={
|
||||
"Authorization": "Bearer authenticated-token",
|
||||
"X-Hermes-Relay-Chat-Session": "chat-1",
|
||||
"X-Hermes-Relay-Run-Id": "run-1",
|
||||
},
|
||||
)
|
||||
request.json = AsyncMock(return_value={"device": "desktop-1"})
|
||||
paired_session = Mock(device_id="paired-phone-1")
|
||||
|
||||
with patch(
|
||||
"plugin.relay.server._require_bearer_session",
|
||||
return_value=(server, paired_session),
|
||||
):
|
||||
response = await handle_desktop_dispatch(request)
|
||||
|
||||
self.assertEqual(response.status, 200)
|
||||
requester = desktop.handle_command.await_args.kwargs["requester"]
|
||||
self.assertEqual(requester.requester_device_id, "paired-phone-1")
|
||||
self.assertEqual(requester.chat_session_id, "chat-1")
|
||||
self.assertEqual(requester.run_id, "run-1")
|
||||
|
||||
async def test_computer_control_session_is_server_owned_stable_and_request_bound(self) -> None:
|
||||
handler, office, _laptop = await _register_two()
|
||||
requester = DesktopRequesterContext(
|
||||
requester_device_id="paired-agent-1",
|
||||
chat_session_id="chat-1",
|
||||
run_id="run-1",
|
||||
profile="default",
|
||||
)
|
||||
|
||||
async def dispatch_once() -> dict[str, Any]:
|
||||
task = asyncio.create_task(
|
||||
handler.handle_command(
|
||||
"desktop_computer_snapshot",
|
||||
{},
|
||||
device="desktop-1",
|
||||
requester=requester,
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0)
|
||||
payload = office.sent[-1]["payload"]
|
||||
await handler.handle(
|
||||
office, # type: ignore[arg-type]
|
||||
{
|
||||
"channel": "desktop",
|
||||
"type": "desktop.response",
|
||||
"payload": {"request_id": payload["request_id"], "ok": True, "result": {}},
|
||||
},
|
||||
)
|
||||
await asyncio.wait_for(task, timeout=1)
|
||||
return payload
|
||||
|
||||
first = await dispatch_once()
|
||||
second = await dispatch_once()
|
||||
first_control = first["control_session"]
|
||||
second_control = second["control_session"]
|
||||
self.assertEqual(first_control["version"], 1)
|
||||
self.assertRegex(first_control["id"], r"^control-[0-9a-f-]+$")
|
||||
self.assertEqual(first_control["id"], second_control["id"])
|
||||
self.assertNotEqual(first_control["request_id"], second_control["request_id"])
|
||||
self.assertEqual(first_control["request_id"], first["request_id"])
|
||||
self.assertEqual(first_control["requester_device_id"], "paired-agent-1")
|
||||
self.assertEqual(first_control["target_device_id"], "desktop-1")
|
||||
self.assertEqual(first_control["chat_session_id"], "chat-1")
|
||||
self.assertEqual(first_control["run_id"], "run-1")
|
||||
self.assertEqual(len(handler._control_sessions), 1)
|
||||
await handler.detach_ws(office, "test disconnect") # type: ignore[arg-type]
|
||||
self.assertEqual(handler._control_sessions, {})
|
||||
|
||||
async def test_computer_command_omits_identity_without_authoritative_run(self) -> None:
|
||||
handler, office, _laptop = await _register_two()
|
||||
task = asyncio.create_task(
|
||||
handler.handle_command(
|
||||
"desktop_computer_snapshot",
|
||||
{},
|
||||
device="desktop-1",
|
||||
requester=DesktopRequesterContext(requester_device_id="paired-agent-1"),
|
||||
)
|
||||
)
|
||||
await asyncio.sleep(0)
|
||||
payload = office.sent[-1]["payload"]
|
||||
self.assertNotIn("control_session", payload)
|
||||
await handler.handle(
|
||||
office, # type: ignore[arg-type]
|
||||
{
|
||||
"channel": "desktop",
|
||||
"type": "desktop.response",
|
||||
"payload": {"request_id": payload["request_id"], "ok": True, "result": {}},
|
||||
},
|
||||
)
|
||||
await asyncio.wait_for(task, timeout=1)
|
||||
|
||||
async def test_new_requester_run_ends_prior_control_session(self) -> None:
|
||||
handler, office, _laptop = await _register_two()
|
||||
|
||||
async def dispatch(run_id: str) -> dict[str, Any]:
|
||||
task = asyncio.create_task(handler.handle_command(
|
||||
"desktop_computer_snapshot", {}, device="desktop-1",
|
||||
requester=DesktopRequesterContext(
|
||||
requester_device_id="paired-agent-1",
|
||||
chat_session_id="chat-1",
|
||||
run_id=run_id,
|
||||
profile="default",
|
||||
),
|
||||
))
|
||||
await asyncio.sleep(0)
|
||||
command = next(
|
||||
item for item in reversed(office.sent)
|
||||
if item["type"] == "desktop.command"
|
||||
)
|
||||
await handler.handle(office, { # type: ignore[arg-type]
|
||||
"channel": "desktop", "type": "desktop.response",
|
||||
"payload": {"request_id": command["payload"]["request_id"], "ok": True, "result": {}},
|
||||
})
|
||||
await task
|
||||
return command["payload"]
|
||||
|
||||
first = await dispatch("run-1")
|
||||
second = await dispatch("run-2")
|
||||
ended = next(item for item in office.sent if item["type"] == "desktop.control_session_end")
|
||||
self.assertEqual(ended["payload"]["version"], 1)
|
||||
self.assertEqual(ended["payload"]["id"], first["control_session"]["id"])
|
||||
self.assertEqual(ended["payload"]["target_device_id"], "desktop-1")
|
||||
self.assertEqual(ended["payload"]["reason"], "requester run changed")
|
||||
self.assertNotEqual(first["control_session"]["id"], second["control_session"]["id"])
|
||||
|
||||
async def test_idle_control_session_emits_end_event(self) -> None:
|
||||
with patch("plugin.relay.channels.desktop.CONTROL_SESSION_IDLE_SECONDS", 0.01):
|
||||
handler, office, _laptop = await _register_two()
|
||||
task = asyncio.create_task(handler.handle_command(
|
||||
"desktop_computer_snapshot", {}, device="desktop-1",
|
||||
requester=DesktopRequesterContext(
|
||||
requester_device_id="paired-agent-1", run_id="run-1"
|
||||
),
|
||||
))
|
||||
await asyncio.sleep(0)
|
||||
command = office.sent[-1]
|
||||
await handler.handle(office, { # type: ignore[arg-type]
|
||||
"channel": "desktop", "type": "desktop.response",
|
||||
"payload": {"request_id": command["payload"]["request_id"], "ok": True, "result": {}},
|
||||
})
|
||||
await task
|
||||
await asyncio.sleep(0.03)
|
||||
ended = next(item for item in office.sent if item["type"] == "desktop.control_session_end")
|
||||
self.assertEqual(ended["payload"]["id"], command["payload"]["control_session"]["id"])
|
||||
self.assertEqual(ended["payload"]["reason"], "control session idle timeout")
|
||||
|
||||
async def test_control_session_capacity_eviction_emits_end_event(self) -> None:
|
||||
with patch("plugin.relay.channels.desktop.CONTROL_SESSIONS_MAX", 1):
|
||||
handler, office, _laptop = await _register_two()
|
||||
|
||||
async def dispatch(requester_device_id: str) -> dict[str, Any]:
|
||||
task = asyncio.create_task(handler.handle_command(
|
||||
"desktop_computer_snapshot", {}, device="desktop-1",
|
||||
requester=DesktopRequesterContext(
|
||||
requester_device_id=requester_device_id,
|
||||
run_id="run-1",
|
||||
),
|
||||
))
|
||||
await asyncio.sleep(0)
|
||||
command = next(
|
||||
item for item in reversed(office.sent)
|
||||
if item["type"] == "desktop.command"
|
||||
)
|
||||
await handler.handle(office, { # type: ignore[arg-type]
|
||||
"channel": "desktop", "type": "desktop.response",
|
||||
"payload": {
|
||||
"request_id": command["payload"]["request_id"],
|
||||
"ok": True,
|
||||
"result": {},
|
||||
},
|
||||
})
|
||||
await task
|
||||
return command["payload"]
|
||||
|
||||
first = await dispatch("paired-agent-1")
|
||||
await dispatch("paired-agent-2")
|
||||
ended = next(
|
||||
item for item in office.sent
|
||||
if item["type"] == "desktop.control_session_end"
|
||||
)
|
||||
self.assertEqual(ended["payload"]["version"], 1)
|
||||
self.assertEqual(ended["payload"]["id"], first["control_session"]["id"])
|
||||
self.assertEqual(ended["payload"]["target_device_id"], "desktop-1")
|
||||
self.assertEqual(ended["payload"]["reason"], "control session capacity eviction")
|
||||
|
||||
async def test_untargeted_command_fails_closed_with_multiple_pcs(self) -> None:
|
||||
handler, _office, _laptop = await _register_two()
|
||||
with self.assertRaisesRegex(DesktopError, "pass device or device_id explicitly"):
|
||||
|
||||
@@ -78,6 +78,34 @@ import requests
|
||||
|
||||
|
||||
_DESKTOP_TARGET: ContextVar[str | None] = ContextVar("desktop_target", default=None)
|
||||
_DESKTOP_CALL_CONTEXT: ContextVar[dict[str, str]] = ContextVar(
|
||||
"desktop_call_context", default={}
|
||||
)
|
||||
|
||||
_CONTROL_CONTEXT_HEADERS = {
|
||||
"chat_session_id": "X-Hermes-Relay-Chat-Session",
|
||||
"run_id": "X-Hermes-Relay-Run-Id",
|
||||
"profile": "X-Hermes-Relay-Profile",
|
||||
}
|
||||
|
||||
|
||||
def _bounded_context_value(value: Any, limit: int = 256) -> str | None:
|
||||
if not isinstance(value, str):
|
||||
return None
|
||||
cleaned = "".join(ch for ch in value.strip() if ch.isprintable())
|
||||
return cleaned[:limit] or None
|
||||
|
||||
|
||||
def _trusted_call_context(kwargs: dict[str, Any]) -> dict[str, str]:
|
||||
"""Extract executor-owned identity without consulting model arguments."""
|
||||
|
||||
values = {
|
||||
"chat_session_id": _bounded_context_value(kwargs.get("session_id")),
|
||||
"run_id": _bounded_context_value(kwargs.get("task_id")),
|
||||
"profile": _bounded_context_value(kwargs.get("profile"))
|
||||
or _bounded_context_value(os.getenv("HERMES_PROFILE")),
|
||||
}
|
||||
return {key: value for key, value in values.items() if value is not None}
|
||||
|
||||
|
||||
# ── Config ────────────────────────────────────────────────────────────────────
|
||||
@@ -115,11 +143,16 @@ def _post(path: str, payload: dict, *, timeout: Optional[float] = None) -> dict:
|
||||
target = _DESKTOP_TARGET.get()
|
||||
if target:
|
||||
payload["device"] = target
|
||||
headers = _auth_headers()
|
||||
for field, value in _DESKTOP_CALL_CONTEXT.get().items():
|
||||
header = _CONTROL_CONTEXT_HEADERS.get(field)
|
||||
if header:
|
||||
headers[header] = value
|
||||
try:
|
||||
r = requests.post(
|
||||
f"{_relay_url()}{path}",
|
||||
json=payload,
|
||||
headers=_auth_headers(),
|
||||
headers=headers,
|
||||
timeout=_timeout() if timeout is None else timeout,
|
||||
)
|
||||
except requests.RequestException as exc:
|
||||
@@ -588,8 +621,12 @@ def desktop_computer_screenshot(
|
||||
include_cursor: bool = True,
|
||||
redact_sensitive: bool = True,
|
||||
save_to: Optional[str] = None,
|
||||
pid: Optional[int] = None,
|
||||
window_id: Optional[int] = None,
|
||||
query: Optional[str] = None,
|
||||
include_screenshot: bool = True,
|
||||
) -> str:
|
||||
"""[EXPERIMENTAL] Capture a desktop screenshot in observe mode."""
|
||||
"""[EXPERIMENTAL] Capture a display or a structured CUA window snapshot."""
|
||||
payload: dict[str, Any] = {
|
||||
"display": display,
|
||||
"include_cursor": bool(include_cursor),
|
||||
@@ -599,6 +636,15 @@ def desktop_computer_screenshot(
|
||||
payload["region"] = region
|
||||
if save_to is not None:
|
||||
payload["save_to"] = save_to
|
||||
for key, value in {
|
||||
"pid": pid,
|
||||
"window_id": window_id,
|
||||
"query": query,
|
||||
}.items():
|
||||
if value is not None:
|
||||
payload[key] = value
|
||||
if pid is not None or window_id is not None or not include_screenshot:
|
||||
payload["include_screenshot"] = bool(include_screenshot)
|
||||
data = _post("/desktop/desktop_computer_screenshot", payload)
|
||||
return json.dumps(data)
|
||||
|
||||
@@ -1160,6 +1206,10 @@ _SCHEMAS: dict[str, dict[str, Any]] = {
|
||||
"type": "string",
|
||||
"description": "Optional path on the desktop client to save the PNG instead of returning bytes.",
|
||||
},
|
||||
"pid": {"type": "integer", "minimum": 1},
|
||||
"window_id": {"type": "integer", "minimum": 1},
|
||||
"query": {"type": "string", "description": "Optional accessibility-tree projection query."},
|
||||
"include_screenshot": {"type": "boolean", "default": True},
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -1189,6 +1239,10 @@ _SCHEMAS: dict[str, dict[str, Any]] = {
|
||||
"type",
|
||||
"type_text",
|
||||
"wait",
|
||||
"click_element",
|
||||
"set_value",
|
||||
"press_key",
|
||||
"scroll_element",
|
||||
],
|
||||
},
|
||||
"coordinate": {
|
||||
@@ -1227,6 +1281,13 @@ _SCHEMAS: dict[str, dict[str, Any]] = {
|
||||
"type": "string",
|
||||
"description": "Human-readable reason for the requested action.",
|
||||
},
|
||||
"pid": {"type": "integer", "minimum": 1},
|
||||
"window_id": {"type": "integer", "minimum": 1},
|
||||
"snapshot_token": {"type": "string"},
|
||||
"snapshot_generation": {"type": "string"},
|
||||
"value": {"type": "string"},
|
||||
"direction": {"type": "string", "enum": ["up", "down", "left", "right"]},
|
||||
"amount": {"type": "integer", "minimum": 1, "maximum": 20},
|
||||
},
|
||||
"required": ["action"],
|
||||
"additionalProperties": False,
|
||||
@@ -1333,10 +1394,17 @@ def _targeted_handler(handler: Any) -> Any:
|
||||
def invoke(args: dict[str, Any], **kwargs: Any) -> Any:
|
||||
call_args = dict(args)
|
||||
target = call_args.pop("device", None) or call_args.pop("device_id", None)
|
||||
# Identity is never a model-owned argument. Drop every reserved shape
|
||||
# before dispatch, then obtain authoritative context from Hermes'
|
||||
# executor kwargs instead.
|
||||
call_args.pop("control_session", None)
|
||||
call_args.pop("control_context", None)
|
||||
token = _DESKTOP_TARGET.set(str(target).strip() if target else None)
|
||||
context_token = _DESKTOP_CALL_CONTEXT.set(_trusted_call_context(kwargs))
|
||||
try:
|
||||
return handler(call_args, **kwargs)
|
||||
finally:
|
||||
_DESKTOP_CALL_CONTEXT.reset(context_token)
|
||||
_DESKTOP_TARGET.reset(token)
|
||||
|
||||
return invoke
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "hermes-relay"
|
||||
version = "1.7.0"
|
||||
version = "1.8.0"
|
||||
description = "Hermes-Relay plugin — Android device control toolset, QR pairing CLI, and WSS relay server for hermes-agent"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
|
||||
@@ -129,6 +129,17 @@ The Windows one-liner installs the checksum-verified CLI and UI bundle by defaul
|
||||
|
||||
The CLI section can **Open terminal** at a normal prompt, **Open Hermes CLI** directly into the paired Hermes TUI, **View daemon log**, or **Run diagnostics**. Updates manage the CLI and UI bundle together. **Help & About** shows the UI, CLI, and connected Relay versions and links to the [desktop documentation](https://hermes-relay.dev/docs/desktop/), [troubleshooting guide](https://hermes-relay.dev/docs/desktop/troubleshooting/), [release notes](https://github.com/Codename-11/hermes-relay/releases?q=desktop), logs, and diagnostics. External links open in the default browser.
|
||||
|
||||
**Settings → Computer control** reports whether the preferred CUA Driver engine
|
||||
is absent, incompatible, degraded, or ready. A ready runtime handles structured
|
||||
actions against a named
|
||||
window in the background, and display a separate animated virtual cursor for
|
||||
each agent control session without moving the physical mouse. CUA remains
|
||||
optional: Windows input is the explicit compatibility backend, this release
|
||||
keeps foreground escalation disabled, and the chosen backend cannot change
|
||||
during an active control session. The same card provides explicit **Install**,
|
||||
**Check**, and **Update** actions; none run automatically.
|
||||
See [Computer-use engines](./tools.md#computer-use-engines).
|
||||
|
||||
## Why both shell AND chat modes?
|
||||
|
||||
They're not the same thing:
|
||||
|
||||
@@ -60,6 +60,45 @@ hermes-relay ui # same as: hermes-relay ui open
|
||||
hermes-relay ui status
|
||||
```
|
||||
|
||||
### Preferred CUA Driver
|
||||
|
||||
CUA Driver is an optional Windows computer-control engine and is **not** bundled
|
||||
with either Hermes-Relay installer surface. Hermes resolves only the canonical
|
||||
upstream package at
|
||||
`%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe`; it does not use an
|
||||
arbitrary PATH shim.
|
||||
|
||||
Hermes exposes the verified upstream lifecycle only after an explicit local
|
||||
choice. Read-only status and update checks never install anything:
|
||||
|
||||
```powershell
|
||||
hermes-relay computer-use cua status
|
||||
hermes-relay computer-use cua health
|
||||
hermes-relay computer-use cua check-update
|
||||
hermes-relay computer-use cua install --yes
|
||||
hermes-relay computer-use cua update --yes
|
||||
```
|
||||
|
||||
Install is pinned to the minimum compatible release. Update first asks the
|
||||
installed driver's native update service which release is current, then refuses
|
||||
to apply it if it falls outside Hermes-Relay's supported range (`>=0.19.3,
|
||||
<0.20.0`). Hermes downloads the versioned GitHub release manifest and installer,
|
||||
checks the manifest repository/product/version and the installer's SHA-256, and
|
||||
then verifies the canonical binary path, version, driver manifest, required
|
||||
tools, and permission mode. Accessibility health remains an explicit recheck
|
||||
while the temporary Windows compatibility workaround is active. These are release-metadata and
|
||||
checksum integrity checks, not a Windows publisher signature.
|
||||
|
||||
The UI provides the same explicit **Install**, **Check**, and **Update** actions
|
||||
under **Settings → Computer control**. When the report says CUA is ready it is
|
||||
the preferred structured engine; `hermes-relay computer-use engine legacy`
|
||||
selects the original Windows input path as an explicit compatibility backend.
|
||||
Engine changes apply only to new control sessions. Hermes does not
|
||||
silently install or update CUA Driver, and `hermes-relay update` manages only
|
||||
the Hermes-Relay CLI/UI bundle. Hermes also disables CUA telemetry for every
|
||||
driver process it starts; telemetry is not silently enabled as part of pairing,
|
||||
Full Access, or engine selection.
|
||||
|
||||
### SmartScreen warning on first launch
|
||||
|
||||
The binaries are unsigned during the experimental phase. Windows may show "Windows protected your PC" for the installer or first launch. Click **More info → Run anyway**. For a CLI-only install, you can also pre-allow the executable from PowerShell:
|
||||
|
||||
@@ -292,10 +292,26 @@ hermes-relay computer-use enable # confirm and persist enablement
|
||||
hermes-relay computer-use enable --yes # explicit non-interactive confirmation
|
||||
hermes-relay computer-use cancel # cancel the active task-scoped grant
|
||||
hermes-relay computer-use disable # disable and request active-grant cancellation
|
||||
hermes-relay computer-use engine cua # prefer structured CUA for new sessions
|
||||
hermes-relay computer-use engine legacy # explicit Windows-input compatibility
|
||||
hermes-relay computer-use cursor on # virtual per-session cursor
|
||||
hermes-relay computer-use cua status
|
||||
hermes-relay computer-use cua health # recheck accessibility health
|
||||
hermes-relay computer-use cua check-update
|
||||
hermes-relay computer-use cua install --yes
|
||||
hermes-relay computer-use cua update --yes
|
||||
```
|
||||
|
||||
Enablement is stored in `~/.hermes/desktop-settings.json`. Restart the daemon after changing it; the Windows tray does that automatically while preserving whether the daemon is running as User or Administrator. `--experimental-computer-use` is a one-process enable override, and `--no-computer-use` always suppresses advertisement for that invocation.
|
||||
|
||||
CUA is the preferred/default structured backend; Windows input is an explicit
|
||||
compatibility choice. Backend selection is fixed when a control session starts,
|
||||
so an engine setting change affects only new sessions. CUA lifecycle mutations
|
||||
require `--yes` and never run automatically. Hermes verifies the upstream
|
||||
release manifest and installer checksum, runs the installer with a sanitized
|
||||
environment, and accepts only `>=0.19.3 <0.20.0` under the canonical
|
||||
`%USERPROFILE%\.cua-driver\packages\current` package.
|
||||
|
||||
## `hermes-relay grants`
|
||||
|
||||
Review assist/control requests written by a headless daemon to the local grant bridge:
|
||||
@@ -317,8 +333,16 @@ Show what the remote agent has run on **this** machine through the desktop tools
|
||||
hermes-relay audit # last 50 desktop-tool calls
|
||||
hermes-relay audit --limit 20 # fewer
|
||||
hermes-relay audit --json # raw entries for scripting
|
||||
hermes-relay audit screenshots --json
|
||||
hermes-relay audit screenshots on --days 7 --yes
|
||||
hermes-relay audit screenshots off --yes
|
||||
```
|
||||
|
||||
Screenshot evidence is local and separate from the JSONL log. Retention defaults
|
||||
to seven days and is capped at 20 PNG files and 10 MB per file. Choose 1, 7, or
|
||||
30 days, or turn it off; disabling retention removes existing screenshot
|
||||
evidence without deleting the remaining activity history.
|
||||
|
||||
```
|
||||
Desktop-tool activity (3 most recent)
|
||||
|
||||
|
||||
@@ -221,6 +221,88 @@ Use `hermes-relay computer-use status` to see the persisted preference, daemon p
|
||||
|
||||
Input injection is currently **Windows-only**; `status` / `screenshot` work cross-platform.
|
||||
|
||||
### Computer-use engines
|
||||
|
||||
The `desktop_computer_*` contract stays the same regardless of the local engine.
|
||||
On Windows you can choose:
|
||||
|
||||
| Engine | Behavior |
|
||||
|--------|----------|
|
||||
| **CUA Driver** | Preferred/default structured engine. Targets a named PID and window through UI Automation and background dispatch, and can show an animated virtual cursor without moving the physical pointer. |
|
||||
| **Windows input** | Explicit compatibility backend. Uses the original Windows input path and can depend on foreground focus or move the physical pointer for some actions. |
|
||||
|
||||
Check the detected runtime and current selection from the CLI:
|
||||
|
||||
```powershell
|
||||
hermes-relay computer-use status --json
|
||||
hermes-relay computer-use engine cua
|
||||
hermes-relay computer-use cursor on
|
||||
hermes-relay computer-use cua status
|
||||
hermes-relay computer-use cua health
|
||||
hermes-relay computer-use cua check-update
|
||||
hermes-relay computer-use cua install --yes
|
||||
hermes-relay computer-use cua update --yes
|
||||
```
|
||||
|
||||
The management UI exposes the same choices under **Settings → Computer
|
||||
control**. Selecting `cua` succeeds only when Hermes finds the canonical runtime
|
||||
at `%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe` and verifies its
|
||||
supported version, manifest identity, allowlisted tools, and non-unrestricted
|
||||
permission mode. Accessibility health is an explicit diagnostic while the
|
||||
temporary Windows workaround for trycua/cua#3103 is active. Hermes does not trust whichever `cua-driver.exe`
|
||||
happens to appear first on `PATH`. If the runtime is missing, incompatible, or
|
||||
degraded, the UI explains why and a new control session can use Windows input
|
||||
compatibility. Backend selection is made once per authenticated control session;
|
||||
changing the preference never switches an active session underneath an action.
|
||||
|
||||
CUA sessions follow Hermes control sessions. Hermes derives the driver session
|
||||
identity locally; an agent cannot choose or share a cursor ID. Each action uses
|
||||
a fresh target-window snapshot, and element handles are wrapped in a short-lived,
|
||||
single-use Hermes token bound to the control authority, grant, PID, window, and
|
||||
snapshot generation. The handler returns a post-action snapshot so the caller
|
||||
can verify whether the expected state actually changed. Grant expiry, cancellation, disconnect, policy change,
|
||||
emergency stop, or daemon shutdown revokes the associated local state.
|
||||
|
||||
**Background is mandatory for CUA.** A target that cannot accept background input
|
||||
returns a failure instead of silently bringing itself forward. The foreground
|
||||
escalation control is reserved for a later explicit-authority path; this release
|
||||
always reports it off, even if an older preview saved the preference, and Full
|
||||
Access cannot enable it. The animated agent cursor is optional. It is a visual overlay
|
||||
with a distinct session identity—not another Windows hardware pointer—and the
|
||||
operator's physical cursor remains untouched by CUA actions.
|
||||
|
||||
CUA Driver is a separate optional dependency. Hermes-Relay does not bundle or
|
||||
silently install/update it. `computer-use cua status|check-update` are
|
||||
read-only; `install|update` require explicit `--yes` confirmation. Hermes uses
|
||||
the canonical upstream package, validates versioned GitHub release metadata and
|
||||
installer SHA-256, and refuses updates outside `>=0.19.3,<0.20.0`. These checks
|
||||
do not claim a Windows publisher signature. Hermes executes the verified
|
||||
temporary installer under a sanitized environment, then validates the canonical
|
||||
`packages/current` binary, driver manifest, and permission mode. The UI and
|
||||
`computer-use cua health` can recheck accessibility health without changing
|
||||
the selected backend. Hermes forces CUA telemetry off for
|
||||
its child invocations; any future telemetry opt-in belongs to the local
|
||||
operator. `hermes-relay update` continues to manage only the Hermes-Relay CLI
|
||||
and Windows UI.
|
||||
|
||||
Window-scoped snapshots and element actions use the selected backend. A normal
|
||||
full-display screenshot remains on the separate read-only `system_capture` path;
|
||||
using it does not switch the input backend. The local audit and UI Activity
|
||||
timeline show bounded high-level evidence—backend, background dispatch, control
|
||||
session, target app/window identifiers, action, phase, and verification state.
|
||||
Accessibility text, screenshot bytes, entered values, and raw driver responses
|
||||
are redacted from the activity drilldown.
|
||||
|
||||
::: warning CUA does not sandbox raw commands
|
||||
CUA's PID/window and snapshot boundaries apply to structured computer-control
|
||||
actions. If Commands, PowerShell, or terminal execution is allowed, the agent
|
||||
can still invoke ordinary OS automation through that broader trusted path.
|
||||
Disable raw command access when scoped computer control is part of your threat
|
||||
model. Full Access never bypasses authenticated targeting, sensitive-surface
|
||||
checks, UAC or Windows-session boundaries, audit, health checks, or emergency
|
||||
stop.
|
||||
:::
|
||||
|
||||
## Diagnosing routing
|
||||
|
||||
If the agent says "desktop_terminal is not available" or calls time out immediately:
|
||||
@@ -278,7 +360,14 @@ If `connected: true` but the agent still says the tool is missing:
|
||||
|
||||
`hermes-relay daemon` runs the WSS connection + tool router headless, so the agent can reach your machine while you're in another window or VS Code or off making coffee. Use `hermes-relay daemon start` to run it in the **background** (no console window, survives closing the terminal), `daemon status` to check it, and `daemon stop` to stop it. See [Subcommands → daemon](./subcommands.md#hermes-relay-daemon) for full lifecycle/log details.
|
||||
|
||||
Want to see what the agent actually ran on your machine? `hermes-relay audit` lists recent `desktop_*` activity from a local log. The management UI previews the latest three events and opens each event into bounded request, stdout, stderr, result, exit, timing, and truncation details. Sensitive request inputs are excluded.
|
||||
Want to see what the agent actually ran on your machine? `hermes-relay audit` lists recent `desktop_*` activity from a local log. The management UI previews the latest three events and opens each event into a lifecycle stepper plus bounded request, stdout, stderr, result, exit, timing, and truncation details. Errors have a dedicated failure panel. Screenshot events may retain bounded local evidence for a larger viewer; Settings controls Off/1-day/7-day/30-day retention and clearing Activity removes it. Sensitive request inputs remain excluded from the JSON audit log.
|
||||
|
||||
The daemon records one interruption event when automatic reconnect begins and a
|
||||
recovery event when it succeeds rather than adding one row per backoff attempt.
|
||||
The Overview shows retry attempt/timing and supports **Retry now** or an explicit
|
||||
disconnect. When the management UI is hidden, connection loss and restoration
|
||||
use the same compact local-card language as permission requests; no duplicate
|
||||
card appears while the UI is already open.
|
||||
|
||||
`daemon start` covers "background, this session." On Windows, **Start UI at sign-in** registers the optional tray as a per-user login entry. **Start daemon with UI** separately opts into connecting remote access when the tray launches and defaults off for existing installs. Neither is a Windows service. For Linux/macOS or a machine-level lifetime, wrap foreground `hermes-relay daemon` with your service manager of choice.
|
||||
|
||||
|
||||
@@ -84,6 +84,42 @@ hermes-relay computer-use cancel
|
||||
|
||||
Switching the host to **Restricted** or using emergency stop also requests cancellation. Without a local approval response, a headless request times out and input remains blocked.
|
||||
|
||||
## CUA Driver is unavailable, incompatible, or degraded
|
||||
|
||||
Inspect the machine-local engine report:
|
||||
|
||||
```powershell
|
||||
hermes-relay computer-use status --json
|
||||
```
|
||||
|
||||
- **Not installed** means the canonical package was not found at
|
||||
`%USERPROFILE%\.cua-driver\packages\current\cua-driver.exe`. A PATH-only shim
|
||||
is intentionally ignored.
|
||||
- **Incompatible** means the executable, manifest, supported version, required
|
||||
tool set, or permission mode did not match the Hermes adapter contract.
|
||||
- **Accessibility health** is a separate, explicit diagnostic on Windows. Use
|
||||
**Recheck** in the UI or run `hermes-relay computer-use cua health`. A
|
||||
degraded result does not disable the runtime while the temporary workaround
|
||||
for the upstream fixed-timeout issue is active; canonical runtime checks and
|
||||
each structured action still fail closed. Confirm the driver is running in
|
||||
the interactive user's logon session rather than Session 0 before deeper
|
||||
diagnosis.
|
||||
|
||||
CUA is preferred for new structured-control sessions. If its executable,
|
||||
manifest, required tool set, daemon status, or safe permission mode is not
|
||||
ready before a session begins, Hermes can select the Windows input compatibility backend;
|
||||
it never changes backend in the middle of a control session. Re-check with
|
||||
`hermes-relay computer-use cua status`, repair explicitly with
|
||||
`computer-use cua install --yes`, or use `computer-use cua check-update` followed
|
||||
by `computer-use cua update --yes`. The UI exposes matching Install, Check, and
|
||||
Update actions. None run automatically.
|
||||
|
||||
If a CUA action reports that background delivery is unavailable, Hermes does
|
||||
not silently foreground the target. Use another structured action or complete
|
||||
that step manually. **Allow foreground escalation** is reserved in this release:
|
||||
runtime status remains off even if an older preview saved the preference, and
|
||||
Full Access does not turn it on.
|
||||
|
||||
## `auth timed out after 15000ms`
|
||||
|
||||
The relay subprocess takes 15–30 seconds on first attach because Hermes initializes the full agent. Bump the timeout for slow first connects:
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Dashboard Plugin
|
||||
# Dashboard and official Desktop plugins
|
||||
|
||||
A hermes-agent dashboard plugin that surfaces relay-specific state in the gateway's web UI. Paired devices, bridge command history, push delivery (future), and active inbound-media tokens — all in one "Relay" tab, no SSH required.
|
||||
Hermes-Relay surfaces its Relay-specific state in the web Dashboard and, when
|
||||
explicitly enabled, in official Hermes Desktop. Both use the same existing
|
||||
profile-scoped backend; neither creates a second Relay service or state store.
|
||||
|
||||
## What It Is
|
||||
|
||||
@@ -20,6 +22,19 @@ The plugin is a thin observer — it never modifies state, never writes to your
|
||||
|
||||
Nothing. The dashboard plugin renders in your browser against the Hermes server — the phone is the subject of observation, not a participant.
|
||||
|
||||
**For official Hermes Desktop:** use a build with the runtime Plugin SDK and
|
||||
unified-package discovery. The regular Hermes-Relay plugin install already
|
||||
places `desktop/plugin.js` beside the Dashboard half. Open **Settings →
|
||||
Plugins**, enable **Hermes Relay**, then use the labeled **Relay** sidebar or
|
||||
status-bar item, or **Hermes Relay: Open** in the command palette.
|
||||
|
||||
Enabling or loading the Desktop plugin does not open its pane. App startup,
|
||||
reconnect, profile changes, navigation restoration, updates, and background
|
||||
events also leave it closed. After an explicit open, close and move/dock work
|
||||
through the native Desktop pane controls. Desktop currently offers no supported
|
||||
agent command for focusing contributed panes and no programmatic move-coordinate
|
||||
API; Hermes-Relay does not use private hooks to imitate those features.
|
||||
|
||||
## Accessing the Dashboard
|
||||
|
||||
Open the hermes-agent dashboard in your browser (default: `http://localhost:<dashboard_port>`). The **Relay** tab sits between Skills and whatever you have next in your nav order — click it and you land on the four-tab shell.
|
||||
|
||||
Reference in New Issue
Block a user