Hermes-Relay · Architecture Reference

Connection Paths & Chat Transport Resolution

How the Android app splits into a Vanilla Hermes path that runs on unmodified upstream Hermes and an additive Relay plugin path — and how, within Vanilla Hermes chat, it climbs a tier ladder of transports and falls back gracefully. A separate build-flavor gate decides which capabilities ship in the APK at all.

Vanilla Hermes = vanilla upstream only Relay = optional plugin Sideload = Device Control gate v1.0.0 · updated 2026-06-18
Vanilla Hermes / upstream surface Relay plugin surface Runtime decision
01

The two paths & the flavor gate

Every capability in the app belongs to exactly one of two paths. The dividing line is a hard product rule: the Vanilla Hermes path must work against unmodified upstream hermes-agent — the app ships on Google Play to servers we don't control. Anything needing server-side code lives behind the opt-in Relay plugin (or goes upstream as a PR with graceful degradation).

🟢 Vanilla Hermes vanilla upstream

No pairing. Works the moment you point at a Hermes server + dashboard.

  • Chat — gateway WS, then API-server SSE (tiered, see §3)
  • Manage — config, profiles, model, env, MCP (dashboard)
  • Vanilla Hermes voice — dashboard /api/audio/*
  • Sessions / history — native /api/sessions
Auth: API-server bearer key + dashboard cookie (Manage sign-in).

🟣 Relay optional plugin

Requires pairing a Relay. Purely additive power features.

  • Terminal — server shell over WSS
  • Bridge — phone/device control (sideload)
  • Enhanced / relay voice — streaming TTS, realtime agent
  • Desktop tools · notification companion · remote access
Auth: paired Relay session token (in-memory; re-pair after relay restart).
Chat never crosses into Relay. Vanilla Hermes chat "mixes" only within the Vanilla Hermes path — between the gateway WS transport and the API-server SSE transports. The Relay plugin adds terminal, bridge, and voice surfaces, but it does not carry Vanilla Hermes chat. That keeps the Play-Store default fully functional with zero plugin installed.

The third axis — build flavor (capability gate)

Path (Vanilla Hermes vs Relay) is about which server surface you talk to. Flavor is a different axis entirely: a capability ceiling compiled into the APK, independent of any server or pairing. Both flavors ship the full Relay client — only sideload compiles in phone Device Control (the AccessibilityService "hands"). The active track shows in Settings → About as a badge.

googlePlay Bridge Core

Conservative track. No phone Device Control surface.

  • Full Vanilla Hermes path — chat, Manage, voice
  • Relay: terminal, relay voice, notification companion, media, session grants
  • No screen reading, taps, typing, screenshots, overlays, or unattended control
Gated android_* tools return a sideload-only 403 instead of crashing; the Bridge phone-control screen isn't present.

sideload full Device Control

Full-capability track. Adds the AccessibilityService bridge.

  • Everything googlePlay ships, plus…
  • Device Control hands: tap · type · navigate · screen-read · screenshot · overlay
  • Unattended access · Tier-C tools (call / SMS / contacts / location)
Device Control tiers 1–6 (baseline → screen context → voice-first → vision-first → safety rails → future) are all sideload-gated.
Bridge is two-layered. Relay pairing unlocks the bridge channel; the sideload flavor unlocks the Device Control hands within it. A paired Relay on a googlePlay build still cannot tap or type — the capability simply isn't compiled in. BuildFlavor · data/FeatureFlags.kt:92
02

Four network endpoints

The phone talks to up to four distinct surfaces. The first three are upstream; the fourth is ours.

TransportTargetOwnerCarries
WS Hermes dashboard :9119 /api/ws Upstream (tui_gateway) Vanilla Hermes gateway chat · live thinking/reasoning deltas
HTTP/SSE Hermes API server :8642 Upstream (api_server) Vanilla Hermes chat fallback · sessions · runs · capabilities
HTTP Hermes dashboard :9119 /api/* Upstream (web_server) Manage · Vanilla Hermes voice /api/audio/* · auth/ws-ticket
WSS/HTTP Relay plugin/server :8767 Hermes-Relay plugin Bridge · terminal · relay voice · desktop tools · media

API-server bearer auth and dashboard cookie auth are separate. See docs/upstream-surface-matrix.md for the full route-ownership contract.

03

Chat transport tiers

Vanilla Hermes chat has four transports, ordered best-to-fallback. The resolver prefers the gateway (it's the only surface with live reasoning), then degrades down the SSE ladder based on what the server actually advertises.

#TierEndpointWhy prefer itTool / reasoning fidelity
1 Gateway WS /api/ws (dashboard) Only surface with live reasoning.delta / thinking.delta; full byte-upload attach Structured tool events + live thinking
2 Sessions SSE /api/sessions/{id}/chat/stream Native upstream, session-persisted; preferred SSE when probed present Structured SSE events; reasoning post-hoc
3 Completions SSE /v1/chat/completions OpenAI-compatible; always-available stateless fallback Inline markdown tool annotations only
4 Runs SSE /v1/runs → /events Structured run lifecycle when sessions absent but runs advertised Structured tool events (tool.started/completed)
Voice turns are SSE-only. The gateway's prompt.submit RPC has no system-message slot, so per-turn ephemeral instructions (voice interface context) can't ride it. A "gateway" preference is force-downgraded to SSE for those turns. ChatViewModel.send() · effectiveEndpoint
04

How a path is chosen — the flowchart

Resolution happens in two moments. First, at connect/probe time, a pure function picks a preference. Second, at send time, ChatViewModel re-checks live state and can still downgrade to SSE. This is why a turn whose preference reads "gateway" can quietly complete over SSE.

Phase A · Connect-time preference

resolveStreamingEndpointPreference(preference, gateway, capabilities) — GatewayModels.kt:63

streamingEndpoint preference
user setting: "auto" (default) · or a manual pin
Is the preference a manual pin?
"gateway" / "sessions" / "completions" / "runs"
Yes — manual
Use it verbatim
Manual selection owns a new chat; existing bindings do not migrate.
No — "auto"
Saved connection owner?
Standard
→ "gateway"
API-only
capabilities
.preferredChatEndpoint()
sessions › completions › runs

Phase B · Send-time dispatch

ChatViewModel.send() — runtime re-check & fallback wiring

User sends a message
effectiveEndpoint = resolved preference
Bound owner == "gateway"?
Direct API owner
dispatchSse(endpoint)
Gateway owner
gateway client live?
null
Preserve + Retry
sign in or reconnect; no owner change
yes
gateway.sendTurn()
gateway.sendTurn → outcome
streams OK
Gateway turn
live thinking + structured tools
onPreflightFailure
Preserve + Retry
nothing started server-side; conversation stays Gateway-owned
Transport affinity. Gateway and Direct API sessions live in different stores. Sign-in expiry or route loss never authorizes Android to resubmit the turn through another owner.
05

Owner, readiness, and capabilities

Saved connection state chooses the owner. Availability reports whether that owner is ready; Direct API capabilities choose an endpoint only inside an API-owned conversation.

GatewayAvailability

GatewayModels.kt:22 · set by the dashboard /api/status + /api/auth/me probe

  • Unknown — no probe yet (startup / connection switch)
  • Ready — reachable + authenticated (or no auth) → Gateway can send
  • SignInRequired — reachable but gated; Manage sign-in unlocks it
  • Unreachable — /api/status didn't answer
  • Unsupported — sticky: WS upgrade/ticket got 404/403 (build predates embedded chat)

ServerCapabilities

HermesApiClient.probeCapabilities() · probe order below

GET /health
unreachable → DISCONNECTED, stop
GET /v1/capabilities
present → parse & return (authoritative)
HEAD probes (fallback)
/api/sessions · …/chat/stream · /v1/chat/completions · /v1/runs

preferredChatEndpoint() = sessionsChatStream ? "sessions" : portable ? "completions" : runs ? "runs" : "sessions"

06

Feature → path matrix

Which path unlocks each feature, and what it degrades to when that path is unavailable.

FeaturePathSurfaceDegrades to
Chat (live thinking)Vanilla HermesGateway WSSign in or retry on the same owner
Direct API chatVanilla HermesAPI-server SSEInline-annotation parser
Session history / CRUDVanilla Hermes/api/sessionsStateless completions (no persistence)
Manage (config/profiles/model/env/MCP)Vanilla HermesDashboard /api/*— (hidden if dashboard down)
Vanilla Hermes voice (STT/TTS)Vanilla HermesDashboard /api/audio/*Relay voice if paired (Auto route)
TerminalRelayWSS terminal.*Unavailable — "Pair Relay"
Bridge — Device Control (tap/type/read/screenshot/overlay)Relay + sideloadAccessibilityService + WSS bridge.commandgooglePlay: surface absent
Unattended accessRelay + sideloadBridge unattended modegooglePlay: not shipped
Tier-C tools (call / SMS / contacts / location)Relay + sideloadandroid_* toolsgooglePlay: 403 sideload_only
Enhanced / streaming voiceRelay/voice/output/*, /voice/synthesizeVanilla Hermes voice (basic)
Realtime agent voiceRelay/voice/realtime-agent/*Unavailable (experimental)
Notification companionRelayWSS notifications channelUnavailable
Desktop toolsRelayWSS desktop.commandUnavailable
07

Auth model per path

Three independent credential systems. None substitutes for another — this is why "connected" is not one boolean.

PathCredentialSourceLifetime
Gateway WS Dashboard cookie + WS ticket Manage sign-in → /api/auth/ws-ticket Ticket 30s, one-use (fresh per connect); cookie = session
API-server SSE Bearer API key Stored per connection (SessionTokenStore) Until rotated
Dashboard / Manage / voice Dashboard cookie Manage sign-in (password / Nous OIDC) Server session; wiped on dashboard restart
Relay plugin Paired session token QR / 6-char pairing → auth.ok Persisted — survives relay restart (sessions file + trusted-device refresh); no re-pair needed
Do not proxy dashboard auth or admin APIs over the Relay. Manage and Vanilla Hermes voice ride the dashboard surface directly with the dashboard cookie. The Relay carries only Relay-owned capabilities.
08

Where status is surfaced today

Inventory of the surfaces a user can read connection / path / feature health from. Status coverage is broad; the gap is a single consolidated diagnostic view and any visibility into the capability snapshot that actually drives the decision.

Connections settings — active card rich

ConnectionsSettingsScreen · ActiveConnectionSections
  • API / Dashboard / Voice / Relay / Terminal / Secure-proxy status rows
  • Route section: per-endpoint probe outcome (LAN/Tailscale/Public)
  • Transport security badge · keystore · relay session count

Settings root — exception pills good

SettingsScreen · ActiveAgentCard
  • "connection · model · personality" subtitle
  • Pills appear only when action needed (API offline, Dashboard sign-in, Relay stale, Plugin offline)

Chat / Terminal headers good

ConnectionStatusBadge · RelayUiState
  • 4-state animated dot (Connected/Connecting/Probing/Disconnected)
  • Relay row: Connected · Stale · Expired + endpoint role ("· Tailscale")

Voice settings good

VoiceSettingsScreen
  • Engine · STT/TTS route (Auto/Vanilla Hermes/Relay) each Ready/Offline/Checking
  • Render path: streaming vs basic synthesize

Bridge good

BridgeScreen · BridgeStatusCard
  • Relay-paired vs "Relay not connected" warning
  • Device / battery / screen / current-app / a11y telemetry

Info sheets (tap-to-detail) fragmented

ApiServer / Relay / Session InfoSheet + DiagnosticsLogPanel
  • Per-surface URL, reachability, recent diagnostic-log entries (6–8)
  • Three separate sheets — no single combined view

Diagnostic gaps vs. hermes relay doctor

What's missingTodayStatus
Active transport "you are on X" indicatorPreference resolved silently; user can't see gateway vs which SSE tier is livegap
Capability snapshot (sessions/completions/runs/health)Probed and used internally; never showngap
"Why auto picked X" explanationNo surfaced reasoning (e.g. "gateway SignInRequired → sessions")gap
Turn latency (TTFE / TTFT / done)TurnLatencyTracer → logcat onlygap
Unified single-screen health viewScattered across 3 info sheets + 2 settings screenspartial
Per-route reachability + reasonShown in Routes section, but not all failure reasonspartial
Recommendation. A single Connection Diagnostics screen (the in-app analogue of hermes relay doctor --json) would consolidate: active chat transport + tier, the capability probe table with the resolver's reasoning, per-route probe outcomes, auth state for all three credential systems, and the last turn's latency marks. Everything it needs already exists in ServerCapabilities, GatewayAvailability, DiagnosticsLog, and TurnLatencyTracer — it's an aggregation surface, not new plumbing.