Compare commits

..
Author SHA1 Message Date
Bailey Dixon 6a93d13ecb Merge origin/dev into fix/android-compaction-watchdog
# Conflicts:
#	CHANGELOG.md
2026-08-31 16:56:15 -04:00
Bailey Dixon cbc02bb407 Merge pull request #514 from Codename-11/fix/android-emulator-testing
test(android): add on-demand emulator coverage
2026-08-31 16:49:11 -04:00
Bailey Dixon 525f6b5fc0 Merge origin/dev into fix/android-compaction-watchdog 2026-08-31 16:41:54 -04:00
Bailey Dixon 6a66710763 Merge origin/dev into fix/android-compaction-watchdog 2026-08-31 16:22:08 -04:00
Bailey Dixon 8625963846 Merge origin/dev into fix/android-compaction-watchdog 2026-08-31 16:00:20 -04:00
Bailey Dixon e0b726de85 Merge origin/dev into fix/android-compaction-watchdog 2026-08-31 15:51:12 -04:00
Bailey Dixon d26bf6c25b Merge origin/dev into fix/android-compaction-watchdog
# Conflicts:
#	DEVLOG.md
2026-08-31 14:10:59 -04:00
Bailey Dixon 181e10f2ad test(android): cover compaction watchdog leases 2026-08-31 14:09:57 -04:00
Bailey Dixon 857a1551f3 Merge origin/dev into fix/android-compaction-watchdog 2026-08-31 13:17:04 -04:00
JackandClaude Opus 5 00052d20d9 watchdog: arm longer leash on compacting status instead of killing silent compaction
Server-side context compaction summarizes the transcript with NO deltas
or tool events flowing until it finishes. Near the context ceiling that
silence routinely exceeds TURN_TIMEOUT_MS (180s), so the idle watchdog
fired session.interrupt on a healthy compression, rolled back its work,
and retriggered on the next prompt — an infinite 'Operation interrupted'
loop that makes near-full sessions permanently unresponsive from mobile.

Observed against a live gateway: four turns killed at exactly
prompt+~251s (visible tool work + 180.0s of silent compaction), server
compression telemetry aborted at 180469/180233/180219ms with
failure_class=explicit_interrupt.

Fix: treat a status.update event with kind 'compacting' like the ask
events that already arm longer leashes (same watchdogTimeoutFor seam):
arm COMPACTING_TIMEOUT_MS (600s) instead of the 180s default. Any
regular event rearms TURN_TIMEOUT_MS as before. Pairs with the gateway
fix that emits periodic compacting heartbeats during compression
(NousResearch/hermes-agent#98371); a single compacting event at
compaction start already engages the longer leash on current gateways.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-31 13:14:35 -04:00
7 changed files with 166 additions and 5 deletions
+1
View File
@@ -14,6 +14,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this
### Fixed
- **Android keeps completed chat text visible when Dashboard sign-in expires.** Generic and reason-coded history `401` responses settle the local turn, preserve its transcript, and surface the existing sign-in recovery without reading another profile's API history.
- **Android keeps long-running context compaction alive.** A client-visible compaction status extends and refreshes the Gateway turn watchdog instead of interrupting healthy compression after the ordinary idle window. (Supersedes #484.)
- **Android Bot Chats render loaded history immediately.** Route-owned chat screens observe their own handler state from first composition, including fast history loads that settle before another frame. (Supersedes #453.)
- **Supervised Gateway setup stays parent-owned.** Add Gateway is single-flight and checks live parent authority before allocating a draft, relock/back cancels the exact pending setup, and the locked Chat footer no longer attempts protected navigation.
- **Generated images stay visible and use their intended Chat animation.** Completed image media survives a marker-lagging history refresh, and both the built-in `image_generate` tool and profile tools ending in `_create_image` use the image-generation presentation.
+13
View File
@@ -1,5 +1,18 @@
# Hermes-Relay — Dev Log
## 2026-08-31 — Android Gateway compaction watchdog lease
Gateway turns now recognize the upstream `status.update` payload kind
`compacting` and arm a ten-minute idle lease instead of the ordinary
three-minute watchdog. A single current-Gateway status protects silent
compaction, while repeated status heartbeats from newer gateways refresh the
same lease. Other status payloads retain the ordinary watchdog.
Focused Gateway client coverage uses shortened timeout seams to prove the
single-status, repeated-heartbeat, ordinary-silence, and payload-fencing paths
without waiting production minutes. The declarative vanilla-Gateway fixture
also models repeated compaction status before terminal completion.
## 2026-08-31 — Complete, readable Android release notes
Android release metadata now keeps one overall title and summary plus a complete
@@ -119,6 +119,8 @@ class GatewayChatClient(
private val promptSubmitTimeoutMs: Long = PROMPT_SUBMIT_REQUEST_TIMEOUT_MS,
/** Test seam — idle-progress watchdog base. Production keeps [TURN_TIMEOUT_MS]. */
private val turnIdleTimeoutMs: Long = TURN_TIMEOUT_MS,
/** Test seam — compaction idle lease. Production keeps [COMPACTING_TIMEOUT_MS]. */
private val compactingTimeoutMs: Long = COMPACTING_TIMEOUT_MS,
/** Random source for ordinary reconnect full-jitter. */
private val reconnectJitterUnit: () -> Double = { kotlin.random.Random.nextDouble() },
) : GatewayProfileEditorClient {
@@ -159,6 +161,19 @@ class GatewayChatClient(
private const val ASK_SUDO_TIMEOUT_MS = 150_000L
private const val ASK_UNBOUNDED_TIMEOUT_MS = 600_000L
/**
* Server-side context compaction summarizes the transcript through a
* (possibly slow) model with NO deltas or tool events flowing until it
* finishes — near the context ceiling that silence routinely exceeds
* [TURN_TIMEOUT_MS], so the idle watchdog would `session.interrupt` a
* healthy compression, roll back its work, and retrigger on the next
* prompt forever. A `status.update` event with kind `compacting`
* (emitted at compaction start, and periodically by newer gateways)
* arms this longer leash instead; any regular event rearms
* [TURN_TIMEOUT_MS].
*/
private const val COMPACTING_TIMEOUT_MS = 600_000L
private const val RPC_TIMEOUT_MS = 15_000L
const val PROFILE_AVATAR_MAX_BYTES = 2_000_000
@@ -4226,10 +4241,12 @@ class GatewayChatClient(
// ------------------------------------------------------------------
/** Per-event idle-watchdog duration — asks block server-side with no events, so they arm longer. */
private fun watchdogTimeoutFor(eventType: String): Long = when (eventType) {
"clarify.request", "secret.request" -> ASK_CLARIFY_SECRET_TIMEOUT_MS
"sudo.request" -> ASK_SUDO_TIMEOUT_MS
"approval.request" -> ASK_UNBOUNDED_TIMEOUT_MS
private fun watchdogTimeoutFor(eventType: String, payload: JsonObject? = null): Long = when {
eventType == "clarify.request" || eventType == "secret.request" -> ASK_CLARIFY_SECRET_TIMEOUT_MS
eventType == "sudo.request" -> ASK_SUDO_TIMEOUT_MS
eventType == "approval.request" -> ASK_UNBOUNDED_TIMEOUT_MS
eventType == "status.update" &&
payload?.stringField("kind") == "compacting" -> compactingTimeoutMs
else -> turnIdleTimeoutMs
}
@@ -4348,7 +4365,7 @@ class GatewayChatClient(
// Reset on every event — long tool runs keep the turn alive.
// Ask requests block with no further events, so they arm with
// their own (longer) duration via watchdogTimeoutFor.
armWatchdog(watchdogTimeoutFor(type))
armWatchdog(watchdogTimeoutFor(type, payload))
// Queue this immediately before the terminal callbacks. Both are
// marshalled through the same dispatcher, preserving callback order
// even when the WebSocket reader and reconnect coroutine differ.
@@ -803,6 +803,7 @@ class GatewayChatClientTest {
rpcTimeoutMs: Long = 15_000L,
promptSubmitTimeoutMs: Long = 1_800_000L,
turnIdleTimeoutMs: Long = 180_000L,
compactingTimeoutMs: Long = 600_000L,
callbackDispatcher: (block: () -> Unit) -> Unit = { it() },
ticketTimeoutMs: Long = 8_000L,
) = GatewayChatClient(
@@ -824,6 +825,7 @@ class GatewayChatClientTest {
rpcTimeoutMs = rpcTimeoutMs,
promptSubmitTimeoutMs = promptSubmitTimeoutMs,
turnIdleTimeoutMs = turnIdleTimeoutMs,
compactingTimeoutMs = compactingTimeoutMs,
)
private fun awaitCondition(
@@ -846,6 +848,7 @@ class GatewayChatClientTest {
rpcTimeoutMs: Long = 15_000L,
promptSubmitTimeoutMs: Long = 1_800_000L,
turnIdleTimeoutMs: Long = 180_000L,
compactingTimeoutMs: Long = 600_000L,
ticketTimeoutMs: Long = 8_000L,
) {
client.shutdown()
@@ -854,6 +857,7 @@ class GatewayChatClientTest {
rpcTimeoutMs = rpcTimeoutMs,
promptSubmitTimeoutMs = promptSubmitTimeoutMs,
turnIdleTimeoutMs = turnIdleTimeoutMs,
compactingTimeoutMs = compactingTimeoutMs,
ticketTimeoutMs = ticketTimeoutMs,
)
}
@@ -4738,6 +4742,85 @@ class GatewayChatClientTest {
)
}
@Test
fun `compacting status extends watchdog until a later completion`() {
rebuildClient(turnIdleTimeoutMs = 250L, compactingTimeoutMs = 1_000L)
val r = Recorder()
client.sendTurn(null, "compact once", null, r.callbacks) { r.preflightFailures += it }
val serverWs = harness.awaitServerSocket()
harness.awaitRpc("prompt.submit")
serverWs.send(
harness.eventFrame(
"status.update",
buildJsonObject { put("kind", "compacting") },
"live-1",
),
)
Thread.sleep(500)
assertTrue("normal idle watchdog fired during compaction: ${r.errors}", r.errors.isEmpty())
assertTrue(harness.rpcLog.none { it.first == "session.interrupt" })
serverWs.send(
harness.eventFrame("message.complete", buildJsonObject { put("text", "done") }, "live-1"),
)
assertTrue("turn never completed", r.completeLatch.await(5, TimeUnit.SECONDS))
assertTrue(r.errors.isEmpty())
assertTrue(r.preflightFailures.isEmpty())
}
@Test
fun `compacting heartbeats rearm watchdog beyond one compaction lease`() {
rebuildClient(turnIdleTimeoutMs = 200L, compactingTimeoutMs = 500L)
val r = Recorder()
client.sendTurn(null, "compact with heartbeats", null, r.callbacks) { r.preflightFailures += it }
val serverWs = harness.awaitServerSocket()
harness.awaitRpc("prompt.submit")
repeat(3) {
serverWs.send(
harness.eventFrame(
"status.update",
buildJsonObject { put("kind", "compacting") },
"live-1",
),
)
Thread.sleep(300)
}
assertTrue("compaction lease was not rearmed: ${r.errors}", r.errors.isEmpty())
assertTrue(harness.rpcLog.none { it.first == "session.interrupt" })
serverWs.send(
harness.eventFrame("message.complete", buildJsonObject { put("text", "done") }, "live-1"),
)
assertTrue("turn never completed", r.completeLatch.await(5, TimeUnit.SECONDS))
assertTrue(r.errors.isEmpty())
assertTrue(r.preflightFailures.isEmpty())
}
@Test
fun `non compacting status keeps the ordinary watchdog`() {
rebuildClient(turnIdleTimeoutMs = 250L, compactingTimeoutMs = 2_000L)
val r = Recorder()
client.sendTurn(null, "ordinary status", null, r.callbacks) { r.preflightFailures += it }
val serverWs = harness.awaitServerSocket()
harness.awaitRpc("prompt.submit")
serverWs.send(
harness.eventFrame(
"status.update",
buildJsonObject { put("kind", "process") },
"live-1",
),
)
assertTrue("ordinary watchdog never fired", r.completeLatch.await(5, TimeUnit.SECONDS))
assertTrue("expected a stream error from the watchdog", r.errors.isNotEmpty())
assertTrue(r.preflightFailures.isEmpty())
harness.awaitRpc("session.interrupt")
}
@Test
fun `idle watchdog fires when events stop flowing`() {
rebuildClient(turnIdleTimeoutMs = 500L)
+1
View File
@@ -68,6 +68,7 @@ the upstream contract identifiers it depends on.
|---|---|
| `initial_history_bind` | Durable, profile-scoped history is already available when the client resumes and first binds its rendered transcript |
| `ordinary_turn` | Normal message start, deltas, completion, and persisted history |
| `compaction_status` | Compaction status is client-visible before terminal completion and may repeat as a heartbeat |
| `rapid_tools_interims` | Rapid chunks, reasoning, tool activity, and interim assistant boundaries |
| `queued_follow_up` | Two explicitly owned turns and ordered queue drainage |
| `scope_rejection_inputs` | Exact, foreign, and unscoped event inputs |
@@ -123,6 +123,23 @@ class FixtureTestCase(unittest.IsolatedAsyncioTestCase):
self.assertEqual(["user", "assistant"], [row["role"] for row in history["messages"]])
self.assertEqual(2, history["pagination"]["returned"])
async def test_compaction_status_repeats_before_terminal_completion(self) -> None:
fixture, base_url = await self.start("compaction_status")
ws, _ = await self.connect(base_url)
await self.rpc(ws, 1, "prompt.submit", {"text": "fixture"})
frames = await self.frames_until(
ws,
lambda frame: frame.get("params", {}).get("type") == "message.complete",
)
events = [frame["params"] for frame in frames if frame.get("method") == "event"]
compacting = [
event for event in events
if event.get("type") == "status.update"
and event.get("payload", {}).get("kind") == "compacting"
]
self.assertEqual(2, len(compacting))
self.assertEqual("message.complete", events[-1]["type"])
async def test_cross_client_observer_never_claims_or_interrupts_producer(self) -> None:
fixture, base_url = await self.start("cross_client_observation")
producer, _ = await self.connect(base_url)
@@ -0,0 +1,29 @@
{
"name": "compaction_status",
"live_session_id": "fixture-live-1",
"stored_session_id": "20260831_120000_compaction",
"profile": "default",
"contract_requirements": [
"gateway.message_complete"
],
"initial_history": [],
"turns": [
{
"steps": [
{"op": "event", "type": "message.start"},
{"op": "event", "type": "status.update", "payload": {"kind": "compacting", "text": "Compacting context"}},
{"op": "sleep", "milliseconds": 50},
{"op": "event", "type": "status.update", "payload": {"kind": "compacting", "text": "Compacting context"}},
{
"op": "persist",
"messages": [
{"id": 1, "role": "user", "content": "Exercise compaction status.", "timestamp": 1.0},
{"id": 2, "role": "assistant", "content": "Compaction finished.", "timestamp": 2.0, "finish_reason": "stop"}
]
},
{"op": "set_running", "value": false},
{"op": "event", "type": "message.complete", "payload": {"text": "Compaction finished.", "status": "complete"}}
]
}
]
}