feat(android): absorb upstream voice and wake parity
This commit is contained in:
@@ -6,6 +6,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **Android can opt into an experimental local “Hey Hermes” wake word.** A user-started microphone foreground service performs sherpa-onnx detection on the phone, keeps pre-activation audio local, shows an ongoing Stop notification, and safely hands microphone ownership to the existing voice flow.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Voice interruption covers the complete active response.** One calibrated VAD listener spans generation and playback on Standard and Realtime paths, rejects stale callbacks and speaker bleed, and keeps promoted background tasks alive when speech alone is stopped.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Windows-trusted certificates work in the desktop CLI.** The packaged Windows binary and newer Node runtimes add the Windows certificate store without dropping bundled or operator-supplied roots, while TLS verification and Relay certificate pinning remain enforced.
|
||||
|
||||
@@ -1,5 +1,32 @@
|
||||
# Hermes-Relay — Dev Log
|
||||
|
||||
## 2026-07-29 — Full-turn voice interruption and local wake-word preview
|
||||
|
||||
Android barge-in now owns one microphone/VAD listener from response generation
|
||||
through playback drain for both Standard and Realtime voice. Quiet-room RMS
|
||||
calibration freezes before output begins, playback receives a grace interval,
|
||||
and model-confirmed majority filtering separates actual interruption from raw
|
||||
ducking hints. Turn epochs, stream cancellation, late-delta suppression, and
|
||||
an awaited microphone handoff keep an interrupted response from speaking again
|
||||
or racing the replacement recording. Exact stop/pause intent is phase-aware,
|
||||
while explicit background-task cancellation remains separate from silencing.
|
||||
|
||||
An opt-in Android-local “Hey Hermes” preview uses sherpa-onnx in a user-started
|
||||
microphone foreground service. Its approximately 6 MB English model is
|
||||
downloaded and hash-verified on first enable rather than bundled. The service
|
||||
keeps pre-activation audio local, exposes an ongoing Stop notification, pauses
|
||||
for active voice, and shares a process-wide single-microphone ownership
|
||||
contract with voice recording, barge-in, and realtime diagnostics. The stored
|
||||
configuration includes strictness, confirmation frames, new-session behavior,
|
||||
and a deliberately inactive future profile-routing shape.
|
||||
|
||||
Focused JVM coverage exercises calibration, grace, listener teardown,
|
||||
Thinking-to-Speaking ownership, generation/playback interruption, command
|
||||
gating, wake preferences, activation, and microphone exclusion. Android
|
||||
compilation for both distribution flavors, sideload lint, and sideload APK
|
||||
packaging pass with all four supported ABIs. On-device acoustic, foreground
|
||||
service, and lifecycle checks remain the corresponding validation gates.
|
||||
|
||||
## 2026-07-28 — Android 1.5.2 production release
|
||||
|
||||
Android 1.5.2 shipped from the approved `dev` to `main` release tree as
|
||||
|
||||
@@ -1042,11 +1042,12 @@ to tool state, safety prompts, or the current task.
|
||||
playback-synchronized amplitude through `shouldMarkRealtimeOutputActive`,
|
||||
matching the basic-TTS path. Confirm visually on-device with the 1.4.1 batch.
|
||||
|
||||
- **Voice command layer — initial 1.4.1 subset code-complete; live verify and
|
||||
- **Voice command layer — phase-aware stop/pause code-complete; live verify and
|
||||
navigation residuals remain.** Exact final transcripts can stop speech,
|
||||
explicitly cancel the active background task, pause/resume Continuous mode,
|
||||
repeat a settled background answer, and start a new Standard chat. Bare `stop`
|
||||
and `cancel`, partial transcripts, and command-like ordinary prompts stay on the
|
||||
repeat a settled background answer, and start a new Standard chat. Bare
|
||||
`stop`/`pause` are commands only after barge-in interrupted an active response;
|
||||
`cancel`, partial transcripts, and command-like ordinary prompts stay on the
|
||||
normal Hermes route. Realtime `new chat` remains gated on a clean websocket
|
||||
session-rebind boundary; `open overlay` and `return to Hermes` remain future
|
||||
navigation commands. Verify barge-in Stop, pause during a background run, local
|
||||
@@ -1082,11 +1083,21 @@ and whether the agent is waiting on the user.
|
||||
experimental barge-in choice. Relay update is server-first; local Voice/barge-in
|
||||
values share one DataStore transaction, with relay rollback on local failure.
|
||||
|
||||
- **Barge-in hardening** — keep barge-in experimental until echo/self-recording
|
||||
- **Barge-in hardening — code complete; on-device matrix remains.** Full-turn
|
||||
listener ownership, AEC/noise suppression, quiet-room calibration,
|
||||
playback grace, duck/cut behavior, late-delta fencing, and single-microphone
|
||||
handoff are implemented. Keep the feature experimental until phone testing
|
||||
covers speakerphone/headphones, quiet/noisy rooms, Standard/Realtime
|
||||
generation and playback, stop/pause, and resume-after-interruption.
|
||||
|
||||
is solved. The target path is proper AEC, playback-ducking, and a rule that
|
||||
|
||||
output audio can never become a user turn.
|
||||
- **Experimental wake word — on-device validation.** Verify first-enable model
|
||||
installation and integrity failure recovery, all supported ABIs, Android
|
||||
notification/microphone permission variants, background-start restrictions,
|
||||
task recreation from the detection notification, acoustic false-positive and
|
||||
false-negative rates, battery impact, stop action, and wake→voice→wake
|
||||
microphone handoff. The first release remains fixed to “Hey Hermes”; do not
|
||||
expose profile-specific phrases until routing and acoustic behavior are
|
||||
implemented and validated.
|
||||
|
||||
- **Audio quality guardrails** — normalize output volume across realtime and
|
||||
|
||||
|
||||
@@ -164,6 +164,21 @@ android {
|
||||
}
|
||||
}
|
||||
|
||||
packaging {
|
||||
jniLibs {
|
||||
// sherpa-onnx v1.13.4 and the Silero VAD both use ONNX Runtime.
|
||||
// Keep them on sherpa's 1.27.0 baseline and package one shared core.
|
||||
pickFirsts += "**/libonnxruntime.so"
|
||||
|
||||
// The Android app calls only sherpa's JNI facade. These native C/C++
|
||||
// API facades are development surfaces and are not loaded by the app.
|
||||
excludes += setOf(
|
||||
"**/libsherpa-onnx-c-api.so",
|
||||
"**/libsherpa-onnx-cxx-api.so",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// JVM unit tests run against the stubbed Android SDK jar, where every
|
||||
// platform API method throws RuntimeException("... not mocked") by
|
||||
// default. With returnDefaultValues = true, those stubs instead
|
||||
@@ -262,6 +277,12 @@ dependencies {
|
||||
// Bundled ONNX Silero model (~2.2 MB); pulled from JitPack.
|
||||
implementation(libs.android.vad.silero)
|
||||
|
||||
// Experimental, opt-in local keyword spotting. Models are downloaded only
|
||||
// after the user enables the feature; no model binary is bundled in APKs.
|
||||
// Keep the shared runtime aligned with sherpa-onnx v1.13.4.
|
||||
implementation(libs.onnxruntime.android)
|
||||
implementation(libs.sherpa.onnx)
|
||||
|
||||
// Google Play In-App Update — googlePlay flavor ONLY (FLEXIBLE flow).
|
||||
// Scoped via the `googlePlayImplementation` configuration so it never
|
||||
// ships in the sideload APK, which updates via the GitHub-releases
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
for the device-control bridge service; the merger dedups.) -->
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_SPECIAL_USE" />
|
||||
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
|
||||
|
||||
<uses-feature android:name="android.hardware.camera" android:required="false" />
|
||||
|
||||
@@ -104,6 +105,14 @@
|
||||
android:value="Keeps user-started Hermes turns connected until they finish or need input, and optionally keeps idle connections responsive when the user enables Persistent connection." />
|
||||
</service>
|
||||
|
||||
<!-- Experimental, explicitly user-started on-device wake-word listener.
|
||||
Audio remains local and the service is never boot/restart started. -->
|
||||
<service
|
||||
android:name=".wake.WakeWordForegroundService"
|
||||
android:exported="false"
|
||||
android:foregroundServiceType="microphone"
|
||||
android:stopWithTask="false" />
|
||||
|
||||
</application>
|
||||
|
||||
</manifest>
|
||||
|
||||
@@ -8,11 +8,16 @@ import android.media.MediaRecorder
|
||||
import android.media.audiofx.AcousticEchoCanceler
|
||||
import android.media.audiofx.NoiseSuppressor
|
||||
import android.util.Log
|
||||
import com.hermesandroid.relay.wake.MicrophoneLease
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwner
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwnershipCoordinator
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.CoroutineDispatcher
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.NonCancellable
|
||||
import kotlinx.coroutines.cancelAndJoin
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.flow.MutableSharedFlow
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
@@ -22,23 +27,26 @@ import kotlinx.coroutines.flow.asSharedFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
import kotlinx.coroutines.isActive
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.coroutines.yield
|
||||
import kotlin.math.max
|
||||
|
||||
/**
|
||||
* Duplex audio capture for voice barge-in (plan unit B3).
|
||||
*
|
||||
* While TTS is playing, this listener continuously pulls 32 ms / 512-sample
|
||||
* PCM frames off the microphone and feeds them to [VadEngine]. It emits two
|
||||
* SharedFlows that B4 will wire into the voice state machine:
|
||||
* During response generation and playback, this listener continuously pulls
|
||||
* 32 ms / 512-sample PCM frames off the microphone and feeds them to
|
||||
* [VadEngine]. One instance owns the full active turn. It emits two
|
||||
* SharedFlows wired into the voice state machine:
|
||||
*
|
||||
* - [maybeSpeech] fires on the **first** positive raw-VAD frame — before the
|
||||
* second-layer debouncer latches. B4 uses this to softly [VoicePlayer.duck]
|
||||
* the TTS so the user's voice has acoustic headroom while we decide whether
|
||||
* to cut off.
|
||||
*
|
||||
* - [bargeInDetected] fires when [VadEngine] confirms speech post-hysteresis.
|
||||
* B4 uses this to call `interruptSpeaking()` and flip state to Listening.
|
||||
* - [bargeInDetected] fires when [VadEngine] confirms speech post-hysteresis
|
||||
* and the calibrated RMS majority gate accepts it. The owner uses this to
|
||||
* interrupt generation/playback and flip state to Listening.
|
||||
*
|
||||
* ### Acoustic echo cancellation
|
||||
*
|
||||
@@ -140,8 +148,28 @@ class BargeInListener internal constructor(
|
||||
private val frameBuffer: ShortArray = ShortArray(VadEngine.FRAME_SIZE_SAMPLES)
|
||||
|
||||
@Volatile private var readerJob: Job? = null
|
||||
@Volatile private var microphoneLease: MicrophoneLease? = null
|
||||
@Volatile private var aec: AcousticEchoCanceler? = null
|
||||
@Volatile private var noiseSuppressor: NoiseSuppressor? = null
|
||||
private val rmsGate = RmsBargeInGate()
|
||||
@Volatile private var playbackGraceMs: Long = RmsBargeInGate.DEFAULT_PLAYBACK_GRACE_MS
|
||||
|
||||
/** Apply the user-facing barge-in sensitivity to the quiet-room RMS gate. */
|
||||
fun setThresholdMultiplier(multiplier: Float) {
|
||||
rmsGate.thresholdMultiplier = multiplier
|
||||
}
|
||||
|
||||
/**
|
||||
* Freeze quiet-room calibration and begin the playback-only grace window.
|
||||
* Idempotent so every renderer may call it at its first audible chunk.
|
||||
*/
|
||||
fun markPlaybackStarted(
|
||||
nowMs: Long = System.currentTimeMillis(),
|
||||
graceMs: Long = RmsBargeInGate.DEFAULT_PLAYBACK_GRACE_MS,
|
||||
) {
|
||||
playbackGraceMs = graceMs.coerceAtLeast(0L)
|
||||
rmsGate.markPlaybackStarted(nowMs)
|
||||
}
|
||||
|
||||
/**
|
||||
* Allocate the audio pipeline and begin reading frames into [vadEngine].
|
||||
@@ -163,6 +191,12 @@ class BargeInListener internal constructor(
|
||||
return
|
||||
}
|
||||
|
||||
val lease = MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.BargeIn)
|
||||
if (lease == null) {
|
||||
Log.i(TAG, "Barge-in listener inactive — microphone is owned by another voice surface")
|
||||
return
|
||||
}
|
||||
microphoneLease = lease
|
||||
if (!audioSource.initialize()) {
|
||||
Log.w(
|
||||
TAG,
|
||||
@@ -170,11 +204,15 @@ class BargeInListener internal constructor(
|
||||
"(missing RECORD_AUDIO permission or mic busy) — listener inactive",
|
||||
)
|
||||
_aecAttached.value = false
|
||||
MicrophoneOwnershipCoordinator.release(lease)
|
||||
microphoneLease = null
|
||||
return
|
||||
}
|
||||
|
||||
_aecAttached.value = false
|
||||
rmsGate.reset()
|
||||
readerJob = scope.launch(readerDispatcher) {
|
||||
var effectsJob: Job? = null
|
||||
try {
|
||||
try {
|
||||
audioSource.start()
|
||||
@@ -185,7 +223,10 @@ class BargeInListener internal constructor(
|
||||
return@launch
|
||||
}
|
||||
Log.i(TAG, "Barge-in AudioRecord reader started")
|
||||
maybeAttachEffects()
|
||||
// Do not block generation-phase listening while waiting for an
|
||||
// AudioTrack session that does not exist until playback. The
|
||||
// effects attach races harmlessly beside the reader.
|
||||
effectsJob = launch { maybeAttachEffects() }
|
||||
|
||||
while (isActive) {
|
||||
val read = try {
|
||||
@@ -222,10 +263,17 @@ class BargeInListener internal constructor(
|
||||
Log.w(TAG, "VadEngine.analyze failed; stopping reader: ${t.message}")
|
||||
break
|
||||
}
|
||||
if (result.probability > 0f) {
|
||||
val gated = rmsGate.observe(
|
||||
frame = frameBuffer,
|
||||
rawSpeech = result.probability > 0f,
|
||||
nowMs = System.currentTimeMillis(),
|
||||
playbackGraceMs = playbackGraceMs,
|
||||
confirmedSpeech = result.isSpeech,
|
||||
)
|
||||
if (gated.maybeSpeech) {
|
||||
_maybeSpeech.tryEmit(Unit)
|
||||
}
|
||||
if (result.isSpeech) {
|
||||
if (gated.detected) {
|
||||
_bargeInDetected.tryEmit(Unit)
|
||||
}
|
||||
// Give the dispatcher a chance to observe cancellation
|
||||
@@ -237,11 +285,19 @@ class BargeInListener internal constructor(
|
||||
yield()
|
||||
}
|
||||
} finally {
|
||||
// The reader reaches this block with its Job cancelled.
|
||||
// Teardown still has to wait for the sibling AEC poll before
|
||||
// releasing the AudioRecord and microphone lease.
|
||||
withContext(NonCancellable) {
|
||||
effectsJob?.cancelAndJoin()
|
||||
}
|
||||
// Release effects + AudioRecord in the reverse of attach order
|
||||
// so the AudioSessionId is still valid when AEC teardown runs.
|
||||
releaseEffects()
|
||||
runCatching { audioSource.stop() }
|
||||
runCatching { audioSource.release() }
|
||||
microphoneLease?.let(MicrophoneOwnershipCoordinator::release)
|
||||
microphoneLease = null
|
||||
_aecAttached.value = false
|
||||
}
|
||||
}
|
||||
@@ -258,8 +314,16 @@ class BargeInListener internal constructor(
|
||||
if (job?.isActive == true) {
|
||||
Log.i(TAG, "Stopping barge-in AudioRecord reader")
|
||||
}
|
||||
// AudioRecord.read() may be blocked in native code, so stop the source
|
||||
// before cancellation to make the reader observe shutdown promptly.
|
||||
runCatching { audioSource.stop() }
|
||||
job?.cancel()
|
||||
readerJob = null
|
||||
if (job == null) {
|
||||
runCatching { audioSource.release() }
|
||||
microphoneLease?.let(MicrophoneOwnershipCoordinator::release)
|
||||
microphoneLease = null
|
||||
}
|
||||
return job
|
||||
}
|
||||
|
||||
|
||||
@@ -4,6 +4,8 @@ import android.annotation.SuppressLint
|
||||
import android.media.AudioFormat
|
||||
import android.media.AudioRecord
|
||||
import android.media.MediaRecorder
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwner
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwnershipCoordinator
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import java.io.ByteArrayOutputStream
|
||||
@@ -40,24 +42,37 @@ class RealtimePcmRecorder(
|
||||
maxDurationMs: Long = 15_000,
|
||||
onLevel: ((Float) -> Unit)? = null,
|
||||
): ByteArray = withContext(Dispatchers.IO) {
|
||||
val minBuffer = AudioRecord.getMinBufferSize(
|
||||
sampleRate,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(sampleRate / 10 * 2)
|
||||
val microphoneLease =
|
||||
MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.RealtimeDiagnostics)
|
||||
?: error("Microphone is in use by another voice feature")
|
||||
val minBuffer = try {
|
||||
AudioRecord.getMinBufferSize(
|
||||
sampleRate,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(sampleRate / 10 * 2)
|
||||
} catch (t: Throwable) {
|
||||
MicrophoneOwnershipCoordinator.release(microphoneLease)
|
||||
throw t
|
||||
}
|
||||
val maxBytes = ((sampleRate * maxDurationMs) / 1000L * 2L).toInt()
|
||||
|
||||
val recorder = AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(sampleRate)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer)
|
||||
.build()
|
||||
val recorder = try {
|
||||
AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(sampleRate)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer)
|
||||
.build()
|
||||
} catch (t: Throwable) {
|
||||
MicrophoneOwnershipCoordinator.release(microphoneLease)
|
||||
throw t
|
||||
}
|
||||
|
||||
val out = ByteArrayOutputStream(minBuffer * 4)
|
||||
val buffer = ByteArray(minBuffer)
|
||||
@@ -77,32 +92,46 @@ class RealtimePcmRecorder(
|
||||
capturing = false
|
||||
try { recorder.stop() } catch (_: Exception) { }
|
||||
recorder.release()
|
||||
MicrophoneOwnershipCoordinator.release(microphoneLease)
|
||||
}
|
||||
out.toByteArray()
|
||||
}
|
||||
|
||||
@SuppressLint("MissingPermission")
|
||||
suspend fun capture(durationMs: Long = 800): ByteArray = withContext(Dispatchers.IO) {
|
||||
val minBuffer = AudioRecord.getMinBufferSize(
|
||||
sampleRate,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(sampleRate / 10 * 2)
|
||||
val microphoneLease =
|
||||
MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.RealtimeDiagnostics)
|
||||
?: error("Microphone is in use by another voice feature")
|
||||
val minBuffer = try {
|
||||
AudioRecord.getMinBufferSize(
|
||||
sampleRate,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(sampleRate / 10 * 2)
|
||||
} catch (t: Throwable) {
|
||||
MicrophoneOwnershipCoordinator.release(microphoneLease)
|
||||
throw t
|
||||
}
|
||||
val targetBytes = ((sampleRate * durationMs) / 1000L * 2L)
|
||||
.toInt()
|
||||
.coerceAtLeast(minBuffer)
|
||||
|
||||
val recorder = AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(sampleRate)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer)
|
||||
.build()
|
||||
val recorder = try {
|
||||
AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(sampleRate)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer)
|
||||
.build()
|
||||
} catch (t: Throwable) {
|
||||
MicrophoneOwnershipCoordinator.release(microphoneLease)
|
||||
throw t
|
||||
}
|
||||
|
||||
val out = ByteArrayOutputStream(targetBytes)
|
||||
val buffer = ByteArray(minBuffer)
|
||||
@@ -123,6 +152,7 @@ class RealtimePcmRecorder(
|
||||
} finally {
|
||||
try { recorder.stop() } catch (_: Exception) { }
|
||||
recorder.release()
|
||||
MicrophoneOwnershipCoordinator.release(microphoneLease)
|
||||
}
|
||||
out.toByteArray()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
package com.hermesandroid.relay.audio
|
||||
|
||||
import kotlin.math.sqrt
|
||||
|
||||
/**
|
||||
* Turn-scoped RMS gate layered in front of the model VAD.
|
||||
*
|
||||
* The first quiet frames establish a room floor before playback. That floor is
|
||||
* frozen as soon as playback begins so speaker output can never teach the gate
|
||||
* to ignore the user. Detection uses a majority window rather than requiring
|
||||
* perfectly consecutive frames, which tolerates short consonant/syllable dips.
|
||||
*/
|
||||
internal class RmsBargeInGate(
|
||||
private val calibrationFrames: Int = DEFAULT_CALIBRATION_FRAMES,
|
||||
private val decisionWindowFrames: Int = DEFAULT_DECISION_WINDOW_FRAMES,
|
||||
private val requiredWindowRatio: Float = DEFAULT_REQUIRED_WINDOW_RATIO,
|
||||
) {
|
||||
private val calibration = ArrayList<Float>(calibrationFrames)
|
||||
private val decisions = ArrayDeque<Boolean>(decisionWindowFrames)
|
||||
|
||||
private var frozenFloor: Float? = null
|
||||
private var playbackStartedAtMs: Long? = null
|
||||
|
||||
var thresholdMultiplier: Float = DEFAULT_THRESHOLD_MULTIPLIER
|
||||
set(value) {
|
||||
field = value.coerceIn(MIN_THRESHOLD_MULTIPLIER, MAX_THRESHOLD_MULTIPLIER)
|
||||
}
|
||||
|
||||
fun reset() {
|
||||
calibration.clear()
|
||||
decisions.clear()
|
||||
frozenFloor = null
|
||||
playbackStartedAtMs = null
|
||||
}
|
||||
|
||||
fun markPlaybackStarted(nowMs: Long) {
|
||||
if (playbackStartedAtMs == null) {
|
||||
freezeCalibration()
|
||||
playbackStartedAtMs = nowMs
|
||||
decisions.clear()
|
||||
}
|
||||
}
|
||||
|
||||
fun observe(
|
||||
frame: ShortArray,
|
||||
rawSpeech: Boolean,
|
||||
nowMs: Long,
|
||||
playbackGraceMs: Long,
|
||||
confirmedSpeech: Boolean = rawSpeech,
|
||||
): RmsGateResult {
|
||||
val rms = rms(frame)
|
||||
val playback = playbackStartedAtMs != null
|
||||
|
||||
if (!playback && frozenFloor == null && calibration.size < calibrationFrames) {
|
||||
calibration += rms
|
||||
if (calibration.size == calibrationFrames) freezeCalibration()
|
||||
}
|
||||
|
||||
val floor = frozenFloor ?: robustFloor(calibration)
|
||||
val scaled = (floor * thresholdMultiplier).coerceAtLeast(MIN_GENERATION_THRESHOLD_RMS)
|
||||
val threshold = if (playback) {
|
||||
scaled.coerceIn(MIN_PLAYBACK_THRESHOLD_RMS, MAX_PLAYBACK_THRESHOLD_RMS)
|
||||
} else {
|
||||
scaled.coerceAtMost(MAX_PLAYBACK_THRESHOLD_RMS)
|
||||
}
|
||||
val inPlaybackGrace = playbackStartedAtMs?.let { nowMs - it < playbackGraceMs } == true
|
||||
val aboveRaw = rawSpeech && rms >= threshold && !inPlaybackGrace
|
||||
val aboveConfirmed = confirmedSpeech && rms >= threshold && !inPlaybackGrace
|
||||
|
||||
decisions.addLast(aboveConfirmed)
|
||||
while (decisions.size > decisionWindowFrames) decisions.removeFirst()
|
||||
val required = (decisionWindowFrames * requiredWindowRatio).toInt()
|
||||
val detected = decisions.size == decisionWindowFrames && decisions.count { it } >= required
|
||||
|
||||
return RmsGateResult(
|
||||
maybeSpeech = aboveRaw,
|
||||
detected = detected,
|
||||
rms = rms,
|
||||
floor = floor,
|
||||
threshold = threshold,
|
||||
calibrating = !playback && frozenFloor == null,
|
||||
playbackGrace = inPlaybackGrace,
|
||||
)
|
||||
}
|
||||
|
||||
private fun freezeCalibration() {
|
||||
if (frozenFloor == null) {
|
||||
frozenFloor = robustFloor(calibration)
|
||||
}
|
||||
}
|
||||
|
||||
private fun robustFloor(values: List<Float>): Float {
|
||||
if (values.isEmpty()) return DEFAULT_QUIET_FLOOR_RMS
|
||||
// Use the quieter half so a cough or chair noise during turn setup
|
||||
// cannot permanently deafen the listener.
|
||||
val sorted = values.sorted()
|
||||
val quietHalf = sorted.take((sorted.size / 2).coerceAtLeast(1))
|
||||
return quietHalf.average().toFloat().coerceAtLeast(MIN_QUIET_FLOOR_RMS)
|
||||
}
|
||||
|
||||
private fun rms(frame: ShortArray): Float {
|
||||
if (frame.isEmpty()) return 0f
|
||||
var sum = 0.0
|
||||
frame.forEach { sample ->
|
||||
val value = sample.toDouble()
|
||||
sum += value * value
|
||||
}
|
||||
return sqrt(sum / frame.size).toFloat()
|
||||
}
|
||||
|
||||
companion object {
|
||||
const val DEFAULT_THRESHOLD_MULTIPLIER = 3f
|
||||
const val DEFAULT_PLAYBACK_GRACE_MS = 500L
|
||||
internal const val DEFAULT_CALIBRATION_FRAMES = 10
|
||||
internal const val DEFAULT_DECISION_WINDOW_FRAMES = 10
|
||||
internal const val DEFAULT_REQUIRED_WINDOW_RATIO = 0.8f
|
||||
internal const val MIN_PLAYBACK_THRESHOLD_RMS = 1_500f
|
||||
internal const val MAX_PLAYBACK_THRESHOLD_RMS = 4_000f
|
||||
internal const val MIN_GENERATION_THRESHOLD_RMS = 300f
|
||||
internal const val DEFAULT_QUIET_FLOOR_RMS = 500f
|
||||
internal const val MIN_QUIET_FLOOR_RMS = 50f
|
||||
internal const val MIN_THRESHOLD_MULTIPLIER = 1.5f
|
||||
internal const val MAX_THRESHOLD_MULTIPLIER = 8f
|
||||
}
|
||||
}
|
||||
|
||||
internal data class RmsGateResult(
|
||||
val maybeSpeech: Boolean,
|
||||
val detected: Boolean,
|
||||
val rms: Float,
|
||||
val floor: Float,
|
||||
val threshold: Float,
|
||||
val calibrating: Boolean,
|
||||
val playbackGrace: Boolean,
|
||||
)
|
||||
@@ -8,6 +8,9 @@ import android.media.MediaRecorder
|
||||
import android.media.audiofx.AcousticEchoCanceler
|
||||
import android.media.audiofx.NoiseSuppressor
|
||||
import android.util.Log
|
||||
import com.hermesandroid.relay.wake.MicrophoneLease
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwner
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwnershipCoordinator
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
@@ -57,6 +60,7 @@ class VoiceRecorder(
|
||||
private val bufferLock = Any()
|
||||
private val stopRequested = AtomicBoolean(false)
|
||||
private var audioRecord: AudioRecord? = null
|
||||
private var microphoneLease: MicrophoneLease? = null
|
||||
private var echoCanceler: AcousticEchoCanceler? = null
|
||||
private var noiseSuppressor: NoiseSuppressor? = null
|
||||
private var currentOutputFile: File? = null
|
||||
@@ -79,12 +83,21 @@ class VoiceRecorder(
|
||||
releaseRecorder()
|
||||
}
|
||||
}
|
||||
val lease = MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.VoiceCapture)
|
||||
?: throw IllegalStateException("Microphone is in use by another voice feature")
|
||||
microphoneLease = lease
|
||||
|
||||
val minBuffer = AudioRecord.getMinBufferSize(
|
||||
val minBuffer = try {
|
||||
AudioRecord.getMinBufferSize(
|
||||
SAMPLE_RATE,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(SAMPLE_RATE / 10 * BYTES_PER_SAMPLE)
|
||||
).coerceAtLeast(SAMPLE_RATE / 10 * BYTES_PER_SAMPLE)
|
||||
} catch (t: Throwable) {
|
||||
MicrophoneOwnershipCoordinator.release(lease)
|
||||
microphoneLease = null
|
||||
throw t
|
||||
}
|
||||
|
||||
val outFile = File(context.cacheDir, "voice_rec_${System.currentTimeMillis()}.wav")
|
||||
currentOutputFile = outFile
|
||||
@@ -95,21 +108,29 @@ class VoiceRecorder(
|
||||
stopRequested.set(false)
|
||||
_amplitude.value = 0f
|
||||
|
||||
val recorder = AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(SAMPLE_RATE)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer * 2)
|
||||
.build()
|
||||
val recorder = try {
|
||||
AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.MIC)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setSampleRate(SAMPLE_RATE)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer * 2)
|
||||
.build()
|
||||
} catch (t: Throwable) {
|
||||
MicrophoneOwnershipCoordinator.release(lease)
|
||||
microphoneLease = null
|
||||
throw t
|
||||
}
|
||||
|
||||
if (recorder.state != AudioRecord.STATE_INITIALIZED) {
|
||||
recorder.release()
|
||||
currentOutputFile = null
|
||||
MicrophoneOwnershipCoordinator.release(lease)
|
||||
microphoneLease = null
|
||||
throw IllegalStateException("AudioRecord failed to initialize")
|
||||
}
|
||||
|
||||
@@ -118,6 +139,8 @@ class VoiceRecorder(
|
||||
} catch (e: Exception) {
|
||||
recorder.release()
|
||||
currentOutputFile = null
|
||||
MicrophoneOwnershipCoordinator.release(lease)
|
||||
microphoneLease = null
|
||||
throw e
|
||||
}
|
||||
|
||||
@@ -281,6 +304,8 @@ class VoiceRecorder(
|
||||
try { record.release() } catch (_: Exception) { }
|
||||
}
|
||||
audioRecord = null
|
||||
microphoneLease?.let(MicrophoneOwnershipCoordinator::release)
|
||||
microphoneLease = null
|
||||
readThread = null
|
||||
readDone = null
|
||||
}
|
||||
|
||||
@@ -1244,6 +1244,36 @@ fun RelayApp() {
|
||||
// bottom navigation bar so the voice overlay can own the entire screen
|
||||
// without the Chat/Terminal/Bridge/Settings tabs peeking through below.
|
||||
val voiceUiState by voiceViewModel.uiState.collectAsState()
|
||||
val wakeActivation by
|
||||
com.hermesandroid.relay.wake.WakeWordActivationCoordinator.pending.collectAsState()
|
||||
val appIsForeground by
|
||||
com.hermesandroid.relay.util.AppForegroundTracker.isForeground.collectAsState()
|
||||
|
||||
// A background detection stays pending behind the actionable
|
||||
// notification. Only a visible Hermes activity may enter voice.
|
||||
LaunchedEffect(wakeActivation?.id, appIsForeground) {
|
||||
val activation = wakeActivation ?: return@LaunchedEffect
|
||||
if (!appIsForeground) return@LaunchedEffect
|
||||
com.hermesandroid.relay.wake.WakeWordForegroundService.prepareForVoice()
|
||||
if (activation.startNewSession) {
|
||||
chatViewModel.createNewChat()
|
||||
}
|
||||
navController.navigate(Screen.Chat.route(openAgentSheet = false)) {
|
||||
launchSingleTop = true
|
||||
}
|
||||
voiceViewModel.enterVoiceMode()
|
||||
// Let the existing RelayApp initialization and Chat destination
|
||||
// settle before opening VoiceRecorder on a cold task recreation.
|
||||
delay(120L)
|
||||
voiceViewModel.startListening()
|
||||
com.hermesandroid.relay.wake.WakeWordActivationCoordinator.consume(activation.id)
|
||||
}
|
||||
|
||||
LaunchedEffect(voiceUiState.voiceMode) {
|
||||
com.hermesandroid.relay.wake.WakeWordForegroundService.setVoiceSessionActive(
|
||||
voiceUiState.voiceMode
|
||||
)
|
||||
}
|
||||
val postResumeQuiet by connectionViewModel.postResumeQuiet.collectAsState()
|
||||
val apiHealth by connectionViewModel.apiServerHealth.collectAsState()
|
||||
val activeConnection by connectionViewModel.activeConnection.collectAsState()
|
||||
|
||||
@@ -522,6 +522,7 @@ fun ChatScreen(
|
||||
micPermissionDenied = false
|
||||
if (pendingVoiceEnter && !isDemoMode) {
|
||||
pendingVoiceEnter = false
|
||||
com.hermesandroid.relay.wake.WakeWordForegroundService.prepareForVoice()
|
||||
voiceViewModel.enterVoiceMode()
|
||||
} else {
|
||||
pendingVoiceEnter = false
|
||||
@@ -538,6 +539,7 @@ fun ChatScreen(
|
||||
) == android.content.pm.PackageManager.PERMISSION_GRANTED
|
||||
if (granted) {
|
||||
micPermissionDenied = false
|
||||
com.hermesandroid.relay.wake.WakeWordForegroundService.prepareForVoice()
|
||||
voiceViewModel.enterVoiceMode()
|
||||
} else {
|
||||
pendingVoiceEnter = true
|
||||
|
||||
@@ -2,6 +2,11 @@
|
||||
|
||||
package com.hermesandroid.relay.ui.screens
|
||||
|
||||
import android.Manifest
|
||||
import android.content.pm.PackageManager
|
||||
import android.os.Build
|
||||
import androidx.activity.compose.rememberLauncherForActivityResult
|
||||
import androidx.activity.result.contract.ActivityResultContracts
|
||||
import androidx.compose.foundation.background
|
||||
import androidx.compose.foundation.BorderStroke
|
||||
import androidx.compose.foundation.clickable
|
||||
@@ -89,6 +94,7 @@ import androidx.compose.ui.text.style.TextOverflow
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.LayoutDirection
|
||||
import androidx.lifecycle.viewmodel.compose.viewModel
|
||||
import androidx.core.content.ContextCompat
|
||||
import com.hermesandroid.relay.R
|
||||
import com.hermesandroid.relay.data.AgentDisplay
|
||||
import com.hermesandroid.relay.data.BargeInPreferences
|
||||
@@ -135,6 +141,8 @@ import com.hermesandroid.relay.viewmodel.VoiceSettingsViewModel
|
||||
import com.hermesandroid.relay.viewmodel.VoiceViewModel
|
||||
import com.hermesandroid.relay.viewmodel.VoiceState
|
||||
import com.hermesandroid.relay.viewmodel.VoicePreviewUiState
|
||||
import com.hermesandroid.relay.wake.WakeWordPreferences
|
||||
import com.hermesandroid.relay.wake.WakeWordRuntimeState
|
||||
import kotlinx.coroutines.launch
|
||||
|
||||
internal enum class VoiceSettingsSection { Output, Listening, Advanced }
|
||||
@@ -201,6 +209,53 @@ fun VoiceSettingsScreen(
|
||||
|
||||
val bargeInPrefs by settingsViewModel.bargeInPrefs.collectAsState()
|
||||
val aecAvailable = settingsViewModel.aecAvailable
|
||||
val wakeWordPrefs by settingsViewModel.wakeWordPrefs.collectAsState()
|
||||
val wakeWordRuntimeState by settingsViewModel.wakeWordRuntimeState.collectAsState()
|
||||
val wakeWordInstallState by settingsViewModel.wakeWordInstallState.collectAsState()
|
||||
|
||||
var wakeWordPermissionError by remember { mutableStateOf<String?>(null) }
|
||||
val notificationPermissionLauncher = rememberLauncherForActivityResult(
|
||||
ActivityResultContracts.RequestPermission(),
|
||||
) { granted ->
|
||||
if (granted || Build.VERSION.SDK_INT < Build.VERSION_CODES.TIRAMISU) {
|
||||
wakeWordPermissionError = null
|
||||
settingsViewModel.setWakeWordEnabled(true)
|
||||
} else {
|
||||
wakeWordPermissionError =
|
||||
context.getString(R.string.wake_word_notification_permission_required)
|
||||
}
|
||||
}
|
||||
val microphonePermissionLauncher = rememberLauncherForActivityResult(
|
||||
ActivityResultContracts.RequestPermission(),
|
||||
) { granted ->
|
||||
if (!granted) {
|
||||
wakeWordPermissionError =
|
||||
context.getString(R.string.wake_word_microphone_permission_required)
|
||||
} else if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU &&
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.POST_NOTIFICATIONS) !=
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
) {
|
||||
notificationPermissionLauncher.launch(Manifest.permission.POST_NOTIFICATIONS)
|
||||
} else {
|
||||
wakeWordPermissionError = null
|
||||
settingsViewModel.setWakeWordEnabled(true)
|
||||
}
|
||||
}
|
||||
val requestWakeWordEnable: () -> Unit = {
|
||||
when {
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) !=
|
||||
PackageManager.PERMISSION_GRANTED ->
|
||||
microphonePermissionLauncher.launch(Manifest.permission.RECORD_AUDIO)
|
||||
Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU &&
|
||||
ContextCompat.checkSelfPermission(context, Manifest.permission.POST_NOTIFICATIONS) !=
|
||||
PackageManager.PERMISSION_GRANTED ->
|
||||
notificationPermissionLauncher.launch(Manifest.permission.POST_NOTIFICATIONS)
|
||||
else -> {
|
||||
wakeWordPermissionError = null
|
||||
settingsViewModel.setWakeWordEnabled(true)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Authoritative relay voice config now lives in the VM (WP-V3). The screen
|
||||
// just observes it; the editor cards push saves back through the VM.
|
||||
@@ -445,6 +500,19 @@ fun VoiceSettingsScreen(
|
||||
prefsRepo = prefsRepo,
|
||||
voiceViewModel = voiceViewModel,
|
||||
)
|
||||
WakeWordCard(
|
||||
preferences = wakeWordPrefs,
|
||||
runtimeState = wakeWordRuntimeState,
|
||||
installing = wakeWordInstallState.installing,
|
||||
error = wakeWordPermissionError ?: wakeWordInstallState.error,
|
||||
onEnable = requestWakeWordEnable,
|
||||
onDisable = { settingsViewModel.setWakeWordEnabled(false) },
|
||||
onSensitivityChange = settingsViewModel::setWakeWordSensitivity,
|
||||
onConfirmationFramesChange =
|
||||
settingsViewModel::setWakeWordConfirmationFrames,
|
||||
onStartNewSessionChange =
|
||||
settingsViewModel::setWakeWordStartNewSession,
|
||||
)
|
||||
BargeInCard(
|
||||
bargeInPrefs = bargeInPrefs,
|
||||
aecAvailable = aecAvailable,
|
||||
@@ -3300,6 +3368,188 @@ private fun GlobalVoiceControlsCard(
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Experimental local wake word.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@Composable
|
||||
private fun WakeWordCard(
|
||||
preferences: WakeWordPreferences,
|
||||
runtimeState: WakeWordRuntimeState,
|
||||
installing: Boolean,
|
||||
error: String?,
|
||||
onEnable: () -> Unit,
|
||||
onDisable: () -> Unit,
|
||||
onSensitivityChange: (Float) -> Unit,
|
||||
onConfirmationFramesChange: (Int) -> Unit,
|
||||
onStartNewSessionChange: (Boolean) -> Unit,
|
||||
) {
|
||||
val abiSupported = Build.SUPPORTED_ABIS.any {
|
||||
it == "arm64-v8a" || it == "armeabi-v7a" || it == "x86_64" || it == "x86"
|
||||
}
|
||||
var sensitivityDraft by remember(preferences.sensitivity) {
|
||||
mutableStateOf(preferences.sensitivity)
|
||||
}
|
||||
var confirmationFramesDraft by remember(preferences.confirmationFrames) {
|
||||
mutableStateOf(preferences.confirmationFrames)
|
||||
}
|
||||
SectionCard(
|
||||
title = stringResource(R.string.wake_word_title),
|
||||
badge = stringResource(R.string.voice_settings_experimental),
|
||||
) {
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.SpaceBetween,
|
||||
) {
|
||||
Column(modifier = Modifier.weight(1f)) {
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_enable),
|
||||
style = MaterialTheme.typography.bodyLarge,
|
||||
)
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_enable_desc),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
if (installing) {
|
||||
CircularProgressIndicator(modifier = Modifier.size(28.dp), strokeWidth = 3.dp)
|
||||
} else {
|
||||
Switch(
|
||||
checked = preferences.enabled,
|
||||
enabled = abiSupported,
|
||||
onCheckedChange = { enabled ->
|
||||
if (enabled) onEnable() else onDisable()
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (!abiSupported) {
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_unsupported_abi),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.error,
|
||||
)
|
||||
}
|
||||
if (installing) {
|
||||
Spacer(Modifier.height(8.dp))
|
||||
LinearProgressIndicator(modifier = Modifier.fillMaxWidth())
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_installing),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
error?.let {
|
||||
Spacer(Modifier.height(8.dp))
|
||||
Text(
|
||||
text = it,
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.error,
|
||||
)
|
||||
}
|
||||
|
||||
if (preferences.enabled) {
|
||||
HorizontalDivider(modifier = Modifier.padding(vertical = 8.dp))
|
||||
ProviderRow(
|
||||
label = stringResource(R.string.wake_word_phrase),
|
||||
value = preferences.phrase,
|
||||
)
|
||||
ProviderRow(
|
||||
label = stringResource(R.string.wake_word_status),
|
||||
value = when (runtimeState) {
|
||||
WakeWordRuntimeState.Stopped ->
|
||||
stringResource(R.string.wake_word_status_stopped)
|
||||
WakeWordRuntimeState.Starting ->
|
||||
stringResource(R.string.wake_word_status_starting)
|
||||
WakeWordRuntimeState.Listening ->
|
||||
stringResource(R.string.wake_word_status_listening)
|
||||
WakeWordRuntimeState.PausedForVoice ->
|
||||
stringResource(R.string.wake_word_status_paused)
|
||||
WakeWordRuntimeState.AwaitingUser ->
|
||||
stringResource(R.string.wake_word_status_detected)
|
||||
WakeWordRuntimeState.Error ->
|
||||
stringResource(R.string.wake_word_status_error)
|
||||
},
|
||||
)
|
||||
|
||||
Spacer(Modifier.height(8.dp))
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_sensitivity),
|
||||
style = MaterialTheme.typography.labelLarge,
|
||||
)
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_sensitivity_desc),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
Slider(
|
||||
value = sensitivityDraft,
|
||||
onValueChange = { sensitivityDraft = it },
|
||||
onValueChangeFinished = { onSensitivityChange(sensitivityDraft) },
|
||||
valueRange = 0.2f..0.9f,
|
||||
steps = 6,
|
||||
)
|
||||
|
||||
Text(
|
||||
text = stringResource(
|
||||
R.string.wake_word_confirmation_frames,
|
||||
confirmationFramesDraft,
|
||||
),
|
||||
style = MaterialTheme.typography.labelLarge,
|
||||
)
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_confirmation_frames_desc),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
Slider(
|
||||
value = confirmationFramesDraft.toFloat(),
|
||||
onValueChange = {
|
||||
confirmationFramesDraft = it.toInt().coerceIn(1, 5)
|
||||
},
|
||||
onValueChangeFinished = {
|
||||
onConfirmationFramesChange(confirmationFramesDraft)
|
||||
},
|
||||
valueRange = 1f..5f,
|
||||
steps = 3,
|
||||
)
|
||||
|
||||
HorizontalDivider(modifier = Modifier.padding(vertical = 8.dp))
|
||||
Row(
|
||||
modifier = Modifier.fillMaxWidth(),
|
||||
verticalAlignment = Alignment.CenterVertically,
|
||||
horizontalArrangement = Arrangement.SpaceBetween,
|
||||
) {
|
||||
Column(modifier = Modifier.weight(1f)) {
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_new_session),
|
||||
style = MaterialTheme.typography.bodyLarge,
|
||||
)
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_new_session_desc),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
Switch(
|
||||
checked = preferences.startNewSession,
|
||||
onCheckedChange = onStartNewSessionChange,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
HorizontalDivider(modifier = Modifier.padding(vertical = 8.dp))
|
||||
Text(
|
||||
text = stringResource(R.string.wake_word_privacy_battery),
|
||||
style = MaterialTheme.typography.bodySmall,
|
||||
color = MaterialTheme.colorScheme.onSurfaceVariant,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Barge-in.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -15,6 +15,11 @@ import com.hermesandroid.relay.network.relay.VoiceConfig
|
||||
import com.hermesandroid.relay.network.relay.VoiceOutputConfig
|
||||
import com.hermesandroid.relay.util.HumanError
|
||||
import com.hermesandroid.relay.util.classifyError
|
||||
import com.hermesandroid.relay.wake.WakeWordForegroundService
|
||||
import com.hermesandroid.relay.wake.WakeWordModelInstaller
|
||||
import com.hermesandroid.relay.wake.WakeWordPreferences
|
||||
import com.hermesandroid.relay.wake.WakeWordPreferencesRepository
|
||||
import com.hermesandroid.relay.wake.WakeWordRuntimeState
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.channels.BufferOverflow
|
||||
import kotlinx.coroutines.flow.MutableSharedFlow
|
||||
@@ -62,6 +67,11 @@ data class VoiceConfigUiState(
|
||||
val realtimeOptionsStatus: String? = null,
|
||||
)
|
||||
|
||||
data class WakeWordInstallUiState(
|
||||
val installing: Boolean = false,
|
||||
val error: String? = null,
|
||||
)
|
||||
|
||||
/**
|
||||
* View-model backing the Voice Settings screen.
|
||||
*
|
||||
@@ -73,6 +83,7 @@ data class VoiceConfigUiState(
|
||||
class VoiceSettingsViewModel(application: Application) : AndroidViewModel(application) {
|
||||
|
||||
private val bargeInRepo = BargeInPreferencesRepository(application)
|
||||
private val wakeWordRepo = WakeWordPreferencesRepository(application)
|
||||
|
||||
/** Current barge-in preferences — mirrors [BargeInPreferencesRepository.flow]. */
|
||||
val bargeInPrefs: StateFlow<BargeInPreferences> = bargeInRepo.flow.stateIn(
|
||||
@@ -81,6 +92,19 @@ class VoiceSettingsViewModel(application: Application) : AndroidViewModel(applic
|
||||
initialValue = BargeInPreferences(),
|
||||
)
|
||||
|
||||
val wakeWordPrefs: StateFlow<WakeWordPreferences> = wakeWordRepo.flow.stateIn(
|
||||
scope = viewModelScope,
|
||||
started = SharingStarted.WhileSubscribed(5_000),
|
||||
initialValue = WakeWordPreferences(),
|
||||
)
|
||||
|
||||
val wakeWordRuntimeState: StateFlow<WakeWordRuntimeState> =
|
||||
WakeWordForegroundService.runtimeState
|
||||
|
||||
private val _wakeWordInstallState = MutableStateFlow(WakeWordInstallUiState())
|
||||
val wakeWordInstallState: StateFlow<WakeWordInstallUiState> =
|
||||
_wakeWordInstallState.asStateFlow()
|
||||
|
||||
/**
|
||||
* One-shot probe of [AcousticEchoCanceler.isAvailable] captured at VM
|
||||
* construction. The value never changes at runtime on a given device, so
|
||||
@@ -123,6 +147,66 @@ class VoiceSettingsViewModel(application: Application) : AndroidViewModel(applic
|
||||
viewModelScope.launch { bargeInRepo.setResumeAfterInterruption(enabled) }
|
||||
}
|
||||
|
||||
fun setWakeWordEnabled(enabled: Boolean) {
|
||||
if (!enabled) {
|
||||
viewModelScope.launch { wakeWordRepo.setEnabled(false) }
|
||||
WakeWordForegroundService.stop(getApplication())
|
||||
_wakeWordInstallState.value = WakeWordInstallUiState()
|
||||
return
|
||||
}
|
||||
if (_wakeWordInstallState.value.installing) return
|
||||
_wakeWordInstallState.value = WakeWordInstallUiState(installing = true)
|
||||
viewModelScope.launch {
|
||||
val result = WakeWordModelInstaller(getApplication()).ensureInstalled()
|
||||
result.fold(
|
||||
onSuccess = {
|
||||
runCatching {
|
||||
wakeWordRepo.setEnabled(true)
|
||||
WakeWordForegroundService.start(getApplication())
|
||||
}.onSuccess {
|
||||
_wakeWordInstallState.value = WakeWordInstallUiState()
|
||||
}.onFailure { error ->
|
||||
wakeWordRepo.setEnabled(false)
|
||||
_wakeWordInstallState.value = WakeWordInstallUiState(
|
||||
error = error.message ?: "Could not start wake-word listening",
|
||||
)
|
||||
}
|
||||
},
|
||||
onFailure = { error ->
|
||||
wakeWordRepo.setEnabled(false)
|
||||
_wakeWordInstallState.value = WakeWordInstallUiState(
|
||||
error = error.message ?: "Could not install the wake-word model",
|
||||
)
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fun setWakeWordSensitivity(value: Float) {
|
||||
viewModelScope.launch {
|
||||
wakeWordRepo.setSensitivity(value)
|
||||
WakeWordForegroundService.reloadSettings()
|
||||
}
|
||||
}
|
||||
|
||||
fun setWakeWordConfirmationFrames(value: Int) {
|
||||
viewModelScope.launch {
|
||||
wakeWordRepo.setConfirmationFrames(value)
|
||||
WakeWordForegroundService.reloadSettings()
|
||||
}
|
||||
}
|
||||
|
||||
fun setWakeWordStartNewSession(enabled: Boolean) {
|
||||
viewModelScope.launch {
|
||||
wakeWordRepo.setStartNewSession(enabled)
|
||||
WakeWordForegroundService.reloadSettings()
|
||||
}
|
||||
}
|
||||
|
||||
fun clearWakeWordError() {
|
||||
_wakeWordInstallState.update { it.copy(error = null) }
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the relay voice config (global fallback TTS/STT, streaming voice
|
||||
* output, realtime agent) plus the advertised options for each config's
|
||||
|
||||
@@ -10,6 +10,7 @@ import androidx.lifecycle.viewModelScope
|
||||
import com.hermesandroid.relay.R
|
||||
import com.hermesandroid.relay.audio.BargeInListener
|
||||
import com.hermesandroid.relay.audio.RealtimePcmPlayer
|
||||
import com.hermesandroid.relay.audio.RmsBargeInGate
|
||||
import com.hermesandroid.relay.audio.VadEngine
|
||||
import com.hermesandroid.relay.audio.VoicePlayer
|
||||
import com.hermesandroid.relay.audio.VoiceRecorder
|
||||
@@ -910,15 +911,14 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
// B4 barge-in state (voice-barge-in 2026-04-17)
|
||||
// ---------------------------------------------------------------------
|
||||
//
|
||||
// BargeInListener is created lazily on the transition into Speaking when
|
||||
// the user has barge-in enabled + a non-Off sensitivity, then torn down
|
||||
// on Speaking-exit. The lifecycle is "per Speaking turn" — one listener
|
||||
// per response the agent gives, not one for the lifetime of voice mode.
|
||||
// BargeInListener is created once when a foreground response enters
|
||||
// Thinking and remains alive through Speaking and final audio drain.
|
||||
// A monotonically increasing epoch fences callbacks from a listener whose
|
||||
// AudioRecord teardown completed after the next turn began.
|
||||
//
|
||||
// The listener owns an AudioRecord under the hood; bracketing it around
|
||||
// Speaking keeps the mic permission footprint tight, avoids contesting
|
||||
// with the VoiceRecorder during Listening, and means "barge-in = off"
|
||||
// genuinely means no mic is ever opened during Speaking.
|
||||
// the active response keeps it out of Listening, avoids contesting with
|
||||
// VoiceRecorder, and means "barge-in = off" never opens a response mic.
|
||||
//
|
||||
// [bargeInPreferences] is initialized from [initialize] and mirrors the
|
||||
// datastore value via [viewModelScope]. Null before initialize — which
|
||||
@@ -943,6 +943,8 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
private var bargeInListener: BargeInListener? = null
|
||||
private var bargeInListenerJob: Job? = null
|
||||
private var bargeInVadEngine: VadEngine? = null
|
||||
private val bargeInTurnEpoch = AtomicLong(0L)
|
||||
@Volatile private var activeBargeInTurnEpoch: Long = 0L
|
||||
|
||||
/**
|
||||
* The ordered list of sentence chunks the synth worker has seen during
|
||||
@@ -2162,10 +2164,10 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
* upstream SSE stream kept generating and the observer kept pushing
|
||||
* fresh deltas, so playback resumed on the next sentence. We now
|
||||
* cancel the stream, tear down the observer + turn job, reset all
|
||||
* per-turn state, and go back to Idle (not Listening — Bailey's
|
||||
* mental model is "stop" = ready to start a new turn on mic tap).
|
||||
* per-turn state, and go back to Idle ("stop" means ready to start a
|
||||
* new turn on the next mic tap).
|
||||
*/
|
||||
fun interruptSpeaking() {
|
||||
fun interruptSpeaking(): Job? {
|
||||
Log.i(
|
||||
TAG,
|
||||
"Interrupting speech pipeline",
|
||||
@@ -2193,7 +2195,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
// double-trigger on the ducking watchdog or emit another
|
||||
// bargeInDetected while the resume watchdog is deliberating.
|
||||
// stopBargeInListener is null-safe.
|
||||
stopBargeInListener()
|
||||
val bargeInReaderRelease = stopBargeInListener()
|
||||
cancelStandardSpeechStream("speech interrupted")
|
||||
// 2026-04-18: the silence watchdog only runs during Listening, but
|
||||
// cancel defensively so a stale job from the prior turn can't
|
||||
@@ -2250,8 +2252,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
speakEnvelope = 0f
|
||||
// Deliberately do NOT clear responseText here. "Stop" should freeze
|
||||
// the visible response so the user can read whatever was already
|
||||
// said before they hit stop — Bailey hit this and pointed out the
|
||||
// old behavior felt like the screen evaporated under his hand. The
|
||||
// said before they hit stop. The
|
||||
// chat history was always preserved server-side; the bug was the
|
||||
// voice overlay's local copy getting blanked. The next [startListening]
|
||||
// resets responseText at the moment the user explicitly starts a
|
||||
@@ -2265,6 +2266,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
hermesConfirmation = null,
|
||||
)
|
||||
}
|
||||
return bargeInReaderRelease
|
||||
}
|
||||
|
||||
fun clearError() {
|
||||
@@ -2765,6 +2767,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
|
||||
return VoiceCommandContext(
|
||||
responseActive = responseActive,
|
||||
interruptedActiveResponse = responseWasInterrupted,
|
||||
backgroundTaskActive = backgroundTaskActive,
|
||||
backgroundAnswerAvailable = backgroundPhase == BackgroundRunPhase.DONE &&
|
||||
realtimeAgentControl != null,
|
||||
@@ -3184,6 +3187,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
val submittedUserUiKey =
|
||||
chatVm.sendVoiceMessage(userText, STABLE_VOICE_INTERFACE_CONTEXT)
|
||||
voiceTurnSessionFence?.bindSubmittedUser(submittedUserUiKey)
|
||||
beginBargeInTurnIfEnabled()
|
||||
startStreamObserver(chatVm)
|
||||
}
|
||||
|
||||
@@ -3260,6 +3264,9 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
responseText = "",
|
||||
)
|
||||
}
|
||||
beginBargeInTurnIfEnabled(
|
||||
audioSessionIdProvider = { realtimePcmPlayer?.audioSessionId ?: 0 },
|
||||
)
|
||||
}
|
||||
|
||||
// Per-turn event state is hoisted to fields so one session-lived callback
|
||||
@@ -4023,6 +4030,9 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
responseText = "",
|
||||
)
|
||||
}
|
||||
beginBargeInTurnIfEnabled(
|
||||
audioSessionIdProvider = { realtimePcmPlayer?.audioSessionId ?: 0 },
|
||||
)
|
||||
val deliveryResult = CompletableDeferred<Result<Unit>>()
|
||||
val queued = channel.trySend(
|
||||
RealtimeTurnInput(
|
||||
@@ -4927,13 +4937,20 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
TAG,
|
||||
"$source audio delta bytes=${audio.size} sampleRate=$sampleRate",
|
||||
)
|
||||
val firstAudioChunk = bargeInStarted.compareAndSet(false, true)
|
||||
if (firstAudioChunk) {
|
||||
if (bargeInListener == null) {
|
||||
startBargeInListenerIfEnabled(
|
||||
audioSessionIdProvider = { pcmPlayer.audioSessionId },
|
||||
)
|
||||
}
|
||||
// Freeze quiet-room calibration before the first PCM write can
|
||||
// reach AudioTrack and leak speaker output into the noise floor.
|
||||
markBargeInPlaybackStarted(REALTIME_BARGE_IN_STARTUP_GUARD_MS)
|
||||
}
|
||||
val level = pcmPlayer.write(audio, sampleRate)
|
||||
scheduleRealtimeAmplitudeRelease(audio.size, sampleRate, lastRealtimeAudioDeltaAtMs)
|
||||
if (bargeInStarted.compareAndSet(false, true)) {
|
||||
startBargeInListenerIfEnabled(
|
||||
audioSessionIdProvider = { pcmPlayer.audioSessionId },
|
||||
startupGuardMs = REALTIME_BARGE_IN_STARTUP_GUARD_MS,
|
||||
)
|
||||
if (firstAudioChunk) {
|
||||
startRealtimePlaybackWatchdog()
|
||||
}
|
||||
// Audio arriving IS the speech signal, whatever produced it: a
|
||||
@@ -5195,10 +5212,11 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
// "currently playing" from the moment play() returns.
|
||||
_currentPlayingChunkIndex.value = _currentPlayingChunkIndex.value + 1
|
||||
trackTtsFile(file)
|
||||
// B4: first chunk of a Speaking run → spin up the listener
|
||||
// if the user has it enabled. Idempotent — startBargeInListener
|
||||
// no-ops if already active.
|
||||
startBargeInListenerIfEnabled()
|
||||
// The turn listener normally started in Thinking. Keep this as
|
||||
// a safety net for standalone speech, then freeze quiet-room
|
||||
// calibration before speaker output reaches the microphone.
|
||||
if (bargeInListener == null) startBargeInListenerIfEnabled()
|
||||
markBargeInPlaybackStarted(RmsBargeInGate.DEFAULT_PLAYBACK_GRACE_MS)
|
||||
},
|
||||
pendingFiles = pendingTtsFiles,
|
||||
onQueueDrained = {
|
||||
@@ -5418,9 +5436,33 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
* listener's internal poll loop can watch it flip from 0 to non-zero
|
||||
* as playback begins.
|
||||
*/
|
||||
private fun beginBargeInTurnIfEnabled(
|
||||
audioSessionIdProvider: (() -> Int)? = null,
|
||||
) {
|
||||
val previousReader = stopBargeInListener()
|
||||
val epoch = bargeInTurnEpoch.incrementAndGet()
|
||||
activeBargeInTurnEpoch = epoch
|
||||
if (previousReader == null) {
|
||||
startBargeInListenerIfEnabled(
|
||||
audioSessionIdProvider = audioSessionIdProvider,
|
||||
epoch = epoch,
|
||||
)
|
||||
} else {
|
||||
viewModelScope.launch {
|
||||
previousReader.join()
|
||||
if (activeBargeInTurnEpoch == epoch) {
|
||||
startBargeInListenerIfEnabled(
|
||||
audioSessionIdProvider = audioSessionIdProvider,
|
||||
epoch = epoch,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun startBargeInListenerIfEnabled(
|
||||
audioSessionIdProvider: (() -> Int)? = null,
|
||||
startupGuardMs: Long = 0L,
|
||||
epoch: Long = bargeInTurnEpoch.incrementAndGet(),
|
||||
) {
|
||||
// Already running → no-op. We bracket per Speaking turn, not per
|
||||
// chunk within a turn.
|
||||
@@ -5473,16 +5515,14 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
}
|
||||
|
||||
bargeInListener = listener
|
||||
bargeInIgnoreUntilMs = if (startupGuardMs > 0L) {
|
||||
System.currentTimeMillis() + startupGuardMs
|
||||
} else {
|
||||
0L
|
||||
}
|
||||
activeBargeInTurnEpoch = epoch
|
||||
listener.setThresholdMultiplier(bargeInThresholdMultiplier(prefs.sensitivity))
|
||||
bargeInIgnoreUntilMs = 0L
|
||||
bargeInGuardLogged = false
|
||||
bargeInListenerJob = viewModelScope.launch {
|
||||
// Fan out the two event flows on child coroutines of this job.
|
||||
launch { listener.maybeSpeech.collect { onMaybeSpeech() } }
|
||||
launch { listener.bargeInDetected.collect { onBargeInDetected() } }
|
||||
launch { listener.maybeSpeech.collect { onMaybeSpeech(epoch) } }
|
||||
launch { listener.bargeInDetected.collect { onBargeInDetected(epoch) } }
|
||||
}
|
||||
Log.i(
|
||||
TAG,
|
||||
@@ -5503,12 +5543,13 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
* unduck the player (in case a ducking watchdog hadn't yet restored
|
||||
* volume), and release the owned VAD engine. Idempotent.
|
||||
*/
|
||||
private fun stopBargeInListener() {
|
||||
private fun stopBargeInListener(): Job? {
|
||||
if (bargeInListener != null) {
|
||||
Log.i(TAG, "Stopping barge-in listener")
|
||||
}
|
||||
bargeInIgnoreUntilMs = 0L
|
||||
bargeInGuardLogged = false
|
||||
activeBargeInTurnEpoch = 0L
|
||||
bargeInListenerJob?.cancel(); bargeInListenerJob = null
|
||||
val stoppedReaderJob = try { bargeInListener?.stop() } catch (_: Throwable) { null }
|
||||
bargeInListener = null
|
||||
@@ -5530,6 +5571,21 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
try { realtimePcmPlayer?.unduck() } catch (_: Throwable) { /* ignore */ }
|
||||
isDucked = false
|
||||
}
|
||||
return stoppedReaderJob
|
||||
}
|
||||
|
||||
private fun markBargeInPlaybackStarted(graceMs: Long) {
|
||||
val now = System.currentTimeMillis()
|
||||
bargeInIgnoreUntilMs = now + graceMs.coerceAtLeast(0L)
|
||||
bargeInGuardLogged = false
|
||||
bargeInListener?.markPlaybackStarted(nowMs = now, graceMs = graceMs)
|
||||
}
|
||||
|
||||
private fun bargeInThresholdMultiplier(sensitivity: BargeInSensitivity): Float = when (sensitivity) {
|
||||
BargeInSensitivity.Off -> 8f
|
||||
BargeInSensitivity.Low -> 4.5f
|
||||
BargeInSensitivity.Default -> 3f
|
||||
BargeInSensitivity.High -> 2f
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -5552,6 +5608,14 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
scheduleDuckingWatchdog()
|
||||
}
|
||||
|
||||
private fun onMaybeSpeech(epoch: Long) {
|
||||
if (epoch != activeBargeInTurnEpoch) {
|
||||
Log.i(TAG, "Ignoring stale barge-in maybe-speech epoch=$epoch active=$activeBargeInTurnEpoch")
|
||||
return
|
||||
}
|
||||
onMaybeSpeech()
|
||||
}
|
||||
|
||||
/**
|
||||
* Post-hysteresis VAD fire — the user is actually speaking. Capture
|
||||
* the current playing chunk for resume, call [interruptSpeaking] to
|
||||
@@ -5583,7 +5647,7 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
// bargeInDetected emissions are impossible until the next
|
||||
// Speaking entry. Belt-and-braces: we also cancel the subscriber
|
||||
// job as part of stopBargeInListener which interruptSpeaking calls.
|
||||
interruptSpeaking()
|
||||
val microphoneRelease = interruptSpeaking()
|
||||
|
||||
// interruptSpeaking landed us in Idle — flip to Listening and
|
||||
// pre-warm the recorder so the first ~100 ms of user speech
|
||||
@@ -5597,19 +5661,28 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
responseText = "",
|
||||
)
|
||||
}
|
||||
val rec = recorder
|
||||
if (rec != null && !rec.isRecording()) {
|
||||
viewModelScope.launch {
|
||||
try {
|
||||
rec.startRecording()
|
||||
microphoneRelease?.join()
|
||||
val rec = recorder
|
||||
if (rec != null && !rec.isRecording()) {
|
||||
rec.startRecording()
|
||||
}
|
||||
scheduleResumeWatchdog()
|
||||
} catch (t: Throwable) {
|
||||
responseInterruptedForVoiceCommand = false
|
||||
Log.w(TAG, "barge-in pre-warm recorder failed: ${t.message}")
|
||||
Log.w(TAG, "barge-in microphone handoff failed: ${t.message}")
|
||||
surfaceError(t, context = "record")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
scheduleResumeWatchdog()
|
||||
private fun onBargeInDetected(epoch: Long) {
|
||||
if (epoch != activeBargeInTurnEpoch) {
|
||||
Log.i(TAG, "Ignoring stale barge-in detection epoch=$epoch active=$activeBargeInTurnEpoch")
|
||||
return
|
||||
}
|
||||
onBargeInDetected()
|
||||
}
|
||||
|
||||
private fun isBargeInStartupGuardActive(): Boolean {
|
||||
@@ -5910,6 +5983,19 @@ class VoiceViewModel(application: Application) : AndroidViewModel(application) {
|
||||
startBargeInListenerIfEnabled()
|
||||
}
|
||||
|
||||
@androidx.annotation.VisibleForTesting
|
||||
internal fun beginBargeInTurnForTest() {
|
||||
_uiState.update { it.copy(state = VoiceState.Thinking) }
|
||||
beginBargeInTurnIfEnabled()
|
||||
}
|
||||
|
||||
@androidx.annotation.VisibleForTesting
|
||||
internal fun markBargeInPlaybackStartedForTest() {
|
||||
_uiState.update { it.copy(state = VoiceState.Speaking) }
|
||||
startBargeInListenerIfEnabled()
|
||||
markBargeInPlaybackStarted(RmsBargeInGate.DEFAULT_PLAYBACK_GRACE_MS)
|
||||
}
|
||||
|
||||
/**
|
||||
* Test hook: seed the Speaking state + spoken-chunk ledger so a
|
||||
* synthetic `onBargeInDetected()` can exercise the resume-watchdog
|
||||
|
||||
@@ -22,6 +22,7 @@ internal enum class VoiceCommandAction {
|
||||
/** State gates that keep an exact command phrase from becoming a global hotword. */
|
||||
internal data class VoiceCommandContext(
|
||||
val responseActive: Boolean = false,
|
||||
val interruptedActiveResponse: Boolean = false,
|
||||
val backgroundTaskActive: Boolean = false,
|
||||
val backgroundAnswerAvailable: Boolean = false,
|
||||
val continuousModeSelected: Boolean = false,
|
||||
@@ -40,6 +41,7 @@ internal data class VoiceCommandContext(
|
||||
*/
|
||||
internal object VoiceCommandInterpreter {
|
||||
private val stopResponsePhrases = setOf(
|
||||
"stop",
|
||||
"stop speaking",
|
||||
"stop talking",
|
||||
"stop the response",
|
||||
@@ -84,8 +86,13 @@ internal object VoiceCommandInterpreter {
|
||||
return when {
|
||||
context.backgroundTaskActive && phrase in cancelBackgroundTaskPhrases ->
|
||||
VoiceCommandAction.CancelBackgroundTask
|
||||
context.responseActive && phrase in stopResponsePhrases ->
|
||||
context.responseActive &&
|
||||
phrase in stopResponsePhrases &&
|
||||
(phrase != "stop" || context.interruptedActiveResponse) ->
|
||||
VoiceCommandAction.StopResponse
|
||||
context.continuousModeSelected &&
|
||||
context.interruptedActiveResponse &&
|
||||
phrase in pauseContinuousPhrases -> VoiceCommandAction.PauseContinuousListening
|
||||
context.continuousModeSelected &&
|
||||
context.continuousListeningActive &&
|
||||
phrase in pauseContinuousPhrases -> VoiceCommandAction.PauseContinuousListening
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import java.util.UUID
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
|
||||
enum class MicrophoneOwner {
|
||||
WakeWord,
|
||||
VoiceCapture,
|
||||
BargeIn,
|
||||
RealtimeDiagnostics,
|
||||
}
|
||||
|
||||
class MicrophoneLease internal constructor(
|
||||
val owner: MicrophoneOwner,
|
||||
internal val token: String,
|
||||
)
|
||||
|
||||
/**
|
||||
* Process-wide ownership seam for every Android AudioRecord path.
|
||||
*
|
||||
* Wake word, foreground capture, full-turn barge-in, and realtime diagnostics
|
||||
* all acquire this lease. The wake service also makes an explicit synchronous
|
||||
* handoff before entering the existing voice flow.
|
||||
*/
|
||||
object MicrophoneOwnershipCoordinator {
|
||||
private val lock = Any()
|
||||
private var activeLease: MicrophoneLease? = null
|
||||
private val _owner = MutableStateFlow<MicrophoneOwner?>(null)
|
||||
val owner: StateFlow<MicrophoneOwner?> = _owner.asStateFlow()
|
||||
|
||||
fun tryAcquire(owner: MicrophoneOwner): MicrophoneLease? = synchronized(lock) {
|
||||
if (activeLease != null) return null
|
||||
MicrophoneLease(owner, UUID.randomUUID().toString()).also {
|
||||
activeLease = it
|
||||
_owner.value = owner
|
||||
}
|
||||
}
|
||||
|
||||
fun release(lease: MicrophoneLease): Boolean = synchronized(lock) {
|
||||
if (activeLease?.token != lease.token) return false
|
||||
activeLease = null
|
||||
_owner.value = null
|
||||
true
|
||||
}
|
||||
|
||||
internal fun resetForTest() = synchronized(lock) {
|
||||
activeLease = null
|
||||
_owner.value = null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import java.util.UUID
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
|
||||
data class WakeWordActivation(
|
||||
val id: String = UUID.randomUUID().toString(),
|
||||
val startNewSession: Boolean,
|
||||
val profileRouting: WakeWordProfileRouting,
|
||||
)
|
||||
|
||||
/**
|
||||
* Durable-within-process handoff between the microphone service and Compose.
|
||||
* StateFlow (instead of an event-only SharedFlow) lets a background detection
|
||||
* remain pending until the user opens the actionable notification.
|
||||
*/
|
||||
object WakeWordActivationCoordinator {
|
||||
private val _pending = MutableStateFlow<WakeWordActivation?>(null)
|
||||
val pending: StateFlow<WakeWordActivation?> = _pending.asStateFlow()
|
||||
|
||||
fun request(activation: WakeWordActivation) {
|
||||
_pending.value = activation
|
||||
}
|
||||
|
||||
fun consume(id: String): Boolean {
|
||||
val current = _pending.value ?: return false
|
||||
if (current.id != id) return false
|
||||
_pending.value = null
|
||||
return true
|
||||
}
|
||||
|
||||
internal fun resetForTest() {
|
||||
_pending.value = null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import com.k2fsa.sherpa.onnx.FeatureConfig
|
||||
import com.k2fsa.sherpa.onnx.KeywordSpotter
|
||||
import com.k2fsa.sherpa.onnx.KeywordSpotterConfig
|
||||
import com.k2fsa.sherpa.onnx.OnlineModelConfig
|
||||
import com.k2fsa.sherpa.onnx.OnlineStream
|
||||
import com.k2fsa.sherpa.onnx.OnlineTransducerModelConfig
|
||||
|
||||
interface WakeWordDetector : AutoCloseable {
|
||||
/**
|
||||
* Accept a 16 kHz mono PCM16 frame. Returns true exactly once when the
|
||||
* configured confirmation count is met.
|
||||
*/
|
||||
fun accept(samples: ShortArray, count: Int): Boolean
|
||||
}
|
||||
|
||||
fun interface WakeWordDetectorFactory {
|
||||
fun create(
|
||||
files: WakeWordModelFiles,
|
||||
sensitivity: Float,
|
||||
confirmationFrames: Int,
|
||||
): WakeWordDetector
|
||||
}
|
||||
|
||||
internal class WakeWordConfirmationGate(private val requiredFrames: Int) {
|
||||
private var matchingFrames = 0
|
||||
private var fired = false
|
||||
|
||||
fun update(matches: Boolean): Boolean {
|
||||
if (fired) return false
|
||||
matchingFrames = if (matches) matchingFrames + 1 else 0
|
||||
if (matchingFrames < requiredFrames.coerceIn(1, 5)) return false
|
||||
fired = true
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
object WakeWordTuning {
|
||||
/** sherpa threshold is 0..1 and higher is harder to trigger. */
|
||||
fun threshold(sensitivity: Float): Float = sensitivity.coerceIn(0.2f, 0.9f)
|
||||
}
|
||||
|
||||
class SherpaWakeWordDetector(
|
||||
files: WakeWordModelFiles,
|
||||
sensitivity: Float,
|
||||
confirmationFrames: Int,
|
||||
) : WakeWordDetector {
|
||||
private val spotter = KeywordSpotter(
|
||||
config = KeywordSpotterConfig(
|
||||
featConfig = FeatureConfig(sampleRate = 16_000, featureDim = 80),
|
||||
modelConfig = OnlineModelConfig(
|
||||
transducer = OnlineTransducerModelConfig(
|
||||
encoder = files.encoder.absolutePath,
|
||||
decoder = files.decoder.absolutePath,
|
||||
joiner = files.joiner.absolutePath,
|
||||
),
|
||||
tokens = files.tokens.absolutePath,
|
||||
numThreads = 2,
|
||||
provider = "cpu",
|
||||
modelType = "zipformer2",
|
||||
),
|
||||
keywordsFile = files.keywords.absolutePath,
|
||||
keywordsScore = 1.5f,
|
||||
keywordsThreshold = WakeWordTuning.threshold(sensitivity),
|
||||
numTrailingBlanks = 2,
|
||||
),
|
||||
)
|
||||
private val stream: OnlineStream = spotter.createStream()
|
||||
private val confirmationGate = WakeWordConfirmationGate(confirmationFrames)
|
||||
private var closed = false
|
||||
|
||||
override fun accept(samples: ShortArray, count: Int): Boolean {
|
||||
if (closed || count <= 0) return false
|
||||
val normalized = FloatArray(count) { index -> samples[index] / 32768.0f }
|
||||
stream.acceptWaveform(normalized, sampleRate = 16_000)
|
||||
var detected = false
|
||||
while (spotter.isReady(stream)) {
|
||||
spotter.decode(stream)
|
||||
val matches = spotter.getResult(stream).keyword
|
||||
.replace('_', ' ')
|
||||
.trim()
|
||||
.equals(DEFAULT_WAKE_PHRASE, ignoreCase = true)
|
||||
if (confirmationGate.update(matches)) {
|
||||
detected = true
|
||||
break
|
||||
}
|
||||
}
|
||||
return detected
|
||||
}
|
||||
|
||||
override fun close() {
|
||||
if (closed) return
|
||||
closed = true
|
||||
stream.release()
|
||||
spotter.release()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import android.Manifest
|
||||
import android.annotation.SuppressLint
|
||||
import android.app.Notification
|
||||
import android.app.NotificationChannel
|
||||
import android.app.NotificationManager
|
||||
import android.app.PendingIntent
|
||||
import android.app.Service
|
||||
import android.content.Context
|
||||
import android.content.Intent
|
||||
import android.content.pm.PackageManager
|
||||
import android.content.pm.ServiceInfo
|
||||
import android.media.AudioFormat
|
||||
import android.media.AudioRecord
|
||||
import android.media.MediaRecorder
|
||||
import android.os.Build
|
||||
import android.os.IBinder
|
||||
import android.util.Log
|
||||
import androidx.core.app.NotificationCompat
|
||||
import androidx.core.content.ContextCompat
|
||||
import com.hermesandroid.relay.MainActivity
|
||||
import com.hermesandroid.relay.R
|
||||
import java.util.concurrent.atomic.AtomicBoolean
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
import kotlinx.coroutines.cancel
|
||||
import kotlinx.coroutines.flow.MutableStateFlow
|
||||
import kotlinx.coroutines.flow.StateFlow
|
||||
import kotlinx.coroutines.flow.asStateFlow
|
||||
import kotlinx.coroutines.flow.first
|
||||
import kotlinx.coroutines.launch
|
||||
|
||||
enum class WakeWordRuntimeState {
|
||||
Stopped,
|
||||
Starting,
|
||||
Listening,
|
||||
PausedForVoice,
|
||||
AwaitingUser,
|
||||
Error,
|
||||
}
|
||||
|
||||
/**
|
||||
* User-started microphone foreground service for experimental local wake word.
|
||||
*
|
||||
* The service never sends captured audio over a network. Network access is
|
||||
* used only by the explicit first-enable model install before this service is
|
||||
* started.
|
||||
*/
|
||||
class WakeWordForegroundService : Service() {
|
||||
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
|
||||
private val resourceLock = Any()
|
||||
private val stopRequested = AtomicBoolean(false)
|
||||
@Volatile private var recognitionJob: Job? = null
|
||||
private var recorder: AudioRecord? = null
|
||||
private var detector: WakeWordDetector? = null
|
||||
private var microphoneLease: MicrophoneLease? = null
|
||||
@Volatile private var voiceSessionActive = false
|
||||
@Volatile private var currentPreferences = WakeWordPreferences()
|
||||
|
||||
override fun onCreate() {
|
||||
super.onCreate()
|
||||
runningInstance = this
|
||||
}
|
||||
|
||||
override fun onBind(intent: Intent?): IBinder? = null
|
||||
|
||||
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
|
||||
// Satisfy the modern five-second watchdog before any action branch.
|
||||
startForegroundNotification(runtimeState.value)
|
||||
when (intent?.action) {
|
||||
ACTION_STOP -> {
|
||||
stopRecognition()
|
||||
setRuntimeState(WakeWordRuntimeState.Stopped)
|
||||
scope.launch {
|
||||
runCatching {
|
||||
WakeWordPreferencesRepository(applicationContext).setEnabled(false)
|
||||
}
|
||||
stopForeground(STOP_FOREGROUND_REMOVE)
|
||||
stopSelf()
|
||||
}
|
||||
}
|
||||
ACTION_START, null -> startFromPersistedSettings()
|
||||
}
|
||||
return START_NOT_STICKY
|
||||
}
|
||||
|
||||
override fun onDestroy() {
|
||||
stopRecognition()
|
||||
if (runningInstance === this) runningInstance = null
|
||||
_runtimeState.value = WakeWordRuntimeState.Stopped
|
||||
scope.cancel()
|
||||
super.onDestroy()
|
||||
}
|
||||
|
||||
private fun startFromPersistedSettings() {
|
||||
if (recognitionJob?.isActive == true || voiceSessionActive) return
|
||||
setRuntimeState(WakeWordRuntimeState.Starting)
|
||||
scope.launch {
|
||||
val prefs = WakeWordPreferencesRepository(applicationContext).flow.first()
|
||||
currentPreferences = prefs
|
||||
if (!prefs.enabled) {
|
||||
setRuntimeState(WakeWordRuntimeState.Stopped)
|
||||
stopForeground(STOP_FOREGROUND_REMOVE)
|
||||
stopSelf()
|
||||
return@launch
|
||||
}
|
||||
startRecognition(prefs)
|
||||
}
|
||||
}
|
||||
|
||||
@SuppressLint("MissingPermission")
|
||||
private fun startRecognition(preferences: WakeWordPreferences) {
|
||||
if (voiceSessionActive || recognitionJob?.isActive == true) return
|
||||
if (ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) !=
|
||||
PackageManager.PERMISSION_GRANTED
|
||||
) {
|
||||
fail("Microphone permission is required")
|
||||
return
|
||||
}
|
||||
val files = WakeWordModelInstaller(this).installedFiles()
|
||||
if (files == null) {
|
||||
fail("Wake-word model is not installed")
|
||||
return
|
||||
}
|
||||
val lease = MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.WakeWord)
|
||||
if (lease == null) {
|
||||
setRuntimeState(WakeWordRuntimeState.PausedForVoice)
|
||||
return
|
||||
}
|
||||
microphoneLease = lease
|
||||
stopRequested.set(false)
|
||||
recognitionJob = scope.launch {
|
||||
var detected = false
|
||||
var unattachedDetector: WakeWordDetector? = null
|
||||
try {
|
||||
val createdDetector = SherpaWakeWordDetector(
|
||||
files = files,
|
||||
sensitivity = preferences.sensitivity,
|
||||
confirmationFrames = preferences.confirmationFrames,
|
||||
)
|
||||
unattachedDetector = createdDetector
|
||||
val minBuffer = AudioRecord.getMinBufferSize(
|
||||
SAMPLE_RATE,
|
||||
AudioFormat.CHANNEL_IN_MONO,
|
||||
AudioFormat.ENCODING_PCM_16BIT,
|
||||
).coerceAtLeast(SAMPLE_RATE / 5 * 2)
|
||||
val createdRecorder = AudioRecord.Builder()
|
||||
.setAudioSource(MediaRecorder.AudioSource.VOICE_RECOGNITION)
|
||||
.setAudioFormat(
|
||||
AudioFormat.Builder()
|
||||
.setSampleRate(SAMPLE_RATE)
|
||||
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
|
||||
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
|
||||
.build()
|
||||
)
|
||||
.setBufferSizeInBytes(minBuffer * 2)
|
||||
.build()
|
||||
if (createdRecorder.state != AudioRecord.STATE_INITIALIZED) {
|
||||
createdRecorder.release()
|
||||
throw IllegalStateException("Wake-word microphone failed to initialize")
|
||||
}
|
||||
synchronized(resourceLock) {
|
||||
if (stopRequested.get()) {
|
||||
createdRecorder.release()
|
||||
return@launch
|
||||
}
|
||||
detector = createdDetector
|
||||
recorder = createdRecorder
|
||||
unattachedDetector = null
|
||||
}
|
||||
createdRecorder.startRecording()
|
||||
setRuntimeState(WakeWordRuntimeState.Listening)
|
||||
|
||||
val samples = ShortArray(FRAME_SAMPLES)
|
||||
while (!stopRequested.get()) {
|
||||
val count = createdRecorder.read(samples, 0, samples.size)
|
||||
if (count < 0) throw IllegalStateException("Wake-word microphone read failed: $count")
|
||||
if (count > 0 && createdDetector.accept(samples, count)) {
|
||||
detected = true
|
||||
break
|
||||
}
|
||||
}
|
||||
} catch (t: Throwable) {
|
||||
if (!stopRequested.get()) {
|
||||
Log.w(TAG, "wake listening failed", t)
|
||||
fail(t.message ?: "Wake-word listener failed")
|
||||
}
|
||||
} finally {
|
||||
runCatching { unattachedDetector?.close() }
|
||||
releaseRecognitionResources()
|
||||
recognitionJob = null
|
||||
}
|
||||
if (detected && !stopRequested.get()) {
|
||||
onWakeDetected(preferences)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Synchronous mic handoff: stop and release AudioRecord before any caller
|
||||
* enters the existing voice capture path.
|
||||
*/
|
||||
private fun pauseForVoice() {
|
||||
voiceSessionActive = true
|
||||
stopRecognition()
|
||||
setRuntimeState(WakeWordRuntimeState.PausedForVoice)
|
||||
}
|
||||
|
||||
private fun setVoiceSessionActive(active: Boolean) {
|
||||
voiceSessionActive = active
|
||||
if (active) {
|
||||
pauseForVoice()
|
||||
} else {
|
||||
val previousJob = recognitionJob
|
||||
scope.launch {
|
||||
// The cancelled reader's finally block owns detector teardown.
|
||||
// Waiting prevents it from closing a newly created detector or
|
||||
// releasing the new listener's microphone lease as stale work.
|
||||
previousJob?.join()
|
||||
if (!voiceSessionActive &&
|
||||
currentPreferences.enabled &&
|
||||
WakeWordActivationCoordinator.pending.value == null
|
||||
) {
|
||||
startRecognition(currentPreferences)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun reloadSettings() {
|
||||
val previousJob = recognitionJob
|
||||
stopRecognition()
|
||||
scope.launch {
|
||||
// The previous job owns the JNI detector teardown. Do not create a
|
||||
// replacement until that teardown has completed.
|
||||
previousJob?.join()
|
||||
currentPreferences = WakeWordPreferencesRepository(applicationContext).flow.first()
|
||||
if (!voiceSessionActive && currentPreferences.enabled) {
|
||||
startRecognition(currentPreferences)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private fun stopRecognition() {
|
||||
stopRequested.set(true)
|
||||
synchronized(resourceLock) {
|
||||
runCatching { recorder?.stop() }
|
||||
runCatching { recorder?.release() }
|
||||
recorder = null
|
||||
microphoneLease?.let(MicrophoneOwnershipCoordinator::release)
|
||||
microphoneLease = null
|
||||
}
|
||||
// Closing the native detector concurrently with accept() can race in
|
||||
// JNI. Stopping AudioRecord unblocks the reader; its finally block owns
|
||||
// detector close after accept() has returned.
|
||||
recognitionJob?.cancel()
|
||||
}
|
||||
|
||||
private fun releaseRecognitionResources() {
|
||||
synchronized(resourceLock) {
|
||||
runCatching { recorder?.stop() }
|
||||
runCatching { recorder?.release() }
|
||||
recorder = null
|
||||
runCatching { detector?.close() }
|
||||
detector = null
|
||||
microphoneLease?.let(MicrophoneOwnershipCoordinator::release)
|
||||
microphoneLease = null
|
||||
}
|
||||
}
|
||||
|
||||
private fun onWakeDetected(preferences: WakeWordPreferences) {
|
||||
// releaseRecognitionResources() has completed before this callback.
|
||||
WakeWordActivationCoordinator.request(
|
||||
WakeWordActivation(
|
||||
startNewSession = preferences.startNewSession,
|
||||
profileRouting = preferences.profileRouting,
|
||||
)
|
||||
)
|
||||
setRuntimeState(WakeWordRuntimeState.AwaitingUser)
|
||||
}
|
||||
|
||||
private fun fail(message: String) {
|
||||
Log.w(TAG, message)
|
||||
setRuntimeState(WakeWordRuntimeState.Error)
|
||||
}
|
||||
|
||||
private fun setRuntimeState(state: WakeWordRuntimeState) {
|
||||
_runtimeState.value = state
|
||||
startForegroundNotification(state)
|
||||
}
|
||||
|
||||
@SuppressLint("ForegroundServiceType")
|
||||
private fun startForegroundNotification(state: WakeWordRuntimeState) {
|
||||
ensureChannel()
|
||||
val notification = buildNotification(state)
|
||||
try {
|
||||
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
|
||||
startForeground(
|
||||
NOTIFICATION_ID,
|
||||
notification,
|
||||
ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE,
|
||||
)
|
||||
} else {
|
||||
startForeground(NOTIFICATION_ID, notification)
|
||||
}
|
||||
} catch (t: Throwable) {
|
||||
Log.w(TAG, "Could not foreground wake-word microphone service", t)
|
||||
stopSelf()
|
||||
}
|
||||
}
|
||||
|
||||
private fun buildNotification(state: WakeWordRuntimeState): Notification {
|
||||
val launchIntent = Intent(this, MainActivity::class.java).apply {
|
||||
flags = Intent.FLAG_ACTIVITY_CLEAR_TOP or Intent.FLAG_ACTIVITY_SINGLE_TOP
|
||||
}
|
||||
val immutableUpdate = PendingIntent.FLAG_UPDATE_CURRENT or PendingIntent.FLAG_IMMUTABLE
|
||||
val launchPending = PendingIntent.getActivity(this, 0, launchIntent, immutableUpdate)
|
||||
val stopPending = PendingIntent.getService(
|
||||
this,
|
||||
1,
|
||||
Intent(this, WakeWordForegroundService::class.java).setAction(ACTION_STOP),
|
||||
immutableUpdate,
|
||||
)
|
||||
val text = when (state) {
|
||||
WakeWordRuntimeState.Starting -> getString(R.string.wake_word_notification_starting)
|
||||
WakeWordRuntimeState.Listening -> getString(R.string.wake_word_notification_listening)
|
||||
WakeWordRuntimeState.PausedForVoice ->
|
||||
getString(R.string.wake_word_notification_paused)
|
||||
WakeWordRuntimeState.AwaitingUser ->
|
||||
getString(R.string.wake_word_notification_detected)
|
||||
WakeWordRuntimeState.Error -> getString(R.string.wake_word_notification_error)
|
||||
WakeWordRuntimeState.Stopped -> getString(R.string.wake_word_notification_stopped)
|
||||
}
|
||||
return NotificationCompat.Builder(this, CHANNEL_ID)
|
||||
.setSmallIcon(R.mipmap.ic_launcher)
|
||||
.setContentTitle(getString(R.string.wake_word_notification_title))
|
||||
.setContentText(text)
|
||||
.setContentIntent(launchPending)
|
||||
.setOngoing(true)
|
||||
.setOnlyAlertOnce(state != WakeWordRuntimeState.AwaitingUser)
|
||||
.setPriority(
|
||||
if (state == WakeWordRuntimeState.AwaitingUser) {
|
||||
NotificationCompat.PRIORITY_HIGH
|
||||
} else {
|
||||
NotificationCompat.PRIORITY_LOW
|
||||
}
|
||||
)
|
||||
.setCategory(NotificationCompat.CATEGORY_SERVICE)
|
||||
.addAction(0, getString(R.string.wake_word_notification_stop), stopPending)
|
||||
.apply {
|
||||
if (state == WakeWordRuntimeState.AwaitingUser) {
|
||||
addAction(0, getString(R.string.wake_word_notification_open), launchPending)
|
||||
}
|
||||
}
|
||||
.build()
|
||||
}
|
||||
|
||||
private fun ensureChannel() {
|
||||
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.O) return
|
||||
val manager = getSystemService(NotificationManager::class.java) ?: return
|
||||
if (manager.getNotificationChannel(CHANNEL_ID) != null) return
|
||||
manager.createNotificationChannel(
|
||||
NotificationChannel(
|
||||
CHANNEL_ID,
|
||||
getString(R.string.wake_word_notification_channel),
|
||||
NotificationManager.IMPORTANCE_LOW,
|
||||
).apply {
|
||||
description = getString(R.string.wake_word_notification_channel_desc)
|
||||
setShowBadge(false)
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
companion object {
|
||||
private const val TAG = "WakeWordService"
|
||||
const val CHANNEL_ID = "wake_word_microphone"
|
||||
const val NOTIFICATION_ID = 4714
|
||||
const val ACTION_START = "com.hermesandroid.relay.wake.START"
|
||||
const val ACTION_STOP = "com.hermesandroid.relay.wake.STOP"
|
||||
private const val SAMPLE_RATE = 16_000
|
||||
private const val FRAME_SAMPLES = 1_600
|
||||
|
||||
private val _runtimeState = MutableStateFlow(WakeWordRuntimeState.Stopped)
|
||||
val runtimeState: StateFlow<WakeWordRuntimeState> = _runtimeState.asStateFlow()
|
||||
|
||||
@Volatile
|
||||
private var runningInstance: WakeWordForegroundService? = null
|
||||
|
||||
fun start(context: Context) {
|
||||
val appContext = context.applicationContext
|
||||
val intent = Intent(appContext, WakeWordForegroundService::class.java)
|
||||
.setAction(ACTION_START)
|
||||
ContextCompat.startForegroundService(appContext, intent)
|
||||
}
|
||||
|
||||
fun stop(context: Context) {
|
||||
context.applicationContext.stopService(
|
||||
Intent(context.applicationContext, WakeWordForegroundService::class.java)
|
||||
)
|
||||
_runtimeState.value = WakeWordRuntimeState.Stopped
|
||||
}
|
||||
|
||||
fun prepareForVoice() {
|
||||
runningInstance?.pauseForVoice()
|
||||
}
|
||||
|
||||
fun setVoiceSessionActive(active: Boolean) {
|
||||
runningInstance?.setVoiceSessionActive(active)
|
||||
}
|
||||
|
||||
fun reloadSettings() {
|
||||
runningInstance?.reloadSettings()
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import android.content.Context
|
||||
import java.io.File
|
||||
import java.io.IOException
|
||||
import java.security.MessageDigest
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.Request
|
||||
|
||||
data class WakeWordModelFiles(
|
||||
val directory: File,
|
||||
val encoder: File,
|
||||
val decoder: File,
|
||||
val joiner: File,
|
||||
val tokens: File,
|
||||
val keywords: File,
|
||||
)
|
||||
|
||||
/**
|
||||
* Installs only the four runtime files needed by the fixed English phrase.
|
||||
*
|
||||
* Files come from the model repository linked by sherpa-onnx's official KWS
|
||||
* documentation. Every download is pinned by byte length and SHA-256 before
|
||||
* the install directory is promoted atomically.
|
||||
*/
|
||||
class WakeWordModelInstaller(
|
||||
context: Context,
|
||||
private val client: OkHttpClient = OkHttpClient(),
|
||||
) {
|
||||
private val appContext = context.applicationContext
|
||||
private val modelRoot = File(appContext.filesDir, MODEL_DIRECTORY)
|
||||
|
||||
suspend fun ensureInstalled(): Result<WakeWordModelFiles> = withContext(Dispatchers.IO) {
|
||||
runCatching {
|
||||
installedFiles()?.let { return@runCatching it }
|
||||
|
||||
val staging = File(appContext.filesDir, "$MODEL_DIRECTORY.installing")
|
||||
staging.deleteRecursively()
|
||||
check(staging.mkdirs()) { "Could not create wake-word model directory" }
|
||||
|
||||
try {
|
||||
MODEL_ARTIFACTS.forEach { artifact ->
|
||||
downloadVerified(artifact, File(staging, artifact.fileName))
|
||||
}
|
||||
File(staging, KEYWORDS_FILE).writeText(KEYWORDS_CONTENT, Charsets.UTF_8)
|
||||
File(staging, INSTALL_MARKER).writeText(MODEL_VERSION, Charsets.UTF_8)
|
||||
|
||||
modelRoot.deleteRecursively()
|
||||
if (!staging.renameTo(modelRoot)) {
|
||||
throw IOException("Could not activate downloaded wake-word model")
|
||||
}
|
||||
installedFiles()
|
||||
?: throw IOException("Wake-word model failed post-install verification")
|
||||
} catch (t: Throwable) {
|
||||
staging.deleteRecursively()
|
||||
throw t
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fun installedFiles(): WakeWordModelFiles? {
|
||||
if (File(modelRoot, INSTALL_MARKER).takeIf { it.isFile }?.readText()?.trim() != MODEL_VERSION) {
|
||||
return null
|
||||
}
|
||||
val byName = MODEL_ARTIFACTS.associate { artifact ->
|
||||
val file = File(modelRoot, artifact.fileName)
|
||||
if (!file.isFile || file.length() != artifact.byteLength) return null
|
||||
artifact.fileName to file
|
||||
}
|
||||
val keywords = File(modelRoot, KEYWORDS_FILE)
|
||||
if (!keywords.isFile || keywords.readText(Charsets.UTF_8) != KEYWORDS_CONTENT) return null
|
||||
return WakeWordModelFiles(
|
||||
directory = modelRoot,
|
||||
encoder = byName.getValue(ENCODER_FILE),
|
||||
decoder = byName.getValue(DECODER_FILE),
|
||||
joiner = byName.getValue(JOINER_FILE),
|
||||
tokens = byName.getValue(TOKENS_FILE),
|
||||
keywords = keywords,
|
||||
)
|
||||
}
|
||||
|
||||
private fun downloadVerified(artifact: ModelArtifact, destination: File) {
|
||||
val request = Request.Builder().url("$MODEL_BASE_URL/${artifact.fileName}").get().build()
|
||||
client.newCall(request).execute().use { response ->
|
||||
if (!response.isSuccessful) {
|
||||
throw IOException("Wake-word model download failed (${response.code})")
|
||||
}
|
||||
val body = response.body
|
||||
val digest = MessageDigest.getInstance("SHA-256")
|
||||
var written = 0L
|
||||
destination.outputStream().buffered().use { output ->
|
||||
body.byteStream().use { input ->
|
||||
val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
|
||||
while (true) {
|
||||
val count = input.read(buffer)
|
||||
if (count < 0) break
|
||||
if (count == 0) continue
|
||||
output.write(buffer, 0, count)
|
||||
digest.update(buffer, 0, count)
|
||||
written += count
|
||||
if (written > artifact.byteLength) {
|
||||
throw IOException("Wake-word model download exceeded expected size")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
val actualHash = digest.digest().joinToString("") { "%02x".format(it) }
|
||||
if (written != artifact.byteLength ||
|
||||
!actualHash.equals(artifact.sha256, ignoreCase = true)
|
||||
) {
|
||||
destination.delete()
|
||||
throw IOException("Wake-word model integrity check failed for ${artifact.fileName}")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private data class ModelArtifact(
|
||||
val fileName: String,
|
||||
val byteLength: Long,
|
||||
val sha256: String,
|
||||
)
|
||||
|
||||
companion object {
|
||||
const val MODEL_VERSION = "gigaspeech-3.3m-2024-01-01-int8-v1"
|
||||
private const val MODEL_DIRECTORY = "wake-word/$MODEL_VERSION"
|
||||
private const val INSTALL_MARKER = "installed.version"
|
||||
private const val KEYWORDS_FILE = "keywords.txt"
|
||||
private const val MODEL_BASE_URL =
|
||||
"https://www.modelscope.cn/models/pkufool/" +
|
||||
"sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01/resolve/master"
|
||||
|
||||
const val ENCODER_FILE = "encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx"
|
||||
const val DECODER_FILE = "decoder-epoch-12-avg-2-chunk-16-left-64.onnx"
|
||||
const val JOINER_FILE = "joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx"
|
||||
const val TOKENS_FILE = "tokens.txt"
|
||||
|
||||
// Generated with this model's published SentencePiece vocabulary.
|
||||
// No arbitrary-phrase tokenizer is bundled in the first release.
|
||||
const val KEYWORDS_CONTENT = "▁HE Y ▁HER ME S @HEY_HERMES\n"
|
||||
|
||||
private val MODEL_ARTIFACTS = listOf(
|
||||
ModelArtifact(
|
||||
ENCODER_FILE,
|
||||
4_807_159,
|
||||
"1e721676515bcd42a186979733981213c66c80db680e1cc582dfedf3be76e678",
|
||||
),
|
||||
ModelArtifact(
|
||||
DECODER_FILE,
|
||||
1_063_189,
|
||||
"f61ebd3eed3773a44d088d53dfae92dbb6aec4839f4dcaee2d402414741663a3",
|
||||
),
|
||||
ModelArtifact(
|
||||
JOINER_FILE,
|
||||
163_380,
|
||||
"eae9da0c7e1e6c6a3f4cc42d167899c388f6c6701b94cb96320e4f55df79624c",
|
||||
),
|
||||
ModelArtifact(
|
||||
TOKENS_FILE,
|
||||
5_006,
|
||||
"fd2ded4050a55d2b1578870ba8697d02371980217806b7558bd0a5cc60f3ba53",
|
||||
),
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import android.content.Context
|
||||
import androidx.datastore.core.DataStore
|
||||
import androidx.datastore.preferences.core.Preferences
|
||||
import androidx.datastore.preferences.core.booleanPreferencesKey
|
||||
import androidx.datastore.preferences.core.edit
|
||||
import androidx.datastore.preferences.core.floatPreferencesKey
|
||||
import androidx.datastore.preferences.core.intPreferencesKey
|
||||
import androidx.datastore.preferences.core.stringPreferencesKey
|
||||
import com.hermesandroid.relay.data.relayDataStore
|
||||
import kotlinx.coroutines.flow.Flow
|
||||
import kotlinx.coroutines.flow.distinctUntilChanged
|
||||
import kotlinx.coroutines.flow.map
|
||||
|
||||
const val DEFAULT_WAKE_PHRASE = "Hey Hermes"
|
||||
|
||||
enum class WakeWordProfileRouteMode(val storageValue: String) {
|
||||
Active("active"),
|
||||
Specific("specific");
|
||||
|
||||
companion object {
|
||||
fun fromStorage(value: String?): WakeWordProfileRouteMode =
|
||||
entries.firstOrNull { it.storageValue == value } ?: Active
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Future-safe routing shape. The first release deliberately supports only
|
||||
* [WakeWordProfileRouteMode.Active], so wake activation preserves the profile,
|
||||
* provider, and model already selected in the app.
|
||||
*/
|
||||
data class WakeWordProfileRouting(
|
||||
val mode: WakeWordProfileRouteMode = WakeWordProfileRouteMode.Active,
|
||||
val profileName: String? = null,
|
||||
)
|
||||
|
||||
data class WakeWordPreferences(
|
||||
val enabled: Boolean = false,
|
||||
val phrase: String = DEFAULT_WAKE_PHRASE,
|
||||
/** Higher is stricter (fewer false activations), matching upstream. */
|
||||
val sensitivity: Float = 0.6f,
|
||||
val confirmationFrames: Int = 3,
|
||||
val startNewSession: Boolean = true,
|
||||
val profileRouting: WakeWordProfileRouting = WakeWordProfileRouting(),
|
||||
)
|
||||
|
||||
class WakeWordPreferencesRepository(
|
||||
private val dataStore: DataStore<Preferences>,
|
||||
) {
|
||||
constructor(context: Context) : this(context.relayDataStore)
|
||||
|
||||
private companion object {
|
||||
val KEY_ENABLED = booleanPreferencesKey("wake_word_enabled")
|
||||
val KEY_PHRASE = stringPreferencesKey("wake_word_phrase")
|
||||
val KEY_SENSITIVITY = floatPreferencesKey("wake_word_sensitivity")
|
||||
val KEY_CONFIRMATION_FRAMES = intPreferencesKey("wake_word_confirmation_frames")
|
||||
val KEY_START_NEW_SESSION = booleanPreferencesKey("wake_word_start_new_session")
|
||||
val KEY_PROFILE_ROUTE_MODE = stringPreferencesKey("wake_word_profile_route_mode")
|
||||
val KEY_PROFILE_NAME = stringPreferencesKey("wake_word_profile_name")
|
||||
}
|
||||
|
||||
val flow: Flow<WakeWordPreferences> = dataStore.data
|
||||
.map { prefs ->
|
||||
val routeMode = WakeWordProfileRouteMode.fromStorage(prefs[KEY_PROFILE_ROUTE_MODE])
|
||||
WakeWordPreferences(
|
||||
enabled = prefs[KEY_ENABLED] ?: false,
|
||||
// Only one phrase has been validated. Ignore stale/future values
|
||||
// until the product actually exposes multi-phrase support.
|
||||
phrase = DEFAULT_WAKE_PHRASE,
|
||||
sensitivity = (prefs[KEY_SENSITIVITY] ?: 0.6f).coerceIn(0.2f, 0.9f),
|
||||
confirmationFrames = (prefs[KEY_CONFIRMATION_FRAMES] ?: 3).coerceIn(1, 5),
|
||||
startNewSession = prefs[KEY_START_NEW_SESSION] ?: true,
|
||||
profileRouting = WakeWordProfileRouting(
|
||||
mode = routeMode,
|
||||
profileName = prefs[KEY_PROFILE_NAME]
|
||||
?.trim()
|
||||
?.takeIf { it.isNotEmpty() },
|
||||
),
|
||||
)
|
||||
}
|
||||
.distinctUntilChanged()
|
||||
|
||||
suspend fun setEnabled(enabled: Boolean) {
|
||||
dataStore.edit {
|
||||
it[KEY_ENABLED] = enabled
|
||||
if (enabled) it[KEY_PHRASE] = DEFAULT_WAKE_PHRASE
|
||||
}
|
||||
}
|
||||
|
||||
suspend fun setSensitivity(sensitivity: Float) {
|
||||
dataStore.edit { it[KEY_SENSITIVITY] = sensitivity.coerceIn(0.2f, 0.9f) }
|
||||
}
|
||||
|
||||
suspend fun setConfirmationFrames(frames: Int) {
|
||||
dataStore.edit { it[KEY_CONFIRMATION_FRAMES] = frames.coerceIn(1, 5) }
|
||||
}
|
||||
|
||||
suspend fun setStartNewSession(enabled: Boolean) {
|
||||
dataStore.edit { it[KEY_START_NEW_SESSION] = enabled }
|
||||
}
|
||||
|
||||
/**
|
||||
* Stored now so a future profile-specific phrase UI can migrate without a
|
||||
* schema rewrite. Product code intentionally writes Active in this release.
|
||||
*/
|
||||
suspend fun setProfileRouting(routing: WakeWordProfileRouting) {
|
||||
dataStore.edit { prefs ->
|
||||
prefs[KEY_PROFILE_ROUTE_MODE] = routing.mode.storageValue
|
||||
val profileName = routing.profileName?.trim()?.takeIf { it.isNotEmpty() }
|
||||
if (profileName == null) {
|
||||
prefs.remove(KEY_PROFILE_NAME)
|
||||
} else {
|
||||
prefs[KEY_PROFILE_NAME] = profileName
|
||||
}
|
||||
prefs[KEY_PHRASE] = DEFAULT_WAKE_PHRASE
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3251,4 +3251,39 @@
|
||||
<string name="conn_info_profile_api_key_saved">Chave de API do perfil salva</string>
|
||||
<string name="conn_info_profile_api_key_cleared">Chave de API do perfil removida</string>
|
||||
<string name="conn_info_profile_api_key_save_failed">Não foi possível salvar a chave de API do perfil</string>
|
||||
|
||||
<!-- Palavra de ativação experimental no dispositivo -->
|
||||
<string name="wake_word_title">Palavra de ativação</string>
|
||||
<string name="wake_word_enable">Detectar “Hey Hermes”</string>
|
||||
<string name="wake_word_enable_desc">Detecção experimental e opcional que funciona inteiramente neste telefone.</string>
|
||||
<string name="wake_word_phrase">Frase</string>
|
||||
<string name="wake_word_status">Status</string>
|
||||
<string name="wake_word_status_stopped">Parado</string>
|
||||
<string name="wake_word_status_starting">Iniciando…</string>
|
||||
<string name="wake_word_status_listening">Escutando localmente</string>
|
||||
<string name="wake_word_status_paused">Pausado enquanto a voz está ativa</string>
|
||||
<string name="wake_word_status_detected">Detectado — abra a notificação</string>
|
||||
<string name="wake_word_status_error">Requer atenção</string>
|
||||
<string name="wake_word_installing">Baixando e verificando o modelo em inglês no dispositivo (cerca de 6 MB)…</string>
|
||||
<string name="wake_word_unsupported_abi">A palavra de ativação requer um dispositivo arm64-v8a, armeabi-v7a, x86_64 ou x86.</string>
|
||||
<string name="wake_word_sensitivity">Rigor</string>
|
||||
<string name="wake_word_sensitivity_desc">Valores mais altos são mais rigorosos e reduzem ativações falsas, mas podem não detectar fala mais baixa.</string>
|
||||
<string name="wake_word_confirmation_frames">Quadros de confirmação: %1$d</string>
|
||||
<string name="wake_word_confirmation_frames_desc">Exige a frase em mais quadros do decodificador antes da ativação.</string>
|
||||
<string name="wake_word_new_session">Iniciar nova sessão</string>
|
||||
<string name="wake_word_new_session_desc">Cria uma conversa antes de abrir a voz. Desative para continuar a sessão atual do perfil selecionado.</string>
|
||||
<string name="wake_word_privacy_battery">O áudio do microfone permanece neste telefone antes da ativação e nunca é enviado pela detecção. A escuta contínua consome mais bateria. O Android mostra uma notificação permanente do microfone com a ação Parar.</string>
|
||||
<string name="wake_word_microphone_permission_required">A permissão do microfone é necessária para detectar a palavra de ativação.</string>
|
||||
<string name="wake_word_notification_permission_required">A permissão de notificações é necessária para que o Android mostre o controle permanente de privacidade do microfone.</string>
|
||||
<string name="wake_word_notification_channel">Microfone da palavra de ativação</string>
|
||||
<string name="wake_word_notification_channel_desc">Controles permanentes de privacidade e parada enquanto a detecção experimental está ativa.</string>
|
||||
<string name="wake_word_notification_title">Detectando “Hey Hermes”</string>
|
||||
<string name="wake_word_notification_starting">Iniciando o detector no dispositivo…</string>
|
||||
<string name="wake_word_notification_listening">Escutando localmente. O áudio não é enviado.</string>
|
||||
<string name="wake_word_notification_paused">Pausado enquanto uma sessão de voz do Hermes usa o microfone.</string>
|
||||
<string name="wake_word_notification_detected">Frase detectada. Toque para abrir a voz do Hermes.</string>
|
||||
<string name="wake_word_notification_error">A detecção da palavra de ativação requer atenção. Abra as configurações de Voz.</string>
|
||||
<string name="wake_word_notification_stopped">A detecção da palavra de ativação está parando.</string>
|
||||
<string name="wake_word_notification_stop">Parar</string>
|
||||
<string name="wake_word_notification_open">Abrir voz</string>
|
||||
</resources>
|
||||
|
||||
@@ -3344,4 +3344,39 @@
|
||||
<string name="conn_info_profile_api_key_saved">已保存配置文件 API 密钥</string>
|
||||
<string name="conn_info_profile_api_key_cleared">已清除配置文件 API 密钥</string>
|
||||
<string name="conn_info_profile_api_key_save_failed">无法保存配置文件 API 密钥</string>
|
||||
|
||||
<!-- 实验性设备端唤醒词 -->
|
||||
<string name="wake_word_title">唤醒词</string>
|
||||
<string name="wake_word_enable">监听“Hey Hermes”</string>
|
||||
<string name="wake_word_enable_desc">完全在此手机上运行的实验性自愿启用检测。</string>
|
||||
<string name="wake_word_phrase">短语</string>
|
||||
<string name="wake_word_status">状态</string>
|
||||
<string name="wake_word_status_stopped">已停止</string>
|
||||
<string name="wake_word_status_starting">正在启动…</string>
|
||||
<string name="wake_word_status_listening">正在本地监听</string>
|
||||
<string name="wake_word_status_paused">语音功能使用期间已暂停</string>
|
||||
<string name="wake_word_status_detected">已检测到 — 请打开通知</string>
|
||||
<string name="wake_word_status_error">需要处理</string>
|
||||
<string name="wake_word_installing">正在下载并验证设备端英语模型(约 6 MB)…</string>
|
||||
<string name="wake_word_unsupported_abi">唤醒词需要 arm64-v8a、armeabi-v7a、x86_64 或 x86 设备。</string>
|
||||
<string name="wake_word_sensitivity">严格程度</string>
|
||||
<string name="wake_word_sensitivity_desc">数值越高,判定越严格、误唤醒越少,但可能无法检测较轻的语音。</string>
|
||||
<string name="wake_word_confirmation_frames">确认帧数:%1$d</string>
|
||||
<string name="wake_word_confirmation_frames_desc">激活前要求解码器在更多帧中检测到该短语。</string>
|
||||
<string name="wake_word_new_session">开始新会话</string>
|
||||
<string name="wake_word_new_session_desc">打开语音前创建新聊天。关闭后将继续所选配置文件的当前会话。</string>
|
||||
<string name="wake_word_privacy_battery">激活前的麦克风音频仅保留在此手机上,唤醒检测绝不会上传音频。持续监听会增加耗电量。Android 会持续显示带“停止”操作的麦克风通知。</string>
|
||||
<string name="wake_word_microphone_permission_required">唤醒词监听需要麦克风权限。</string>
|
||||
<string name="wake_word_notification_permission_required">需要通知权限,以便 Android 持续显示麦克风隐私控制。</string>
|
||||
<string name="wake_word_notification_channel">唤醒词麦克风</string>
|
||||
<string name="wake_word_notification_channel_desc">实验性唤醒词监听期间持续显示隐私和停止控制。</string>
|
||||
<string name="wake_word_notification_title">正在监听“Hey Hermes”</string>
|
||||
<string name="wake_word_notification_starting">正在启动设备端检测器…</string>
|
||||
<string name="wake_word_notification_listening">正在本地监听。不会上传音频。</string>
|
||||
<string name="wake_word_notification_paused">Hermes 语音会话正在使用麦克风,监听已暂停。</string>
|
||||
<string name="wake_word_notification_detected">已检测到短语。点按即可打开 Hermes 语音。</string>
|
||||
<string name="wake_word_notification_error">唤醒词监听需要处理。请打开语音设置。</string>
|
||||
<string name="wake_word_notification_stopped">正在停止唤醒词监听。</string>
|
||||
<string name="wake_word_notification_stop">停止</string>
|
||||
<string name="wake_word_notification_open">打开语音</string>
|
||||
</resources>
|
||||
|
||||
@@ -3411,4 +3411,39 @@
|
||||
<string name="conn_info_profile_api_key_saved">Profil-API-Schlüssel gespeichert</string>
|
||||
<string name="conn_info_profile_api_key_cleared">Profil-API-Schlüssel gelöscht</string>
|
||||
<string name="conn_info_profile_api_key_save_failed">Profil-API-Schlüssel konnte nicht gespeichert werden</string>
|
||||
|
||||
<!-- Experimentelles Weckwort auf dem Gerät -->
|
||||
<string name="wake_word_title">Weckwort</string>
|
||||
<string name="wake_word_enable">Auf „Hey Hermes“ hören</string>
|
||||
<string name="wake_word_enable_desc">Experimentelle, freiwillig aktivierbare Erkennung, die vollständig auf diesem Telefon läuft.</string>
|
||||
<string name="wake_word_phrase">Phrase</string>
|
||||
<string name="wake_word_status">Status</string>
|
||||
<string name="wake_word_status_stopped">Angehalten</string>
|
||||
<string name="wake_word_status_starting">Wird gestartet…</string>
|
||||
<string name="wake_word_status_listening">Hört lokal zu</string>
|
||||
<string name="wake_word_status_paused">Pausiert, während Sprache aktiv ist</string>
|
||||
<string name="wake_word_status_detected">Erkannt — Benachrichtigung öffnen</string>
|
||||
<string name="wake_word_status_error">Aktion erforderlich</string>
|
||||
<string name="wake_word_installing">Das englische Gerätemodell wird heruntergeladen und geprüft (ca. 6 MB)…</string>
|
||||
<string name="wake_word_unsupported_abi">Das Weckwort erfordert ein arm64-v8a-, armeabi-v7a-, x86_64- oder x86-Gerät.</string>
|
||||
<string name="wake_word_sensitivity">Strenge</string>
|
||||
<string name="wake_word_sensitivity_desc">Höhere Werte sind strenger und reduzieren Fehlauslösungen, können aber leisere Sprache überhören.</string>
|
||||
<string name="wake_word_confirmation_frames">Bestätigungs-Frames: %1$d</string>
|
||||
<string name="wake_word_confirmation_frames_desc">Die Phrase muss vor der Aktivierung in mehr Decoder-Frames vorkommen.</string>
|
||||
<string name="wake_word_new_session">Neue Sitzung starten</string>
|
||||
<string name="wake_word_new_session_desc">Vor dem Öffnen der Sprache einen neuen Chat erstellen. Deaktivieren, um die aktuelle Sitzung des ausgewählten Profils fortzusetzen.</string>
|
||||
<string name="wake_word_privacy_battery">Mikrofon-Audio bleibt vor der Aktivierung auf diesem Telefon und wird von der Weckworterkennung nie hochgeladen. Ständiges Zuhören verbraucht mehr Akku. Android zeigt eine dauerhafte Mikrofonbenachrichtigung mit einer Stopp-Aktion.</string>
|
||||
<string name="wake_word_microphone_permission_required">Für das Weckwort ist die Mikrofonberechtigung erforderlich.</string>
|
||||
<string name="wake_word_notification_permission_required">Die Benachrichtigungsberechtigung ist erforderlich, damit Android die dauerhafte Mikrofon-Datenschutzsteuerung anzeigen kann.</string>
|
||||
<string name="wake_word_notification_channel">Weckwort-Mikrofon</string>
|
||||
<string name="wake_word_notification_channel_desc">Dauerhafte Datenschutz- und Stoppsteuerung bei aktiver experimenteller Weckworterkennung.</string>
|
||||
<string name="wake_word_notification_title">Hört auf „Hey Hermes“</string>
|
||||
<string name="wake_word_notification_starting">Geräteinterne Erkennung wird gestartet…</string>
|
||||
<string name="wake_word_notification_listening">Hört lokal zu. Audio wird nicht hochgeladen.</string>
|
||||
<string name="wake_word_notification_paused">Pausiert, während eine Hermes-Sprachsitzung das Mikrofon verwendet.</string>
|
||||
<string name="wake_word_notification_detected">Phrase erkannt. Tippen, um Hermes Voice zu öffnen.</string>
|
||||
<string name="wake_word_notification_error">Die Weckworterkennung erfordert Aufmerksamkeit. Spracheinstellungen öffnen.</string>
|
||||
<string name="wake_word_notification_stopped">Die Weckworterkennung wird beendet.</string>
|
||||
<string name="wake_word_notification_stop">Stopp</string>
|
||||
<string name="wake_word_notification_open">Sprache öffnen</string>
|
||||
</resources>
|
||||
|
||||
@@ -3096,4 +3096,39 @@
|
||||
<string name="conn_info_profile_api_key_saved">Clave API del perfil guardada</string>
|
||||
<string name="conn_info_profile_api_key_cleared">Clave API del perfil borrada</string>
|
||||
<string name="conn_info_profile_api_key_save_failed">No se pudo guardar la clave API del perfil</string>
|
||||
|
||||
<!-- Palabra de activación experimental en el dispositivo -->
|
||||
<string name="wake_word_title">Palabra de activación</string>
|
||||
<string name="wake_word_enable">Detectar “Hey Hermes”</string>
|
||||
<string name="wake_word_enable_desc">Detección experimental y opcional que se ejecuta íntegramente en este teléfono.</string>
|
||||
<string name="wake_word_phrase">Frase</string>
|
||||
<string name="wake_word_status">Estado</string>
|
||||
<string name="wake_word_status_stopped">Detenido</string>
|
||||
<string name="wake_word_status_starting">Iniciando…</string>
|
||||
<string name="wake_word_status_listening">Escuchando localmente</string>
|
||||
<string name="wake_word_status_paused">En pausa mientras la voz está activa</string>
|
||||
<string name="wake_word_status_detected">Detectado — abre la notificación</string>
|
||||
<string name="wake_word_status_error">Requiere atención</string>
|
||||
<string name="wake_word_installing">Descargando y verificando el modelo de inglés en el dispositivo (unos 6 MB)…</string>
|
||||
<string name="wake_word_unsupported_abi">La palabra de activación requiere un dispositivo arm64-v8a, armeabi-v7a, x86_64 o x86.</string>
|
||||
<string name="wake_word_sensitivity">Rigor</string>
|
||||
<string name="wake_word_sensitivity_desc">Los valores más altos son más estrictos y reducen las activaciones falsas, pero pueden omitir voces más bajas.</string>
|
||||
<string name="wake_word_confirmation_frames">Fotogramas de confirmación: %1$d</string>
|
||||
<string name="wake_word_confirmation_frames_desc">Exige la frase en más fotogramas del decodificador antes de la activación.</string>
|
||||
<string name="wake_word_new_session">Iniciar una sesión nueva</string>
|
||||
<string name="wake_word_new_session_desc">Crea un chat nuevo antes de abrir la voz. Desactívalo para continuar la sesión actual del perfil seleccionado.</string>
|
||||
<string name="wake_word_privacy_battery">El audio del micrófono permanece en este teléfono antes de la activación y la detección nunca lo sube. La escucha continua consume más batería. Android muestra una notificación permanente del micrófono con una acción para detenerla.</string>
|
||||
<string name="wake_word_microphone_permission_required">Se requiere permiso para usar el micrófono al detectar la palabra de activación.</string>
|
||||
<string name="wake_word_notification_permission_required">Se requiere permiso para mostrar notificaciones a fin de que Android muestre el control permanente de privacidad del micrófono.</string>
|
||||
<string name="wake_word_notification_channel">Micrófono de la palabra de activación</string>
|
||||
<string name="wake_word_notification_channel_desc">Controles permanentes de privacidad y detención mientras la detección experimental está activa.</string>
|
||||
<string name="wake_word_notification_title">Detectando “Hey Hermes”</string>
|
||||
<string name="wake_word_notification_starting">Iniciando el detector en el dispositivo…</string>
|
||||
<string name="wake_word_notification_listening">Escuchando localmente. El audio no se sube.</string>
|
||||
<string name="wake_word_notification_paused">En pausa mientras una sesión de voz de Hermes usa el micrófono.</string>
|
||||
<string name="wake_word_notification_detected">Frase detectada. Toca para abrir la voz de Hermes.</string>
|
||||
<string name="wake_word_notification_error">La detección de la palabra de activación requiere atención. Abre los ajustes de Voz.</string>
|
||||
<string name="wake_word_notification_stopped">La detección de la palabra de activación se está deteniendo.</string>
|
||||
<string name="wake_word_notification_stop">Detener</string>
|
||||
<string name="wake_word_notification_open">Abrir voz</string>
|
||||
</resources>
|
||||
|
||||
@@ -3410,4 +3410,39 @@
|
||||
<string name="conn_info_profile_api_key_saved">プロファイル API キーを保存しました</string>
|
||||
<string name="conn_info_profile_api_key_cleared">プロファイル API キーを消去しました</string>
|
||||
<string name="conn_info_profile_api_key_save_failed">プロファイル API キーを保存できませんでした</string>
|
||||
|
||||
<!-- 実験的な端末内ウェイクワード -->
|
||||
<string name="wake_word_title">ウェイクワード</string>
|
||||
<string name="wake_word_enable">「Hey Hermes」を検出</string>
|
||||
<string name="wake_word_enable_desc">この端末内だけで動作する、実験的なオプトイン検出機能です。</string>
|
||||
<string name="wake_word_phrase">フレーズ</string>
|
||||
<string name="wake_word_status">状態</string>
|
||||
<string name="wake_word_status_stopped">停止中</string>
|
||||
<string name="wake_word_status_starting">起動中…</string>
|
||||
<string name="wake_word_status_listening">端末内で検出中</string>
|
||||
<string name="wake_word_status_paused">音声機能の使用中は一時停止</string>
|
||||
<string name="wake_word_status_detected">検出しました — 通知を開いてください</string>
|
||||
<string name="wake_word_status_error">確認が必要です</string>
|
||||
<string name="wake_word_installing">端末内の英語モデルをダウンロードして検証しています(約 6 MB)…</string>
|
||||
<string name="wake_word_unsupported_abi">ウェイクワードには arm64-v8a、armeabi-v7a、x86_64、または x86 端末が必要です。</string>
|
||||
<string name="wake_word_sensitivity">判定の厳しさ</string>
|
||||
<string name="wake_word_sensitivity_desc">値を高くすると誤検出は減りますが、小さな声を検出できない場合があります。</string>
|
||||
<string name="wake_word_confirmation_frames">確認フレーム数: %1$d</string>
|
||||
<string name="wake_word_confirmation_frames_desc">有効化する前に、より多くのデコーダーフレームでフレーズを確認します。</string>
|
||||
<string name="wake_word_new_session">新しいセッションを開始</string>
|
||||
<string name="wake_word_new_session_desc">音声機能を開く前に新しいチャットを作成します。選択中のプロファイルの現在のセッションを続けるにはオフにします。</string>
|
||||
<string name="wake_word_privacy_battery">有効化前のマイク音声はこの端末内に留まり、ウェイクワード検出によってアップロードされることはありません。常時検出はバッテリーを多く消費します。Android は停止操作付きのマイク通知を常時表示します。</string>
|
||||
<string name="wake_word_microphone_permission_required">ウェイクワードの検出にはマイクの権限が必要です。</string>
|
||||
<string name="wake_word_notification_permission_required">Android がマイクのプライバシー操作を常時表示するには、通知の権限が必要です。</string>
|
||||
<string name="wake_word_notification_channel">ウェイクワード用マイク</string>
|
||||
<string name="wake_word_notification_channel_desc">実験的なウェイクワード検出中に、プライバシーと停止の操作を常時表示します。</string>
|
||||
<string name="wake_word_notification_title">「Hey Hermes」を検出中</string>
|
||||
<string name="wake_word_notification_starting">端末内の検出機能を起動しています…</string>
|
||||
<string name="wake_word_notification_listening">端末内で検出中です。音声はアップロードされません。</string>
|
||||
<string name="wake_word_notification_paused">Hermes の音声セッションがマイクを使用しているため一時停止中です。</string>
|
||||
<string name="wake_word_notification_detected">フレーズを検出しました。タップして Hermes の音声機能を開きます。</string>
|
||||
<string name="wake_word_notification_error">ウェイクワード検出を確認してください。音声設定を開いてください。</string>
|
||||
<string name="wake_word_notification_stopped">ウェイクワード検出を停止しています。</string>
|
||||
<string name="wake_word_notification_stop">停止</string>
|
||||
<string name="wake_word_notification_open">音声機能を開く</string>
|
||||
</resources>
|
||||
|
||||
@@ -3417,4 +3417,39 @@
|
||||
<string name="conn_info_profile_api_key_saved">Profile API key saved</string>
|
||||
<string name="conn_info_profile_api_key_cleared">Profile API key cleared</string>
|
||||
<string name="conn_info_profile_api_key_save_failed">Could not save the profile API key</string>
|
||||
|
||||
<!-- Experimental on-device wake word -->
|
||||
<string name="wake_word_title">Wake word</string>
|
||||
<string name="wake_word_enable">Listen for “Hey Hermes”</string>
|
||||
<string name="wake_word_enable_desc">Experimental, opt-in detection that runs entirely on this phone.</string>
|
||||
<string name="wake_word_phrase">Phrase</string>
|
||||
<string name="wake_word_status">Status</string>
|
||||
<string name="wake_word_status_stopped">Stopped</string>
|
||||
<string name="wake_word_status_starting">Starting…</string>
|
||||
<string name="wake_word_status_listening">Listening locally</string>
|
||||
<string name="wake_word_status_paused">Paused while voice is active</string>
|
||||
<string name="wake_word_status_detected">Detected — open the notification</string>
|
||||
<string name="wake_word_status_error">Needs attention</string>
|
||||
<string name="wake_word_installing">Downloading and verifying the on-device English model (about 6 MB)…</string>
|
||||
<string name="wake_word_unsupported_abi">Wake word requires an arm64-v8a, armeabi-v7a, x86_64, or x86 device.</string>
|
||||
<string name="wake_word_sensitivity">Strictness</string>
|
||||
<string name="wake_word_sensitivity_desc">Higher values are stricter and reduce false activations, but may miss quieter speech.</string>
|
||||
<string name="wake_word_confirmation_frames">Confirmation frames: %1$d</string>
|
||||
<string name="wake_word_confirmation_frames_desc">Require the phrase across more decoder frames before activation.</string>
|
||||
<string name="wake_word_new_session">Start a new session</string>
|
||||
<string name="wake_word_new_session_desc">Create a fresh chat before opening voice. Turn this off to continue the selected profile’s current session.</string>
|
||||
<string name="wake_word_privacy_battery">Microphone audio stays on this phone before activation and is never uploaded by wake detection. Continuous listening uses extra battery. Android shows an ongoing microphone notification with a Stop action.</string>
|
||||
<string name="wake_word_microphone_permission_required">Microphone permission is required for wake-word listening.</string>
|
||||
<string name="wake_word_notification_permission_required">Notification permission is required so Android can show the ongoing microphone privacy control.</string>
|
||||
<string name="wake_word_notification_channel">Wake-word microphone</string>
|
||||
<string name="wake_word_notification_channel_desc">Ongoing privacy and stop controls while experimental wake-word listening is active.</string>
|
||||
<string name="wake_word_notification_title">“Hey Hermes” listening</string>
|
||||
<string name="wake_word_notification_starting">Starting the on-device detector…</string>
|
||||
<string name="wake_word_notification_listening">Listening locally. Audio is not uploaded.</string>
|
||||
<string name="wake_word_notification_paused">Paused while a Hermes voice session owns the microphone.</string>
|
||||
<string name="wake_word_notification_detected">Phrase detected. Tap to open Hermes voice.</string>
|
||||
<string name="wake_word_notification_error">Wake-word listening needs attention. Open Voice settings.</string>
|
||||
<string name="wake_word_notification_stopped">Wake-word listening is stopping.</string>
|
||||
<string name="wake_word_notification_stop">Stop</string>
|
||||
<string name="wake_word_notification_open">Open voice</string>
|
||||
</resources>
|
||||
|
||||
@@ -3,6 +3,7 @@ package com.hermesandroid.relay.audio
|
||||
import android.media.audiofx.AcousticEchoCanceler
|
||||
import android.media.audiofx.NoiseSuppressor
|
||||
import android.util.Log
|
||||
import com.hermesandroid.relay.wake.MicrophoneOwnershipCoordinator
|
||||
import io.mockk.every
|
||||
import io.mockk.mockk
|
||||
import io.mockk.mockkStatic
|
||||
@@ -54,6 +55,7 @@ class BargeInListenerTest {
|
||||
|
||||
@Before
|
||||
fun setUp() {
|
||||
MicrophoneOwnershipCoordinator.resetForTest()
|
||||
// android.util.Log + android.media.audiofx.* are JVM-side stubs on
|
||||
// the unit-test classpath — their native methods throw "Method not
|
||||
// mocked" unless explicitly stubbed. [BargeInListener] touches
|
||||
@@ -78,19 +80,19 @@ class BargeInListenerTest {
|
||||
|
||||
@After
|
||||
fun tearDown() {
|
||||
MicrophoneOwnershipCoordinator.resetForTest()
|
||||
unmockkStatic(Log::class)
|
||||
unmockkStatic(AcousticEchoCanceler::class)
|
||||
unmockkStatic(NoiseSuppressor::class)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `bargeInDetected fires when VadEngine reports speech`() = runTest {
|
||||
fun `bargeInDetected fires after sustained model speech crosses RMS window`() = runTest {
|
||||
val vadEngine = mockk<VadEngine>()
|
||||
every { vadEngine.analyze(any()) } returnsMany listOf(
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
)
|
||||
every { vadEngine.analyze(any()) } returns
|
||||
VadResult(isSpeech = true, probability = 1f)
|
||||
|
||||
val source = ScriptedAudioSource(listOf(makeFrame()))
|
||||
val source = ScriptedAudioSource(List(10) { makeFrame(3_000) })
|
||||
val listener = BargeInListener(
|
||||
audioSource = source,
|
||||
vadEngine = vadEngine,
|
||||
@@ -101,6 +103,7 @@ class BargeInListenerTest {
|
||||
val collector = listener.bargeInDetected.asCollector(this)
|
||||
|
||||
listener.start(this)
|
||||
listener.markPlaybackStarted(nowMs = 0L, graceMs = 0L)
|
||||
// Advance enough for the single frame to be read + analyzed + emitted.
|
||||
// Bounded advance rather than advanceUntilIdle: the reader loops on
|
||||
// delay(5) after the scripted frames are drained, which would make
|
||||
@@ -113,7 +116,7 @@ class BargeInListenerTest {
|
||||
runCurrent()
|
||||
|
||||
assertEquals(
|
||||
"bargeInDetected should fire exactly once for a single-frame speech verdict",
|
||||
"bargeInDetected should fire once the 300ms majority window is satisfied",
|
||||
1,
|
||||
collector.events.get(),
|
||||
)
|
||||
@@ -133,7 +136,7 @@ class BargeInListenerTest {
|
||||
every { vadEngine.analyze(any()) } returns
|
||||
VadResult(isSpeech = false, probability = 1f)
|
||||
|
||||
val source = ScriptedAudioSource(listOf(makeFrame()))
|
||||
val source = ScriptedAudioSource(listOf(makeFrame(3_000)))
|
||||
val listener = BargeInListener(
|
||||
audioSource = source,
|
||||
vadEngine = vadEngine,
|
||||
@@ -145,6 +148,7 @@ class BargeInListenerTest {
|
||||
val bargeCollector = listener.bargeInDetected.asCollector(this)
|
||||
|
||||
listener.start(this)
|
||||
listener.markPlaybackStarted(nowMs = 0L, graceMs = 0L)
|
||||
advanceTimeBy(50)
|
||||
runCurrent()
|
||||
|
||||
@@ -167,18 +171,22 @@ class BargeInListenerTest {
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `scripted speech silence speech cadence fires bargeInDetected twice`() = runTest {
|
||||
// 5-frame script: speech, silence, silence, speech, silence.
|
||||
fun `window tolerates short dips and still fires`() = runTest {
|
||||
val vadEngine = mockk<VadEngine>()
|
||||
every { vadEngine.analyze(any()) } returnsMany listOf(
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = false, probability = 0f),
|
||||
VadResult(isSpeech = false, probability = 0f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = false, probability = 0f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
VadResult(isSpeech = false, probability = 0f),
|
||||
VadResult(isSpeech = true, probability = 1f),
|
||||
)
|
||||
|
||||
val frames = List(5) { makeFrame() }
|
||||
val frames = List(10) { makeFrame(3_000) }
|
||||
val source = ScriptedAudioSource(frames)
|
||||
val listener = BargeInListener(
|
||||
audioSource = source,
|
||||
@@ -190,7 +198,8 @@ class BargeInListenerTest {
|
||||
val bargeCollector = listener.bargeInDetected.asCollector(this)
|
||||
|
||||
listener.start(this)
|
||||
// 5 scripted frames consumed + tail delay loop. Bounded advance so
|
||||
listener.markPlaybackStarted(nowMs = 0L, graceMs = 0L)
|
||||
// Scripted frames consumed + tail delay loop. Bounded advance so
|
||||
// the subsequent delay(5) retry loop doesn't spin forever.
|
||||
advanceTimeBy(100)
|
||||
runCurrent()
|
||||
@@ -200,8 +209,8 @@ class BargeInListenerTest {
|
||||
runCurrent()
|
||||
|
||||
assertEquals(
|
||||
"two scripted speech frames → two barge-in events",
|
||||
2,
|
||||
"80% speech across the decision window should trigger once",
|
||||
1,
|
||||
bargeCollector.events.get(),
|
||||
)
|
||||
|
||||
@@ -229,11 +238,11 @@ class BargeInListenerTest {
|
||||
runCurrent()
|
||||
assertTrue("reader should have issued at least one read()", source.readCount.get() > 0)
|
||||
|
||||
listener.stop()
|
||||
// Within the 500 ms plan budget — we use virtual time so this is
|
||||
// deterministic rather than wall-clock.
|
||||
advanceTimeBy(500)
|
||||
runCurrent()
|
||||
val stoppedReader = listener.stop()
|
||||
// stop() is deliberately non-blocking for production callers. Joining
|
||||
// the returned reader job proves its finally block completes promptly
|
||||
// and makes the ownership-release assertion deterministic.
|
||||
stoppedReader?.join()
|
||||
|
||||
// After stop(), the finally block must have released the source.
|
||||
assertTrue("source should have been released by stop()", source.released)
|
||||
@@ -285,7 +294,8 @@ class BargeInListenerTest {
|
||||
// ─── Helpers ──────────────────────────────────────────────────────────
|
||||
|
||||
/** A zero-filled PCM frame at the VadEngine's required length. */
|
||||
private fun makeFrame(): ShortArray = ShortArray(VadEngine.FRAME_SIZE_SAMPLES)
|
||||
private fun makeFrame(amplitude: Int = 0): ShortArray =
|
||||
ShortArray(VadEngine.FRAME_SIZE_SAMPLES) { amplitude.toShort() }
|
||||
|
||||
/**
|
||||
* Hand-rolled SharedFlow collector for the test's TestScope. Using a
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
package com.hermesandroid.relay.audio
|
||||
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
class RmsBargeInGateTest {
|
||||
@Test
|
||||
fun `quiet calibration is frozen before playback and speaker bleed alone does not trip`() {
|
||||
val gate = RmsBargeInGate()
|
||||
repeat(10) { gate.observe(frame(200), rawSpeech = false, nowMs = it * 32L, playbackGraceMs = 500) }
|
||||
|
||||
gate.markPlaybackStarted(400)
|
||||
repeat(20) { index ->
|
||||
val result = gate.observe(
|
||||
frame = frame(1_200),
|
||||
rawSpeech = true,
|
||||
nowMs = 1_000L + index * 32L,
|
||||
playbackGraceMs = 500,
|
||||
)
|
||||
assertFalse(result.detected)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `speech over playback crosses held floor after grace and majority window`() {
|
||||
val gate = RmsBargeInGate()
|
||||
repeat(10) { gate.observe(frame(250), rawSpeech = false, nowMs = it * 32L, playbackGraceMs = 500) }
|
||||
gate.markPlaybackStarted(400)
|
||||
|
||||
var detected = false
|
||||
repeat(10) { index ->
|
||||
detected = gate.observe(
|
||||
frame = frame(if (index == 4) 1_000 else 3_000),
|
||||
rawSpeech = index != 4,
|
||||
nowMs = 1_000L + index * 32L,
|
||||
playbackGraceMs = 500,
|
||||
).detected
|
||||
}
|
||||
|
||||
assertTrue(detected)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `higher multiplier is stricter`() {
|
||||
val eager = RmsBargeInGate().apply { thresholdMultiplier = 2f }
|
||||
val strict = RmsBargeInGate().apply { thresholdMultiplier = 8f }
|
||||
repeat(10) {
|
||||
eager.observe(frame(500), false, it * 32L, 0)
|
||||
strict.observe(frame(500), false, it * 32L, 0)
|
||||
}
|
||||
|
||||
var eagerDetected = false
|
||||
var strictDetected = false
|
||||
repeat(10) { index ->
|
||||
eagerDetected = eager.observe(frame(1_500), true, 500L + index * 32L, 0).detected
|
||||
strictDetected = strict.observe(frame(1_500), true, 500L + index * 32L, 0).detected
|
||||
}
|
||||
|
||||
assertTrue(eagerDetected)
|
||||
assertFalse(strictDetected)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `playback grace suppresses early detection`() {
|
||||
val gate = RmsBargeInGate()
|
||||
repeat(10) { gate.observe(frame(100), false, it * 32L, 500) }
|
||||
gate.markPlaybackStarted(400)
|
||||
|
||||
repeat(12) { index ->
|
||||
val result = gate.observe(frame(4_000), true, 410L + index * 32L, 500)
|
||||
assertFalse(result.detected)
|
||||
}
|
||||
}
|
||||
|
||||
private fun frame(amplitude: Int): ShortArray =
|
||||
ShortArray(VadEngine.FRAME_SIZE_SAMPLES) { amplitude.toShort() }
|
||||
}
|
||||
@@ -69,6 +69,50 @@ class VoiceCommandInterpreterTest {
|
||||
assertNull(VoiceCommandInterpreter.interpretFinalTranscript("stop", bothActive))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `bare stop is accepted only after an active response barge in`() {
|
||||
assertEquals(
|
||||
VoiceCommandAction.StopResponse,
|
||||
VoiceCommandInterpreter.interpretFinalTranscript(
|
||||
"stop",
|
||||
VoiceCommandContext(
|
||||
responseActive = true,
|
||||
interruptedActiveResponse = true,
|
||||
),
|
||||
),
|
||||
)
|
||||
assertNull(
|
||||
VoiceCommandInterpreter.interpretFinalTranscript(
|
||||
"stop",
|
||||
VoiceCommandContext(responseActive = true),
|
||||
),
|
||||
)
|
||||
assertNull(
|
||||
VoiceCommandInterpreter.interpretFinalTranscript(
|
||||
"stop the container",
|
||||
VoiceCommandContext(
|
||||
responseActive = true,
|
||||
interruptedActiveResponse = true,
|
||||
),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `bare pause after barge in pauses continuous mode in generation or playback`() {
|
||||
assertEquals(
|
||||
VoiceCommandAction.PauseContinuousListening,
|
||||
VoiceCommandInterpreter.interpretFinalTranscript(
|
||||
"pause",
|
||||
VoiceCommandContext(
|
||||
responseActive = true,
|
||||
interruptedActiveResponse = true,
|
||||
continuousModeSelected = true,
|
||||
),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `state gates stop and background cancellation`() {
|
||||
assertNull(
|
||||
|
||||
@@ -159,6 +159,40 @@ class VoiceViewModelBargeInTest {
|
||||
return vm
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------
|
||||
// Full-turn ownership — Thinking -> Speaking uses one listener
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `one listener spans Thinking into Speaking without rearm`() = runTest {
|
||||
val vm = buildViewModel()
|
||||
|
||||
vm.beginBargeInTurnForTest()
|
||||
runCurrent()
|
||||
assertEquals(VoiceState.Thinking, vm.uiState.value.state)
|
||||
verify(exactly = 1) { bargeInListener.start(any()) }
|
||||
|
||||
vm.markBargeInPlaybackStartedForTest()
|
||||
runCurrent()
|
||||
assertEquals(VoiceState.Speaking, vm.uiState.value.state)
|
||||
verify(exactly = 1) { bargeInListener.start(any()) }
|
||||
verify(exactly = 1) { bargeInListener.markPlaybackStarted(any(), any()) }
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `bargeInDetected during Thinking interrupts generation and captures replacement`() = runTest {
|
||||
val vm = buildViewModel()
|
||||
vm.beginBargeInTurnForTest()
|
||||
runCurrent()
|
||||
|
||||
bargeInFlow.emit(Unit)
|
||||
runCurrent()
|
||||
|
||||
assertEquals(VoiceState.Listening, vm.uiState.value.state)
|
||||
verify(atLeast = 1) { chatViewModel.cancelStream() }
|
||||
verify(atLeast = 1) { recorder.startRecording() }
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------
|
||||
// Test 1 — bargeInDetected while Speaking → interrupt + Listening
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import org.junit.After
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertNull
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Before
|
||||
import org.junit.Test
|
||||
|
||||
class WakeWordCoreTest {
|
||||
@Before
|
||||
fun setUp() {
|
||||
MicrophoneOwnershipCoordinator.resetForTest()
|
||||
WakeWordActivationCoordinator.resetForTest()
|
||||
}
|
||||
|
||||
@After
|
||||
fun tearDown() {
|
||||
MicrophoneOwnershipCoordinator.resetForTest()
|
||||
WakeWordActivationCoordinator.resetForTest()
|
||||
}
|
||||
|
||||
@Test
|
||||
fun microphoneLease_isExclusiveAndRejectsStaleRelease() {
|
||||
val wakeLease = MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.WakeWord)
|
||||
requireNotNull(wakeLease)
|
||||
assertEquals(MicrophoneOwner.WakeWord, MicrophoneOwnershipCoordinator.owner.value)
|
||||
assertNull(MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.VoiceCapture))
|
||||
|
||||
val stale = MicrophoneLease(MicrophoneOwner.WakeWord, "stale")
|
||||
assertFalse(MicrophoneOwnershipCoordinator.release(stale))
|
||||
assertEquals(MicrophoneOwner.WakeWord, MicrophoneOwnershipCoordinator.owner.value)
|
||||
|
||||
assertTrue(MicrophoneOwnershipCoordinator.release(wakeLease))
|
||||
assertNull(MicrophoneOwnershipCoordinator.owner.value)
|
||||
assertEquals(
|
||||
MicrophoneOwner.VoiceCapture,
|
||||
MicrophoneOwnershipCoordinator.tryAcquire(MicrophoneOwner.VoiceCapture)?.owner,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun activation_remainsPendingUntilMatchingConsumer() {
|
||||
val activation = WakeWordActivation(
|
||||
id = "activation-1",
|
||||
startNewSession = true,
|
||||
profileRouting = WakeWordProfileRouting(),
|
||||
)
|
||||
WakeWordActivationCoordinator.request(activation)
|
||||
|
||||
assertEquals(activation, WakeWordActivationCoordinator.pending.value)
|
||||
assertFalse(WakeWordActivationCoordinator.consume("other"))
|
||||
assertEquals(activation, WakeWordActivationCoordinator.pending.value)
|
||||
assertTrue(WakeWordActivationCoordinator.consume("activation-1"))
|
||||
assertNull(WakeWordActivationCoordinator.pending.value)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun confirmationGate_firesOnceAfterRequiredConsecutiveFrames() {
|
||||
val gate = WakeWordConfirmationGate(requiredFrames = 3)
|
||||
assertFalse(gate.update(true))
|
||||
assertFalse(gate.update(false))
|
||||
assertFalse(gate.update(true))
|
||||
assertFalse(gate.update(true))
|
||||
assertTrue(gate.update(true))
|
||||
assertFalse(gate.update(true))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun sensitivityThreshold_higherMeansStricterAndClamps() {
|
||||
assertTrue(WakeWordTuning.threshold(0.8f) > WakeWordTuning.threshold(0.3f))
|
||||
assertEquals(0.2f, WakeWordTuning.threshold(-1f))
|
||||
assertEquals(0.9f, WakeWordTuning.threshold(2f))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package com.hermesandroid.relay.wake
|
||||
|
||||
import androidx.datastore.core.DataStore
|
||||
import androidx.datastore.preferences.core.Preferences
|
||||
import androidx.datastore.preferences.core.PreferenceDataStoreFactory
|
||||
import java.io.File
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.cancel
|
||||
import kotlinx.coroutines.flow.first
|
||||
import kotlinx.coroutines.test.runTest
|
||||
import org.junit.After
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertNull
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Before
|
||||
import org.junit.Rule
|
||||
import org.junit.Test
|
||||
import org.junit.rules.TemporaryFolder
|
||||
|
||||
class WakeWordPreferencesTest {
|
||||
@get:Rule
|
||||
val tempFolder = TemporaryFolder()
|
||||
|
||||
private lateinit var scope: CoroutineScope
|
||||
private lateinit var dataStore: DataStore<Preferences>
|
||||
private lateinit var repository: WakeWordPreferencesRepository
|
||||
|
||||
@Before
|
||||
fun setUp() {
|
||||
scope = CoroutineScope(Dispatchers.IO + Job())
|
||||
val file: File = tempFolder.newFile("wake_preferences_test.preferences_pb")
|
||||
if (file.exists()) file.delete()
|
||||
dataStore = PreferenceDataStoreFactory.create(scope = scope, produceFile = { file })
|
||||
repository = WakeWordPreferencesRepository(dataStore)
|
||||
}
|
||||
|
||||
@After
|
||||
fun tearDown() {
|
||||
scope.cancel()
|
||||
}
|
||||
|
||||
@Test
|
||||
fun defaults_areOptInAndUseOnlyValidatedPhrase() = runTest {
|
||||
val preferences = repository.flow.first()
|
||||
assertFalse(preferences.enabled)
|
||||
assertEquals(DEFAULT_WAKE_PHRASE, preferences.phrase)
|
||||
assertEquals(0.6f, preferences.sensitivity)
|
||||
assertEquals(3, preferences.confirmationFrames)
|
||||
assertTrue(preferences.startNewSession)
|
||||
assertEquals(WakeWordProfileRouteMode.Active, preferences.profileRouting.mode)
|
||||
assertNull(preferences.profileRouting.profileName)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun enabled_roundTrips() = runTest {
|
||||
repository.setEnabled(true)
|
||||
val preferences = repository.flow.first()
|
||||
assertTrue(preferences.enabled)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun sensitivity_roundTripsAndClamps() = runTest {
|
||||
repository.setSensitivity(2f)
|
||||
val preferences = repository.flow.first()
|
||||
assertEquals(0.9f, preferences.sensitivity)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun confirmationFrames_roundTripsAndClamps() = runTest {
|
||||
repository.setConfirmationFrames(9)
|
||||
val preferences = repository.flow.first()
|
||||
assertEquals(5, preferences.confirmationFrames)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun startNewSession_roundTrips() = runTest {
|
||||
repository.setStartNewSession(false)
|
||||
val preferences = repository.flow.first()
|
||||
assertFalse(preferences.startNewSession)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun futureProfileRoutingShape_roundTripsWithoutEnablingIt() = runTest {
|
||||
repository.setProfileRouting(
|
||||
WakeWordProfileRouting(
|
||||
mode = WakeWordProfileRouteMode.Specific,
|
||||
profileName = "research",
|
||||
)
|
||||
)
|
||||
|
||||
val preferences = repository.flow.first()
|
||||
assertEquals(WakeWordProfileRouteMode.Specific, preferences.profileRouting.mode)
|
||||
assertEquals("research", preferences.profileRouting.profileName)
|
||||
assertEquals(DEFAULT_WAKE_PHRASE, preferences.phrase)
|
||||
}
|
||||
}
|
||||
@@ -2257,3 +2257,57 @@ active private route.
|
||||
- Android retains a full-screen embedded WebView for compatible dashboard
|
||||
cookie providers, while providers that prohibit embedding use the explicit
|
||||
brokered native route.
|
||||
|
||||
---
|
||||
|
||||
## ADR 41 — Android owns full-turn interruption and experimental wake detection
|
||||
|
||||
**Status:** Accepted (2026-07-29).
|
||||
|
||||
**Context.** Android barge-in previously armed only when speech playback began.
|
||||
That left agent generation non-interruptible and recreated the microphone/VAD
|
||||
pipeline at the Thinking-to-Speaking boundary. Upstream voice work established
|
||||
a safer full-turn lifecycle, quiet calibration before playback, and phase-aware
|
||||
bare stop behavior. Upstream wake listening is host-local, but enabling that
|
||||
server listener from Android would capture audio on the wrong machine and
|
||||
couple Standard voice to non-standard server behavior.
|
||||
|
||||
**Decision.**
|
||||
|
||||
- Android owns one barge-in listener per active voice response, spanning
|
||||
`Thinking`, `Speaking`, and final audio drain on Standard and Realtime paths.
|
||||
A turn epoch fences callbacks, and teardown completes before replacement
|
||||
capture or another listener can acquire the microphone.
|
||||
- Quiet-room RMS calibration occurs before output and freezes at playback
|
||||
start. Sensitivity scales the threshold; playback adds a grace interval and
|
||||
bounded threshold; majority filtering requires model-confirmed speech.
|
||||
- Interruption uses the existing active-turn cancellation seam. Late Standard
|
||||
stream content and Realtime audio are suppressed. Silencing does not cancel
|
||||
a promoted Hermes task; explicit background-task cancellation remains the
|
||||
separate destructive intent.
|
||||
- Bare `stop` and `pause` are voice commands only when they are the exact final
|
||||
transcript captured after an active response interruption. Longer ordinary
|
||||
requests remain agent input.
|
||||
- Wake-word detection is Android-local, experimental, and off by default. A
|
||||
user-started microphone foreground service runs sherpa-onnx for the single
|
||||
validated “Hey Hermes” phrase, with an ongoing notification and Stop action.
|
||||
No pre-activation PCM leaves the phone.
|
||||
- Wake and voice share a process-wide microphone lease. Detection releases its
|
||||
recorder before entering the existing voice flow and resumes only after voice
|
||||
exits. There is no boot receiver or server wake-listener control.
|
||||
- The KWS model is downloaded and SHA-256 verified on first enable. Preferences
|
||||
store enabled state, fixed phrase, strictness, confirmation frames,
|
||||
start-new-session behavior, and a future-safe profile-routing shape. Only
|
||||
active-profile preservation is implemented; profile-specific phrases are
|
||||
intentionally not claimed.
|
||||
|
||||
**Consequences.**
|
||||
|
||||
- Standard voice remains Dashboard/Gateway-backed and works against unmodified
|
||||
upstream Hermes. Local detection is an Android input affordance, not a Relay
|
||||
server dependency.
|
||||
- The sherpa runtime increases Android artifacts for each packaged ABI, while
|
||||
the approximately 6 MB model is device storage rather than APK payload.
|
||||
- Continuous wake listening has visible microphone and battery cost and
|
||||
requires explicit device/acoustic validation before the experimental label
|
||||
can be reconsidered.
|
||||
|
||||
+4
-1
@@ -867,7 +867,8 @@ utilities.
|
||||
- `RealtimePcmPlayer` streams `/voice/output/*`, `/voice/realtime/*`, and `/voice/realtime-agent/*` PCM deltas directly to `AudioTrack`.
|
||||
- `VoiceViewModel` state machine (`Idle / Listening / Transcribing / Thinking / Speaking / Error`). Assistant text is sanitized (markdown / tool-annotations / URLs / emoji-set stripped) on each delta before a coalescing chunker (`MIN_COALESCE_LEN=40`, `MAX_BUFFER_LEN=400` secondary-break escape, 800 ms timer flush) emits sentence-scale chunks. The observer aggregates every assistant bubble created by one Hermes run, including interim tool handoffs and the final answer, and finishes speech only when the run-level stream ends. Stable bubble identity and submitted-turn/session fences prevent StateFlow/history reconciliation or a pending-new-chat session switch from speaking stale or duplicate text. Bubble boundaries flush incomplete prior text so adjacent narration cannot run together. The default queue calls `/voice/output/*` for exact renderer PCM playback; failed output turns fall back to the existing `/voice/synthesize` synth/play workers. The same stream observer watches Hermes-owned `ToolCall` state and speaks bounded status lines for running tools; execution, approval, and tool results remain in the Hermes chat/relay loop.
|
||||
- Server-side, `/voice/synthesize` runs a matching sanitizer (`plugin/relay/tts_sanitizer.py`) before handing text to the upstream `text_to_speech_tool` — defense-in-depth for any client that doesn't pre-sanitize.
|
||||
- **Barge-in** (opt-in, default off). While in `Speaking`, a `BargeInListener` runs a duplex `AudioRecord` (16 kHz mono PCM, `VOICE_COMMUNICATION` source) feeding 32 ms frames through a Silero VAD (`com.github.gkonovalov:android-vad:silero`). `AcousticEchoCanceler` + `NoiseSuppressor` attach to the ExoPlayer audio session so TTS output doesn't retrigger VAD. A single raw speech frame → `VoicePlayer.duck()` (volume 0.3f) with a 500 ms un-duck watchdog. `N` consecutive frames (2–3, sensitivity-tuned) → `interruptSpeaking()` (same cancellation path V4 wired for user taps). A 600 ms watchdog on `VoiceRecorder.amplitude` then decides: if the user keeps talking, new turn proceeds normally; if silence wins AND `resumeAfterInterruption=true`, `VoiceViewModel` re-enqueues the unplayed chunks from `spokenChunks[lastInterruptedAtChunkIndex+1..]` and flips back to `Speaking`. Settings UI exposes `BargeInPreferences` (enabled / sensitivity ∈ `Off/Low/Default/High` / resume) with an `AcousticEchoCanceler.isAvailable()`-driven compatibility badge.
|
||||
- **Full-turn barge-in** (opt-in, default off). One turn-scoped `BargeInListener` starts when the submitted voice turn enters `Thinking` and remains the sole listener through generation, `Speaking`, and audio drain on both Standard and Realtime paths. Its duplex `AudioRecord` (16 kHz mono PCM, `VOICE_COMMUNICATION` source) feeds 32 ms frames through Silero VAD plus a quiet-room RMS calibration that freezes before playback, a sensitivity-scaled threshold, a 500 ms playback grace period, and an 80%-majority decision window. Raw probable speech ducks playback; only model-confirmed speech above the RMS gate interrupts. `AcousticEchoCanceler` + `NoiseSuppressor` attach when a playback session becomes available. Detection uses the existing gateway/provider interrupt seam, fences stale callbacks and late audio/text deltas, waits for microphone release, then captures the replacement utterance. A 600 ms watchdog preserves the existing resume-after-interruption behavior for playback. Exact bare `stop`/`pause` transcripts are commands only after an active response was interrupted; ordinary requests such as “stop the container” remain agent input. Silencing a promoted background run leaves the Hermes task alive unless the user explicitly requests background-task cancellation.
|
||||
- **Experimental local wake word** (opt-in, default off). Android runs sherpa-onnx keyword spotting for the single validated phrase “Hey Hermes” inside a user-started microphone foreground service. Pre-activation PCM never leaves the phone. Voice settings expose strictness (higher is harder to trigger), confirmation frames, and start-new-session behavior; the stored routing shape reserves future profile-specific selection while this release deliberately preserves the currently selected profile. The first enable downloads and SHA-256 verifies the approximately 6 MB English KWS model rather than bundling it in the APK. Detection releases the wake microphone before entering the existing voice flow, pauses wake listening while voice owns the microphone, and resumes only after voice exits. Android’s ongoing microphone notification provides the persistent privacy status and Stop action; there is no boot or background auto-start.
|
||||
- Stable voice integrates with `ChatViewModel` by **observing** `messages: StateFlow`; transcribed text goes through normal `chatVm.sendMessage(text)` so voice utterances appear as regular user messages in chat history. Experimental Realtime Agent creates a mirrored chat turn and applies broker events directly so tool state, transcript text, assistant deltas, and final responses appear without leaving voice mode.
|
||||
- `VoiceModeOverlay` — full-screen UI with the MorphingSphere at 60% height in `voiceMode=true`, transcribed + response text, mic button supporting Tap / Hold / Continuous interaction modes.
|
||||
- `MorphingSphere` gains `SphereState.Listening` (soft blue/purple, subtle wobble with user amplitude) and `SphereState.Speaking` (vivid green/teal, dramatic core-warmth pulse with agent amplitude). Additive changes — existing call sites unchanged via defaulted `voiceAmplitude` / `voiceMode` params.
|
||||
@@ -923,6 +924,8 @@ Current Android dependency versions. Source of truth is `gradle/libs.versions.to
|
||||
| Haze | 1.7.2 | Glassmorphism blur |
|
||||
| ML Kit Barcode Scanning | 17.3.0 | QR pairing scan |
|
||||
| CameraX | 1.6.0 | QR camera preview |
|
||||
| ONNX Runtime Android | 1.27.0 | Shared Silero VAD and sherpa KWS runtime |
|
||||
| sherpa-onnx | 1.13.4 | Experimental on-device keyword spotting |
|
||||
| xterm.js | 5.x | Terminal emulator (WebView) |
|
||||
| aiohttp | 3.14.1+ | Server relay |
|
||||
| libtmux | 0.37+ | tmux session management |
|
||||
|
||||
@@ -28,6 +28,8 @@ camera = "1.6.1"
|
||||
play-publisher = "4.0.0"
|
||||
media3 = "1.10.1"
|
||||
androidVad = "2.0.10"
|
||||
sherpaOnnx = "v1.13.4"
|
||||
onnxRuntime = "1.27.0"
|
||||
spatialsdk = "0.13.2"
|
||||
play-app-update = "2.1.0"
|
||||
|
||||
@@ -105,6 +107,8 @@ meta-spatial-sdk-mruk = { group = "com.meta.spatial", name = "meta-spatial-sdk-m
|
||||
|
||||
# android-vad (Silero) — on-device voice activity detection for barge-in (B2)
|
||||
android-vad-silero = { group = "com.github.gkonovalov.android-vad", name = "silero", version.ref = "androidVad" }
|
||||
sherpa-onnx = { group = "com.github.k2-fsa", name = "sherpa-onnx", version.ref = "sherpaOnnx" }
|
||||
onnxruntime-android = { group = "com.microsoft.onnxruntime", name = "onnxruntime-android", version.ref = "onnxRuntime" }
|
||||
|
||||
# Google Play In-App Update — googlePlay flavor ONLY (FLEXIBLE flow).
|
||||
# Sideload uses the GitHub releases UpdateChecker instead and must NOT pull
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# sherpa-onnx and English keyword-spotting model
|
||||
|
||||
Hermes-Relay's optional experimental Android wake-word feature uses:
|
||||
|
||||
- **sherpa-onnx v1.13.4**, Copyright 2024 Xiaomi Corporation and sherpa-onnx
|
||||
contributors, licensed under the Apache License 2.0:
|
||||
<https://github.com/k2-fsa/sherpa-onnx/blob/v1.13.4/LICENSE>
|
||||
- **sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01**, published by the
|
||||
sherpa-onnx project for English keyword spotting. The upstream model listing
|
||||
identifies the model as Apache License 2.0:
|
||||
<https://www.modelscope.cn/models/pkufool/sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01/summary>
|
||||
|
||||
The application downloads only the int8 encoder, decoder, int8 joiner, and
|
||||
token table when the user explicitly enables wake word. Downloads use the
|
||||
publisher's `resolve/master` URLs and are pinned in
|
||||
`WakeWordModelInstaller.kt` by byte length and SHA-256:
|
||||
|
||||
| File | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx` | 4,807,159 | `1e721676515bcd42a186979733981213c66c80db680e1cc582dfedf3be76e678` |
|
||||
| `decoder-epoch-12-avg-2-chunk-16-left-64.onnx` | 1,063,189 | `f61ebd3eed3773a44d088d53dfae92dbb6aec4839f4dcaee2d402414741663a3` |
|
||||
| `joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx` | 163,380 | `eae9da0c7e1e6c6a3f4cc42d167899c388f6c6701b94cb96320e4f55df79624c` |
|
||||
| `tokens.txt` | 5,006 | `fd2ded4050a55d2b1578870ba8697d02371980217806b7558bd0a5cc60f3ba53` |
|
||||
|
||||
The sherpa-onnx project notes that model licenses can differ from the runtime
|
||||
license. Redistribution and production release of these weights must therefore
|
||||
retain this provenance record and be reviewed independently of the
|
||||
sherpa-onnx runtime dependency.
|
||||
|
||||
## Apache License 2.0
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
|
||||
these works except in compliance with the License. You may obtain a copy at:
|
||||
|
||||
<https://www.apache.org/licenses/LICENSE-2.0>
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed
|
||||
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
|
||||
CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
@@ -177,7 +177,11 @@ These controls always show and apply to both engines:
|
||||
|
||||
### Barge-in
|
||||
|
||||
New as of 2026-04-17. Lets you interrupt the agent by speaking while it's replying — the same turn-taking pattern ChatGPT and Siri use. **Default off** because echo-cancellation quality varies widely across Android phones; opt in once and the setting persists.
|
||||
Lets you interrupt the agent by speaking while it is thinking or talking. One
|
||||
listener stays active across the whole response, so the microphone does not
|
||||
re-arm between generation and playback. **Default off** because
|
||||
echo-cancellation quality varies across Android phones; opt in once and the
|
||||
setting persists.
|
||||
|
||||
- **Interrupt when I speak** — master toggle. Default off.
|
||||
- **Sensitivity** — `Off / Low / Default / High`. Higher values fire on quieter / shorter speech. Start with Default; drop to Low if your phone false-triggers on its own TTS, raise to High if you find yourself having to speak up.
|
||||
@@ -185,7 +189,40 @@ New as of 2026-04-17. Lets you interrupt the agent by speaking while it's replyi
|
||||
|
||||
**Device compatibility.** If your phone doesn't support hardware echo cancellation (`AcousticEchoCanceler`), you'll see a warning badge next to the master toggle: *"Your device may have limited echo cancellation. Barge-in quality will vary."* You can still enable barge-in, but expect more false triggers from the phone's own speaker feeding back into the mic. **Using headphones fixes this entirely** — the mic never hears the TTS output, so VAD has nothing to confuse.
|
||||
|
||||
**How it feels in practice.** As soon as you start speaking, the agent's voice briefly ducks in volume (about 30 %) — that's the app acknowledging "I think I heard something" before committing. If you keep speaking, it stops entirely within a fraction of a second and you're back in recording mode. If it was a false trigger (one stray frame of background noise), the volume pops back up to full after ~500 ms with no interruption.
|
||||
Before playback, the app samples the room noise and freezes that calibration so
|
||||
the phone's speaker cannot teach the detector the wrong noise floor. Playback
|
||||
also has a short grace period. As soon as you start speaking, the agent's voice
|
||||
briefly ducks in volume (about 30%). Sustained, model-confirmed speech stops the
|
||||
active generation or playback and captures your replacement request. A false
|
||||
trigger returns to full volume after about 500 ms.
|
||||
|
||||
After an interruption, exact “stop” or “pause” utterances control the active
|
||||
voice response. Longer requests such as “stop the container” still go to the
|
||||
agent normally. Stopping speech does not cancel a promoted background task; use
|
||||
the task's explicit cancel action or say the explicit background-task
|
||||
cancellation command.
|
||||
|
||||
### Experimental wake word
|
||||
|
||||
Under **Voice Settings → Listening**, enable **Listen for “Hey Hermes”** to use
|
||||
the Android-local wake-word preview. It is off by default. The first enable
|
||||
downloads and verifies an English keyword model of about 6 MB. The APK includes
|
||||
the sherpa-onnx runtime but not the model.
|
||||
|
||||
Detection runs on the phone: microphone audio is not uploaded before the phrase
|
||||
activates voice. Android requires a user-started microphone foreground service,
|
||||
shows an ongoing privacy notification, and provides a **Stop** action. The
|
||||
listener does not start at boot. When the phrase is detected, the wake listener
|
||||
releases the microphone before the normal voice flow records anything, stays
|
||||
paused for the voice session, then resumes after voice exits.
|
||||
|
||||
The initial preview supports one phrase, “Hey Hermes.” **Strictness** controls
|
||||
false activations (higher is stricter), **Confirmation frames** controls how
|
||||
many matching decoder frames are required, and **Start a new session** chooses
|
||||
between a fresh chat and the selected profile's current session.
|
||||
Profile-specific wake phrases and routing are not supported yet. Continuous
|
||||
local listening uses additional battery and still needs device-specific
|
||||
acoustic testing.
|
||||
|
||||
### Hermes Chat + Voice Output
|
||||
|
||||
|
||||
Reference in New Issue
Block a user