Hermes-Relay Voice Lab

Realtime Voice / Streaming TTS CLI Readme

Use this provider lab to test voice providers, save WAV output, capture JSONL event logs, compare expression settings, and separate deterministic streaming TTS renderers from realtime voice-agent modes.

Standalone CLI + relay voice routes OpenAI + Grok + ElevenLabs E2E approach eval Streaming TTS target Realtime agent test mode Outputs ignored by git

Android Dev Build Test

Start the relay voice routes

The Android dev path uses /voice/output/* for exact assistant speech and keeps /voice/realtime/* for provider-agent lab testing. Existing /voice/config, /voice/transcribe, and /voice/synthesize stay available as basic fallback utilities.

.\scripts\voice-relay-dev.ps1 -Provider grok

Pair the Android Studio dev build

Use the normal relay pairing flow so the app has a relay session token with the voice:realtime grant.

Open voice mode or the dev testbench

The main chat voice overlay uses the relay voice output path when available. For isolated provider checks, open Settings > Voice. Voice Output exposes profile-scoped provider/model/voice dropdowns from relay metadata, refreshes the selected provider's options on the relay before saving, and keeps advanced manual entry plus Save & test. The experimental realtime provider editor is shown only when Developer options are unlocked. For the standalone dev testbench, open Settings > Developer options > Realtime voice lab.

Fast Path

Check the local setup

Run this from the repo root.

cd C:\Users\Bailey\Desktop\Open-Projects\hermes-relay
python -m pip install -e ".[voice-lab]"
.\scripts\voice-lab.ps1 -Mode doctor

List providers

.\scripts\voice-lab.ps1 -Mode providers

Run a free local smoke test

.\scripts\voice-lab.ps1 -Provider stub

Run Grok Voice Agent

SuperGrok/Premium+ works through the lab-owned xAI OAuth flow, or you can use a direct xAI API key.

.\scripts\voice-lab.ps1 -Mode auth -Provider grok
.\scripts\voice-lab.ps1 -Provider grok -Text "Hermes voice lab online."

Run OpenAI Realtime

This spends API quota. The launcher uses VOICE_TOOLS_OPENAI_KEY or OPENAI_API_KEY from the environment or VOICE_LAB_HOME\.env.

.\scripts\voice-lab.ps1 -Provider openai -Text "Hermes voice lab online."

Run the E2E approach eval

This compares realtime providers and streaming TTS with the same acknowledgement, simulated tool wait, pronunciation, and volume-consistency prompts.

.\scripts\voice-lab.ps1 `
  -Mode eval `
  -Providers "xai_realtime,openai_realtime,elevenlabs_tts,stub" `
  -OutputDir voice-lab-runs\e2e `
  -Visual off `
  -Json

What The Launcher Does

scripts\voice-lab.ps1 wraps python -m plugin.voice_lab with test-friendly defaults. It picks a provider, creates an output directory, passes expression settings, and writes run artifacts.

Option	Default	Use
`-Mode`	`run`	`doctor`, `providers`, `auth`, `tui`, `run`, `test`, `stt`, `s2s`, `bench`, or `eval`.
`-Providers`	`xai_realtime,openai_realtime,elevenlabs_tts,stub`	Provider matrix for `-Mode eval`.
`-Provider`	`auto`	Uses Grok when lab-owned xAI OAuth or an xAI key is available, then OpenAI, otherwise `stub`.
`-OutputDir`	`voice-lab-runs`	Where WAV files and JSONL event logs are written.
`-InputAudio`	none	Required for `-Mode stt` and `-Mode s2s`.
`-Visual`	`on`	Uses the Textual TUI for interactive `-Mode tui` sessions and shows the PCM waveform meter for one-shot runs.
`-Play`	`false`	Plays the generated WAV from the terminal after the provider run completes.
`-Model`	`gpt-realtime-2`	OpenAI Realtime model override.
`-Voice`	`marin`	OpenAI Realtime voice override.
`-XAIModel`	`grok-voice-latest`	xAI/Grok Voice Agent model override.
`-XAIVoice`	`eve`	xAI/Grok Voice Agent voice override.
`-ElevenLabsModel`	`eleven_flash_v2_5`	ElevenLabs streaming TTS model override.
`-ElevenLabsVoice`	`JBFqnCBsd6RMkjVDRZzb`	ElevenLabs voice ID override.

Common Commands

Persistent TUI lab

.\scripts\voice-lab.ps1 -Mode tui -Provider grok

Grok with SuperGrok OAuth

.\scripts\voice-lab.ps1 -Mode auth -Provider grok

.\scripts\voice-lab.ps1 `
  -Provider grok `
  -Text "Testing Grok realtime speech output."

Grok with a one-off xAI key

.\scripts\voice-lab.ps1 `
  -Provider grok `
  -XAIKey "xai-..." `
  -Text "Testing Grok realtime speech output."

Grok with tool-call event scaffolding

.\scripts\voice-lab.ps1 `
  -Provider grok `
  -ProviderOption tool_scaffold=true `
  -Text "Run a voice tool-call test."

OpenAI with a one-off key

.\scripts\voice-lab.ps1 `
  -Provider openai `
  -OpenAIKey "sk-..." `
  -Text "Testing realtime speech output."

OpenAI with a different voice

.\scripts\voice-lab.ps1 `
  -Provider openai `
  -Voice marin `
  -Tone calm `
  -Intensity 0.35 `
  -Text "Read this in a precise, steady tone."

ElevenLabs streaming TTS

.\scripts\voice-lab.ps1 `
  -Provider elevenlabs `
  -ElevenLabsKey "..." `
  -Text "Testing crisp streamed TTS output."

E2E approach evaluation

.\scripts\voice-lab.ps1 `
  -Mode eval `
  -Providers "xai_realtime,openai_realtime,elevenlabs_tts,stub" `
  -OutputDir voice-lab-runs\e2e `
  -Visual off `
  -Json

Disable the waveform

.\scripts\voice-lab.ps1 -Provider grok -Visual off

Play the rendered WAV

.\scripts\voice-lab.ps1 -Provider grok -Text "Play this after rendering." -Play

Emit JSON for automation

.\scripts\voice-lab.ps1 -Provider stub -Json -Visual off

STT scaffold

.\scripts\voice-lab.ps1 `
  -Mode stt `
  -Provider stub `
  -InputAudio voice-lab-runs\sample.wav `
  -ExpectedText "expected transcript"

Speech-to-speech scaffold

.\scripts\voice-lab.ps1 `
  -Mode s2s `
  -Provider stub `
  -InputAudio voice-lab-runs\sample.wav `
  -Text "I heard the sample and am responding."

Batch Runs

Create a text file with one prompt per non-empty line, then run -Mode bench.

.\scripts\voice-lab.ps1 `
  -Mode bench `
  -Provider grok `
  -ScriptFile .\scripts\voice-lab-phrases.txt `
  -OutputDir voice-lab-runs `
  -Visual on

Event logs are written under voice-lab-runs\events unless -NoEventLog is passed.

TUI Mode

The persistent TUI uses a Textual-backed, PulseForge-style keyboard interface in an interactive terminal. Piped runs, -Json, and -Visual off fall back to the simpler console loop so automation stays clean.

.\scripts\voice-lab.ps1 -Mode tui -Provider grok

Reuse provider auth across turns.
Tab through dropdowns for provider, model, voice, emotion, tone, pace, style, and command execution.
Use footer keys: F2 provider, F3 command, F4 prompt, F5 emotion, F6 tone, F7 tool, F8 play, F9 replay, F10 rerun, Ctrl+Q quit.
Use the command dropdown for run, replay, rerun, save, log, open log, provider option, play, tool, JSON, help, prompt focus, and quit.
Show generated PCM waveform, latency, provider status, and artifact paths per turn.
Fallback command mode still supports :provider, :voice, :model, :emotion, :replay, and :save.
Keep non-interactive -Json -Visual off commands stable for automation.

Voice Output Framing

Normal assistant narration should be a speech renderer: final Hermes text in, exact audio out. Realtime voice-agent providers stay separate for speech-to-speech, provider turn-taking, and tool-event experiments.

streaming_tts_renderer: primary target for Grok TTS, OpenAI TTS, ElevenLabs, Hermes fallback, and short tool-status lines.
realtime_agent: provider-agent mode for Grok/OpenAI Realtime experiments and non-executing tool-call scaffolds.
Hermes remains the owner of chat text, tool execution, approvals, and final answers.
Android sends the active profile to voice config/session routes so named profiles can keep separate voice_output and realtime_voice choices.
Provider option refreshes stay server-side, return grouped/searchable voice metadata, and validate model/voice/sample-rate compatibility before saving.

Status

The Android voice overlay uses relay-mediated streaming TTS for deterministic speech rendering and keeps realtime providers as voice-agent test modes.

Current Android dev voice uses /voice/output/config, /voice/output/session, and /voice/output/{session_id} when available.
Basic fallback remains /voice/config, /voice/transcribe, and /voice/synthesize.
OpenAI Realtime and xAI/Grok Realtime are live provider adapters.
ElevenLabs streaming TTS is a live cascaded-voice comparison adapter.
xai_tts and openai_tts are first-class streaming TTS renderers.
xAI custom voices are fetched with pagination and option results are cached briefly by the relay; OpenAI voices remain static from official docs.
stt and s2s are standalone lab scaffolds, not relay route changes.
stub is a local deterministic WAV generator.

Credentials

The OpenAI provider reads VOICE_TOOLS_OPENAI_KEY first, then OPENAI_API_KEY. Both can live in the shell environment or VOICE_LAB_HOME\.env.

VOICE_TOOLS_OPENAI_KEY=sk-...

The Grok provider supports lab-owned xAI OAuth from VOICE_LAB_HOME\auth\xai-oauth.json. It also accepts VOICE_TOOLS_XAI_KEY, XAI_API_KEY, or an ephemeral xAI realtime token.

.\scripts\voice-lab.ps1 -Mode auth -Provider grok
VOICE_TOOLS_XAI_KEY=xai-...

The ElevenLabs provider reads VOICE_TOOLS_ELEVENLABS_KEY first, then ELEVENLABS_API_KEY.

VOICE_TOOLS_ELEVENLABS_KEY=...

Do not commit API keys. Do not scrape Grok.com or X browser/session cookies; use the lab-owned OAuth flow for subscription auth.

References: xAI TTS, xAI Voice Agent, OpenAI TTS, OpenAI Realtime, xAI OAuth discovery, and xAI SuperGrok surface announcement.

Waveform

The CLI shows a compact terminal meter from generated PCM chunk RMS levels. Before audio arrives, it idles low so provider connection state is still visible.

Artifacts

*.wav contains the rendered audio.
*.jsonl contains raw provider event summaries.
summary.json, runs.jsonl, and report.md are written by -Mode eval.
-Play plays the rendered WAV after the run.
voice-lab-runs/ and realtime-voice-runs/ are ignored by git.

Troubleshooting

Run .\scripts\voice-lab.ps1 -Mode doctor first.
If Grok fails immediately, run .\scripts\voice-lab.ps1 -Mode auth -Provider grok, or set XAI_API_KEY / VOICE_TOOLS_XAI_KEY.
If OpenAI fails immediately, verify the key exists and is not committed.
If provider output is noisy, use -Visual off.
If automation needs clean output, use -Json -Visual off.

Source docs: realtime-voice-poc.md. Current standalone package: plugin.voice_lab.