Second step of the HTTPS front, deliberately its own generation: pointing
ROOT_URL at an HTTPS name that served nothing would have broken the UI's
absolute links. The vhost (40ae3e00) is proven — ACME issued over HTTP-01 and
https://gitea.wingard.pro/ serves 200 with a chain that validates against the
system trust store, so it is a real Let's Encrypt cert rather than self-signed.
Measured before flipping: Gitea already answers the registry Bearer realm on
https, because it derives the scheme from X-Forwarded-Proto. So Docker's auth
flow was already fully TLS and this flip is for the web UI's own links, so
browsing over https stops emitting http URLs.
DOMAIN stays 10.0.1.49: it only affects newly copied SSH clone URLs, and
existing remotes should not churn.
Adds a Let's Encrypt-backed vhost in front of Gitea's port 3010, which carries
both the web UI and the Docker registry (/v2/). This is what lets containers
push and pull the local image WITHOUT insecure-registries: that daemon option
needs a config change plus a daemon restart on every client that pulls,
including the arch-vm guest, and what it really buys is skipping cert
verification. A real cert on a name clients already resolve needs no client
configuration at all. Body and timeout limits are raised because image layers
are GB-scale.
Also adds gitea.wingard.pro to networking.hosts and to the Pi-hole records: the
host's own resolver goes to Comcast (*.wingard.pro -> WAN) and the router does
not hairpin, so the host cannot reach its own services by name.
ROOT_URL is deliberately still the plain-HTTP address; it flips to the HTTPS
name in a follow-up once the certificate is proven to issue and serve.
Two unrelated things land together because they were already in the same working tree.
Mine (this session): 8770 added to networking.firewall.interfaces.wg0.allowedTCPPorts
so the WireGuard-only front door of the gpu-cockpit console is reachable from the
phone. Deliberately NOT in the global allowedTCPPorts list - it follows the same
pattern as 8790 (Roman's WebUI), which is scoped to wg0 for the same reason.
Pre-existing, uncommitted when I got here (file mtime 00:02 today), committed rather
than left dangling so the tree is not silently dirty during a rebuild:
- the SillyTavern reverse proxy vhost on 8744, including the libvirt/:8000 hop and the
duplicate-forwarded-header / IP-whitelist traps documented inline
- 8744 added to both the wg0 and eno1 firewall lists
I have not reviewed the SillyTavern half; it is recorded here so the history matches
the file that gets built.
Serve the roleplay model to the rp-tavern libvirt guest on virbr0:11434,
scoped to the bridge so 11434 is not exposed on eno1 or wg0.
CPU-only: --device none is load-bearing. Default device selection offloads the
mmproj vision projector onto the 4060 (~1.8 GB) even with -ngl 0, so the model
looks CPU-resident while 1.8 GB of the card is gone. Measured 3237 MiB used
with the projector on the card vs 1436 MiB with --device none, vision intact.
Other choices: 32K context / single slot; --chat-template-kwargs disables
Gemma-4 thinking, which otherwise spends the whole token budget in the
reasoning channel and returns empty content. Not enabled -- ~17 GB resident,
start on demand.
Measured in service: 128K decodes at 44.7 short / 42.9 after a 6.7K prompt,
against 44.5/43.1 at 64K, at the same VRAM. Doubling the window is free here.
256K costs ~30% with q8 V-cache but only ~13% with -ctv q4_0.
Corrects the earlier standalone llama-cli probe, which reported 18 tok/s at
128K and was wrong below 256K -- context cost must be measured in service.
Order the unit after llama-server-local and speech-to-speech. Conflicts= stops
them but adds no ordering, so --fit could sample free VRAM while the voice
stack was still releasing it and under-allocate: measured 16.7 tok/s stale vs
42-46 tok/s with a free card, a 2.7x swing from ordering alone.
-t 16 --no-reasoning-preserve: thread count measured as non-bottleneck
(t8 42.3 / t16 45.1 / t24 43.4 / t32 44.3, within noise); reasoning stays on
because 3B active params benefits from it, but the trace is not carried across
turns, keeping prior-turn thinking out of a 64K window.
Second local llama-server unit for heavier agent work (35B MoE, 3B active),
alias local-qwen35, 64K context at ~32 tok/s on the 8GB RTX 4060.
unitConfig.Conflicts against llama-server-local and speech-to-speech because
the card cannot hold this model and the voice stack simultaneously; the
switch is explicit and reversible rather than a silent CUDA OOM. Not enabled
at boot - a boot-time winner between the two would be nondeterministic.
LimitMEMLOCK=infinity pairs with `-lm mmap+mlock`: llama.cpp falls back to
pageable memory silently when the lock fails.
- mnemosyneSitePkgs listed only top-level paths, so fastembed's runtime deps
were never on the interpreter path: vector search was dead (vec_* empty),
recall silently FTS5-only.
- extraPythonPackages cannot be used: fastembed's closure reaches
huggingface-hub, which the sealed venv ships, and the wrapper's collision
guard rejects that build.
- hermes-desktop.service and hermes-webui.service spawn the venv python with
PYTHONPATH=/opt/hermes-agent-standalone only, so Mnemosyne never
instantiated there (warning every turn since 2026-09-10). The desktop now
reads environment.etc."hermes-desktop-env" via a drop-in EnvironmentFile;
the webui gets the same string inline.
- nginx vhost joplin.wingard.pro:8444 + ACME HTTP-01 vhost, real Let's Encrypt cert
- systemd.services.joplin: docker compose stack (joplin/server 3.7.2 + Postgres 16), /opt/joplin
- firewall: 8444 on wg0 + eno1 only (8444 is NOT port-forwarded)
- Pi-hole record: 10.0.1.49 joplin.wingard.pro
- networking.hosts: this host now resolves its own *.wingard.pro service names. It queries
Comcast DNS directly, never its own Pi-hole, so the names resolved to the WAN IP and the
LAN-only ports refused the connection (joplin CLI: ECONNREFUSED 71.61.169.167:8444).
- environment.systemPackages: joplin-cli -- the headless agent interface. Joplin's REST Data
API on :41184 is hosted by the DESKTOP app and does not exist on a headless server.
Verified: gen ri08579xg21bxwkgssw5sjn5yzzhljf8 live; /api/ping 200 with a verified TLS chain
using the plain service name; 2 notes + 1 notebook synced from the CLI and confirmed as rows
in the server's items table.
The hermes-agent input was locked to the narHash of a shallow fetch
(revCount 1). After nix-collect-garbage reaped the cached source store
path, the next re-fetch produced the full-history tree hash and the lock
failed to validate, breaking all nixos-rebuild evaluation.
Input repo unshallowed; lock re-resolved against the same rev
(d337b736aa1e8ebecfab043842d13e4a2d2f48a3) - no version change.
Adds systemd.services.immich: oneshot running 'docker compose up -d' from
/opt/immich, wired the same way as karakeep/pia-qbit.
requires = [ "unlock-8tb.service" ] deliberately, NOT storage-8TB.mount: that
unit is fragmentless on this host, so a Requires= on it can never be satisfied
at boot and the unit is silently skipped instead of failing loudly (the bug that
killed three units' autostart until 2026-09-16).
Immich's database AND media both live on the 8TB pool, so RequiresMountsFor
covers the mount-shadow trap -- without it a boot with the pool absent would
resolve the bind sources onto the root filesystem and start Immich against empty
storage with a fresh database.
A 24h audit found ~2,350 failed root logins/day from a rotating botnet,
accounting for ~75% of all journal error lines on the host. sshd already
refuses every one (PasswordAuthentication = false; the generated PAM auth
stack is a bare pam_deny), so this was never an intrusion risk -- it was
unbounded retries plus log spam, with no ban layer at all since crowdsec
was removed on 2026-09-12 for being silently inert.
bantime = 1h, maxretry = 3, findtime = 3600
bantime-increment on (rndtime 8m, maxtime 48h)
ignoreIP = every trusted route (LAN, WireGuard, Palantir, libvirt, docker)
findtime is deliberately 1h rather than the 600s default: measured traffic
shows most attacking IPs do only 1-4 attempts per 10 minutes but 8-26 per
hour, so the default window let the drip through. Simulated on the real 24h
sample, coverage goes from 90% (4/600) to 98% (3/3600).
banaction stays at the module default -> iptables-multiport, since
networking.nftables.enable is false on this host.
RemainAfterExit=true left the unit active(exited), so the timer's plain Start was
a no-op and the 15-min re-assert never ran. Dropped it (unit now ends inactive)
and switched OnUnitActiveSec -> OnCalendar='*:0/15', which is wall-clock and the
right pairing for a oneshot that exits.
Caught by re-testing with the primitive the timer actually uses (systemctl start,
not restart) and by simulating SAB faithfully (os.chmod syscall on the release dir
too, not a shell mkdir under umask 022).
SAB applies permissions=775 via os.chmod(), a bit-exact syscall that clears
setgid. A category dir left in SAB's primary group makes every release folder
inside inherit 'sabnzbd' instead of 'media', so sonarr/radarr can read+copy but
cannot unlink the source -- unlink needs write+execute on the parent dir -- and
imports die with UnauthorizedAccessException, churning every ~90s.
Oneshot + 15-min timer re-asserts chgrp media + chmod 2775 on complete/ and each
category dir beneath it. Requires unlock-20tb.service (pool is fragmentless, no
.mount unit). Deliberately not a tmpfiles 'd' rule (root-fs shadowing risk).
Measured journal volume on this host is ~19.5 MB/h, so 1G held under two days of
history and left the journal pinned at its ceiling (986M/37.9M free on 09-17).
On 2026-09-17 01:33-08:40 a runaway container (hexstrike, removed that day) logged
~150k msgs/h through docker's journald driver, 99.8% of that boot's docker volume,
and in that window systemd SIGABRT'd journald twice for missing its 3-minute
watchdog. Flood source gone; this removes the cap pressure. 4G ~= 8 days of
history. Mechanism not fully pinned (no assertion in the cores) - the comment
block records the evidence and the open question.
The live speech-to-speech stack spoke with bf_emma (UK) while every generated
episode and voice reply uses af_bella (US). Harley asked for one consistent
voice everywhere. Both flags move together: af_* voices need lang code 'a'.
Adds the two services that give hermes-live-voice a local `local` provider,
so the realtime path no longer needs an OpenAI/Gemini key.
llama-server-local : Qwen3-4B-Instruct-2507 Q4_K_M on :8080, OpenAI-compatible
speech-to-speech : Silero VAD -> Parakeet TDT -> llama-server -> Kokoro TTS
on ws://127.0.0.1:8765/v1/realtime
Hermes remains the agent brain; the realtime model only does turn-taking and
calls the gateway tools. Notable constraints, all verified on the box:
- llama-cpp-cuda (new systemPackage) with --jinja, required for tool calls.
- LD_LIBRARY_PATH needs BOTH /run/opengl-driver/lib (libcuda; without it torch
reports cuda_available=False and silently uses CPU) and gcc's libstdc++
(pip wheels have no RPATH, Nix binaries do).
- Context 4096 + q8_0 KV and Kokoro pinned to CPU to fit an 8 GB RTX 4060
alongside the desktop: ~276 + ~1990 + ~2870 MiB, ~1170 MiB free.
- Both units require unlock-8tb.service (never storage-8TB.mount) as the
weights live on /storage/8TB.
Measured: STT -> LLM -> TTS round trip ~2s.
Second WebUI instance pinned to Roman's own profile via
HERMES_WEBUI_ISOLATED_PROFILE, running as the roman system user with
HERMES_HOME outside Harley's shared .hermes home. Firewall scoped to
wg0 + eno1 like Vaultwarden. No new packages: reuses the existing
hermes-agent venv.
storage-8TB.mount has no fragment (/storage/8TB is deliberately not a
fileSystems entry) and only exists once unlock-8tb.service has mounted the
pool -- after the boot transaction is built. A hard Requires= on it created
no start job, so pia-qbit, karakeep and docker-camofox-browser were silently
discarded at every CLEAN boot: enabled, no journal entry, no failed unit.
Require unlock-8tb.service instead: it has a fragment, is satisfiable at
boot, and still fails LOUDLY if the pool never unlocks.
The Exos peer is the only wg0 peer whose endpoint is a hostname
(briggs.wiki:51820), so its generated start script needs DNS resolved at unit
start. network-online.target is hollow on this host: NetworkManager-wait-online
is force-disabled for boot speed, so the target is "reached" instantly and the
peer unit raced the network. On 2026-09-14 23:23 it lost that race, exited 1,
and Type=oneshot + Restart=no made the loss permanent: /storage/Exos stayed
dead for 11h, the CIFS automount failed on every trigger, and the no-SSH
transport to Roman went blind with it (only noticed when Harley tried to SSH
in and could not).
pihole-ftl-setup.service exited 1 in the same second for the same reason (it
curls ftl.pi-hole.net and talks to the FTL API before the network is usable)
and had sat in `systemctl --failed` ever since.
- wg0 Exos peer unit: Restart=on-failure + RestartSec=5s. Legal on Type=oneshot
(only Restart=always/on-success are forbidden), so it retries ~every 6s until
DNS answers instead of dying once and staying dead. No boot delay, and
NetworkManager-wait-online stays disabled as Harley intended.
- pihole-ftl-setup: Restart=on-failure + RestartSec=20s, deliberately bounded
by StartLimitIntervalSec=600 + StartLimitBurst=5, so a permanently broken run
cannot hammer `pihole -g` blocklist downloads.
Verified after switch (generation
/nix/store/dhgrxdiz4zg14z8qr71xvlb64yr7lvz4-nixos-system-utumno-26.11.20260910.8ce4ef6):
switch rc=0, `readlink -f /run/current-system` moved, generated units read back
with Restart=on-failure / RestartSec=5s (peer) and StartLimitBurst=5 +
RestartSec=20s (pihole), live `systemctl show` matches, `systemctl --failed`
lists 0 units, wg0 back to 5/5 peers with a live Exos handshake, /storage/Exos
readable, and the no-SSH transport answered a fresh read-only request in
1574ms. pihole-ftl-setup was re-run by hand to clear its stale failure: exit 0,
gravity rebuilt.
Diagnosis in one command, should this ever recur:
/run/wrappers/bin/sudo -n wg show wg0 peers # must list 5 peers
These three stacks bind paths under /storage/8TB. RequiresMountsFor alone produced
only an ORDERING dependency on systemd 261 (verified: -p Requires came back empty
and list-dependencies --reverse listed no dependents), so an absent pool would
still have started them against root-resident shadow directories -- the trap that
filled the 468G root SSD on 2026-09-13. Add an explicit requires= on the mount.
The unit only ordered After=docker.service/network-online.target. With the 8TB
pool absent, the compose bind /storage/8TB/Downloads resolved to a directory on
the root filesystem and filled the 468G root SSD (root hit 0 bytes free 2026-09-13).
Also carries a pre-existing uncommitted change: python312 wrapped with pyserial
so python3 -c 'import serial' works for serial-monitor scripts.
Port 3010 sat in networking.firewall.allowedTCPPorts, which accepts it on EVERY
interface, in BOTH the IPv4 and IPv6 chains. eno1 carries this host's global IPv6
addresses (2601:540:cc03:4760::/64), so the host itself was not what limited
reachability -- and interface scoping would NOT have fixed it either, because
internet traffic arrives on eno1 exactly the way LAN traffic does. Allowlisting the
SOURCE networks is what actually closes it.
Replaced with source-scoped accepts via networking.firewall.extraCommands. Confirmed
against the nixpkgs module source that extraCommands renders BEFORE the final
nixos-fw-log-refuse rule, so the accepts are reachable and not dead code:
127.0.0.0/8 host-local
10.0.1.0/24 LAN
10.10.10.0/24 WireGuard
192.168.2.0/24 Palantir's segment
192.168.122.0/24 libvirt VM network
172.16.0.0/12 docker bridges (host-local only)
Deliberately NO IPv6 accept: Gitea is reached as http://10.0.1.49:3010, and the LAN
IPv6 prefix is globally routable, so any v6 accept would re-open the same hole.
Verified live after switch: iptables `nixos-fw` carries the six scoped accepts at
rules 61-66 directly ahead of the refuse rule at 67; ip6tables `nixos-fw` has NO 3010
rule at all; Gitea still returns HTTP 200 on 127.0.0.1 and 10.0.1.49; `/api/v1/version`
=> 1.27.3; `git ls-remote` over 3010 still works. Switch exit 0.
Diagnostic if this bites: a blocked client appears in the journal as
'refused connection:' -- add its subnet here rather than reverting the scoping.
CrowdSec is gone, at Harley's call. It had been non-functional for days -- the
firewall bouncer never applied bans at nftables, and the agent itself failed on
restart because /var/lib/crowdsec is a symlink to private/crowdsec and a tmpfiles
`d` rule creates a directory but will NOT replace an existing symlink. Rather than
carry a broken IPS, it is removed: activation drops the user/group and the
/etc/tmpfiles.d symlink, and its left-over state (/etc/crowdsec, /var/lib/crowdsec,
/var/lib/private/crowdsec) was deleted by hand.
!! This host now has NO intrusion-prevention layer. That is deliberate.
!! To restore: services.crowdsec + services.crowdsec-firewall-bouncer, and give
!! StateDirectory a REAL directory (tmpfiles type `R`, then `d`) not a symlink.
Two capacity fixes that were owed after the 2026-09-11 collapse:
* nix.settings.max-jobs = 6, cores = 6. The defaults (auto / 0) resolve on this
32-thread host to 32 builders EACH running a fully parallel make -- ~1024
concurrent writers on the single dm-crypt+btrfs root device, which is why no
build could finish (load 100, PSI 92%, 61 D-state, journald killed 34x/8h).
Bound by hand the same advance built clean on 2026-09-12: 1862 derivations,
0 errors, peak PSI ~12%. Now declarative, so nobody has to remember the flag.
* services.journald.settings.Journal.SystemMaxUse = "1G". Uncapped, journald
defaults to 10% of the FILESYSTEM (up to ~46 GiB on this root) on the same
device as /nix/store, and was being watchdog-killed under build I/O.
NOTE: this nixpkgs renamed the option -- `services.journald.extraConfig` now
trips a dead-option assertion, hence the settings form.
Result: the system is now `running` with 0 failed units (was `degraded` with 3-4).
Fixes CVE-2026-60004 (Gitea < 1.27.1 RCE). The new snapshot carries Gitea
1.27.3 -- verified live after activation, /api/v1/version -> 1.27.3, web UI
HTTP 200. Registration is now closed (DISABLE_REGISTRATION = true), which
shuts the account-acquisition path the CVE needs.
HOLDS: new `nixpkgs-held` flake input pinned to rev 241313f -- the revision
the running system was built with -- so held packages reproduce their
already-built derivations byte-for-byte and Nix reuses the existing outputs:
* sunshine: upstream deleted the pinned tag v2026.826.1804 (unfetchable)
* rpcs3: new snapshot's 0.0.42-unstable-2026-08-15 fails with a GCC ICE
Both verified before switching: `nix-store -q --deriver <installed path>`
matched `nix eval .drvPath` exactly for each, so neither package recompiled.
(Sunshine's security-wrapper and unit file ARE regenerated by the new
nixpkgs, so the user unit restarts on activation; the binary and its
behaviour are unchanged.)
BUILD METHOD: bounded with --max-jobs 6 --cores 6. The 2026-09-11 attempt
used max-jobs=auto (32 builders) x cores=0 (each builder fully parallel on
32 threads) -> ~1024 concurrent writers on the single dm-crypt+btrfs root
device, which collapsed it: load 100, PSI 92% some / 88% full, 61 tasks in
D-state, systemd-journald killed 34x in 8h, and the build could not finish.
This run: peak PSI ~12%, D-state 0, 1862 derivations, 0 errors.
NOTE: `nixpkgs-held` also removes the reliance on a previous attempt's
unrooted store paths. Those are GC-able (a plain nix-collect-garbage reaped
them on 2026-09-12, resetting an apparent "0 to fetch" back to a full 8.3
GiB download). Keeping the old revision as a first-class flake input makes
the reuse explicit and reproducible instead of incidental.
The stealth-browser fallback container was created by hand with docker run,
so no rebuild could reproduce it and browser-profile state was discarded on
every recreate. Declare it under virtualisation.oci-containers (docker
backend) instead:
- image camofox-browser:152.0.4-x86_64, pull = never (locally built, no registry)
- ports 9377:9377 on 0.0.0.0 (ESP32 status dashboard probes it over the LAN)
- volume state -> /root/.camofox so sessions survive a recreate
- extraOptions --restart=unless-stopped, matching every other Docker workload
- autoRemoveOnStop = false: --rm conflicts with --restart (docker refuses both)
Verified: docker-camofox-browser.service enabled+active, systemd restart
recreates the container end-to-end, /health + LAN 200, 15/15 REST checks.
Gives the agent a real CDP browser backend. browser_exec (Browser Use CLI)
resolves browser.cdp_url before falling back to a local Chrome, so this
service is what makes it work on this host. Loopback-only; Camofox on 9377
stays the stealth fallback for anti-bot pages.
SIGNUPS_ALLOWED true -> false. The setting was explicitly documented as
temporary ("open only until the first account is registered, then set false
and rebuild"); the account now exists. INVITATIONS_ALLOWED was already false,
so no new account can be created without an invite.
Verified live: built env file SIGNUPS_ALLOWED=false; /api/config ->
settings.disableUserRegistration=true.
- services.vaultwarden on 127.0.0.1:8222, sqlite, DOMAIN includes :8443
- nginx vhost vault.wingard.pro listens on 8443 only, which is NOT
port-forwarded; firewall opens 8443 on eno1 + wg0 only, so the vault is
reachable over LAN and WireGuard but never via the *.wingard.pro wildcard
- separate port-80 "vault-acme" vhost serves ACME HTTP-01 tokens only
- security.acme now issues/renews vault, jellyfin and speedtest certs
automatically, replacing the hand-copied /etc/letsencrypt/live certs that
had no renewal automation (jellyfin's expired 2026-09-23)
- Pi-hole resolves vault.wingard.pro -> 10.0.1.49 for LAN clients
- daily sqlite-consistent backup to /var/backup/vaultwarden
SIGNUPS_ALLOWED is temporarily true so the first account can be registered;
flip to false afterwards.
ssh + linux collections, LAPI on loopback, whitelist for trusted LAN/WG
ranges. tmpfiles rule keeps /var/lib/crowdsec a real directory (the module's
StateDirectory symlink is unwritable through ProtectSystem=strict).
This was already live in the working tree; committing it on its own so the
vaultwarden change stays a separate, reviewable commit.
Tux profile is gone; the dashboard was bound to 10.10.10.1 (WG-only) and referenced
the removed 'tux' profile, so it couldn't start and Hermes-Relay clients got
connection-refused on the LAN IP. Bind 0.0.0.0, run -p nyra, workdir=nyra.
The wg0 interface is now configured via networking.wireguard.interfaces in
configuration.nix (privateKeyFile wg0-key.key). The hand-managed wg0.conf is no
longer read by the system; the old file content is preserved as
wg0.conf.UNUSED-legacy-not-read-by-nixos for reference. The unused legacy
config is intentionally not read by NixOS.
- add zha + zhaquirks (overlay, from prebuilt wheel) + universal-silabs-flasher
+ ha-silabs-firmware-client to services.home-assistant.extraPackages
- put the hass service user in the dialout group
- grant the HA unit access to /dev/ttyUSB* via systemd DeviceAllow (the module's
DevicePolicy=closed otherwise blocks it -> EPERM and ZHA cannot connect)
Also bundles the pre-existing arch-vm wireguard peer rework that was already in
the working tree.
Serve the WebUI from the sealed uv2nix venv (passthru.hermesVenv) with
HERMES_WEBUI_PYTHON pinned so subprocess agent/gateway runs use the venv
python, not system python3. HERMES_WEBUI_AGENT_DIR points at the source
checkout for run_agent.py + hermes_cli. Localhost-only bind (sessions+memory).
Harley