10 Commits
Author SHA1 Message Date
nyra 2f4614f353 android: Compose client for the cockpit, over WireGuard only
Built by Codex (gpt-6-luna) from android/API-CONTRACT.md and android/AGENTS.md; reviewed,
verified and fixed by Nyra.

The ten-item v1: settings with a Test-connection ping, the card gauge keyed to
pct_of_usable, per-pipeline state/health including the external-holder explanation,
switch/stop/restart with confirmation, the protected 409 -> explicit force path, model
pickers that post their selection, job following that shows the server's own steps,
free-the-card, 3s refresh plus resume and pull-to-refresh, and 401/tunnel-aware errors.

Verified, not taken on trust: ./gradlew --rerun-tasks testDebugUnitTest assembleDebug ->
BUILD SUCCESSFUL, test XML tests=1 skipped=0 failures=0 errors=0, and all 40 tasks
executed rather than reported up-to-date.

Fix applied on review: Job.steps was typed List<String> while the server appends
{"ts","text","ok"} objects, so Gson threw on any job that had taken a step and the UI
surfaced it as a tunnel failure. Now a JobStep type, and the panel shows the step history.
The contract said "plus a steps array" without naming the element type; it now does,
including that steps exist only on /api/jobs/<id>. Also gated the 3s poll on the
foreground so a backgrounded app stops hitting the tunnel.
2026-09-27 16:39:37 +00:00
nyra 6fa69c4a31 android: API contract, agent rules and a real state fixture for the client build 2026-09-27 16:24:14 +00:00
nyra-lab 6029bfed64 open a token-gated front door on the WireGuard address, and only there
The console was loopback-only, which is right for the desktop GUI and the TUI but leaves
the phone with nothing to talk to. --host now takes a comma-separated list and
--token-file names the secret:

- loopback needs no token (it is the machine itself)
- any non-loopback listener requires one, and the server REFUSES TO START if the token
  file is missing, so a front door cannot be opened by accident
- the extra listener retries until its interface exists - wg0 can come up after this
  service does, and a boot race must not look like an auth failure from the phone
- /api/state reports "listeners", so which doors are actually open is visible
- new /api/ping for clients that just want to test the connection

Nine new cases: the token decision table (loopback, ipv6, missing/wrong/right bearer,
query token), token-file reading, and the refusal to open an unauthenticated door.
2026-09-27 16:17:19 +00:00
nyra-lab cb4d250eb2 audit the HTTP reclaim too - it frees VRAM like any command does
The reclaim recipe's HTTP branch called urllib directly and left no row in
audit.jsonl, so the one action that actually frees the card was invisible in the
trail. Runner.note() records side effects that have no subprocess, and the failure
message now goes through friendly_error like every other status line.
2026-09-27 15:27:12 +00:00
nyra-lab 86055b2e69 GUI: stop letting the poll eat button feedback, and shrink the logo
Every button reported its outcome ("job ... accepted", or the error) by writing the
header host line - which render() rewrites on every poll, so the message vanished
inside a second and the buttons looked dead. Notices now have their own element and
linger for 12s.

Also: the logo ships at 160x160 (20 KB) instead of the raw 1024x1024 (523 KB), and a
.gitignore keeps __pycache__/.venv out of the checkout.
2026-09-27 15:26:13 +00:00
nyra-lab eb217ea3e1 make the TUI actually run, and stop leaking developer text into status lines
The TUI defined refresh(), which collides with Textual Widget.refresh(*, repaint).
App.__init__ died on construction, so the terminal front end had never once run -
the lab had no textual installed, so nothing exercised that path. Renamed it to
reload_state, added action_reload for the r binding, and confirmed live frames on
Utumno against textual 8.2.8.

Also in this pass:
- health failures read "connection refused" / "timed out after 3s" instead of
  <urlopen error [Errno 111] Connection refused> (probes.friendly_error)
- the GUI header crops the logo into a rounded badge and gains a favicon
- the process table shows the memory not attributable to any process, so the rows
  reconcile with the card total
2026-09-27 15:24:47 +00:00
nyra-lab 5ad1e02168 switching must not be blocked by a process we did not start
ComfyUI is normally started by the Pictures page, so the cockpit correctly sees it
as external and refuses to kill it - which also blocked every switch. Implicit
stops (the card switch) are now best-effort: report it, leave it running, and let
the reclaim + VRAM wait decide. Explicit stops stay strict and fail loudly.
Three new cases cover it, plus the protected-stop wall and a portable fixture.
2026-09-27 15:18:48 +00:00
nyra-lab 8af2e04866 tests: do not assume the dev VM has no GPU - accept the VRAM verdict (unavailable/wait/below), found by running the suite on the real host 2026-09-27 15:16:33 +00:00
nyra-lab ff11e0189f gpu-cockpit: registry-driven GPU console (server, GUI, TUI, tests)
Built by Codex (nyra-lab, gpt-6-luna, effort high) from the brief in
runs/gpu-cockpit-v1/prompt.md; nyra reviewed the whole tree and fixed three
things before this commit:

- unit probe now asks systemd for LoadState, so a typo'd unit reads
  unknown instead of down
- a protected pipeline can no longer be stopped by a mis-click: explicit
  stop/restart of a live protected pipeline returns 409 and needs force
  (the GUI asks for a deliberate "Force stop"), plus two tests for it
- read-only probes (systemctl show, journalctl, nvidia-smi) still run under
  --dry-run, so dry-run shows real state while changing nothing

Tests: 24 cases pass in the dev VM (no GPU, stub systemd) and on NixOS.
2026-09-27 15:16:20 +00:00
nyra-lab 152db1098e AGENTS.md: repo conventions, commands, boundaries, definition of done 2026-09-27 14:50:08 +00:00
39 changed files with 2783 additions and 1 deletions
+3
View File
@@ -0,0 +1,3 @@
__pycache__/
*.pyc
.venv/
+54
View File
@@ -0,0 +1,54 @@
# gpu-cockpit — agent notes
A registry-driven console for a **single shared inference GPU**. One server owns state and command
execution; the web GUI and the terminal TUI are thin clients of its HTTP API. Nothing in the code
knows about any specific service — that all lives in the TOML registry.
## Layout
- `cockpit/` — the package: registry, runner, GPU probe, unit/proc control, health, state,
arbitration, HTTP server, and the inline GUI under `cockpit/gui/`. **stdlib only.**
- `tui/cockpit.py` — terminal client. The only place `textual` may be imported.
- `tests/`, `scripts/selftest.sh` — fixture-driven; must pass with no GPU and no real systemd.
- `registry.example.toml` — documentation by example. Real registries are deployment data.
## Commands
```bash
python3 -m cockpit --registry registry.example.toml --check # validate, no server
python3 -m cockpit --registry registry.example.toml --port 8770
python3 -m cockpit --registry registry.example.toml --dry-run # executes nothing, reports intent
bash scripts/selftest.sh # the acceptance suite
python3 -m compileall -q cockpit tui
python3 tui/cockpit.py --print # no TTY, no textual needed
```
Run `bash scripts/selftest.sh` before claiming anything works. It is the contract, not a smoke test.
## Boundaries
**Always do**
- Route every executed command through the single runner, so the audit log stays complete.
- Treat a health probe as the only evidence a pipeline is up; a `0` exit from a start command is not.
- Bound every wait, and report a timeout as a failure with the real observed state.
- Bind `127.0.0.1` unless `--host` is passed explicitly.
- Keep registry problems as validation errors/warnings shown in the UI, never a crash at boot.
**Ask first**
- Adding any dependency, including to `tui/`.
- Changing the registry schema or an existing field's meaning.
- Renaming an API route or a key inside `/api/state` (both front ends depend on them).
**Never do**
- `shell=True`, or building a command from an HTTP request's contents. The API accepts pipeline
ids, action names and option ids, and validates each against the registry.
- Stop a `protected = true` pipeline implicitly, or infer it from a `0` exit code.
- Write to the registry file from the API. Selection state is the most the server may write.
- Edit files outside this repository, or assume a deployment host's paths exist.
## Definition of done
1. `bash scripts/selftest.sh` exits 0 and prints one `PASS:` line per case.
2. `python3 -m compileall -q cockpit tui` is silent.
3. `python3 -m cockpit --registry registry.example.toml --check` reports `0 errors, 0 warnings`.
4. The README documents how to add a new pipeline without touching code — and it is true.
+112 -1
View File
@@ -1,3 +1,114 @@
# gpu-cockpit
GPU + pipeline cockpit: registry-driven VRAM/state console with TUI and web GUI
`gpu-cockpit` is a registry-driven console for one shared inference GPU. A single Python server
owns host probes, pipeline state, process/unit control, action jobs and the audit trail. The browser
GUI and terminal TUI are HTTP clients; service-specific details stay in TOML.
## Run
Requires Python 3.11+ (stdlib only for the server and GUI). Copy
[`registry.example.toml`](registry.example.toml) to `~/.config/gpu-cockpit/registry.toml`, then:
```sh
python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml --check
python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml --port 8770
```
The server binds to `127.0.0.1` by default. Set `--host` to bind elsewhere deliberately. State,
logs, selections and audit records go in `~/.local/state/gpu-cockpit` by default. `COCKPIT_REGISTRY`
can provide the default registry path. `--dry-run` serves the API and performs read-only health
checks while recording intended commands without starting or stopping processes.
Open `http://127.0.0.1:8770/` for the responsive web GUI. The TUI uses the same API:
```sh
python3 tui/cockpit.py --url http://127.0.0.1:8770
python3 tui/cockpit.py --print --url http://127.0.0.1:8770
```
Install `textual` to use the interactive TUI (`pip install textual`). `--print` needs no optional
dependency or terminal.
## Add a pipeline
Add another `[[pipeline]]` block in the registry. No Python change is required. Each pipeline needs
one or more components. Components start in listed order and stop in reverse order. A proc component
is spawned and supervised by the server; a unit component is operated through `systemctl`.
`health` can use `http`, `tcp`, `command` or `none`.
For example, a CPU-only local service can run alongside a card owner:
```toml
[[pipeline]]
id = "cpu-llm"
label = "CPU language model"
description = "A local inference endpoint that does not claim the GPU"
requires_card = false
vram_mib = 0
serving = "Small instruct model"
[[pipeline.component]]
kind = "proc"
id = "server"
label = "Inference server"
argv = ["/opt/local-llm/bin/server", "--port", "8090", "{model}"]
cwd = "/opt/local-llm"
env = { OMP_NUM_THREADS = "8" }
ready_timeout_s = 30
stop_timeout_s = 10
health = { kind = "tcp", host = "127.0.0.1", port = 8090, timeout_s = 2 }
[[pipeline.option]]
id = "model"
label = "Model"
kind = "argv"
default = "small"
[[pipeline.option.choice]]
id = "small"
label = "Small model"
vars = { model = "/models/small.gguf" }
```
Placeholders are substituted within argv elements using values from `argv` options. An empty value
drops the entire argv element. Every placeholder must be declared by an option, and every choice
must provide each variable used by that option. Options can also be `note` kind for display-only
choices. Run `--check` before restarting the server; validation problems remain visible in the API
and GUI.
## Safety and operation
- A pipeline with `requires_card = true` owns the exclusive card group. Starting another such
pipeline stops active peers, in reverse component order, before reclaiming and starting it.
- `protected = true` blocks implicit stops. A user must make a deliberate forced switch.
- Starts count as successful only after their configured health probe passes. Timeouts fail the job
and include the recent pipeline log tail. GPU waits are bounded and report unavailable probes or
timeout conditions honestly.
- Only one action job runs globally at a time. Jobs expose progress in `/api/jobs/<id>` and state.
- Proc pidfiles include Linux process start identity; stale files and reused PIDs do not imply an
owned process. An externally started healthy service is shown as external and cannot be killed.
- Executed commands and dry-run intent are recorded under the state directory. The API never edits
the registry; option selection is persisted in `selected.toml`.
## API
| Route | Purpose |
|---|---|
| `GET /` | Web GUI |
| `GET /static/<file>` | Allowlisted GUI assets |
| `GET /api/state` | Full card, component, pipeline, job and validation snapshot |
| `GET /api/events` | Server-sent state events and heartbeat (`?once=1` for one event) |
| `POST /api/pipelines/<id>/action` | Start, stop or restart; returns a job id |
| `POST /api/card/free` | Run the registry reclaim recipe as a job |
| `GET /api/jobs/<id>` | Job state, progress steps and failure detail |
| `GET /api/logs/<id>?n=200` | Tail proc log or unit journal |
| `GET /api/audit?n=100` | Recent command audit records |
| `GET /api/registry` | Parsed registry and validation report |
| `POST /api/pipelines/<id>/options` | Validate and persist option selection |
## Development checks
```sh
python3 -m compileall -q cockpit tui
python3 -m cockpit --registry registry.example.toml --check
bash scripts/selftest.sh
```
+4
View File
@@ -0,0 +1,4 @@
local.properties
.gradle/
.kotlin/
**/build/
+92
View File
@@ -0,0 +1,92 @@
# AGENTS.md — `android/` (GPU Cockpit mobile client)
Scope: everything under `android/`. This is an **Android app** that talks to the gpu-cockpit
HTTP server over WireGuard. It is a thin client: the server owns all state and executes
everything. The app renders and requests; it never decides whether a pipeline is up.
## Read first
* `android/API-CONTRACT.md` — the authoritative API. **Never invent an endpoint, field or enum
value.** If the contract is missing something, say so instead of guessing.
* The server itself is `../cockpit/server.py` (read-only for you — see Boundaries).
## Build & verify
```bash
cd android
./gradlew assembleDebug # must succeed
./gradlew testDebugUnitTest # must pass
```
* APK: `android/app/build/outputs/apk/debug/app-debug.apk`
* `ANDROID_HOME=/home/nyra/android-sdk` (platform-tools + cmdline-tools present, platforms 34/36,
build-tools 34.0.0/36.0.0). `local.properties` must contain `sdk.dir=/home/nyra/android-sdk`
and **must not be committed**.
* Gradle wrapper: `gradle-8.10.2-bin.zip` — already in `~/.gradle/wrapper/dists`, do not change
the version (a different one means a ~130 MB download for no reason).
* `JAVA_HOME` — a JDK 17 is on the machine; use whatever `./gradlew` resolves, but set
`compileOptions`/`kotlinOptions` to **Java 17**.
## Toolchain pins (do not "upgrade" these)
| Thing | Version |
|---|---|
| AGP (`com.android.application`) | 8.8.2 |
| Kotlin (`org.jetbrains.kotlin.android`) | 2.0.0 |
| Compose compiler plugin (`org.jetbrains.kotlin.plugin.compose`) | 2.0.0 |
| Compose BOM (`androidx.compose:compose-bom`) | 2024.06.00 |
| compileSdk / targetSdk | 34 |
| minSdk | 26 |
| coroutines | 1.8.1 (`kotlinx-coroutines-android`) |
Dependencies: **Gson only** (`com.google.code.gson:gson:2.10.1`) on top of the Compose/AndroidX
set. No OkHttp, no Hilt, no Room, no Kotlin serialization, no DI framework. Use
`java.net.HttpURLConnection` for HTTP (it also does the SSE stream) and coroutines for threading.
Every dependency is a failure mode; this app needs three.
## Hard requirements that are easy to get wrong
1. **Cleartext HTTP.** The server is plain HTTP on `10.10.10.1`. Without
`android:usesCleartextTraffic="true"` (or a network-security-config scoped to that host)
every request fails with `CLEARTEXT communication to 10.10.10.1 not permitted`.
2. **`android.permission.INTERNET`** in the manifest. Obvious, still forgotten.
3. **Auth header on every request**: `Authorization: Bearer <token>`. A wrong/missing token is a
`401` — surface it as "token rejected" with a path to Settings, not a generic error.
4. **Keep the parsing layer pure Kotlin.** The model classes + JSON parsing must not touch any
`android.*` API, so `testDebugUnitTest` (JVM) can parse `fixtures/state-sample.json`. Keep
Android-specific code (prefs, UI, network) in separate files.
5. **Settings storage is a fixed contract** (automation seeds it):
`SharedPreferences` file **`cockpit`**, keys **`base_url`** (default
`http://10.10.10.1:8770`) and **`token`**. Do not rename them, do not add obfuscation.
6. **No client-side job state.** A switch takes 15–45 s and keeps running on the server if the app
is closed or the screen locks. Model it by re-reading `/api/state` (or `GET /api/jobs/<id>`),
never by assuming.
7. **Never send `force: true` without the user asking.** Protected pipelines need it and the
server explains why in its 409 message; show that message and let the user choose.
8. **Never claim success the server did not report.** Show the server's step text and, on failure,
its message. The server is deliberately honest (it reports a real log tail when a start fails);
the app must not paper over that with a green checkmark.
## UI
Material 3, dark, matching the web GUI exactly:
```
bg #0c1118 panel #141d27 line #293747 text #e7edf4 muted #91a2b5
green #47d7a0 amber #f3b95f red #ff6c72 blue #71b7ff
```
Mobile-first, one hand, thumb-reachable actions, works on a 360 dp-wide screen. Show:
card gauge (use `pct_of_usable`), the pipeline cards with real state, who is holding the card,
and a clear "this is a tunnel; is WireGuard up?" message when the connection fails (the phone
cannot reach `10.10.10.1` unless the WireGuard app is connected — say that instead of "network
error").
## Boundaries
* **Touch only files under `android/`.** The Python server, GUI, TUI, registry and tests belong to
a different workstream; changing them will break a live system. If you believe the server needs
a change, write it in your summary instead — do not edit it.
* No secrets in the repo. The token is typed by the user at runtime; never hardcode, never log,
never print it.
* `local.properties`, `build/`, `.gradle/` are not committed (add them to `.gitignore`).
+158
View File
@@ -0,0 +1,158 @@
# gpu-cockpit HTTP API — contract for the Android client
Authoritative. Written from the running server (`cockpit/server.py`) and a live capture of
`/api/state` on Utumno. **Do not invent endpoints, fields, or enum values** — if something is
missing here, ask rather than guess.
## Transport
* Plain HTTP on the **WireGuard address only**: default base URL `http://10.10.10.1:8770`.
Nothing is listening on the LAN interface, so an off-tunnel call simply fails.
* **Auth**: bearer token in a header on every request:
```
Authorization: Bearer <token>
```
A `?token=<token>` query parameter is accepted as a fallback, but the header is the normal path.
* Missing/incorrect token on the tunnel → **HTTP 401** `{"error": "unauthorized"}`.
* Loopback callers (the desktop GUI, the TUI) bypass the token. That is irrelevant to the app.
* Requests are plain HTTP → the app **must** permit cleartext to this host
(`android:usesCleartextTraffic="true"` or a network-security-config scoped to `10.10.10.1`).
Symptom if forgotten: `CLEARTEXT communication to 10.10.10.1 not permitted`.
## GET endpoints
| Path | Returns |
|---|---|
| `/api/ping` | `{"ok":true,"host":"utumno","version":"0.1.0","now":"<iso8601>"}` |
| `/api/state` | the full snapshot (below) — the app's main data source |
| `/api/registry` | `{"registry":{…},"errors":[],"warnings":[]}` — the raw TOML registry as parsed |
| `/api/audit?n=100` | last N exec records: `[{"ts","pipeline","action","argv":[…],"rc","duration_ms","dry_run"}]` |
| `/api/logs/<pipeline_id>?n=200` | **list of strings** — journal lines for that pipeline's units, or one string saying the journal was unavailable |
| `/api/jobs/<job_id>` | the full job record. **This is the only place `steps` exists** — the snapshot's `job` object carries just `{id,pipeline,state,step,started}`, no steps. 404 `{"error":"job not found"}` if unknown |
| `/api/events` | **SSE**. First frame is `event: state` + the snapshot; then one frame per change, `: heartbeat` every 15 s. `?once=1` sends a single frame and closes — handy as a cheap liveness check |
## POST endpoints
All POST bodies are JSON. Success → **HTTP 202** `{"job_id":"j-<millis>-<pid>"}`.
| Path | Body | Meaning |
|---|---|---|
| `/api/card/free` | `{}` | run the configured reclaim recipe to free the card (currently ComfyUI `/free`) |
| `/api/pipelines/<id>/action` | `{"action":"start"\|"stop"\|"restart","options":{},"force":false}` | start / stop / restart that pipeline |
| `/api/pipelines/<id>/options` | `{"options":{…}}` | **persist a selection only**, no action. Returns `{"selected":{…}}` |
Error replies (JSON `{"error": "…"}`):
* `401` — `unauthorized`
* `400` — `invalid JSON`, `invalid action`, `unknown pipeline`, `options must be an object`,
`unknown option value <key>=<value>`
* `409` — protected pipeline and no `force` (e.g. `protected pipeline train is active; force is
required to stop it`), or a `ValueError` from preflight (card busy / insufficient VRAM)
* `500` — anything unexpected; the message is human-readable
**`action` is only ever `start` / `stop` / `restart`.** A "switch to this pipeline" is an
`action: "start"` — the server stops whatever else is holding the card itself, as part of the job,
and reports each step. There is no `switch` verb.
### `options`
Only pipelines that expose choices have a non-empty `options` array. Choices are validated
server-side; an unknown value is a 400. Example (the `llama` pipeline):
```json
{"options":[{"id":"model","label":"Model","kind":"argv","selected":"qwen3-4b-q4km",
"choices":[{"id":"qwen3-4b-q4km","label":"Qwen3-4B-Instruct-2507 (Q4_K_M, 2.4 GB)",
"note":"The voice stack's verbalizer. Fits whole, fastest thing here.",
"verified":"2026-09-25","is_default":true}, …]}]}
```
So a selection is `{"<option.id>":"<choice.id>"}` → send it as the `options` object of the action
and/or POST it to `/api/pipelines/<id>/options` to remember it. `selected` on the pipeline is the
current value. Render `label`, show `note` as secondary text, and prefer `is_default` when the
user has never chosen. `verified` is a date the combination was last proven — show it if present,
it is the difference between "we think this fits" and "we measured it".
## The snapshot (`/api/state`)
```json
{
"host": "utumno", "now": "<iso8601>", "version": "0.1.0",
"card": {
"available": true, "name": "NVIDIA GeForce RTX 4060",
"used_mib": 6149, "total_mib": 8188, "util_pct": 0, "temp_c": 48,
"free_mib": 2039, "idle_floor_mib": 1700, "usable_mib": 6488, "pct_of_usable": 95,
"processes": [{"pid": 3024502, "name": "llama-server", "used_mib": 2874, "pipeline": "voice"}],
"history": [{"t": "<iso8601>", "used_mib": 1151}]
},
"pipelines": [{
"id": "voice", "label": "Realtime voice",
"description": "Talk to the local voice stack: 4B verbalizer, Parakeet STT, Kokoro TTS.",
"state": "up", "health": "ok",
"requires_card": true, "vram_mib": 4900, "protected": false,
"serving": "Qwen3-4B-Instruct-2507 Q4_K_M (verbalizer) + Parakeet-TDT (STT) + Kokoro af_bella (TTS)",
"open_url": "http://127.0.0.1:8788",
"note": "Boot default - both units are enabled, so this is what comes back after a reboot. …",
"last_change": null,
"components": [{"id":"s2s","kind":"unit","label":"Speech-to-speech (:8765)","state":"up",
"detail":"pid 3024507","health":{"ok":true,"detail":"TCP connected","latency_ms":2},
"port":null,"unit":"speech-to-speech.service"}],
"selected": {"model": "qwen3-4b-q4km"},
"options": [],
"job": {"id":"j-…","state":"running","step":"stopping Image generation (releasing the card)…"}
}],
"job": {"id":"j-…","pipeline":"image","state":"running","step":"waiting for VRAM (used 5100 of 2000 MiB)…","started":"<iso8601>"},
"registry": {"path":"/home/harley/.config/gpu-cockpit/registry.toml","pipeline_count":6,"errors":[],"warnings":[]},
"listeners": ["127.0.0.1:8770","10.10.10.1:8770"]
}
```
Field notes the UI depends on:
* `pipeline.state` ∈ `up` · `down` · `error` · `starting` · `stopping`.
`pipeline.health` ∈ `ok` · `unknown` · `fail`. `up`+`ok` is genuinely healthy; `up`+`error`
means the process is there but its health probe fails — say that, do not show a green light.
* `component.state` may be `up` with `detail: "external"` — a process the cockpit did **not**
start (ComfyUI started by the Pictures page). The cockpit will not kill it; stopping the
pipeline is best-effort. Surface this, it explains "why didn't it stop".
* `component.detail` is `"pid <n>"` for owned components, `"external"` otherwise.
* `card.idle_floor_mib` is what the desktop itself uses; `usable_mib` = total − floor. Use
`pct_of_usable` for the gauge, not `used/total`.
* `card.processes[].pipeline` is `null` for processes that belong to no pipeline (desktop, kwin).
Show those greyed; they are why the card is never "0 MiB".
* `job` (top level) and `pipeline.job` are `null` when nothing is running. A job keeps running on
the **server** if the app closes — on reconnect, re-read the state, never resume a client-side
job model.
## Job lifecycle (the part that makes the app honest)
1. POST the action → `202 {"job_id": …}`
2. Poll `GET /api/jobs/<job_id>` (≈1 s) or just re-read `/api/state`; both carry
`state` (`running` → `done` / `error`) and the latest `step`.
The job record is:
```json
{"id":"j-…","pipeline":"image","action":"start","state":"running","started":"<iso8601>",
"error":null,
"steps":[{"ts":"<iso8601>","text":"stopping Realtime voice (releasing the card)…","ok":true}]}
```
**`steps` elements are objects, not strings** — `{"ts","text","ok"}` — oldest first, so `step`
is simply the last one's `text`, and `ok:false` marks a step that reported a problem. `error`
carries the failure message when `state` is `error`, and is `null` otherwise. Both fields are
named exactly as written here; do not invent alternates.
3. Show the real step text verbatim — e.g. `stopping Realtime voice (releasing the card)…`,
`waiting for VRAM (used 5100 of 2000 MiB)…`, `starting ComfyUI server (:8188)…`.
A switch takes ~15–45 s and sometimes reports an honest failure with the log tail
(e.g. the voice stack needs >20 s to load Kokoro). **Do not show a success state the server
did not report.** If the job ends in `error`, show the last step and the server's message.
4. `force: true` is required for a `protected` pipeline stop/restart; without it the server
answers 409 and the app should offer the force path explicitly (the LoRA training pipeline is
protected). Never send `force: true` silently.
## Captured fixtures
* `android/fixtures/state-sample.json` — a real `/api/state` capture (history trimmed), for unit
tests of the parsing layer. Parse it in a JVM test; if a field is renamed, that test should fail.
+46
View File
@@ -0,0 +1,46 @@
plugins {
id("com.android.application")
id("org.jetbrains.kotlin.android")
id("org.jetbrains.kotlin.plugin.compose")
}
android {
namespace = "dev.gpucockpit"
compileSdk = 34
defaultConfig {
applicationId = "dev.gpucockpit"
minSdk = 26
targetSdk = 34
versionCode = 1
versionName = "1.0"
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
kotlinOptions { jvmTarget = "17" }
buildFeatures { compose = true }
testOptions { unitTests.isIncludeAndroidResources = false }
}
dependencies {
implementation(platform("androidx.compose:compose-bom:2024.06.00"))
implementation("androidx.activity:activity-compose:1.9.0")
implementation("androidx.compose.ui:ui")
implementation("androidx.compose.ui:ui-tooling-preview")
implementation("androidx.compose.material3:material3")
implementation("androidx.compose.material:material-icons-extended")
implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.8.3")
implementation("androidx.lifecycle:lifecycle-runtime-compose:2.8.3")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.8.1")
implementation("com.google.code.gson:gson:2.10.1")
debugImplementation("androidx.compose.ui:ui-tooling")
testImplementation("junit:junit:4.13.2")
}
tasks.matching { it.name == "assembleDebug" }.configureEach {
doLast { println("APK: ${layout.buildDirectory.file("outputs/apk/debug/app-debug.apk").get().asFile.absolutePath}") }
}
+11
View File
@@ -0,0 +1,11 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.INTERNET" />
<application android:allowBackup="false" android:label="GPU Cockpit" android:theme="@android:style/Theme.Material.NoActionBar" android:usesCleartextTraffic="true">
<activity android:name=".MainActivity" android:exported="true">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>
@@ -0,0 +1,365 @@
package dev.gpucockpit
import android.content.Context
import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.*
import androidx.compose.foundation.gestures.detectVerticalDragGestures
import androidx.compose.foundation.layout.*
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.items
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.text.KeyboardOptions
import androidx.compose.material3.*
import androidx.compose.runtime.*
import androidx.compose.ui.*
import androidx.compose.ui.draw.clip
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.input.pointer.pointerInput
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.input.KeyboardType
import androidx.compose.ui.text.input.PasswordVisualTransformation
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import androidx.lifecycle.compose.LocalLifecycleOwner
import dev.gpucockpit.data.ApiException
import dev.gpucockpit.data.CockpitApi
import dev.gpucockpit.model.JobStep
import dev.gpucockpit.model.Pipeline
import dev.gpucockpit.model.Snapshot
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import java.text.SimpleDateFormat
import java.util.Date
import java.util.Locale
private val Bg = Color(0xFF0c1118)
private val Panel = Color(0xFF141d27)
private val Line = Color(0xFF293747)
private val Ink = Color(0xFFe7edf4)
private val Muted = Color(0xFF91a2b5)
private val Green = Color(0xFF47d7a0)
private val Amber = Color(0xFFf3b95f)
private val Red = Color(0xFFff6c72)
private val Blue = Color(0xFF71b7ff)
private const val DefaultUrl = "http://10.10.10.1:8770"
class MainActivity : ComponentActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
window.statusBarColor = android.graphics.Color.rgb(12, 17, 24)
window.navigationBarColor = android.graphics.Color.rgb(12, 17, 24)
val prefs = getSharedPreferences("cockpit", Context.MODE_PRIVATE)
setContent { CockpitApp(prefs) }
}
}
@Composable private fun CockpitApp(prefs: android.content.SharedPreferences) {
var baseUrl by remember { mutableStateOf(prefs.getString("base_url", DefaultUrl) ?: DefaultUrl) }
var token by remember { mutableStateOf(prefs.getString("token", "") ?: "") }
var showSettings by remember { mutableStateOf(token.isBlank()) }
var snapshot by remember { mutableStateOf<Snapshot?>(null) }
var connectionError by remember { mutableStateOf<String?>(null) }
var pingMessage by remember { mutableStateOf<String?>(null) }
var pingBusy by remember { mutableStateOf(false) }
var busy by remember { mutableStateOf(false) }
var jobView by remember { mutableStateOf<JobView?>(null) }
var pendingConfirm by remember { mutableStateOf<PendingAction?>(null) }
var protectedPrompt by remember { mutableStateOf<ProtectedPrompt?>(null) }
var expanded by remember { mutableStateOf(setOf<String>()) }
var foreground by remember { mutableStateOf(true) }
val optionsByPipeline = remember { mutableStateMapOf<String, MutableMap<String, String>>() }
val scope = rememberCoroutineScope()
val api = remember(baseUrl, token) { CockpitApi(baseUrl, token) }
val lifecycleOwner = LocalLifecycleOwner.current
suspend fun refresh() {
if (token.isBlank()) return
try {
snapshot = api.state(); connectionError = null
val currentJob = jobView
if (currentJob?.state == "running") {
try {
val latest = api.job(currentJob.id)
jobView = JobView(latest.id, latest.step, latest.state, currentJob.started, latest.error, latest.steps)
} catch (_: Exception) { /* The next foreground refresh will re-read the server job. */ }
}
}
catch (e: ApiException) { connectionError = if (e.code == 401) "token rejected" else e.message }
catch (_: Exception) { connectionError = "is WireGuard connected? this only works over the tunnel" }
}
LaunchedEffect(api, token) {
if (token.isNotBlank()) refresh()
while (true) { delay(3000); if (foreground && !showSettings && token.isNotBlank()) refresh() }
}
DisposableEffect(lifecycleOwner, api, token) {
val observer = LifecycleEventObserver { _, event ->
when (event) {
Lifecycle.Event.ON_RESUME -> { foreground = true; scope.launch { refresh() } }
Lifecycle.Event.ON_PAUSE -> foreground = false
else -> {}
}
}
lifecycleOwner.lifecycle.addObserver(observer)
onDispose { lifecycleOwner.lifecycle.removeObserver(observer) }
}
fun perform(id: String?, action: String, force: Boolean = false) {
scope.launch {
busy = true
try {
val options = id?.let { optionsByPipeline[it]?.toMap().orEmpty() }.orEmpty()
if (id != null && options.isNotEmpty()) api.rememberOptions(id, options)
val jobId = if (id == null) api.freeCard() else api.action(id, action, options, force)
val started = System.currentTimeMillis()
var job: dev.gpucockpit.model.Job? = null
jobView = JobView(jobId, "", "running", started, null)
while (true) {
job = api.job(jobId)
jobView = JobView(jobId, job.step, job.state, started, job.error, job.steps)
refresh()
if (job.state == "done" || job.state == "error") break
delay(1000)
}
refresh()
} catch (e: ApiException) {
if (e.code == 409 && id != null && !force) protectedPrompt = ProtectedPrompt(id, action, e.message)
else connectionError = if (e.code == 401) "token rejected" else e.message
} catch (_: Exception) { connectionError = "is WireGuard connected? this only works over the tunnel" }
finally { busy = false }
}
}
MaterialTheme(colorScheme = darkColorScheme(primary = Green, background = Bg, surface = Panel, onSurface = Ink)) {
if (showSettings) {
SettingsScreen(baseUrl, { baseUrl = it }, token, { token = it }, pingBusy, pingMessage,
onTest = {
scope.launch {
pingBusy = true; pingMessage = null
try {
val result = api.ping()
pingMessage = "Connected · ${result["host"] ?: "unknown host"} · v${result["version"] ?: "?"}"
} catch (e: ApiException) { pingMessage = if (e.code == 401) "token rejected" else e.message }
catch (_: Exception) { pingMessage = "Cannot reach the tunnel. Is WireGuard connected? This only works over the tunnel." }
finally { pingBusy = false }
}
}, onSave = {
prefs.edit().putString("base_url", baseUrl.trim()).putString("token", token).apply()
showSettings = false; snapshot = null; connectionError = null
scope.launch { refresh() }
})
} else {
Column(Modifier.fillMaxSize().background(Bg)) {
Row(Modifier.fillMaxWidth().padding(horizontal = 20.dp, vertical = 14.dp), verticalAlignment = Alignment.CenterVertically) {
Column(Modifier.weight(1f)) {
Text("GPU COCKPIT", color = Green, fontSize = 12.sp, fontWeight = FontWeight.Bold, letterSpacing = 2.sp)
Text(snapshot?.host?.uppercase() ?: "TUNNEL CONSOLE", color = Muted, fontSize = 12.sp)
}
TextButton(onClick = { showSettings = true }) { Text("SETTINGS", color = Blue) }
}
if (connectionError != null) ErrorBanner(connectionError!!, onSettings = { showSettings = true })
jobView?.let { JobPanel(it) { jobView = null } }
LazyColumn(Modifier.fillMaxSize().pointerInput(Unit) {
var drag = 0f
detectVerticalDragGestures(onVerticalDrag = { _, amount -> drag += amount }, onDragEnd = {
if (drag > 70) scope.launch { refresh() }; drag = 0f
})
}, contentPadding = PaddingValues(start = 16.dp, end = 16.dp, top = 4.dp, bottom = 28.dp), verticalArrangement = Arrangement.spacedBy(12.dp)) {
val s = snapshot
if (s == null) item { LoadingCard(connectionError == null) }
else {
item { CardGauge(s, onFree = { pendingConfirm = PendingAction(null, "free") }, enabled = !busy) }
item { Text("PIPELINES · ${s.pipelines.size}", color = Muted, fontSize = 11.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.5.sp, modifier = Modifier.padding(start = 4.dp, top = 6.dp)) }
items(s.pipelines, key = { it.id }) { pipeline ->
PipelineCard(pipeline, expanded.contains(pipeline.id), onExpand = {
expanded = if (expanded.contains(pipeline.id)) expanded - pipeline.id else expanded + pipeline.id
val current = optionsByPipeline[pipeline.id] ?: pipeline.options.associate { option -> option.id to (pipeline.selected[option.id] ?: option.selected ?: option.choices.firstOrNull { it.is_default }?.id.orEmpty()) }.toMutableMap()
optionsByPipeline[pipeline.id] = current
}, selected = optionsByPipeline[pipeline.id].orEmpty(), onSelect = { optionId, choice ->
val current = optionsByPipeline[pipeline.id]?.toMutableMap() ?: mutableMapOf()
current[optionId] = choice; optionsByPipeline[pipeline.id] = current
}, enabled = !busy, onAction = { action -> pendingConfirm = PendingAction(pipeline, action) })
}
}
}
}
}
}
pendingConfirm?.let { pending ->
val p = pending.pipeline
AlertDialog(onDismissRequest = { pendingConfirm = null }, containerColor = Panel,
title = { Text(if (pending.action == "free") "Free GPU card?" else "${pending.action.replaceFirstChar { it.uppercase() }} ${p?.label}?", color = Ink) },
text = { Text(if (pending.action == "free") "Run the server's configured card reclaim recipe." else "The server will manage the pipeline transition and report each step.", color = Muted) },
confirmButton = { TextButton(onClick = { pendingConfirm = null; perform(p?.id, if (pending.action == "free") "free" else pending.action) }) { Text("CONTINUE", color = Green) } },
dismissButton = { TextButton(onClick = { pendingConfirm = null }) { Text("CANCEL", color = Muted) } })
}
protectedPrompt?.let { prompt ->
AlertDialog(onDismissRequest = { protectedPrompt = null }, containerColor = Panel,
title = { Text("Protected pipeline", color = Amber) },
text = { Text("${prompt.serverMessage}\n\nForce ${prompt.action} this pipeline?", color = Ink) },
confirmButton = { TextButton(onClick = { protectedPrompt = null; perform(prompt.id, prompt.action, true) }) { Text("FORCE ${prompt.action.uppercase()}", color = Red) } },
dismissButton = { TextButton(onClick = { protectedPrompt = null }) { Text("CANCEL", color = Muted) } })
}
}
private data class PendingAction(val pipeline: Pipeline?, val action: String)
private data class ProtectedPrompt(val id: String, val action: String, val serverMessage: String)
private data class JobView(val id: String, val step: String, val state: String, val started: Long, val error: String?, val steps: List<JobStep> = emptyList())
@Composable private fun SettingsScreen(url: String, onUrl: (String) -> Unit, token: String, onToken: (String) -> Unit,
busy: Boolean, result: String?, onTest: () -> Unit, onSave: () -> Unit) {
Column(Modifier.fillMaxSize().background(Bg).verticalScroll(rememberScrollState()).padding(20.dp), verticalArrangement = Arrangement.spacedBy(16.dp)) {
Text("CONNECTION", color = Green, fontSize = 12.sp, fontWeight = FontWeight.Bold, letterSpacing = 2.sp)
Text("WireGuard link", color = Ink, fontSize = 28.sp, fontWeight = FontWeight.Bold)
Text("GPU Cockpit is reachable only through the WireGuard tunnel.", color = Muted)
OutlinedTextField(url, onUrl, label = { Text("Base URL") }, singleLine = true, modifier = Modifier.fillMaxWidth(), colors = fieldColors())
OutlinedTextField(token, onToken, label = { Text("Bearer token") }, singleLine = true, visualTransformation = PasswordVisualTransformation(), modifier = Modifier.fillMaxWidth(), colors = fieldColors())
Button(onClick = onTest, enabled = !busy, modifier = Modifier.fillMaxWidth(), colors = ButtonDefaults.buttonColors(containerColor = Line)) { Text(if (busy) "TESTING…" else "TEST CONNECTION", color = Ink) }
result?.let { Text(it, color = if (it.startsWith("Connected")) Green else Red) }
Button(onClick = onSave, modifier = Modifier.fillMaxWidth()) { Text("SAVE AND OPEN DASHBOARD") }
}
}
@Composable private fun CardGauge(snapshot: Snapshot, onFree: () -> Unit, enabled: Boolean) {
val card = snapshot.card
Column(Modifier.fillMaxWidth().clip(RoundedCornerShape(20.dp)).background(Panel).border(1.dp, Line, RoundedCornerShape(20.dp)).padding(18.dp)) {
Row(verticalAlignment = Alignment.CenterVertically) {
Column(Modifier.weight(1f)) {
Text("SHARED GPU", color = Muted, fontSize = 11.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.5.sp)
Text(card.name.ifBlank { "Card unavailable" }, color = Ink, fontSize = 19.sp, fontWeight = FontWeight.SemiBold)
}
Text("${card.pct_of_usable}%", color = if (card.pct_of_usable >= 90) Amber else Green, fontSize = 30.sp, fontWeight = FontWeight.Bold)
}
Spacer(Modifier.height(16.dp))
Box(Modifier.fillMaxWidth().height(10.dp).clip(CircleShape).background(Line)) {
Box(Modifier.fillMaxWidth((card.pct_of_usable.coerceIn(0, 100) / 100f)).fillMaxHeight().clip(CircleShape).background(if (card.pct_of_usable >= 90) Amber else Green))
}
Spacer(Modifier.height(10.dp))
Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.SpaceBetween) {
Text("${card.used_mib} / ${card.usable_mib} MiB usable", color = Ink, fontSize = 13.sp)
Text("${card.free_mib} MiB free", color = Muted, fontSize = 13.sp)
}
Spacer(Modifier.height(14.dp))
Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(8.dp)) {
Metric("TEMP", card.temp_c?.let { "$it° C" } ?: "—", Amber, Modifier.weight(1f))
Metric("UTIL", "${card.util_pct}%", Blue, Modifier.weight(1f))
Metric("CARD", if (card.available) "ONLINE" else "OFFLINE", if (card.available) Green else Red, Modifier.weight(1f))
}
if (card.processes.isNotEmpty()) {
Spacer(Modifier.height(12.dp)); Text("PROCESSES", color = Muted, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
card.processes.forEach { proc ->
Row(Modifier.fillMaxWidth().padding(top = 5.dp), horizontalArrangement = Arrangement.SpaceBetween) {
Text(proc.name + (proc.pipeline?.let { " · $it" } ?: " · unassigned"), color = if (proc.pipeline == null) Muted else Ink, fontSize = 12.sp)
Text("${proc.used_mib} MiB", color = Muted, fontSize = 12.sp)
}
}
}
Spacer(Modifier.height(12.dp))
OutlinedButton(onClick = onFree, enabled = enabled, modifier = Modifier.fillMaxWidth(), border = BorderStroke(1.dp, Line)) { Text("FREE CARD", color = Blue, fontWeight = FontWeight.Bold) }
}
}
@Composable private fun Metric(label: String, value: String, color: Color, modifier: Modifier = Modifier) {
Column(modifier.clip(RoundedCornerShape(12.dp)).background(Bg).padding(10.dp)) {
Text(label, color = Muted, fontSize = 9.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
Text(value, color = color, fontSize = 16.sp, fontWeight = FontWeight.Bold)
}
}
@Composable private fun PipelineCard(p: Pipeline, expanded: Boolean, onExpand: () -> Unit,
selected: Map<String, String>, onSelect: (String, String) -> Unit,
enabled: Boolean, onAction: (String) -> Unit) {
val unhealthy = p.health == "fail" || (p.state == "up" && p.health != "ok")
val stateColor = when { unhealthy || p.state == "error" -> Red; p.state == "up" -> Green; p.state == "starting" || p.state == "stopping" -> Amber; else -> Muted }
Column(Modifier.fillMaxWidth().clip(RoundedCornerShape(18.dp)).background(Panel).border(1.dp, Line, RoundedCornerShape(18.dp)).clickable { onExpand() }.padding(16.dp)) {
Row(verticalAlignment = Alignment.CenterVertically) {
Column(Modifier.weight(1f)) {
Text(p.label, color = Ink, fontSize = 18.sp, fontWeight = FontWeight.SemiBold)
Text(if (unhealthy) "${p.state.uppercase()} · HEALTH FAIL" else "${p.state.uppercase()} · ${p.health.uppercase()}", color = stateColor, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
}
if (p.requires_card) Text("${p.vram_mib} MiB", color = Blue, fontSize = 12.sp)
}
Spacer(Modifier.height(8.dp))
Text(p.description, color = Muted, fontSize = 13.sp, lineHeight = 18.sp)
Spacer(Modifier.height(10.dp))
Text("SERVING", color = Muted, fontSize = 9.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
Text(p.serving, color = Ink, fontSize = 13.sp)
p.last_change?.let { Text("Changed · $it", color = Muted, fontSize = 11.sp, modifier = Modifier.padding(top = 7.dp)) }
if (expanded) {
p.note?.let { Text(it, color = Muted, fontSize = 12.sp, modifier = Modifier.padding(top = 10.dp)) }
if (p.components.isNotEmpty()) {
Spacer(Modifier.height(12.dp)); Text("COMPONENTS", color = Muted, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
p.components.forEach { c ->
Column(Modifier.fillMaxWidth().padding(top = 7.dp).clip(RoundedCornerShape(10.dp)).background(Bg).padding(10.dp)) {
Text("${c.label} · ${c.state.uppercase()}", color = if (c.state == "up" && c.health?.ok == true) Green else if (c.state == "up") Amber else Muted, fontSize = 12.sp, fontWeight = FontWeight.SemiBold)
Text(c.detail, color = Muted, fontSize = 11.sp)
if (c.detail == "external") Text("Started outside Cockpit. Cockpit will not kill a process it did not start.", color = Amber, fontSize = 11.sp)
}
}
}
p.options.forEach { option ->
OptionPicker(option, selected[option.id] ?: option.selected ?: option.choices.firstOrNull { it.is_default }?.id.orEmpty(), { onSelect(option.id, it) })
}
Row(Modifier.fillMaxWidth().padding(top = 14.dp), horizontalArrangement = Arrangement.spacedBy(8.dp)) {
Button(onClick = { onAction("start") }, enabled = enabled, modifier = Modifier.weight(1f)) { Text("SWITCH TO THIS", fontSize = 11.sp) }
OutlinedButton(onClick = { onAction("stop") }, enabled = enabled, modifier = Modifier.weight(.7f), border = BorderStroke(1.dp, Line)) { Text("STOP", color = Ink, fontSize = 11.sp) }
OutlinedButton(onClick = { onAction("restart") }, enabled = enabled, modifier = Modifier.weight(.8f), border = BorderStroke(1.dp, Line)) { Text("RESTART", color = Ink, fontSize = 11.sp) }
}
} else Text("TAP FOR COMPONENTS & CONTROLS", color = Blue, fontSize = 10.sp, fontWeight = FontWeight.Bold, modifier = Modifier.padding(top = 11.dp))
}
}
@Composable private fun OptionPicker(option: dev.gpucockpit.model.PipelineOption, chosen: String, onSelect: (String) -> Unit) {
var open by remember { mutableStateOf(false) }
val choice = option.choices.firstOrNull { it.id == chosen }
Column(Modifier.fillMaxWidth().padding(top = 14.dp)) {
Text(option.label.uppercase(), color = Muted, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
Box {
OutlinedButton(onClick = { open = true }, modifier = Modifier.fillMaxWidth(), border = BorderStroke(1.dp, Line)) {
Text(choice?.label ?: "Choose ${option.label}", color = Ink, fontSize = 12.sp)
}
DropdownMenu(expanded = open, onDismissRequest = { open = false }, modifier = Modifier.background(Panel)) {
option.choices.forEach { c -> DropdownMenuItem(text = { Text(c.label + if (c.is_default) " · DEFAULT" else "", color = Ink, fontSize = 12.sp) }, onClick = { onSelect(c.id); open = false }) }
}
}
choice?.note?.let { Text(it, color = Muted, fontSize = 11.sp) }
if (choice?.verified != null) Text("Verified ${choice.verified}", color = Green, fontSize = 10.sp)
}
}
@Composable private fun JobPanel(job: JobView, onDismiss: () -> Unit) {
val elapsed = ((System.currentTimeMillis() - job.started) / 1000).toInt()
Column(Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 4.dp).clip(RoundedCornerShape(14.dp)).background(Color(0xFF1B2937)).padding(14.dp)) {
Text(if (job.state == "done") "SERVER REPORTED SUCCESS" else if (job.state == "error") "JOB FAILED" else "SERVER JOB · ${elapsed}s", color = if (job.state == "done") Green else if (job.state == "error") Red else Amber, fontSize = 11.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
Text(job.step.ifBlank { "Waiting for first server step…" }, color = Ink, fontSize = 13.sp, modifier = Modifier.padding(top = 5.dp))
job.steps.takeLast(5).forEach { s ->
Text((if (s.ok) "· " else "! ") + s.text, color = if (s.ok) Muted else Amber, fontSize = 11.sp, modifier = Modifier.padding(top = 3.dp))
}
job.error?.let { Text(it, color = Red, fontSize = 12.sp, modifier = Modifier.padding(top = 4.dp)) }
if (job.state == "done" || job.state == "error") TextButton(onClick = onDismiss, modifier = Modifier.align(Alignment.End)) { Text("DISMISS", color = Muted) }
}
}
@Composable private fun ErrorBanner(message: String, onSettings: () -> Unit) {
Row(Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 4.dp).clip(RoundedCornerShape(12.dp)).background(Color(0xFF392126)).padding(12.dp), verticalAlignment = Alignment.CenterVertically) {
Text(message, color = Red, fontSize = 12.sp, modifier = Modifier.weight(1f))
if (message == "token rejected") TextButton(onClick = onSettings) { Text("SETTINGS", color = Blue, fontSize = 10.sp) }
}
}
@Composable private fun LoadingCard(loading: Boolean) {
Column(Modifier.fillMaxWidth().clip(RoundedCornerShape(18.dp)).background(Panel).padding(20.dp)) {
Text(if (loading) "Connecting to Utumno…" else "Dashboard unavailable", color = Ink, fontWeight = FontWeight.SemiBold)
Text(if (loading) "Checking the WireGuard route" else "Check the connection and pull down to retry.", color = Muted, fontSize = 13.sp, modifier = Modifier.padding(top = 6.dp))
}
}
@Composable private fun fieldColors() = OutlinedTextFieldDefaults.colors(
focusedTextColor = Ink, unfocusedTextColor = Ink, focusedBorderColor = Green, unfocusedBorderColor = Line,
focusedLabelColor = Green, unfocusedLabelColor = Muted, cursorColor = Green
)
@@ -0,0 +1,56 @@
package dev.gpucockpit.data
import com.google.gson.Gson
import com.google.gson.reflect.TypeToken
import dev.gpucockpit.model.Job
import dev.gpucockpit.model.Snapshot
import dev.gpucockpit.model.SnapshotParser
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.net.HttpURLConnection
import java.net.URL
class ApiException(val code: Int, override val message: String) : Exception(message)
class CockpitApi(private val baseUrl: String, private val token: String) {
private val gson = Gson()
private fun request(path: String, method: String = "GET", body: Any? = null): String {
val connection = (URL(baseUrl.trimEnd('/') + path).openConnection() as HttpURLConnection)
try {
connection.requestMethod = method
connection.connectTimeout = 5000
connection.readTimeout = 8000
connection.setRequestProperty("Authorization", "Bearer $token")
connection.setRequestProperty("Accept", "application/json")
if (body != null) {
connection.doOutput = true
connection.setRequestProperty("Content-Type", "application/json")
connection.outputStream.use { it.write(gson.toJson(body).toByteArray(Charsets.UTF_8)) }
}
val code = connection.responseCode
val stream = if (code in 200..299) connection.inputStream else connection.errorStream
val text = stream?.bufferedReader()?.use { it.readText() }.orEmpty()
if (code !in 200..299) {
val error = try { gson.fromJson(text, Map::class.java)?.get("error")?.toString() } catch (_: Exception) { null }
throw ApiException(code, error ?: "HTTP $code")
}
return text
} finally { connection.disconnect() }
}
suspend fun ping(): Map<String, Any?> = withContext(Dispatchers.IO) {
val type = object : TypeToken<Map<String, Any?>>() {}.type
gson.fromJson(request("/api/ping"), type)
}
suspend fun state(): Snapshot = withContext(Dispatchers.IO) { SnapshotParser.parse(request("/api/state")) }
suspend fun action(id: String, action: String, options: Map<String, String> = emptyMap(), force: Boolean = false): String = withContext(Dispatchers.IO) {
val response = request("/api/pipelines/$id/action", "POST", mapOf("action" to action, "options" to options, "force" to force))
gson.fromJson(response, Map::class.java)["job_id"].toString()
}
suspend fun rememberOptions(id: String, options: Map<String, String>) = withContext(Dispatchers.IO) {
request("/api/pipelines/$id/options", "POST", mapOf("options" to options))
}
suspend fun freeCard(): String = withContext(Dispatchers.IO) {
val response = request("/api/card/free", "POST", emptyMap<String, Any>())
gson.fromJson(response, Map::class.java)["job_id"].toString()
}
suspend fun job(id: String): Job = withContext(Dispatchers.IO) { gson.fromJson(request("/api/jobs/$id"), Job::class.java) }
}
@@ -0,0 +1,54 @@
package dev.gpucockpit.model
import com.google.gson.Gson
import com.google.gson.annotations.SerializedName
data class Snapshot(
val host: String = "", val now: String = "", val version: String = "",
val card: Card = Card(), val pipelines: List<Pipeline> = emptyList(), val job: Job? = null,
val registry: Registry? = null, val listeners: List<String> = emptyList()
)
data class Card(
val available: Boolean = false, val name: String = "", val used_mib: Int = 0,
val total_mib: Int = 0, val util_pct: Int = 0, val temp_c: Int? = null,
val free_mib: Int = 0, val idle_floor_mib: Int = 0, val usable_mib: Int = 0,
val pct_of_usable: Int = 0, val processes: List<GpuProcess> = emptyList(),
val history: List<HistoryPoint> = emptyList()
)
data class GpuProcess(val pid: Long = 0, val name: String = "", val used_mib: Int = 0, val pipeline: String? = null)
data class HistoryPoint(val t: String = "", val used_mib: Int = 0)
data class Pipeline(
val id: String = "", val label: String = "", val description: String = "", val state: String = "unknown",
val health: String = "unknown", val requires_card: Boolean = false, val vram_mib: Int = 0,
val protected: Boolean = false, val serving: String = "", val open_url: String? = null,
val note: String? = null, val last_change: String? = null,
val components: List<Component> = emptyList(), val selected: Map<String, String> = emptyMap(),
val options: List<PipelineOption> = emptyList(), val job: Job? = null
)
data class Component(
val id: String = "", val kind: String = "", val label: String = "", val state: String = "unknown",
val detail: String = "", val health: ComponentHealth? = null, val port: Int? = null, val unit: String? = null
)
data class ComponentHealth(val ok: Boolean = false, val detail: String = "", val latency_ms: Int? = null)
data class PipelineOption(
val id: String = "", val label: String = "", val kind: String = "", val selected: String? = null,
val choices: List<Choice> = emptyList()
)
data class Choice(
val id: String = "", val label: String = "", val note: String? = null, val verified: String? = null,
val is_default: Boolean = false
)
data class Job(
@SerializedName("id") val id: String = "", val pipeline: String? = null, val state: String = "",
val step: String = "", val started: String? = null, val error: String? = null,
// The server appends {"ts","text","ok"} objects here. Typing this as List<String> made Gson
// throw on any job that had taken a step, which the UI then reported as a tunnel failure.
val steps: List<JobStep> = emptyList()
)
data class JobStep(val ts: String = "", val text: String = "", val ok: Boolean = true)
data class Registry(val path: String = "", val pipeline_count: Int = 0, val errors: List<String> = emptyList(), val warnings: List<String> = emptyList())
object SnapshotParser {
private val gson = Gson()
fun parse(json: String): Snapshot = gson.fromJson(json, Snapshot::class.java)
}
@@ -0,0 +1,20 @@
package dev.gpucockpit.model
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotNull
import org.junit.Test
import java.io.File
class SnapshotParserTest {
@Test fun parsesCapturedSnapshotAndKeyDashboardValues() {
val json = File("../fixtures/state-sample.json").readText()
val state = SnapshotParser.parse(json)
assertEquals("utumno", state.host)
assertEquals(6, state.pipelines.size)
val voice = state.pipelines.first { it.id == "voice" }
assertEquals(2, voice.components.size)
assertEquals(6488, state.card.usable_mib)
assertEquals(95, state.card.pct_of_usable)
assertNotNull(state.pipelines.first { it.id == "llama" }.options.first().choices.first().note)
}
}
+5
View File
@@ -0,0 +1,5 @@
plugins {
id("com.android.application") version "8.8.2" apply false
id("org.jetbrains.kotlin.android") version "2.0.0" apply false
id("org.jetbrains.kotlin.plugin.compose") version "2.0.0" apply false
}
+340
View File
@@ -0,0 +1,340 @@
{
"host": "utumno",
"now": "2026-09-27T16:22:49+00:00",
"version": "0.1.0",
"card": {
"available": true,
"name": "NVIDIA GeForce RTX 4060",
"used_mib": 6149,
"total_mib": 8188,
"util_pct": 0,
"temp_c": 47,
"processes": [
{
"pid": 7632,
"name": "kwin_wayland",
"used_mib": 22,
"pipeline": null
},
{
"pid": 2037014,
"name": "python",
"used_mib": 178,
"pipeline": null
},
{
"pid": 3024502,
"name": "llama-server",
"used_mib": 2874,
"pipeline": "voice"
},
{
"pid": 3024507,
"name": "python",
"used_mib": 2014,
"pipeline": "voice"
}
],
"free_mib": 2039,
"idle_floor_mib": 1700,
"usable_mib": 6488,
"pct_of_usable": 95,
"history": [
{
"t": "2026-09-27T16:19:32+00:00",
"used_mib": 1151
},
{
"t": "2026-09-27T16:19:34+00:00",
"used_mib": 1151
},
{
"t": "2026-09-27T16:19:36+00:00",
"used_mib": 1151
}
]
},
"pipelines": [
{
"id": "voice",
"label": "Realtime voice",
"description": "Talk to the local voice stack: 4B verbalizer, Parakeet STT, Kokoro TTS.",
"state": "up",
"health": "ok",
"requires_card": true,
"vram_mib": 4900,
"protected": false,
"serving": "Qwen3-4B-Instruct-2507 Q4_K_M (verbalizer) + Parakeet-TDT (STT) + Kokoro af_bella (TTS)",
"open_url": "http://127.0.0.1:8788",
"note": "Boot default - both units are enabled, so this is what comes back after a reboot. Stopping it is not disabling it; one Start brings it all back.",
"last_change": null,
"components": [
{
"id": "verbalizer",
"kind": "unit",
"label": "Verbalizer LLM (:8080)",
"state": "up",
"detail": "pid 3024502",
"health": {
"ok": true,
"detail": "HTTP 200",
"latency_ms": 2
},
"port": null,
"unit": "llama-server-local.service"
},
{
"id": "s2s",
"kind": "unit",
"label": "Speech-to-speech (:8765)",
"state": "up",
"detail": "pid 3024507",
"health": {
"ok": true,
"detail": "TCP connected",
"latency_ms": 0
},
"port": 8765,
"unit": "speech-to-speech.service"
}
],
"selected": {},
"options": [],
"job": null
},
{
"id": "image",
"label": "Image generation",
"description": "Z-Image Turbo behind ComfyUI. The Pictures page is the friendly front door.",
"state": "up",
"health": "ok",
"requires_card": true,
"vram_mib": 7019,
"protected": false,
"serving": "Z-Image-Turbo GGUF Q4_K_M + Qwen3-4B text encoder + nyra ladder LoRAs",
"open_url": "http://127.0.0.1:8899",
"note": "Verified peak 7019 MiB of 8188, ~25-30 s a picture at 768. The Pictures page can start ComfyUI by itself - if it did, this panel shows the server as external and will not kill it.",
"last_change": null,
"components": [
{
"id": "comfy",
"kind": "proc",
"label": "ComfyUI server (:8188)",
"state": "up",
"detail": "external",
"health": {
"ok": true,
"detail": "HTTP 200",
"latency_ms": 6
},
"port": null,
"unit": null
}
],
"selected": {},
"options": [],
"job": null
},
{
"id": "llama",
"label": "LLM playground",
"description": "Load any model we have proved onto the card, with its known-good flags.",
"state": "down",
"health": "unknown",
"requires_card": true,
"vram_mib": 6500,
"protected": false,
"serving": "whatever you pick below (OpenAI-compatible on :11435)",
"open_url": null,
"note": "Each entry carries the flags measured on this card. The 13 GB+ MoE models are not here - they live in the Agent and Roleplay pipelines, which leave the card free.",
"last_change": null,
"components": [
{
"id": "server",
"kind": "proc",
"label": "llama-server (:11435)",
"state": "down",
"detail": "not running",
"health": {
"ok": false,
"detail": "connection refused",
"latency_ms": 1
},
"port": null,
"unit": null
}
],
"selected": {
"model": "qwen3-4b-q4km"
},
"options": [
{
"id": "model",
"label": "Model",
"kind": "argv",
"selected": "qwen3-4b-q4km",
"choices": [
{
"id": "qwen3-4b-q4km",
"label": "Qwen3-4B-Instruct-2507 (Q4_K_M, 2.4 GB)",
"note": "The voice stack's verbalizer. Fits whole, fastest thing here.",
"verified": "2026-09-25",
"is_default": true
},
{
"id": "qwen3-4b-q5km",
"label": "Qwen3-4B-Instruct-2507 (Q5_K_M, 2.9 GB)",
"note": "Same model, one quant up. Small quality gain, still fits.",
"verified": null,
"is_default": false
},
{
"id": "gemma4-12b-heretic-iq3xs",
"label": "Gemma 4 12B Heretic uncensored (IQ3_XS, 5.3 GB)",
"note": "Uncensored 12B. Weights fit with headroom; keep the context modest.",
"verified": null,
"is_default": false
}
]
}
],
"job": null
},
{
"id": "agent",
"label": "Agent LLM (Hermes heavy)",
"description": "The big local model Hermes talks to through its llama-local provider.",
"state": "down",
"health": "unknown",
"requires_card": true,
"vram_mib": 1500,
"protected": false,
"serving": "Qwen3.6-35B-A3B UD-IQ3_S, 128K context, q8_0 KV",
"open_url": null,
"note": "Mostly CPU-resident (a 3B-active MoE), so only a small slice of the card. systemd Conflicts= stops the voice pair automatically when this starts. Hermes reaches it as local-qwen35 on :11434.",
"last_change": null,
"components": [
{
"id": "server",
"kind": "unit",
"label": "llama-server-35b (:11434)",
"state": "down",
"detail": "inactive",
"health": null,
"port": null,
"unit": "llama-server-35b.service"
}
],
"selected": {},
"options": [],
"job": null
},
{
"id": "train",
"label": "LoRA training",
"description": "ai-toolkit Z-Image LoRA runs. Hours, not seconds - this is the one to leave alone.",
"state": "down",
"health": "unknown",
"requires_card": true,
"vram_mib": 3200,
"protected": true,
"serving": "the config you pick (warm-cache two-phase run)",
"open_url": null,
"note": "PROTECTED: nothing here will stop it to free the card. A run is hours of work, so stopping it is a deliberate, confirmed action. Progress is in the log tail (~3.4 s/step measured).",
"last_change": null,
"components": [
{
"id": "run",
"kind": "proc",
"label": "run-lora.sh",
"state": "down",
"detail": "not running",
"health": {
"ok": false,
"detail": "no health check",
"latency_ms": 0
},
"port": null,
"unit": null
}
],
"selected": {},
"options": [
{
"id": "config",
"label": "Config",
"kind": "argv",
"selected": "config-emo",
"choices": [
{
"id": "config-emo",
"label": "config-emo.yaml (emo outfit set)",
"note": "146 + 11 barefoot hero images. Completed 2026-09-26.",
"verified": "2026-09-26",
"is_default": true
},
{
"id": "config-sundress",
"label": "config-sundress.yaml (sundress set)",
"note": "92 + 5 hero images. Completed 2026-09-26.",
"verified": "2026-09-26",
"is_default": false
},
{
"id": "config-canon",
"label": "config-canon.yaml (canon set)",
"note": "The first canon run - the one the pet atlas came from.",
"verified": null,
"is_default": false
}
]
}
],
"job": null
},
{
"id": "rp",
"label": "Roleplay guest (CPU)",
"description": "Uncensored Gemma 4 26B served to the lab guest. CPU only, card untouched.",
"state": "up",
"health": "ok",
"requires_card": false,
"vram_mib": 0,
"protected": false,
"serving": "Gemma4-26B-A4B QAT Uncensored Q4_K_M + vision projector, --device none",
"open_url": null,
"note": "Runs entirely on CPU (32 cores, 93 GB RAM) and is exposed on the lab bridge, not loopback. Starting it never disturbs whatever owns the card.",
"last_change": null,
"components": [
{
"id": "server",
"kind": "unit",
"label": "llama-server-rp (:11434 on the lab bridge)",
"state": "up",
"detail": "pid 1034519",
"health": {
"ok": true,
"detail": "HTTP 200",
"latency_ms": 1
},
"port": null,
"unit": "llama-server-rp.service"
}
],
"selected": {},
"options": [],
"job": null
}
],
"job": null,
"registry": {
"path": "/home/harley/.config/gpu-cockpit/registry.toml",
"pipeline_count": 6,
"errors": [],
"warnings": []
},
"listeners": [
"127.0.0.1:8770",
"10.10.10.1:8770"
]
}
+3
View File
@@ -0,0 +1,3 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
android.useAndroidX=true
kotlin.code.style=official
Binary file not shown.
+7
View File
@@ -0,0 +1,7 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
Vendored Executable
+252
View File
@@ -0,0 +1,252 @@
#!/bin/sh
#
# Copyright © 2015-2021 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
' "$PWD" ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-classpath "$CLASSPATH" \
org.gradle.wrapper.GradleWrapperMain \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"
+94
View File
@@ -0,0 +1,94 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem SPDX-License-Identifier: Apache-2.0
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:execute
@rem Setup the command line
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega
+7
View File
@@ -0,0 +1,7 @@
pluginManagement { repositories { google(); mavenCentral(); gradlePluginPortal() } }
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories { google(); mavenCentral() }
}
rootProject.name = "GpuCockpit"
include(":app")
+2
View File
@@ -0,0 +1,2 @@
"""Registry-driven single-GPU pipeline console."""
__version__ = "0.1.0"
+4
View File
@@ -0,0 +1,4 @@
from .server import main
if __name__ == "__main__":
raise SystemExit(main())
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+28
View File
@@ -0,0 +1,28 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>GPU Cockpit</title><link rel="icon" href="/static/logo.png">
<style>
:root{color-scheme:dark;--bg:#0c1118;--panel:#141d27;--line:#293747;--text:#e7edf4;--muted:#91a2b5;--green:#47d7a0;--amber:#f3b95f;--red:#ff6c72;--blue:#71b7ff}*{box-sizing:border-box}body{margin:0;background:radial-gradient(ellipse at 40% -10%,#1a2a3c,var(--bg) 55%);color:var(--text);font:15px/1.45 system-ui,sans-serif}header{padding:18px max(20px,calc((100vw - 1440px)/2));display:flex;gap:14px;align-items:center;border-bottom:1px solid var(--line);position:sticky;top:0;background:#0c1118eF;z-index:4}header .brand{width:38px;height:38px;flex:none;border-radius:11px;overflow:hidden;border:1px solid #2b3a4d;background:#14161f;box-shadow:0 0 12px #71b7ff22}header .brand img{width:180%;height:180%;margin:-40% 0 0 -40%;display:block}.unattributed td{color:var(--muted);font-style:italic}h1{font-size:19px;margin:0}header small,.muted{color:var(--muted)}.right{margin-left:auto;display:flex;align-items:center;gap:14px}.wrap{max-width:1440px;margin:auto;padding:22px}.top{display:grid;grid-template-columns:minmax(300px,1fr) minmax(0,2fr);gap:18px}.panel,.card{background:linear-gradient(145deg,#17222e,#121a23);border:1px solid var(--line);border-radius:15px;padding:18px;box-shadow:0 12px 32px #0002}.panel h2{margin:0 0 14px;font-size:16px}.gauge{display:flex;gap:18px;align-items:center}.ring{width:120px;height:120px;flex:none}.ring text{fill:var(--text);font-size:17px;font-weight:700}.metrics{display:grid;grid-template-columns:repeat(3,1fr);gap:10px;flex:1}.metric{border-left:1px solid var(--line);padding-left:12px}.metric b{display:block;font-size:20px}.metric span{font-size:12px;color:var(--muted)}.spark{width:100%;height:40px;margin-top:10px}.process{width:100%;border-collapse:collapse;font-size:13px}.process td,.process th{text-align:left;padding:7px;border-bottom:1px solid #29374788}.external{color:var(--amber)}.section{display:flex;align-items:end;justify-content:space-between;margin:25px 0 12px}.section h2{margin:0}.grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:14px}.card h3{margin:0;font-size:17px}.title{display:flex;align-items:center;gap:9px}.dot{width:10px;height:10px;border-radius:50%;background:#778493}.dot.up{background:var(--green);box-shadow:0 0 10px #47d7a088}.dot.partial,.dot.starting,.dot.stopping{background:var(--amber)}.dot.error{background:var(--red)}.desc{min-height:42px;color:var(--muted);font-size:13px;margin:7px 0 12px}.tags{display:flex;gap:7px;flex-wrap:wrap;color:var(--muted);font-size:12px}.serving{margin:10px 0 4px;font-weight:600}.note{color:var(--amber);font-size:12px;min-height:18px}.components{border-top:1px solid var(--line);margin-top:12px;padding-top:9px}.comp{display:flex;justify-content:space-between;gap:8px;font-size:12px;padding:3px 0}.selects{margin-top:10px;display:grid;gap:8px}.choice{display:grid;grid-template-columns:auto 1fr;align-items:center;gap:8px;font-size:12px}select,button{font:inherit;color:var(--text);background:#202e3d;border:1px solid #3a4e63;border-radius:8px;padding:8px}select{min-width:0;width:100%}button{cursor:pointer}button:hover{border-color:var(--blue)}button:focus-visible,a:focus-visible,select:focus-visible{outline:2px solid var(--blue);outline-offset:2px}.primary{background:#216b58;border-color:#399b7e;font-weight:650}.danger{background:#45272c;border-color:#704047}.buttons{display:flex;gap:7px;flex-wrap:wrap;margin-top:13px}.buttons button{font-size:13px}.job{color:var(--blue);font-size:12px;min-height:18px;margin-top:8px}.log{margin-top:8px;color:var(--muted);font-size:12px}details summary{cursor:pointer;color:var(--muted)}.banner{display:none;padding:14px 17px;margin-top:16px;border:1px solid #a77e38;background:#382d1c;border-radius:12px}.banner.show{display:flex;align-items:center;gap:12px;flex-wrap:wrap}.banner span{flex:1}.activity{margin-top:20px}.activity table{width:100%;font-size:12px}#activityContent{max-height:320px;overflow:auto}.status{display:inline-flex;align-items:center;gap:7px}.live{color:var(--green)}.offline{color:var(--red)}.notice{color:var(--amber);font-size:13px;max-width:34ch;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.notice.bad{color:var(--red)}a{color:var(--blue)}@media(max-width:1000px){.grid{grid-template-columns:repeat(2,minmax(0,1fr))}.top{grid-template-columns:1fr}}@media(max-width:600px){header{padding:12px 14px}.wrap{padding:14px}.grid{grid-template-columns:1fr}.right{gap:7px}header small{display:none}.gauge{align-items:flex-start}.ring{width:95px;height:95px}.metrics{grid-template-columns:repeat(2,1fr)}.metric b{font-size:17px}}
</style></head><body><header><span class="brand"><img src="/static/logo.png" onerror="this.parentNode.hidden=true" alt=""></span><div><h1>GPU Cockpit</h1><small id="host">Connecting…</small></div><div class="right"><span id="notice" class="notice"></span><span id="connection" class="status offline">● offline</span><span id="stream" class="muted">polling</span><button class="primary" onclick="freeCard()">Free the card</button></div></header>
<main class="wrap"><section class="top"><div class="panel"><h2>Card memory</h2><div class="gauge"><svg class="ring" viewBox="0 0 120 120" aria-label="VRAM usage"><circle cx="60" cy="60" r="48" fill="none" stroke="#293747" stroke-width="10"/><circle id="arc" cx="60" cy="60" r="48" fill="none" stroke="#47d7a0" stroke-width="10" stroke-linecap="round" transform="rotate(-90 60 60)" stroke-dasharray="301.6" stroke-dashoffset="301.6"/><text x="60" y="65" text-anchor="middle" id="pct">—</text></svg><div class="metrics"><div class="metric"><b id="used">—</b><span>used / total MiB</span></div><div class="metric"><b id="util">—</b><span>GPU utilization</span></div><div class="metric"><b id="temp">—</b><span>temperature</span></div></div></div><small class="muted" id="floor"></small><svg class="spark" viewBox="0 0 300 40" preserveAspectRatio="none"><polyline id="spark" fill="none" stroke="#71b7ff" stroke-width="2" points=""/></svg></div><div class="panel"><h2>GPU processes</h2><div id="processes" class="muted">No GPU probe</div></div></section>
<div id="banner" class="banner"><span id="bannerText"></span><button id="switchBtn" class="primary">Switch to this</button><button id="forceBtn" class="danger" hidden>Force switch</button></div><div class="section"><h2>Pipelines</h2><span class="muted" id="regline"></span></div><section id="pipes" class="grid"></section>
<details class="panel activity"><summary>Activity · command audit and registry report</summary><div id="activityContent"><h3>Registry validation</h3><div id="validation"></div><h3>Command audit</h3><div id="audit"></div></div></details></main>
<script>
const $=id=>document.getElementById(id);let state=null,busy=false,evt=null;
async function api(path,opts){let r=await fetch(path,opts);let j=await r.json().catch(()=>({}));if(!r.ok)throw Error(j.error||r.statusText);return j}
function esc(x){return String(x??'').replace(/[&<>"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]))}
function render(s){state=s;$('host').textContent=`${s.host} · ${s.now}`;$('connection').className='status live';$('connection').textContent='● live';let c=s.card;if(!c.available){$('used').textContent='No GPU probe';$('util').textContent='—';$('temp').textContent='—';$('pct').textContent='—';$('arc').style.strokeDashoffset=301.6;$('floor').textContent=c.detail||'nvidia-smi unavailable';$('processes').textContent='No GPU probe';}else{$('used').textContent=`${c.used_mib} / ${c.total_mib}`;$('util').textContent=`${c.util_pct}%`;$('temp').textContent=`${c.temp_c}°C`;$('pct').textContent=`${c.pct_of_usable}%`;$('arc').style.strokeDashoffset=301.6*(1-c.pct_of_usable/100);$('floor').textContent=`Idle floor ${c.idle_floor_mib} MiB · gauge uses ${c.usable_mib} MiB usable`;$('processes').innerHTML=c.processes.length?`<table class="process"><thead><tr><th>PID</th><th>Process</th><th>MiB</th><th>Owner</th></tr></thead><tbody>${c.processes.map(p=>`<tr class="${p.pipeline?'':'external'}"><td>${p.pid}</td><td>${esc(p.name)}</td><td>${p.used_mib}</td><td>${p.pipeline?esc(p.pipeline):'External holder'}</td></tr>`).join('')+((c.used_mib-c.processes.reduce((a,p)=>a+p.used_mib,0))>150?`<tr class="unattributed"><td>—</td><td>desktop &amp; graphics surfaces</td><td>${c.used_mib-c.processes.reduce((a,p)=>a+p.used_mib,0)}</td><td>—</td></tr>`:'')}</tbody></table>`:'No GPU processes reported';let vals=c.history.map(x=>x.used_mib),max=Math.max(1,...vals);$('spark').setAttribute('points',vals.map((v,i)=>`${i*300/Math.max(1,vals.length-1)},${38-v/max*34}`).join(' '));}
$('regline').textContent=`${s.registry.pipeline_count} pipelines · ${s.registry.errors.length} errors · ${s.registry.warnings.length} warnings`;
$('pipes').innerHTML=s.pipelines.map(p=>`<article class="card"><div class="title"><span class="dot ${p.state}"></span><h3>${esc(p.label)}</h3><span class="right muted">${esc(p.state)}</span></div><div class="desc">${esc(p.description)}</div><div class="tags"><span>${p.vram_mib??'—'} MiB expected</span><span>${p.requires_card?'shared card':'CPU / independent'}</span>${p.protected?'<span>protected</span>':''}</div><div class="serving">${esc(p.serving||'')}</div><div class="note">${esc(p.note||'')}</div><div class="components">${p.components.map(c=>`<div class="comp"><span>${esc(c.label)} · ${esc(c.state)}${c.port?` · :${c.port}`:''}</span><span>${esc(c.health?.detail||c.detail||'')}</span></div>`).join('')}</div><div class="selects">${p.options.map(o=>`<label class="choice">${esc(o.label)}<select aria-label="${esc(o.label)}" onchange="choose('${p.id}','${o.id}',this.value)">${o.choices.map(ch=>`<option value="${esc(ch.id)}" ${ch.id===o.selected?'selected':''}>${esc(ch.label)}${ch.verified?' · verified':''}</option>`).join('')}</select>${o.choices.find(x=>x.id===o.selected)?.note?`<small class="muted">${esc(o.choices.find(x=>x.id===o.selected).note)}</small>`:''}</label>`).join('')}</div><div class="buttons"><button class="primary" onclick="action('${p.id}','start')">Start</button><button onclick="stopPipeline('${p.id}')">Stop</button><button onclick="action('${p.id}','restart')">Restart</button><button onclick="switchTo('${p.id}')">Switch to this</button>${p.open_url?`<a href="${esc(p.open_url)}" target="_blank" rel="noopener">Open ↗</a>`:''}</div><div class="job">${p.job?`${esc(p.job.id)} · ${esc(p.job.step)}`:''}</div><details class="log"><summary>Log tail</summary><pre id="log-${p.id}">Open to load</pre></details></article>`).join('');
let active=s.pipelines.find(p=>p.job);busy=!!s.job;$('pipes').querySelectorAll('button,select').forEach(x=>x.disabled=busy);if(active){$('banner').classList.add('show');$('bannerText').textContent=`${active.label} is changing: ${active.job.step}`;$('switchBtn').hidden=true;$('forceBtn').hidden=true;}else{$('banner').classList.remove('show');}
$('validation').innerHTML=[...s.registry.errors.map(x=>`<div class="external">Error: ${esc(x)}</div>`),...s.registry.warnings.map(x=>`<div class="muted">Warning: ${esc(x)}</div>`)].join('')||'<span class="live">No registry problems</span>';api('/api/audit?n=30').then(a=>$('audit').innerHTML=`<table class="process"><tbody>${a.slice().reverse().map(x=>`<tr><td>${esc(x.ts)}</td><td>${esc(x.pipeline)}</td><td>${esc(x.argv.join(' '))}</td><td>${x.rc}</td><td>${x.duration_ms} ms</td></tr>`).join('')}</tbody></table>`).catch(()=>{});
}
async function refresh(){try{render(await api('/api/state'))}catch(e){$('connection').className='status offline';$('connection').textContent='● offline';$('host').textContent='Waiting for server';}}
let noticeTimer=null;
function notice(msg,bad){let n=$('notice');n.textContent=msg;n.className=bad?'notice bad':'notice';clearTimeout(noticeTimer);noticeTimer=setTimeout(()=>{n.textContent=''},12000)}
async function action(id,a,force=false){if(busy)return;let options={};state.pipelines.find(p=>p.id===id)?.options.forEach(o=>options[o.id]=o.selected);try{let j=await api(`/api/pipelines/${id}/action`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({action:a,options,force})});notice(`job ${j.job_id} accepted`);busy=true;refresh()}catch(e){notice(e.message,true);}}
async function choose(pid,oid,value){let p=state.pipelines.find(p=>p.id===pid),options={};p.options.forEach(o=>options[o.id]=o.selected);options[oid]=value;try{await api(`/api/pipelines/${pid}/options`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({options})});refresh()}catch(e){notice(e.message,true)}}
async function switchTo(id){let p=state.pipelines.find(x=>x.id===id),owners=state.pipelines.filter(x=>x.id!==id&&x.requires_card&&x.state!=='down');let blocked=owners.find(x=>x.protected);if(blocked){$('banner').classList.add('show');$('bannerText').textContent=`${blocked.label} is protected and would keep ${p.label} from using the card. Force stops it and loses its current work.`;$('switchBtn').hidden=true;$('forceBtn').hidden=false;$('forceBtn').onclick=()=>action(id,'start',true);return;}if(owners.length){$('banner').classList.add('show');$('bannerText').textContent=`Starting ${p.label} will stop ${owners.map(x=>x.label).join(', ')} and release the card.`;$('switchBtn').hidden=false;$('switchBtn').textContent='Stop them and switch';$('switchBtn').onclick=()=>action(id,'start');}else action(id,'start');}
async function stopPipeline(id){let p=state.pipelines.find(x=>x.id===id);if(p&&p.protected&&p.state!=='down'){$('banner').classList.add('show');$('bannerText').textContent=`${p.label} is protected \u2014 stopping it now would lose work in progress.`;$('switchBtn').hidden=true;$('forceBtn').hidden=false;$('forceBtn').textContent='Force stop';$('forceBtn').onclick=()=>action(id,'stop',true);return;}action(id,'stop');}
async function freeCard(){try{await api('/api/card/free',{method:'POST'});busy=true;refresh()}catch(e){notice(e.message,true)}}
document.addEventListener('click',e=>{let d=e.target.closest('details.log');if(d&&d.open){let id=d.querySelector('pre').id.slice(4);api(`/api/logs/${id}?n=40`).then(x=>$(d.querySelector('pre').id).textContent=x.join('\n')).catch(()=>{})}});
function connect(){if(!('EventSource'in window))return;$('stream').textContent='live stream';evt=new EventSource('/api/events');evt.addEventListener('state',e=>{try{render(JSON.parse(e.data))}catch{}});evt.onerror=()=>{$('stream').textContent='polling fallback';evt.close();evt=null;setTimeout(connect,5000)}}refresh();connect();setInterval(()=>{if(!evt||evt.readyState!==1)refresh()},3000);
</script></body></html>
Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

+63
View File
@@ -0,0 +1,63 @@
"""Read-only host, process and health probes."""
import json, os, re, socket, subprocess, time, urllib.request, urllib.error
def friendly_error(exc,timeout=None):
"""A status line has no room for '<urlopen error [Errno 111] Connection refused>'.
Map the usual socket failures to something a person reads at a glance."""
reason=getattr(exc,"reason",None); errno=getattr(reason,"errno",None) or getattr(exc,"errno",None)
if errno==111: return "connection refused"
if errno==113: return "no route to host"
if errno==98: return "address already in use"
if errno==110 or isinstance(exc,(TimeoutError,socket.timeout)) or "timed out" in str(exc).lower():
return f"timed out after {timeout}s" if timeout else "timed out"
if isinstance(exc,urllib.error.URLError): exc=reason or exc
return str(exc).strip(" <>")[:120]
def health(spec, runner=None, pipeline=None):
if not spec or spec.get("kind")=="none": return {"ok":True,"detail":"no health check","latency_ms":0}
start=time.monotonic(); kind=spec.get("kind"); timeout=min(30,max(.1,float(spec.get("timeout_s",3))))
try:
if kind=="http":
req=urllib.request.Request(spec["url"],method="GET")
with urllib.request.urlopen(req,timeout=timeout) as r: ok=200<=r.status<400; detail=f"HTTP {r.status}"
elif kind=="tcp":
with socket.create_connection((spec["host"],int(spec["port"])),timeout=timeout): pass
ok=True; detail="TCP connected"
elif kind=="command":
if runner:
rc,out,_=runner.run(spec["argv"],pipeline,"health",timeout=timeout,readonly=True)
else:
return {"ok":False,"detail":"command health probe needs runner","latency_ms":0}
ok=rc==0; detail=f"exit {rc}"
else: return {"ok":False,"detail":f"unknown health kind: {kind}","latency_ms":0}
return {"ok":ok,"detail":detail,"latency_ms":round((time.monotonic()-start)*1000)}
except Exception as exc: return {"ok":False,"detail":friendly_error(exc,timeout),"latency_ms":round((time.monotonic()-start)*1000)}
def gpu(runner=None):
try:
if runner:
rc,out,_=runner.run(["nvidia-smi","--query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu","--format=csv,noheader,nounits"],None,"gpu-probe",timeout=3,readonly=True)
else:
p=subprocess.run(["nvidia-smi","--query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu","--format=csv,noheader,nounits"],timeout=3,text=True,capture_output=True); rc,out=p.returncode,p.stdout if p.returncode==0 else p.stderr
if rc: raise RuntimeError(out.strip() or f"exit {rc}")
row=out.strip().splitlines()[0].split(",")
name,used,total,util,temp=[x.strip() for x in row]
procs=[]
if runner: prc,pout,_=runner.run(["nvidia-smi","--query-compute-apps=pid,process_name,used_memory","--format=csv,noheader,nounits"],None,"gpu-process-probe",timeout=3,readonly=True)
else:
pp=subprocess.run(["nvidia-smi","--query-compute-apps=pid,process_name,used_memory","--format=csv,noheader,nounits"],timeout=3,text=True,capture_output=True); prc,pout=pp.returncode,pp.stdout
if prc==0:
for line in pout.splitlines():
x=[z.strip() for z in line.split(",")]
if len(x)>=3:
try: procs.append({"pid":int(x[0]),"name":os.path.basename(x[1]),"used_mib":int(float(x[2])),"pipeline":None})
except ValueError: pass
return {"available":True,"name":name,"used_mib":int(float(used)),"total_mib":int(float(total)),"util_pct":int(float(util)),"temp_c":int(float(temp)),"processes":procs}
except Exception as exc: return {"available":False,"name":None,"used_mib":None,"total_mib":None,"util_pct":None,"temp_c":None,"processes":[],"detail":friendly_error(exc)}
def proc_start(pid):
try:
raw=open(f"/proc/{pid}/stat").read(); fields=raw[raw.rfind(")")+2:].split()
if fields[0] in ("Z","X"): return None
return fields[19]
except Exception: return None
+149
View File
@@ -0,0 +1,149 @@
"""Registry loading and validation. Invalid pipelines are retained for reporting."""
import re
import tomllib
from pathlib import Path
ID = re.compile(r"^[a-z0-9-]+$")
PLACEHOLDER = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*)\}")
def load(path):
errors, warnings = [], []
try:
with open(path, "rb") as f:
data = tomllib.load(f)
except Exception as exc:
return {"card": {}, "pipeline": []}, [f"registry: {exc}"], warnings
card = data.setdefault("card", {})
if not isinstance(card, dict):
errors.append("card must be a table")
card = data["card"] = {}
if "reclaim" in card and not isinstance(card["reclaim"], dict):
errors.append("card.reclaim must be a table")
card.pop("reclaim", None)
for key, default, minimum in (("total_mib", None, 1), ("idle_floor_mib", 0, 0), ("poll_seconds", 2, .1)):
if key not in card:
if default is not None: card[key] = default
continue
try:
value=float(card[key])
if value < minimum: raise ValueError()
card[key]=int(value) if key != "poll_seconds" else value
except (TypeError, ValueError):
errors.append(f"card.{key} must be a number >= {minimum}")
if default is not None: card[key]=default
else: card.pop(key,None)
rec=card.get("reclaim")
if rec and rec.get("kind") not in ("http", "command"):
errors.append("card.reclaim.kind must be http or command")
if rec:
try:
if int(rec.get("wait_timeout_s",45)) < 1: raise ValueError()
except (TypeError,ValueError): errors.append("card.reclaim.wait_timeout_s must be a positive integer")
if rec.get("kind")=="http" and not isinstance(rec.get("url"),str): errors.append("card.reclaim.url is required for http reclaim")
if rec.get("kind")=="command" and (not isinstance(rec.get("argv"),list) or not rec.get("argv")):
errors.append("card.reclaim.argv is required for command reclaim")
pipes = data.setdefault("pipeline", [])
if not isinstance(pipes, list):
errors.append("pipeline must be an array of tables")
pipes = []
data["pipeline"] = pipes
ids = set()
for i, p in enumerate(pipes):
prefix = f"pipeline[{i}]"
if not isinstance(p, dict):
errors.append(f"{prefix}: pipeline entry must be a table")
continue
pid = p.get("id")
if not isinstance(pid, str) or not ID.fullmatch(pid):
errors.append(f"{prefix}: invalid id")
elif pid in ids:
errors.append(f"{prefix}: duplicate id {pid}")
if isinstance(pid, str):
ids.add(pid)
comps = p.get("component", [])
if not isinstance(comps, list):
errors.append(f"{prefix} {pid}: component must be an array of tables")
comps = []
p["component"] = comps
if not comps:
errors.append(f"{prefix} {pid}: at least one component is required")
opts = p.get("option", [])
if not isinstance(opts, list):
errors.append(f"{prefix} {pid}: option must be an array of tables")
opts = []
p["option"] = opts
clean_opts = [o for o in opts if isinstance(o, dict) and isinstance(o.get("id"),str)]
if len(clean_opts) != len(opts):
errors.append(f"{prefix} {pid}: option entries must be tables with string ids")
p["option"] = clean_opts
for o in clean_opts:
if not isinstance(o.get("id"),str): errors.append(f"{prefix} {pid}: option id must be a string")
if o.get("kind") not in ("argv", "note"):
errors.append(f"{prefix} {pid}: option {o.get('id')} kind must be argv or note")
choices = o.get("choice", [])
if not isinstance(choices, list):
errors.append(f"{prefix} {pid}: option {o.get('id')} choices must be an array")
choices = []
choices = [ch for ch in choices if isinstance(ch, dict) and isinstance(ch.get("id"),str)]
o["choice"] = choices
if any(not isinstance(ch.get("id"),str) for ch in choices): errors.append(f"{prefix} {pid}: choice ids must be strings")
if o.get("kind") == "argv":
names = set()
for ch in choices:
vars_ = ch.get("vars", {})
if not isinstance(vars_, dict):
errors.append(f"{prefix} {pid}: choice {ch.get('id')} vars must be a table")
vars_ = ch["vars"] = {}
names.update(vars_)
for ch in choices:
missing = names - set(ch.get("vars", {}))
if missing:
errors.append(f"{prefix} {pid}: choice {ch.get('id')} missing vars: {', '.join(sorted(missing))}")
if o.get("default") not in [ch.get("id") for ch in choices]:
errors.append(f"{prefix} {pid}: option {o.get('id')} has invalid default")
compids = set()
for c in comps:
if not isinstance(c, dict):
errors.append(f"{prefix} {pid}: component entries must be tables")
continue
if c.get("kind") not in ("proc", "unit"):
errors.append(f"{prefix} {pid}: component kind must be proc or unit")
cid = c.get("id")
if isinstance(cid,str) and cid in compids:
errors.append(f"{prefix} {pid}: duplicate component id {cid}")
if isinstance(cid, str):
compids.add(cid)
if c.get("kind")=="unit" and not isinstance(c.get("unit"),str):
errors.append(f"{prefix} {pid}/{cid}: unit name is required")
if c.get("kind") == "proc":
argv = c.get("argv", [])
if not isinstance(argv, list) or not argv or any(not isinstance(x, str) for x in argv):
errors.append(f"{prefix} {pid}/{cid}: proc argv must be a non-empty string array")
argv = []
elif argv[0].startswith("/") and not Path(argv[0]).exists():
warnings.append(f"{pid}/{cid}: executable path does not exist: {argv[0]}")
providers = {name for o in clean_opts if o.get("kind") == "argv" for ch in o.get("choice", []) for name in ch.get("vars", {})}
for arg in argv:
for name in PLACEHOLDER.findall(arg):
if name not in providers:
errors.append(f"{prefix} {pid}: placeholder {{{name}}} has no option value")
if "env" in c and (not isinstance(c["env"], dict) or any(not isinstance(k, str) or not isinstance(v, str) for k,v in c["env"].items())):
errors.append(f"{prefix} {pid}/{cid}: env must map strings to strings")
if "health" in c and not isinstance(c["health"], dict):
errors.append(f"{prefix} {pid}/{cid}: health must be a table")
c["health"] = {"kind": "none"}
health=c.get("health",{"kind":"none"})
if isinstance(health,dict):
if health.get("kind") not in ("http","tcp","command","none"):
errors.append(f"{prefix} {pid}/{cid}: invalid health kind")
if health.get("kind")=="command" and (not isinstance(health.get("argv"),list) or not health.get("argv")):
errors.append(f"{prefix} {pid}/{cid}: command health requires argv")
cwd = c.get("cwd")
if cwd and not Path(cwd).exists():
warnings.append(f"{pid}/{cid}: configured cwd does not exist: {cwd}")
return data, errors, warnings
def problem_count(reg):
return len(reg.get("pipeline", []))
+66
View File
@@ -0,0 +1,66 @@
"""The sole subprocess boundary and command audit trail."""
import json, os, subprocess, threading, time
import signal
from datetime import datetime, timezone
def stamp(): return datetime.now(timezone.utc).isoformat(timespec="seconds")
class Runner:
def __init__(self, state_dir, dry_run=False, systemctl="systemctl", sudo="sudo"):
self.state_dir, self.dry_run, self.systemctl, self.sudo = state_dir, dry_run, systemctl, sudo
self.lock = threading.Lock(); self.audit_path = os.path.join(state_dir, "audit.jsonl")
os.makedirs(state_dir, exist_ok=True)
def run(self, argv, pipeline=None, action="probe", timeout=15, env=None, cwd=None, readonly=False):
start=time.monotonic(); rc=0; out=""
try:
if self.dry_run and not readonly:
out="dry-run: command not executed"
else:
p=subprocess.run(list(map(str,argv)), cwd=cwd, env=env, text=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, timeout=timeout, check=False)
rc=p.returncode; out=p.stdout or ""
except Exception as exc:
rc=127; out=f"{type(exc).__name__}: {exc}"
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":list(argv),"rc":rc,
"duration_ms":round((time.monotonic()-start)*1000),"dry_run":self.dry_run}
with self.lock:
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
return rc,out,row
def note(self,argv,action,pipeline=None,rc=0,duration_ms=0):
"""Audit a side effect that has no subprocess of its own - the HTTP reclaim
frees VRAM on the card like any command does, so the trail must carry it."""
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":list(map(str,argv)),"rc":rc,
"duration_ms":round(duration_ms),"dry_run":self.dry_run}
with self.lock:
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
return row
def audit(self,n=100):
try:
with open(self.audit_path,encoding="utf-8") as f: rows=[json.loads(x) for x in f if x.strip()]
return rows[-max(1,min(1000,n)):]
except OSError: return []
def spawn(self, argv, *, cwd=None, env=None, stdout=None, stderr=None, pipeline=None, action="start"):
"""Start an owned long-lived child; launch intent is audited by run()."""
start=time.monotonic()
try:
child=subprocess.Popen(list(map(str,argv)), cwd=cwd, env=env, stdin=subprocess.DEVNULL,
stdout=stdout, stderr=stderr, start_new_session=True)
except Exception as exc:
self.record_intent(argv,pipeline,action,cwd,rc=127,duration_ms=round((time.monotonic()-start)*1000),error=str(exc))
raise
self.record_intent(argv,pipeline,action,cwd,rc=0,duration_ms=round((time.monotonic()-start)*1000))
return child
def record_intent(self, argv, pipeline=None, action="start", cwd=None, *, rc=0, duration_ms=0, error=None):
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":list(argv),"rc":rc,
"duration_ms":duration_ms,"dry_run":False,"cwd":cwd}
if error: row["error"]=error
with self.lock:
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
def signal(self, pid, sig, pipeline=None, action="stop"):
start=time.monotonic(); rc=0; out=""
try: os.kill(pid,sig)
except ProcessLookupError: rc=1; out="process no longer exists"
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":["signal",signal.Signals(sig).name,str(pid)],"rc":rc,"duration_ms":round((time.monotonic()-start)*1000),"dry_run":False}
with self.lock:
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
return rc,out,row
+423
View File
@@ -0,0 +1,423 @@
"""HTTP API, asynchronous action jobs and snapshot assembly."""
import argparse, hmac, json, os, re, secrets, signal, socket, sys, threading, time, urllib.request
from collections import deque
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlparse, parse_qs, unquote
from . import __version__
from .registry import load, PLACEHOLDER
from .runner import Runner, stamp
from .probes import health, gpu, proc_start, friendly_error
def iso(): return datetime.now(timezone.utc).isoformat(timespec="seconds")
def is_loopback(addr):
"""Loopback is the machine itself: the desktop GUI and the TUI. Everything else has
crossed a network boundary and has to prove it may talk to us."""
a=str(addr or "")
return a=="::1" or a.startswith("127.") or a.startswith("::ffff:127.")
def bearer(headers,query):
got=(headers.get("Authorization") or "").strip()
if got[:7].lower()=="bearer ": return got[7:].strip()
return (query or {}).get("token",[""])[0]
def token_matches(expected,headers,query,client_addr):
"""No token configured == loopback-only: main() refuses to open a front-door listener
without one, so this can never be the thing that leaves a port wide open."""
if not expected: return True
if is_loopback(client_addr): return True
return hmac.compare_digest(bearer(headers,query),expected)
def load_token(path):
if not path: return None
try: return (Path(path).read_text(encoding="utf-8").strip() or None)
except FileNotFoundError: return None
except OSError as exc:
print(f"cannot read token file {path}: {exc}",file=sys.stderr); raise SystemExit(1)
class App:
def __init__(self,args):
self.args=args; self.registry,self.errors,self.warnings=load(args.registry)
self.pipelines={p.get("id"):p for p in self.registry["pipeline"] if isinstance(p,dict) and isinstance(p.get("id"),str) and re.fullmatch(r"[a-z0-9-]+",p.get("id"))}
self.runner=Runner(args.state_dir,args.dry_run,args.systemctl,args.sudo)
self.lock=threading.Lock(); self.action_lock=threading.Lock(); self.job_guard=threading.Lock(); self.jobs={}; self.active_job=None
self.history=deque(maxlen=90); self.last_gpu_sample=0; self.last_change={}; self.selected=self.read_selected()
def read_selected(self):
try:
import tomllib
with open(os.path.join(self.args.state_dir,"selected.toml"),"rb") as f: return tomllib.load(f)
except Exception: return {}
def save_selected(self):
path=os.path.join(self.args.state_dir,"selected.toml"); lines=[]
for pid,opts in self.selected.items():
if not isinstance(opts,dict): continue
lines.append(f'[{json.dumps(pid)}]')
lines.extend(f'{k} = {json.dumps(v)}' for k,v in opts.items())
Path(path).write_text("\n".join(lines)+("\n" if lines else ""),encoding="utf-8")
def component(self,p,c):
result={"id":c.get("id"),"kind":c.get("kind"),"label":c.get("label",c.get("id")),"state":"unknown","detail":"state unavailable","health":None,"port":None,"unit":None}
spec=c.get("health",{}); result["port"]=spec.get("port") if spec.get("kind")=="tcp" else None
if c.get("kind")=="unit":
unit=c.get("unit",""); result["unit"]=unit
rc,out,_=self.runner.run([self.args.systemctl,"show",unit,"--property=LoadState,ActiveState,SubState,ExecMainPID","--no-pager"],p.get("id"),"probe",timeout=4,readonly=True)
vals=dict(x.split("=",1) for x in out.splitlines() if "=" in x)
if rc or not vals: result["state"]="unknown"; result["detail"]=out.strip() or "systemd unit not found"; return result
if vals.get("LoadState")=="not-found": result["state"]="unknown"; result["detail"]=f"systemd unit not found: {unit}"; return result
if vals.get("ActiveState")=="active":
h=health(spec,self.runner,p.get("id")); result.update(state="up" if h["ok"] else "error",detail=f"pid {vals.get('ExecMainPID','?')}",health=h)
else: result.update(state="down",detail=vals.get("ActiveState","inactive"))
return result
pidfile=Path(self.args.state_dir)/"pids"/f"{p.get('id')}-{c.get('id')}.json"
recorded=None
try: recorded=json.loads(pidfile.read_text())
except Exception: pass
if recorded:
pid=recorded.get("pid"); start=proc_start(pid) if isinstance(pid,int) else None
if start is not None and str(start)==str(recorded.get("start")):
h=health(spec,self.runner,p.get("id")); result.update(state="up" if h["ok"] else "error",detail=f"pid {pid}",health=h); return result
h=health(spec,self.runner,p.get("id"))
if spec.get("kind")=="none": h={"ok":False,"detail":"no health check","latency_ms":0}
if h["ok"]: result.update(state="up",detail="external",health=h)
elif recorded: result.update(state="down",detail="stale pidfile; process identity changed",health=h)
else: result.update(state="down",detail="not running",health=h)
return result
def snapshot(self):
g=gpu(self.runner); cardcfg=self.registry.get("card",{}); total=cardcfg.get("total_mib") or g.get("total_mib"); floor=int(cardcfg.get("idle_floor_mib",0)); usable=max(0,(total or 0)-floor)
if g.get("available") and time.monotonic()-self.last_gpu_sample>=max(.1,float(cardcfg.get("poll_seconds",2))):
self.history.append({"t":iso(),"used_mib":g["used_mib"]})
self.last_gpu_sample=time.monotonic()
g.update(total_mib=total,free_mib=max(0,total-g["used_mib"]) if g.get("available") and total else None,idle_floor_mib=floor,usable_mib=usable,pct_of_usable=min(100,round(100*g["used_mib"]/usable)) if g.get("available") and usable else None,history=list(self.history))
pipes=[]; activejob=self.jobs.get(self.active_job) if self.active_job else None
for pid,p in self.pipelines.items():
comps=[self.component(p,c) for c in p.get("component",[])]; ups=sum(c["state"]=="up" for c in comps)
errors=sum(c["state"]=="error" for c in comps)
state="up" if comps and ups==len(comps) else "error" if errors and ups==0 else "partial" if ups else "down"
job=next((j for j in self.jobs.values() if j.get("pipeline")==pid and j.get("state")=="running"),None)
if job: state="stopping" if job.get("action")=="stop" else "starting"
options=[]; selected=self.selected.get(pid,{})
for o in p.get("option",[]):
choice=selected.get(o.get("id"),o.get("default"))
options.append({"id":o.get("id"),"label":o.get("label"),"kind":o.get("kind"),"selected":choice,"choices":[{"id":c.get("id"),"label":c.get("label"),"note":c.get("note"),"verified":c.get("verified"),"is_default":c.get("id")==o.get("default")} for c in o.get("choice",[])]})
pipes.append({"id":pid,"label":p.get("label",pid),"description":p.get("description",""),"state":state,"health":"ok" if state=="up" else "unknown" if state=="down" else "fail","requires_card":p.get("requires_card",False),"vram_mib":p.get("vram_mib"),"protected":p.get("protected",False),"serving":p.get("serving"),"open_url":p.get("open_url"),"note":p.get("note"),"last_change":self.last_change.get(pid),"components":comps,"selected":selected,"options":options,"job":({"id":job["id"],"state":job["state"],"step":job["steps"][-1]["text"] if job["steps"] else "working"} if job else None)})
if g.get("available"):
for pipe in pipes:
for comp in pipe["components"]:
detail=comp.get("detail","")
if comp.get("state")=="up" and detail.startswith("pid "):
try: pidnum=int(detail.split()[1])
except ValueError: continue
for pr in g["processes"]:
if pr["pid"]==pidnum: pr["pipeline"]=pipe["id"]
return {"host":socket.gethostname(),"now":iso(),"version":__version__,"card":g,"pipelines":pipes,"job":({"id":activejob["id"],"pipeline":activejob.get("pipeline"),"state":activejob["state"],"step":activejob["steps"][-1]["text"] if activejob["steps"] else "working","started":activejob["started"]} if activejob else None),"registry":{"path":self.args.registry,"pipeline_count":len(self.pipelines),"errors":self.errors,"warnings":self.warnings},"listeners":list(getattr(self,"listeners",[]))}
def start_job(self,pid,action,options=None,force=False):
with self.job_guard:
if self.active_job:
raise ValueError(f"another action is running ({self.active_job})")
jid=f"j-{int(time.time()*1000)}-{os.getpid()}"; job={"id":jid,"pipeline":pid,"action":action,"state":"running","started":iso(),"steps":[],"error":None}
self.jobs[jid]=job; self.active_job=jid
threading.Thread(target=self.work,args=(job,options or {},force),daemon=True).start(); return jid
def preflight(self,p,action,options,force):
if p.get("protected") and action in ("stop","restart") and not force:
live=[self.component(p,c)["state"] for c in p.get("component",[])]
if any(x in ("up","error","partial") for x in live):
raise PermissionError(f"protected pipeline {p['id']} is active; force is required to stop it")
if action in ("start","restart") and p.get("requires_card") and not force:
for other in self.pipelines.values():
if other["id"] != p["id"] and other.get("requires_card") and other.get("protected"):
if any(self.component(other,c)["state"] in ("up","error","partial") for c in other.get("component",[])):
raise PermissionError(f"protected pipeline {other['id']} is active; force is required")
if action in ("start","restart"):
for opt in p.get("option",[]):
oid=opt.get("id"); value=options.get(oid,self.selected.get(p["id"],{}).get(oid,opt.get("default")))
choice=next((x for x in opt.get("choice",[]) if x.get("id")==value),None)
if choice is None: raise LookupError(f"unknown option value {oid}={value}")
required=set().union(*(set(x.get("vars",{})) for x in opt.get("choice",[]))) if opt.get("kind")=="argv" else set()
if required-set(choice.get("vars",{})): raise LookupError(f"choice {value} is missing required vars")
def step(self,j,text,ok=True): j["steps"].append({"ts":iso(),"text":text,"ok":ok})
def work(self,j,options,force):
try:
if j["action"]=="free":
self.step(j,"asking the configured reclaim recipe to free the card")
self.reclaim(j)
else:
p=self.pipelines[j["pipeline"]]
if j["action"] in ("stop","restart"): self.stop_pipeline(j,p,force)
if j["action"] in ("start","restart"):
self.start_pipeline(j,p,options,force)
j["state"]="done"; self.last_change[j.get("pipeline")]=iso() if j.get("pipeline") else None
except Exception as exc:
j["error"]=str(exc); self.step(j,f"failed: {exc}",False); j["state"]="failed"
finally:
if self.active_job==j["id"]: self.active_job=None
def validate_options(self,p,requested):
out={}; psel=self.selected.setdefault(p["id"],{})
for o in p.get("option",[]):
oid=o.get("id"); choice=requested.get(oid,psel.get(oid,o.get("default")))
found=next((c for c in o.get("choice",[]) if c.get("id")==choice),None)
if not found: raise ValueError(f"unknown option value {oid}={choice}")
allvars=set().union(*(set(c.get("vars",{})) for c in o.get("choice",[]))) if o.get("kind")=="argv" else set()
if allvars-set(found.get("vars",{})): raise ValueError(f"choice {choice} is missing required vars")
psel[oid]=choice
if o.get("kind")=="argv": out.update(found.get("vars",{}))
self.save_selected(); return out
def start_pipeline(self,j,p,requested,force):
vars=self.validate_options(p,requested)
current=[self.component(p,c)["state"] for c in p.get("component",[])]
if current and all(x=="up" for x in current):
self.step(j,f"{p.get('label',p['id'])} is already healthy; no command was run")
return
if p.get("requires_card"):
for other in self.pipelines.values():
if other["id"]==p["id"] or not other.get("requires_card"): continue
state=[self.component(other,c)["state"] for c in other.get("component",[])]
if any(x in ("up","error","partial") for x in state):
if other.get("protected") and not force: raise RuntimeError(f"protected pipeline {other['id']} is active; force is required")
self.step(j,f"stopping {other.get('label',other['id'])} (releasing the card)…")
self.stop_pipeline(j,other,force,implicit=True)
reclaim=self.registry.get("card",{}).get("reclaim")
if reclaim: self.reclaim(j)
threshold=p.get("start_below_mib")
if threshold is not None:
g=gpu(self.runner); used=g.get("used_mib") if g.get("available") else None
deadline=time.monotonic()+min(120,int(reclaim.get("wait_timeout_s",45)) if reclaim else 45)
if used is None: self.step(j,"GPU probe unavailable; cannot verify VRAM before start",False)
else:
while used>=int(threshold) and time.monotonic()<deadline:
self.step(j,f"waiting for used VRAM < {threshold} MiB (now {used})…")
time.sleep(min(max(.1,float(self.registry.get("card",{}).get("poll_seconds",2))),max(0,deadline-time.monotonic())))
g=gpu(self.runner); used=g.get("used_mib") if g.get("available") else None
if used is None: break
if used is None or used>=int(threshold): self.step(j,f"VRAM wait timed out/unavailable (observed {used}); continuing with start",False)
else: self.step(j,f"used VRAM is {used} MiB; below {threshold} MiB")
for c in p.get("component",[]):
self.step(j,f"starting {c.get('label',c.get('id'))}…")
self.control(p,c,"start",vars)
if self.args.dry_run:
self.step(j,f"dry-run: would start {c.get('label',c.get('id'))}; health was not asserted")
continue
timeout=max(1,int(c.get("ready_timeout_s",20))); deadline=time.monotonic()+min(timeout,300); h=health(c.get("health",{}),self.runner,p["id"]); elapsed=0
while not h["ok"] and time.monotonic()<deadline:
time.sleep(min(.5,max(0,deadline-time.monotonic()))); elapsed+=.5; h=health(c.get("health",{}),self.runner,p["id"])
if elapsed and int(elapsed)%5==0: self.step(j,f"waiting for health ({int(elapsed)}s)…")
if not h["ok"]:
tail=self.logs(p["id"],20); raise RuntimeError(f"{c.get('label',c.get('id'))} health failed after {timeout}s: {h['detail']}; last log lines: {' | '.join(tail[-8:]) or 'log file does not exist'}")
self.step(j,f"{c.get('label',c.get('id'))} healthy ({h['detail']})")
def stop_pipeline(self,j,p,force,implicit=False):
if p.get("protected") and implicit and not force: raise RuntimeError(f"protected pipeline {p['id']} is active; force is required")
for c in reversed(p.get("component",[])):
state=self.component(p,c)
if state["state"]=="down": continue
if c.get("kind")=="proc" and state["detail"]=="external":
msg=f"cannot stop {c.get('label')}: not started by the cockpit (no owned pid)"
if implicit:
self.step(j,msg+" - leaving it running; the card must be freed another way",False); continue
raise RuntimeError(msg)
self.step(j,f"stopping {c.get('label',c.get('id'))}…")
self.control(p,c,"stop",{})
states=[self.component(p,c)["state"] for c in p.get("component",[])]
self.step(j,f"stop verification: {', '.join(states)}; GPU probe reports {gpu(self.runner).get('used_mib')} MiB used")
def control(self,p,c,action,vars):
pid=p["id"]; cid=c.get("id"); kind=c.get("kind")
if kind=="unit":
argv=[self.args.systemctl,action,c["unit"]]
if self.args.sudo: argv=[self.args.sudo,"-n",*argv]
rc,out,_=self.runner.run(argv,pid,action,timeout=30)
if rc: raise RuntimeError(f"systemctl {action} {c['unit']} failed: {out.strip()}")
return
pf=Path(self.args.state_dir)/"pids"/f"{pid}-{cid}.json"; pf.parent.mkdir(parents=True,exist_ok=True)
old=None
try: old=json.loads(pf.read_text())
except Exception: pass
if action=="stop":
if self.args.dry_run: self.runner.run(["kill","-TERM",str((old or {}).get("pid","unknown"))],pid,action); return
if not old or not isinstance(old.get("pid"),int) or proc_start(old["pid"]) is None or str(proc_start(old["pid"]))!=str(old.get("start")):
if health(c.get("health",{}),self.runner,pid)["ok"]: raise RuntimeError(f"refusing to stop {c.get('label')}: external process has no verified pid")
return
self.runner.signal(old["pid"],signal.SIGTERM,pid,action)
deadline=time.monotonic()+min(60,int(c.get("stop_timeout_s",20)))
while proc_start(old["pid"]) is not None and time.monotonic()<deadline: time.sleep(.2)
if proc_start(old["pid"]) is not None:
self.runner.signal(old["pid"],signal.SIGKILL,pid,action)
try: pf.unlink()
except OSError: pass
return
argv=[]
for arg in c.get("argv",[]):
v=PLACEHOLDER.sub(lambda m:str(vars.get(m.group(1),"")),str(arg))
if v: argv.append(v)
if self.args.dry_run:
self.runner.run(argv,pid,action,cwd=c.get("cwd")); return
# Start processes through the runner-owned subprocess API, then detach into their logfile.
log=Path(self.args.state_dir)/"logs"/f"{pid}.log"; log.parent.mkdir(parents=True,exist_ok=True)
env=os.environ.copy(); env.update({str(k):str(v) for k,v in c.get("env",{}).items()})
proc=self.runner.spawn(argv,cwd=c.get("cwd"),env=env,stdout=open(log,"ab"),stderr=__import__("subprocess").STDOUT,pipeline=pid,action=action)
time.sleep(.03); start=proc_start(proc.pid)
if start is None: raise RuntimeError(f"process exited during launch; see {log}")
pf.write_text(json.dumps({"pid":proc.pid,"start":start}),encoding="utf-8")
def reclaim(self,j):
rec=self.registry.get("card",{}).get("reclaim")
if not rec: self.step(j,"no card.reclaim recipe is configured",False); return
if rec.get("kind")=="http":
method,url=rec.get("method","POST"),rec.get("url","")
if self.args.dry_run: self.runner.run(["HTTP",method,url],None,"reclaim"); return
started=time.monotonic()
try:
req=urllib.request.Request(url,data=rec.get("body","").encode() if rec.get("body") else None,method=method,headers={"Content-Type":"application/json"})
with urllib.request.urlopen(req,timeout=10) as res:
self.runner.note(["HTTP",method,url],"reclaim",rc=res.status,duration_ms=(time.monotonic()-started)*1000)
self.step(j,f"reclaim endpoint returned HTTP {res.status}")
except Exception as exc:
self.runner.note(["HTTP",method,url],"reclaim",rc=127,duration_ms=(time.monotonic()-started)*1000)
self.step(j,f"reclaim endpoint failed: {friendly_error(exc,10)}",False)
elif rec.get("kind")=="command":
rc,out,_=self.runner.run(rec.get("argv",[]),None,"reclaim",timeout=30)
self.step(j,f"reclaim command exit {rc}: {out.strip()}",rc==0)
else: self.step(j,f"unknown reclaim kind {rec.get('kind')}",False)
if rec.get("wait_below_mib") is not None and not self.args.dry_run:
deadline=time.monotonic()+min(120,int(rec.get("wait_timeout_s",45))); observed=None
while time.monotonic()<deadline:
g=gpu(self.runner); observed=g.get("used_mib") if g.get("available") else None
if observed is not None and observed<int(rec["wait_below_mib"]):
self.step(j,f"used VRAM is {observed} MiB; below reclaim threshold {rec['wait_below_mib']} MiB"); return
if observed is None: break
time.sleep(min(max(.1,float(self.registry.get("card",{}).get("poll_seconds",2))),max(0,deadline-time.monotonic())))
self.step(j,f"reclaim VRAM wait timed out/unavailable (observed {observed} MiB)",False)
def logs(self,pid,n):
p=self.pipelines.get(pid,{})
units=[c.get("unit") for c in p.get("component",[]) if c.get("kind")=="unit" and c.get("unit")]
if units:
result=[]
for unit in units:
rc,out,_=self.runner.run(["journalctl","-u",unit,"-n",str(max(1,min(1000,n))),"--no-pager"],pid,"logs",timeout=8,readonly=True)
if rc: result.append(f"{unit}: journal unavailable: {out.strip()}")
else: result.extend(out.splitlines())
return result[-n:]
path=Path(self.args.state_dir)/"logs"/f"{pid}.log"
try: return path.read_text(errors="replace").splitlines()[-n:]
except OSError: return [f"log file does not exist: {path}"]
class Handler(BaseHTTPRequestHandler):
server_version="gpu-cockpit"
def log_message(self,*a): pass
@property
def app(self): return self.server.app
def sendj(self,obj,status=200):
b=json.dumps(obj).encode(); self.send_response(status); self.send_header("Content-Type","application/json"); self.send_header("Content-Length",str(len(b))); self.end_headers(); self.wfile.write(b)
def do_GET(self):
u=urlparse(self.path); path=u.path
if not token_matches(self.app.auth_token,self.headers,parse_qs(u.query),self.client_address[0]): return self.sendj({"error":"unauthorized"},401)
if path=="/": return self.file(Path(__file__).parent/"gui"/"index.html","text/html; charset=utf-8")
if path.startswith("/static/"):
name=unquote(path[len("/static/"):]); ext=Path(name).suffix.lower()
if "/" in name or ".." in name or ext not in (".png",".svg",".css",".js",".ico"): return self.send_error(404)
return self.file(Path(__file__).parent/"gui"/"static"/name, {".png":"image/png",".svg":"image/svg+xml",".css":"text/css",".js":"text/javascript",".ico":"image/x-icon"}[ext])
if path=="/api/ping": return self.sendj({"ok":True,"host":socket.gethostname(),"version":__version__,"now":iso()})
if path=="/api/state": return self.sendj(self.app.snapshot())
if path=="/api/registry": return self.sendj({"registry":self.app.registry,"errors":self.app.errors,"warnings":self.app.warnings})
if path=="/api/audit": return self.sendj(self.app.runner.audit(int(parse_qs(u.query).get("n",[100])[0])))
if path.startswith("/api/logs/"):
pid=path.rsplit("/",1)[-1]
if pid not in self.app.pipelines: return self.sendj({"error":"unknown pipeline"},404)
return self.sendj(self.app.logs(pid,int(parse_qs(u.query).get("n",[200])[0])))
if path.startswith("/api/jobs/"):
jid=path.rsplit("/",1)[-1]; job=self.app.jobs.get(jid)
return self.sendj(job or {"error":"job not found"},200 if job else 404)
if path=="/api/events":
q=parse_qs(u.query); once=q.get("once")==["1"]; data=json.dumps(self.app.snapshot())
self.send_response(200); self.send_header("Content-Type","text/event-stream"); self.send_header("Cache-Control","no-cache"); self.end_headers()
self.wfile.write(f"event: state\ndata: {data}\n\n".encode()); self.wfile.flush()
if not once:
try:
last=data; heartbeat=time.monotonic()
while True:
time.sleep(1); current=json.dumps(self.app.snapshot())
old_state=json.loads(last); new_state=json.loads(current)
old_state.pop("now",None); new_state.pop("now",None)
if old_state != new_state:
self.wfile.write(f"event: state\ndata: {current}\n\n".encode()); self.wfile.flush(); last=current
elif time.monotonic()-heartbeat>=15:
self.wfile.write(b": heartbeat\n\n"); self.wfile.flush(); heartbeat=time.monotonic()
except (BrokenPipeError,ConnectionResetError): pass
return
self.send_error(404)
def file(self,path,mime):
try: b=path.read_bytes()
except OSError: return self.send_error(404)
self.send_response(200); self.send_header("Content-Type",mime); self.send_header("Content-Length",str(len(b))); self.end_headers(); self.wfile.write(b)
def do_POST(self):
u=urlparse(self.path); path=u.path
if not token_matches(self.app.auth_token,self.headers,parse_qs(u.query),self.client_address[0]): return self.sendj({"error":"unauthorized"},401)
try: body=json.loads(self.rfile.read(int(self.headers.get("Content-Length",0))) or b"{}")
except Exception: return self.sendj({"error":"invalid JSON"},400)
try:
if path=="/api/card/free": jid=self.app.start_job(None,"free")
else:
sm=re.fullmatch(r"/api/pipelines/([a-z0-9-]+)/options",path)
if sm:
pid=sm.group(1); p=self.app.pipelines.get(pid)
if not p: return self.sendj({"error":"unknown pipeline"},404)
opts=body.get("options",{})
if not isinstance(opts,dict): return self.sendj({"error":"options must be an object"},400)
self.app.preflight(p,"start",opts,True)
self.app.validate_options(p,opts)
return self.sendj({"selected":self.app.selected.get(pid,{})})
m=re.fullmatch(r"/api/pipelines/([a-z0-9-]+)/action",path)
if not m: return self.send_error(404)
pid=m.group(1); p=self.app.pipelines.get(pid)
if not p: return self.sendj({"error":"unknown pipeline"},404)
action=body.get("action")
if action not in ("start","stop","restart"): return self.sendj({"error":"invalid action"},400)
options=body.get("options",{})
if not isinstance(options,dict): return self.sendj({"error":"options must be an object"},400)
for opt in p.get("option",[]):
if opt.get("id") in options and options[opt["id"]] not in [x.get("id") for x in opt.get("choice",[])]: return self.sendj({"error":f"unknown option value {opt['id']}={options[opt['id']] }"},400)
self.app.preflight(p,action,options,bool(body.get("force",False)))
jid=self.app.start_job(pid,action,options,bool(body.get("force",False)))
return self.sendj({"job_id":jid},202)
except PermissionError as exc: return self.sendj({"error":str(exc)},409)
except LookupError as exc: return self.sendj({"error":str(exc)},400)
except ValueError as exc: return self.sendj({"error":str(exc)},409)
except Exception as exc: return self.sendj({"error":str(exc)},500)
def serve_extras(app,hosts,port):
"""Bind the front-door listeners, retrying until they exist. A WireGuard address only
appears once wg0 is up, which can happen after this service starts - without the retry a
boot race would close the door permanently and look like an auth problem from the phone."""
pending=list(hosts)
while pending:
for h in list(pending):
try: srv=ThreadingHTTPServer((h,port),Handler)
except OSError as exc:
print(f"front door {h}:{port} not available yet ({exc}) - retrying in 5s",flush=True); continue
srv.app=app; app.listeners.append(f"{h}:{port}")
print(f"listening on http://{h}:{port} (front door)",flush=True)
threading.Thread(target=srv.serve_forever,daemon=True).start()
pending.remove(h)
if pending: time.sleep(5)
def main(argv=None):
home=str(Path.home()); envreg=os.environ.get("COCKPIT_REGISTRY")
ap=argparse.ArgumentParser(); ap.add_argument("--registry",default=envreg or os.path.join(home,".config/gpu-cockpit/registry.toml")); ap.add_argument("--host",default="127.0.0.1",help="comma-separated; anything non-loopback needs a token"); ap.add_argument("--token-file",default=os.path.join(home,".config/gpu-cockpit/token")); ap.add_argument("--port",type=int,default=8770); ap.add_argument("--state-dir",default=os.path.join(home,".local/state/gpu-cockpit")); ap.add_argument("--systemctl",default="systemctl"); ap.add_argument("--sudo",default=None); ap.add_argument("--dry-run",action="store_true"); ap.add_argument("--check",action="store_true"); args=ap.parse_args(argv)
reg,errors,warnings=load(args.registry)
if args.check:
print(f"registry OK: {len(reg.get('pipeline',[]))} pipelines, {len(errors)} errors, {len(warnings)} warnings")
for x in errors: print(f"ERROR: {x}")
for x in warnings: print(f"WARNING: {x}")
return 1 if errors else 0
app=App(args)
app.auth_token=load_token(args.token_file); app.listeners=[]
hosts=[h.strip() for h in str(args.host).split(",") if h.strip()] or ["127.0.0.1"]
extras=[h for h in hosts if not is_loopback(h)]
if extras and not app.auth_token:
print(f"refusing to listen on {', '.join(extras)} with no token ({args.token_file}) - that is an open front door",file=sys.stderr); return 1
try: server=ThreadingHTTPServer((hosts[0],args.port),Handler)
except OSError as exc: print(f"cannot listen on http://{hosts[0]}:{args.port}: {exc}",file=sys.stderr); return 1
server.app=app; app.listeners.append(f"{hosts[0]}:{args.port}")
print(f"listening on http://{hosts[0]}:{args.port}",flush=True)
if extras: threading.Thread(target=serve_extras,args=(app,extras,args.port),daemon=True).start()
try: server.serve_forever()
except KeyboardInterrupt: pass
finally: server.server_close()
return 0
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=GPU Cockpit single-card pipeline console
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/python3 -m cockpit --registry %h/.config/gpu-cockpit/registry.toml --state-dir %h/.local/state/gpu-cockpit --host 127.0.0.1 --port 8770
Restart=on-failure
RestartSec=3
[Install]
WantedBy=default.target
+119
View File
@@ -0,0 +1,119 @@
[card]
label = "RTX 4060"
total_mib = 8188
idle_floor_mib = 1700
poll_seconds = 2
[card.reclaim]
label = "ask the image server to unload models"
kind = "http"
url = "http://127.0.0.1:8188/free"
method = "POST"
body = '{"unload_models": true, "free_memory": true}'
wait_below_mib = 2000
wait_timeout_s = 45
[[pipeline]]
id = "image"
label = "Image generation"
description = "An image generation service on the shared card"
requires_card = true
vram_mib = 7019
protected = false
start_below_mib = 2600
serving = "Z-Image-Turbo Q4_K_M + LoRA ladder"
open_url = "http://127.0.0.1:8188"
note = "Peaks near 7 GB; leave the card alone while it renders."
[[pipeline.component]]
kind = "proc"
id = "image-server"
label = "Image server"
argv = ["/usr/bin/python3", "-m", "http.server", "8188", "--bind", "127.0.0.1", "{extra}"]
cwd = "/tmp"
env = { PYTHONUNBUFFERED = "1" }
ready_timeout_s = 15
stop_timeout_s = 5
health = { kind = "http", url = "http://127.0.0.1:8188/", timeout_s = 2 }
[[pipeline.option]]
id = "model"
label = "Model"
kind = "argv"
default = "turbo"
[[pipeline.option.choice]]
id = "turbo"
label = "Z-Image Turbo"
note = "Fast image generation"
verified = "2026-09-26"
vars = { model = "/models/z-image.gguf", alias = "image-turbo", extra = "" }
[[pipeline.option.choice]]
id = "quality"
label = "Image quality model"
vars = { model = "/models/image-quality.gguf", alias = "image-quality", extra = "--directory ." }
[[pipeline.option]]
id = "profile"
label = "Profile note"
kind = "note"
default = "desktop"
[[pipeline.option.choice]]
id = "desktop"
label = "Desktop preset"
note = "Leaves memory available for the desktop."
[[pipeline.option.choice]]
id = "render"
label = "Render preset"
note = "Use for dedicated render sessions."
[[pipeline]]
id = "voice"
label = "Voice stack"
description = "A systemd managed voice pipeline"
requires_card = true
vram_mib = 3200
protected = false
serving = "Speech and verbalization"
[[pipeline.component]]
kind = "unit"
id = "voice-server"
label = "Voice server"
unit = "gpu-cockpit-example-voice.service"
ready_timeout_s = 20
health = { kind = "command", argv = ["/usr/bin/test", "-e", "/dev/null"], timeout_s = 2 }
[[pipeline]]
id = "trainer"
label = "Trainer"
description = "Explicitly protected card workload"
requires_card = true
protected = true
vram_mib = 7000
serving = "Training job"
[[pipeline.component]]
kind = "proc"
id = "trainer-proc"
label = "Training process"
argv = ["/bin/sleep", "300"]
ready_timeout_s = 2
stop_timeout_s = 2
health = { kind = "none" }
[[pipeline]]
id = "cpu-llm"
label = "CPU model"
description = "Runs without claiming the GPU"
requires_card = false
vram_mib = 0
serving = "CPU inference"
[[pipeline.component]]
kind = "proc"
id = "cpu"
label = "CPU service"
argv = ["/bin/sleep", "300"]
ready_timeout_s = 2
stop_timeout_s = 2
health = { kind = "none" }
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd)"
exec python3 "$root/tests/selftest.py" "$root"
+119
View File
@@ -0,0 +1,119 @@
import json, os, pathlib, socket, subprocess, sys, tempfile, threading, time, urllib.error, urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer
root=pathlib.Path(sys.argv[1]); tmp=pathlib.Path(tempfile.mkdtemp(prefix='cockpit-test-')); cases=0; proc=None
def check(name, test):
global cases
if not test: print(f'FAIL: {name}: assertion failed',flush=True); raise SystemExit(1)
cases+=1; print(f'PASS: {name}',flush=True)
class Reclaim(BaseHTTPRequestHandler):
hits=0
def do_POST(self): Reclaim.hits+=1; self.send_response(200); self.end_headers(); self.wfile.write(b'ok')
def log_message(self,*a): pass
reclaim=HTTPServer(('127.0.0.1',0),Reclaim); threading.Thread(target=reclaim.serve_forever,daemon=True).start()
stub=tmp/'systemctl'; stub.write_text('''#!/usr/bin/env sh
s="$COCKPIT_STUB_STATE"; c="$1"; u="$2"
case "$c" in
show) if grep -qx "$u" "$s" 2>/dev/null; then echo ActiveState=active; echo SubState=running; echo ExecMainPID=1; else echo ActiveState=inactive; echo SubState=dead; echo ExecMainPID=0; fi;;
start|restart) grep -vx "$u" "$s" 2>/dev/null > "$s.tmp" || true; echo "$u" >> "$s.tmp"; mv "$s.tmp" "$s";;
stop) grep -vx "$u" "$s" 2>/dev/null > "$s.tmp" || true; mv "$s.tmp" "$s";;
is-active) grep -qx "$u" "$s" && exit 0 || exit 3;; *) exit 0;; esac
'''); stub.chmod(0o755)
port=socket.socket(); port.bind(('127.0.0.1',0)); pnum=port.getsockname()[1]; port.close()
reg=tmp/'registry.toml'
reg.write_text(f'''[card]\ntotal_mib=8188\nidle_floor_mib=1700\npoll_seconds=0.05\n[card.reclaim]\nkind="http"\nurl="http://127.0.0.1:{reclaim.server_port}/free"\nmethod="POST"\nwait_timeout_s=1\n[[pipeline]]\nid="a"\nlabel="A"\nrequires_card=true\nstart_below_mib=2600\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="A process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline]]\nid="b"\nlabel="B"\nrequires_card=true\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="B process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline]]\nid="protected"\nlabel="Protected"\nrequires_card=true\nprotected=true\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Protected process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline]]\nid="bad-health"\nlabel="Unreachable"\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Bad process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=1\nstop_timeout_s=1\nhealth={{kind="tcp",host="127.0.0.1",port=1,timeout_s=0.1}}\n[[pipeline]]\nid="options"\nlabel="Options"\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Option process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)","{{extra}}"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline.option]]\nid="model"\nlabel="Model"\nkind="argv"\ndefault="one"\n[[pipeline.option.choice]]\nid="one"\nlabel="One"\nvars={{extra=""}}\n[[pipeline.option.choice]]\nid="two"\nlabel="Two"\nvars={{extra="-v"}}\n[[pipeline.option]]\nid="required"\nlabel="Required var"\nkind="argv"\ndefault="present"\n[[pipeline.option.choice]]\nid="present"\nlabel="Present"\nvars={{required="yes"}}\n[[pipeline.option.choice]]\nid="absent"\nlabel="Missing required value"\nvars={{}}\n[[pipeline]]\nid="broken"\nlabel="Broken"\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Broken"\nargv=["{sys.executable}","{{missing}}"]\nhealth={{kind="none"}}\n[[pipeline]]\nid="external"\nlabel="External"\nrequires_card=true\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="External process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nhealth={{kind="tcp",host="127.0.0.1",port={reclaim.server_port},timeout_s=1}}\n''')
state=tmp/'state'; env=os.environ.copy(); env['COCKPIT_STUB_STATE']=str(tmp/'units'); log=open(tmp/'server.log','w+')
proc=subprocess.Popen([sys.executable,'-m','cockpit','--registry',str(reg),'--port',str(pnum),'--state-dir',str(state),'--systemctl',str(stub)],cwd=root,stdout=log,stderr=log,env=env)
base=f'http://127.0.0.1:{pnum}'
def get(path):
with urllib.request.urlopen(base+path,timeout=5) as r:return r.status,r.read().decode(),r.headers
def post(path,obj):
req=urllib.request.Request(base+path,data=json.dumps(obj).encode(),headers={'Content-Type':'application/json'},method='POST')
try:
with urllib.request.urlopen(req,timeout=5) as r:return r.status,json.loads(r.read())
except urllib.error.HTTPError as e:return e.code,json.loads(e.read())
def snapshot():return json.loads(get('/api/state')[1])
def job(jid):
end=time.time()+5
while time.time()<end:
j=json.loads(get('/api/jobs/'+jid)[1])
if j['state']!='running':return j
time.sleep(.05)
raise AssertionError('job timed out')
def action(pid,act='start',**kw):
code,r=post(f'/api/pipelines/{pid}/action',{'action':act,**kw})
return job(r['job_id']) if code==202 else (code,r)
try:
deadline=time.time()+5
while time.time()<deadline:
try: status,raw,_=get('/api/state'); break
except Exception: time.sleep(.05)
check('server starts, listens, and state shape',status==200 and all(k in json.loads(raw) for k in ('host','card','pipelines','registry')))
check('fixture pipeline count',len(json.loads(raw)['pipelines'])==7)
check('down state for not-started proc',next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
check('unreachable health state is down',next(x for x in snapshot()['pipelines'] if x['id']=='bad-health')['state']=='down')
j=action('a'); steps=' '.join(x['text'] for x in j['steps'])
vram_verdict=('GPU probe unavailable' in steps) or ('VRAM wait' in steps) or ('below 2600 MiB' in steps)
check('start verifies health/PID with an honest VRAM verdict',j['state']=='done' and vram_verdict and 'healthy' in steps and next(x for x in snapshot()['pipelines'] if x['id']=='a')['components'][0]['detail'].startswith('pid '))
j=action('a','stop'); check('stop verifies pipeline down',j['state']=='done' and next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
action('a'); j=action('b'); check('arbitration stops A before B',j['state']=='done' and any('stopping A' in x['text'] for x in j['steps']) and next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
action('protected'); status,body=post('/api/pipelines/b/action',{'action':'start'}); check('protected switch returns 409 without change',status==409 and 'protected' in body['error'] and next(x for x in snapshot()['pipelines'] if x['id']=='protected')['state']=='up')
status,body=post('/api/pipelines/protected/action',{'action':'stop'}); check('protected stop requires force',status==409 and 'protected' in body['error'] and next(x for x in snapshot()['pipelines'] if x['id']=='protected')['state']=='up')
j=action('protected','stop',force=True); check('force stop of a protected pipeline works',j['state']=='done' and next(x for x in snapshot()['pipelines'] if x['id']=='protected')['state']=='down')
j=action('b','start',force=True); check('force allows protected switch',j['state']=='done')
check('reclaim HTTP recipe called',Reclaim.hits>=2)
action('b')
ext=next(x for x in snapshot()['pipelines'] if x['id']=='external')
check('unowned but answering component reads up/external',ext['state']=='up' and ext['components'][0]['detail']=='external')
j=action('external','stop'); check('explicit stop of an unowned process fails loudly',j['state']=='failed' and 'not started by the cockpit' in j['error'])
j=action('a','start'); check('a switch leaves an unowned holder running and still proceeds',j['state']=='done' and any('leaving it running' in x['text'] for x in j['steps']))
status,_=post('/api/pipelines/options/action',{'action':'start','options':{'model':'invalid'}}); check('unknown option value returns 400',status==400)
status,_=post('/api/pipelines/options/action',{'action':'start','options':{'required':'absent'}}); check('choice missing a declared var returns 400',status==400)
status,body=post('/api/pipelines/options/options',{'options':{'model':'two'}}); check('option selection persists',status==200 and next(x for x in snapshot()['pipelines'] if x['id']=='options')['selected'].get('model')=='two')
j=action('bad-health'); check('ready timeout fails with log tail',j['state']=='failed' and 'last log lines' in j['error'])
pf=state/'pids'/'a-p.json'; pf.parent.mkdir(parents=True,exist_ok=True); pf.write_text('not json'); check('garbage pidfile reads down',next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
pf.write_text(json.dumps({'pid':os.getpid(),'start':'wrong'})); check('reused pid mismatch reads down',next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
status,raw,_=get('/api/events?once=1'); check('SSE once emits state event',status==200 and 'event: state' in raw)
status,html,_=get('/'); check('GUI markup served',status==200 and 'Free the card' in html)
status,raw,_=get('/api/registry'); check('broken registry validation reported',status==200 and bool(json.loads(raw)['errors']))
# Verify dry-run records intent without creating proc children.
dryport=socket.socket(); dryport.bind(('127.0.0.1',0)); dp=dryport.getsockname()[1]; dryport.close()
dry=subprocess.Popen([sys.executable,'-m','cockpit','--registry',str(reg),'--port',str(dp),'--state-dir',str(tmp/'dry'),'--dry-run'],cwd=root,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True,env=env)
try:
line=dry.stdout.readline(); check('dry-run server starts',line.startswith('listening on'))
req=urllib.request.Request(f'http://127.0.0.1:{dp}/api/pipelines/a/action',data=b'{"action":"start"}',headers={'Content-Type':'application/json'},method='POST')
jid=json.loads(urllib.request.urlopen(req).read())['job_id']; time.sleep(.2)
req=urllib.request.Request(f'http://127.0.0.1:{dp}/api/pipelines/options/action',data=b'{"action":"start","options":{"model":"two"}}',headers={'Content-Type':'application/json'},method='POST')
urllib.request.urlopen(req).read(); time.sleep(.2); rows=[json.loads(x) for x in (tmp/'dry'/'audit.jsonl').read_text().splitlines()]
check('dry-run audits and launches no process',any(x['dry_run'] for x in rows) and not (tmp/'dry'/'pids'/'a-p.json').exists())
check('selected option changes recorded argv',any(x['pipeline']=='options' and '-v' in x['argv'] for x in rows))
finally: dry.terminate(); dry.wait(timeout=3)
# --- the front door: the tunnel listener is token-gated, and cannot be opened without one ---
if str(root) not in sys.path: sys.path.insert(0,str(root))
from cockpit import server as srvmod
check('loopback needs no token (the desktop GUI and the TUI live there)',srvmod.token_matches('sekret',{},{},'127.0.0.1'))
check('ipv6 loopback counts as loopback',srvmod.token_matches('sekret',{},{},'::1'))
check('a tunnel caller with no token is refused',not srvmod.token_matches('sekret',{},{},'192.168.2.1'))
check('a tunnel caller with the right bearer token gets in',srvmod.token_matches('sekret',{'Authorization':'Bearer sekret'},{},'192.168.2.1'))
check('a wrong bearer token is refused',not srvmod.token_matches('sekret',{'Authorization':'Bearer nope'},{},'192.168.2.1'))
check('a query token serves clients that cannot set headers',srvmod.token_matches('sekret',{}, {'token':['sekret']},'192.168.2.1'))
(tmp/'tok').write_text('sekret\n')
check('a token file is read back trimmed',srvmod.load_token(str(tmp/'tok'))=='sekret')
check('a missing token file means no token, not an empty one',srvmod.load_token(str(tmp/'absent')) is None)
spare=socket.socket(); spare.bind(('127.0.0.1',0)); sport=spare.getsockname()[1]; spare.close()
guard=subprocess.run([sys.executable,'-m','cockpit','--registry',str(reg),'--port',str(sport),'--state-dir',str(tmp/'guard'),
'--host','127.0.0.1,10.10.10.1','--token-file',str(tmp/'absent'),'--dry-run'],
cwd=root,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True,env=env,timeout=30)
check('a front door with no token is refused outright, not silently left open',guard.returncode==1 and 'open front door' in guard.stdout)
print(f'ALL TESTS PASSED ({cases} cases)',flush=True)
finally:
if proc:
proc.terminate()
try:proc.wait(timeout=3)
except subprocess.TimeoutExpired:proc.kill();proc.wait()
for f in (state/'pids').glob('*.json') if (state/'pids').exists() else []:
try:
pid=json.loads(f.read_text())['pid']
if pid != os.getpid(): os.kill(pid,15)
except Exception:pass
reclaim.shutdown(); log.close()
+111
View File
@@ -0,0 +1,111 @@
#!/usr/bin/env python3
"""Textual HTTP client for gpu-cockpit; --print needs only the standard library."""
import argparse, json, sys, time, urllib.error, urllib.request
def get(url):
with urllib.request.urlopen(url,timeout=3) as r: return json.loads(r.read())
def plain(url):
try:
s=get(url.rstrip('/')+'/api/state')
except Exception as exc:
print(f"Cannot reach gpu-cockpit at {url}: {exc}. Start it with: python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml")
return 2
c=s['card'];
if c['available']: print(f"{s['host']} · GPU {c['used_mib']}/{c['usable_mib']} MiB usable · {c['util_pct']}% · {c['temp_c']}°C")
else: print(f"{s['host']} · no GPU probe ({c.get('detail','unavailable')})")
for p in s['pipelines']: print(f"{p['state']:9} {p['id']:<16} {p['label']} · {p.get('vram_mib') or 0} MiB")
return 0
def main():
ap=argparse.ArgumentParser(); ap.add_argument('--url',default='http://127.0.0.1:8770'); ap.add_argument('--refresh',type=float,default=2); ap.add_argument('--print',dest='plain',action='store_true'); a=ap.parse_args()
if a.plain: return plain(a.url)
try:
from textual.app import App, ComposeResult
from textual.containers import Horizontal, Vertical
from textual.widgets import Button, DataTable, Footer, Header, Select, Static
except ImportError:
print('Textual is required for the interactive TUI. Install it with: pip install textual',file=sys.stderr); return 2
class Cockpit(App):
TITLE='GPU Cockpit'; BINDINGS=[('q','quit','Quit'),('r','reload','Refresh'),('s','start','Start'),('x','stop','Stop'),('R','restart','Restart'),('f','free','Free card'),('l','logs','Logs'),('m','choose','Choose option')]
CSS='Screen{layout:vertical} #content{height:1fr} #pipes{width:40%} #detail{width:60%;border:round $primary;padding:1} #step{height:3;color:$accent}'
def compose(self)->ComposeResult:
yield Header(); yield Static('Connecting…',id='step')
with Horizontal(id='content'):
yield DataTable(id='pipes'); yield Static('Select a pipeline',id='detail')
yield Footer()
def on_mount(self):
self.query_one('#pipes',DataTable).add_columns('State','ID','Pipeline','VRAM MiB'); self.set_interval(a.refresh,self.reload_state)
def reload_state(self):
try: self.snap=get(a.url.rstrip('/')+'/api/state'); table=self.query_one('#pipes',DataTable); table.clear()
except Exception as exc:
self.snap=None; self.query_one('#step',Static).update(f"Server unavailable at {a.url}. Start: python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml ({exc})"); return
c=self.snap['card']; head=f"{self.snap['host']} · "+(f"GPU {c['used_mib']}/{c['usable_mib']} MiB · {c['util_pct']}% · {c['temp_c']}°C" if c['available'] else 'no GPU probe')
if self.snap.get('job'): head+=f" · {self.snap['job']['step']}"
self.query_one('#step',Static).update(head)
for p in self.snap['pipelines']: table.add_row({'up':'●','down':'○','partial':'◐','starting':'◒','stopping':'◓'}.get(p['state'],'?'),p['id'],p['label'],str(p.get('vram_mib') or 0),key=p['id'])
def on_data_table_row_highlighted(self,event):
if not self.snap:return
p=next((x for x in self.snap['pipelines'] if x['id']==str(event.row_key.value)),None)
if p:
try: logs='\n'.join(get(a.url.rstrip('/')+f"/api/logs/{p['id']}?n=20"))
except Exception as exc: logs=f"Logs unavailable: {exc}"
opts='\n'.join(f"{o['label']}: {o['selected']}" for o in p['options']) or 'No options'
comps='\n'.join(f"{c['label']}: {c['state']} · {c.get('health',{}).get('detail') if c.get('health') else c['detail']}" for c in p['components'])
detail=f"{p['label']} · {p['state']}\n{p['description']}\nServing: {p.get('serving') or '—'}\nVRAM: {p.get('vram_mib') or 0} MiB\n\nComponents\n{comps}\n\nOptions\n{opts}\n\nLast 20 log lines\n{logs}"
if p.get('job'): detail+=f"\n\nJob {p['job']['id']}: {p['job']['step']}"
self.query_one('#detail',Static).update(detail)
def selected(self):
if not self.snap:return None
key=self.query_one('#pipes',DataTable).cursor_row
try: pid=self.query_one('#pipes',DataTable).get_row_at(key)[1]
except Exception:return None
return next((x for x in self.snap['pipelines'] if x['id']==pid),None)
def post(self,path,data):
req=urllib.request.Request(a.url.rstrip('/')+path,data=json.dumps(data).encode(),headers={'Content-Type':'application/json'},method='POST')
try: urllib.request.urlopen(req,timeout=3).read()
except Exception as exc:self.query_one('#step',Static).update(str(exc))
self.set_timer(.2,self.reload_state)
def action(self,name):
p=self.selected()
if p:self.post(f"/api/pipelines/{p['id']}/action",{'action':name,'options':p['selected']})
def action_reload(self):self.reload_state()
def action_start(self):self.action('start')
def action_stop(self):self.action('stop')
def action_restart(self):self.action('restart')
def action_free(self):self.post('/api/card/free',{})
def action_logs(self):
p=self.selected()
if p:
try:self.push_screen(LogsScreen('\n'.join(get(a.url.rstrip('/')+f"/api/logs/{p['id']}?n=20"))))
except Exception as exc:self.query_one('#step',Static).update(str(exc))
def action_choose(self):
p=self.selected()
if p:self.push_screen(OptionScreen(p))
from textual.screen import ModalScreen
class LogsScreen(ModalScreen):
BINDINGS=[('escape','dismiss','Close')]
def __init__(self,text):super().__init__();self.text=text
def compose(self):yield Static(self.text)
class OptionScreen(ModalScreen):
BINDINGS=[('escape','dismiss','Close')]
CSS='OptionScreen{align: center middle} #box{width:70%;height:auto;max-height:80%;border:round $primary;background:$surface;padding:1 2} Select{margin:1 0} Button{margin-top:1}'
def __init__(self,pipeline):super().__init__();self.pipeline=pipeline;self.values=dict(pipeline['selected'])
def compose(self):
with Vertical(id='box'):
yield Static(f"Choose options · {self.pipeline['label']}")
for option in self.pipeline['options']:
values=[(c['label'],c['id']) for c in option['choices']]
if values: yield Select(values,value=option['selected'],prompt=option['label'],id='option-'+option['id'])
yield Button('Save selection',variant='primary',id='save')
def on_select_changed(self,event):
if event.select.id and event.select.id.startswith('option-') and event.value is not Select.BLANK:
self.values[event.select.id[7:]]=event.value
def on_button_pressed(self,event):
if event.button.id=='save':
path=f"/api/pipelines/{self.pipeline['id']}/options"
req=urllib.request.Request(a.url.rstrip('/')+path,data=json.dumps({'options':self.values}).encode(),headers={'Content-Type':'application/json'},method='POST')
try: urllib.request.urlopen(req,timeout=3).read(); self.dismiss(self.values)
except Exception as exc: self.query_one(Static).update(str(exc))
Cockpit().run(); return 0
if __name__=='__main__': raise SystemExit(main())