Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2f4614f353 | ||
|
|
6fa69c4a31 | ||
|
|
6029bfed64 | ||
|
|
cb4d250eb2 | ||
|
|
86055b2e69 | ||
|
|
eb217ea3e1 | ||
|
|
5ad1e02168 | ||
|
|
8af2e04866 | ||
|
|
ff11e0189f | ||
|
|
152db1098e |
@@ -0,0 +1,3 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
@@ -0,0 +1,54 @@
|
||||
# gpu-cockpit — agent notes
|
||||
|
||||
A registry-driven console for a **single shared inference GPU**. One server owns state and command
|
||||
execution; the web GUI and the terminal TUI are thin clients of its HTTP API. Nothing in the code
|
||||
knows about any specific service — that all lives in the TOML registry.
|
||||
|
||||
## Layout
|
||||
|
||||
- `cockpit/` — the package: registry, runner, GPU probe, unit/proc control, health, state,
|
||||
arbitration, HTTP server, and the inline GUI under `cockpit/gui/`. **stdlib only.**
|
||||
- `tui/cockpit.py` — terminal client. The only place `textual` may be imported.
|
||||
- `tests/`, `scripts/selftest.sh` — fixture-driven; must pass with no GPU and no real systemd.
|
||||
- `registry.example.toml` — documentation by example. Real registries are deployment data.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
python3 -m cockpit --registry registry.example.toml --check # validate, no server
|
||||
python3 -m cockpit --registry registry.example.toml --port 8770
|
||||
python3 -m cockpit --registry registry.example.toml --dry-run # executes nothing, reports intent
|
||||
bash scripts/selftest.sh # the acceptance suite
|
||||
python3 -m compileall -q cockpit tui
|
||||
python3 tui/cockpit.py --print # no TTY, no textual needed
|
||||
```
|
||||
|
||||
Run `bash scripts/selftest.sh` before claiming anything works. It is the contract, not a smoke test.
|
||||
|
||||
## Boundaries
|
||||
|
||||
**Always do**
|
||||
- Route every executed command through the single runner, so the audit log stays complete.
|
||||
- Treat a health probe as the only evidence a pipeline is up; a `0` exit from a start command is not.
|
||||
- Bound every wait, and report a timeout as a failure with the real observed state.
|
||||
- Bind `127.0.0.1` unless `--host` is passed explicitly.
|
||||
- Keep registry problems as validation errors/warnings shown in the UI, never a crash at boot.
|
||||
|
||||
**Ask first**
|
||||
- Adding any dependency, including to `tui/`.
|
||||
- Changing the registry schema or an existing field's meaning.
|
||||
- Renaming an API route or a key inside `/api/state` (both front ends depend on them).
|
||||
|
||||
**Never do**
|
||||
- `shell=True`, or building a command from an HTTP request's contents. The API accepts pipeline
|
||||
ids, action names and option ids, and validates each against the registry.
|
||||
- Stop a `protected = true` pipeline implicitly, or infer it from a `0` exit code.
|
||||
- Write to the registry file from the API. Selection state is the most the server may write.
|
||||
- Edit files outside this repository, or assume a deployment host's paths exist.
|
||||
|
||||
## Definition of done
|
||||
|
||||
1. `bash scripts/selftest.sh` exits 0 and prints one `PASS:` line per case.
|
||||
2. `python3 -m compileall -q cockpit tui` is silent.
|
||||
3. `python3 -m cockpit --registry registry.example.toml --check` reports `0 errors, 0 warnings`.
|
||||
4. The README documents how to add a new pipeline without touching code — and it is true.
|
||||
@@ -1,3 +1,114 @@
|
||||
# gpu-cockpit
|
||||
|
||||
GPU + pipeline cockpit: registry-driven VRAM/state console with TUI and web GUI
|
||||
`gpu-cockpit` is a registry-driven console for one shared inference GPU. A single Python server
|
||||
owns host probes, pipeline state, process/unit control, action jobs and the audit trail. The browser
|
||||
GUI and terminal TUI are HTTP clients; service-specific details stay in TOML.
|
||||
|
||||
## Run
|
||||
|
||||
Requires Python 3.11+ (stdlib only for the server and GUI). Copy
|
||||
[`registry.example.toml`](registry.example.toml) to `~/.config/gpu-cockpit/registry.toml`, then:
|
||||
|
||||
```sh
|
||||
python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml --check
|
||||
python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml --port 8770
|
||||
```
|
||||
|
||||
The server binds to `127.0.0.1` by default. Set `--host` to bind elsewhere deliberately. State,
|
||||
logs, selections and audit records go in `~/.local/state/gpu-cockpit` by default. `COCKPIT_REGISTRY`
|
||||
can provide the default registry path. `--dry-run` serves the API and performs read-only health
|
||||
checks while recording intended commands without starting or stopping processes.
|
||||
|
||||
Open `http://127.0.0.1:8770/` for the responsive web GUI. The TUI uses the same API:
|
||||
|
||||
```sh
|
||||
python3 tui/cockpit.py --url http://127.0.0.1:8770
|
||||
python3 tui/cockpit.py --print --url http://127.0.0.1:8770
|
||||
```
|
||||
|
||||
Install `textual` to use the interactive TUI (`pip install textual`). `--print` needs no optional
|
||||
dependency or terminal.
|
||||
|
||||
## Add a pipeline
|
||||
|
||||
Add another `[[pipeline]]` block in the registry. No Python change is required. Each pipeline needs
|
||||
one or more components. Components start in listed order and stop in reverse order. A proc component
|
||||
is spawned and supervised by the server; a unit component is operated through `systemctl`.
|
||||
`health` can use `http`, `tcp`, `command` or `none`.
|
||||
|
||||
For example, a CPU-only local service can run alongside a card owner:
|
||||
|
||||
```toml
|
||||
[[pipeline]]
|
||||
id = "cpu-llm"
|
||||
label = "CPU language model"
|
||||
description = "A local inference endpoint that does not claim the GPU"
|
||||
requires_card = false
|
||||
vram_mib = 0
|
||||
serving = "Small instruct model"
|
||||
|
||||
[[pipeline.component]]
|
||||
kind = "proc"
|
||||
id = "server"
|
||||
label = "Inference server"
|
||||
argv = ["/opt/local-llm/bin/server", "--port", "8090", "{model}"]
|
||||
cwd = "/opt/local-llm"
|
||||
env = { OMP_NUM_THREADS = "8" }
|
||||
ready_timeout_s = 30
|
||||
stop_timeout_s = 10
|
||||
health = { kind = "tcp", host = "127.0.0.1", port = 8090, timeout_s = 2 }
|
||||
|
||||
[[pipeline.option]]
|
||||
id = "model"
|
||||
label = "Model"
|
||||
kind = "argv"
|
||||
default = "small"
|
||||
[[pipeline.option.choice]]
|
||||
id = "small"
|
||||
label = "Small model"
|
||||
vars = { model = "/models/small.gguf" }
|
||||
```
|
||||
|
||||
Placeholders are substituted within argv elements using values from `argv` options. An empty value
|
||||
drops the entire argv element. Every placeholder must be declared by an option, and every choice
|
||||
must provide each variable used by that option. Options can also be `note` kind for display-only
|
||||
choices. Run `--check` before restarting the server; validation problems remain visible in the API
|
||||
and GUI.
|
||||
|
||||
## Safety and operation
|
||||
|
||||
- A pipeline with `requires_card = true` owns the exclusive card group. Starting another such
|
||||
pipeline stops active peers, in reverse component order, before reclaiming and starting it.
|
||||
- `protected = true` blocks implicit stops. A user must make a deliberate forced switch.
|
||||
- Starts count as successful only after their configured health probe passes. Timeouts fail the job
|
||||
and include the recent pipeline log tail. GPU waits are bounded and report unavailable probes or
|
||||
timeout conditions honestly.
|
||||
- Only one action job runs globally at a time. Jobs expose progress in `/api/jobs/<id>` and state.
|
||||
- Proc pidfiles include Linux process start identity; stale files and reused PIDs do not imply an
|
||||
owned process. An externally started healthy service is shown as external and cannot be killed.
|
||||
- Executed commands and dry-run intent are recorded under the state directory. The API never edits
|
||||
the registry; option selection is persisted in `selected.toml`.
|
||||
|
||||
## API
|
||||
|
||||
| Route | Purpose |
|
||||
|---|---|
|
||||
| `GET /` | Web GUI |
|
||||
| `GET /static/<file>` | Allowlisted GUI assets |
|
||||
| `GET /api/state` | Full card, component, pipeline, job and validation snapshot |
|
||||
| `GET /api/events` | Server-sent state events and heartbeat (`?once=1` for one event) |
|
||||
| `POST /api/pipelines/<id>/action` | Start, stop or restart; returns a job id |
|
||||
| `POST /api/card/free` | Run the registry reclaim recipe as a job |
|
||||
| `GET /api/jobs/<id>` | Job state, progress steps and failure detail |
|
||||
| `GET /api/logs/<id>?n=200` | Tail proc log or unit journal |
|
||||
| `GET /api/audit?n=100` | Recent command audit records |
|
||||
| `GET /api/registry` | Parsed registry and validation report |
|
||||
| `POST /api/pipelines/<id>/options` | Validate and persist option selection |
|
||||
|
||||
## Development checks
|
||||
|
||||
```sh
|
||||
python3 -m compileall -q cockpit tui
|
||||
python3 -m cockpit --registry registry.example.toml --check
|
||||
bash scripts/selftest.sh
|
||||
```
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
local.properties
|
||||
.gradle/
|
||||
.kotlin/
|
||||
**/build/
|
||||
@@ -0,0 +1,92 @@
|
||||
# AGENTS.md — `android/` (GPU Cockpit mobile client)
|
||||
|
||||
Scope: everything under `android/`. This is an **Android app** that talks to the gpu-cockpit
|
||||
HTTP server over WireGuard. It is a thin client: the server owns all state and executes
|
||||
everything. The app renders and requests; it never decides whether a pipeline is up.
|
||||
|
||||
## Read first
|
||||
|
||||
* `android/API-CONTRACT.md` — the authoritative API. **Never invent an endpoint, field or enum
|
||||
value.** If the contract is missing something, say so instead of guessing.
|
||||
* The server itself is `../cockpit/server.py` (read-only for you — see Boundaries).
|
||||
|
||||
## Build & verify
|
||||
|
||||
```bash
|
||||
cd android
|
||||
./gradlew assembleDebug # must succeed
|
||||
./gradlew testDebugUnitTest # must pass
|
||||
```
|
||||
|
||||
* APK: `android/app/build/outputs/apk/debug/app-debug.apk`
|
||||
* `ANDROID_HOME=/home/nyra/android-sdk` (platform-tools + cmdline-tools present, platforms 34/36,
|
||||
build-tools 34.0.0/36.0.0). `local.properties` must contain `sdk.dir=/home/nyra/android-sdk`
|
||||
and **must not be committed**.
|
||||
* Gradle wrapper: `gradle-8.10.2-bin.zip` — already in `~/.gradle/wrapper/dists`, do not change
|
||||
the version (a different one means a ~130 MB download for no reason).
|
||||
* `JAVA_HOME` — a JDK 17 is on the machine; use whatever `./gradlew` resolves, but set
|
||||
`compileOptions`/`kotlinOptions` to **Java 17**.
|
||||
|
||||
## Toolchain pins (do not "upgrade" these)
|
||||
|
||||
| Thing | Version |
|
||||
|---|---|
|
||||
| AGP (`com.android.application`) | 8.8.2 |
|
||||
| Kotlin (`org.jetbrains.kotlin.android`) | 2.0.0 |
|
||||
| Compose compiler plugin (`org.jetbrains.kotlin.plugin.compose`) | 2.0.0 |
|
||||
| Compose BOM (`androidx.compose:compose-bom`) | 2024.06.00 |
|
||||
| compileSdk / targetSdk | 34 |
|
||||
| minSdk | 26 |
|
||||
| coroutines | 1.8.1 (`kotlinx-coroutines-android`) |
|
||||
|
||||
Dependencies: **Gson only** (`com.google.code.gson:gson:2.10.1`) on top of the Compose/AndroidX
|
||||
set. No OkHttp, no Hilt, no Room, no Kotlin serialization, no DI framework. Use
|
||||
`java.net.HttpURLConnection` for HTTP (it also does the SSE stream) and coroutines for threading.
|
||||
Every dependency is a failure mode; this app needs three.
|
||||
|
||||
## Hard requirements that are easy to get wrong
|
||||
|
||||
1. **Cleartext HTTP.** The server is plain HTTP on `10.10.10.1`. Without
|
||||
`android:usesCleartextTraffic="true"` (or a network-security-config scoped to that host)
|
||||
every request fails with `CLEARTEXT communication to 10.10.10.1 not permitted`.
|
||||
2. **`android.permission.INTERNET`** in the manifest. Obvious, still forgotten.
|
||||
3. **Auth header on every request**: `Authorization: Bearer <token>`. A wrong/missing token is a
|
||||
`401` — surface it as "token rejected" with a path to Settings, not a generic error.
|
||||
4. **Keep the parsing layer pure Kotlin.** The model classes + JSON parsing must not touch any
|
||||
`android.*` API, so `testDebugUnitTest` (JVM) can parse `fixtures/state-sample.json`. Keep
|
||||
Android-specific code (prefs, UI, network) in separate files.
|
||||
5. **Settings storage is a fixed contract** (automation seeds it):
|
||||
`SharedPreferences` file **`cockpit`**, keys **`base_url`** (default
|
||||
`http://10.10.10.1:8770`) and **`token`**. Do not rename them, do not add obfuscation.
|
||||
6. **No client-side job state.** A switch takes 15–45 s and keeps running on the server if the app
|
||||
is closed or the screen locks. Model it by re-reading `/api/state` (or `GET /api/jobs/<id>`),
|
||||
never by assuming.
|
||||
7. **Never send `force: true` without the user asking.** Protected pipelines need it and the
|
||||
server explains why in its 409 message; show that message and let the user choose.
|
||||
8. **Never claim success the server did not report.** Show the server's step text and, on failure,
|
||||
its message. The server is deliberately honest (it reports a real log tail when a start fails);
|
||||
the app must not paper over that with a green checkmark.
|
||||
|
||||
## UI
|
||||
|
||||
Material 3, dark, matching the web GUI exactly:
|
||||
|
||||
```
|
||||
bg #0c1118 panel #141d27 line #293747 text #e7edf4 muted #91a2b5
|
||||
green #47d7a0 amber #f3b95f red #ff6c72 blue #71b7ff
|
||||
```
|
||||
|
||||
Mobile-first, one hand, thumb-reachable actions, works on a 360 dp-wide screen. Show:
|
||||
card gauge (use `pct_of_usable`), the pipeline cards with real state, who is holding the card,
|
||||
and a clear "this is a tunnel; is WireGuard up?" message when the connection fails (the phone
|
||||
cannot reach `10.10.10.1` unless the WireGuard app is connected — say that instead of "network
|
||||
error").
|
||||
|
||||
## Boundaries
|
||||
|
||||
* **Touch only files under `android/`.** The Python server, GUI, TUI, registry and tests belong to
|
||||
a different workstream; changing them will break a live system. If you believe the server needs
|
||||
a change, write it in your summary instead — do not edit it.
|
||||
* No secrets in the repo. The token is typed by the user at runtime; never hardcode, never log,
|
||||
never print it.
|
||||
* `local.properties`, `build/`, `.gradle/` are not committed (add them to `.gitignore`).
|
||||
@@ -0,0 +1,158 @@
|
||||
# gpu-cockpit HTTP API — contract for the Android client
|
||||
|
||||
Authoritative. Written from the running server (`cockpit/server.py`) and a live capture of
|
||||
`/api/state` on Utumno. **Do not invent endpoints, fields, or enum values** — if something is
|
||||
missing here, ask rather than guess.
|
||||
|
||||
## Transport
|
||||
|
||||
* Plain HTTP on the **WireGuard address only**: default base URL `http://10.10.10.1:8770`.
|
||||
Nothing is listening on the LAN interface, so an off-tunnel call simply fails.
|
||||
* **Auth**: bearer token in a header on every request:
|
||||
|
||||
```
|
||||
Authorization: Bearer <token>
|
||||
```
|
||||
|
||||
A `?token=<token>` query parameter is accepted as a fallback, but the header is the normal path.
|
||||
* Missing/incorrect token on the tunnel → **HTTP 401** `{"error": "unauthorized"}`.
|
||||
* Loopback callers (the desktop GUI, the TUI) bypass the token. That is irrelevant to the app.
|
||||
* Requests are plain HTTP → the app **must** permit cleartext to this host
|
||||
(`android:usesCleartextTraffic="true"` or a network-security-config scoped to `10.10.10.1`).
|
||||
Symptom if forgotten: `CLEARTEXT communication to 10.10.10.1 not permitted`.
|
||||
|
||||
## GET endpoints
|
||||
|
||||
| Path | Returns |
|
||||
|---|---|
|
||||
| `/api/ping` | `{"ok":true,"host":"utumno","version":"0.1.0","now":"<iso8601>"}` |
|
||||
| `/api/state` | the full snapshot (below) — the app's main data source |
|
||||
| `/api/registry` | `{"registry":{…},"errors":[],"warnings":[]}` — the raw TOML registry as parsed |
|
||||
| `/api/audit?n=100` | last N exec records: `[{"ts","pipeline","action","argv":[…],"rc","duration_ms","dry_run"}]` |
|
||||
| `/api/logs/<pipeline_id>?n=200` | **list of strings** — journal lines for that pipeline's units, or one string saying the journal was unavailable |
|
||||
| `/api/jobs/<job_id>` | the full job record. **This is the only place `steps` exists** — the snapshot's `job` object carries just `{id,pipeline,state,step,started}`, no steps. 404 `{"error":"job not found"}` if unknown |
|
||||
| `/api/events` | **SSE**. First frame is `event: state` + the snapshot; then one frame per change, `: heartbeat` every 15 s. `?once=1` sends a single frame and closes — handy as a cheap liveness check |
|
||||
|
||||
## POST endpoints
|
||||
|
||||
All POST bodies are JSON. Success → **HTTP 202** `{"job_id":"j-<millis>-<pid>"}`.
|
||||
|
||||
| Path | Body | Meaning |
|
||||
|---|---|---|
|
||||
| `/api/card/free` | `{}` | run the configured reclaim recipe to free the card (currently ComfyUI `/free`) |
|
||||
| `/api/pipelines/<id>/action` | `{"action":"start"\|"stop"\|"restart","options":{},"force":false}` | start / stop / restart that pipeline |
|
||||
| `/api/pipelines/<id>/options` | `{"options":{…}}` | **persist a selection only**, no action. Returns `{"selected":{…}}` |
|
||||
|
||||
Error replies (JSON `{"error": "…"}`):
|
||||
|
||||
* `401` — `unauthorized`
|
||||
* `400` — `invalid JSON`, `invalid action`, `unknown pipeline`, `options must be an object`,
|
||||
`unknown option value <key>=<value>`
|
||||
* `409` — protected pipeline and no `force` (e.g. `protected pipeline train is active; force is
|
||||
required to stop it`), or a `ValueError` from preflight (card busy / insufficient VRAM)
|
||||
* `500` — anything unexpected; the message is human-readable
|
||||
|
||||
**`action` is only ever `start` / `stop` / `restart`.** A "switch to this pipeline" is an
|
||||
`action: "start"` — the server stops whatever else is holding the card itself, as part of the job,
|
||||
and reports each step. There is no `switch` verb.
|
||||
|
||||
### `options`
|
||||
|
||||
Only pipelines that expose choices have a non-empty `options` array. Choices are validated
|
||||
server-side; an unknown value is a 400. Example (the `llama` pipeline):
|
||||
|
||||
```json
|
||||
{"options":[{"id":"model","label":"Model","kind":"argv","selected":"qwen3-4b-q4km",
|
||||
"choices":[{"id":"qwen3-4b-q4km","label":"Qwen3-4B-Instruct-2507 (Q4_K_M, 2.4 GB)",
|
||||
"note":"The voice stack's verbalizer. Fits whole, fastest thing here.",
|
||||
"verified":"2026-09-25","is_default":true}, …]}]}
|
||||
```
|
||||
|
||||
So a selection is `{"<option.id>":"<choice.id>"}` → send it as the `options` object of the action
|
||||
and/or POST it to `/api/pipelines/<id>/options` to remember it. `selected` on the pipeline is the
|
||||
current value. Render `label`, show `note` as secondary text, and prefer `is_default` when the
|
||||
user has never chosen. `verified` is a date the combination was last proven — show it if present,
|
||||
it is the difference between "we think this fits" and "we measured it".
|
||||
|
||||
## The snapshot (`/api/state`)
|
||||
|
||||
```json
|
||||
{
|
||||
"host": "utumno", "now": "<iso8601>", "version": "0.1.0",
|
||||
"card": {
|
||||
"available": true, "name": "NVIDIA GeForce RTX 4060",
|
||||
"used_mib": 6149, "total_mib": 8188, "util_pct": 0, "temp_c": 48,
|
||||
"free_mib": 2039, "idle_floor_mib": 1700, "usable_mib": 6488, "pct_of_usable": 95,
|
||||
"processes": [{"pid": 3024502, "name": "llama-server", "used_mib": 2874, "pipeline": "voice"}],
|
||||
"history": [{"t": "<iso8601>", "used_mib": 1151}]
|
||||
},
|
||||
"pipelines": [{
|
||||
"id": "voice", "label": "Realtime voice",
|
||||
"description": "Talk to the local voice stack: 4B verbalizer, Parakeet STT, Kokoro TTS.",
|
||||
"state": "up", "health": "ok",
|
||||
"requires_card": true, "vram_mib": 4900, "protected": false,
|
||||
"serving": "Qwen3-4B-Instruct-2507 Q4_K_M (verbalizer) + Parakeet-TDT (STT) + Kokoro af_bella (TTS)",
|
||||
"open_url": "http://127.0.0.1:8788",
|
||||
"note": "Boot default - both units are enabled, so this is what comes back after a reboot. …",
|
||||
"last_change": null,
|
||||
"components": [{"id":"s2s","kind":"unit","label":"Speech-to-speech (:8765)","state":"up",
|
||||
"detail":"pid 3024507","health":{"ok":true,"detail":"TCP connected","latency_ms":2},
|
||||
"port":null,"unit":"speech-to-speech.service"}],
|
||||
"selected": {"model": "qwen3-4b-q4km"},
|
||||
"options": [],
|
||||
"job": {"id":"j-…","state":"running","step":"stopping Image generation (releasing the card)…"}
|
||||
}],
|
||||
"job": {"id":"j-…","pipeline":"image","state":"running","step":"waiting for VRAM (used 5100 of 2000 MiB)…","started":"<iso8601>"},
|
||||
"registry": {"path":"/home/harley/.config/gpu-cockpit/registry.toml","pipeline_count":6,"errors":[],"warnings":[]},
|
||||
"listeners": ["127.0.0.1:8770","10.10.10.1:8770"]
|
||||
}
|
||||
```
|
||||
|
||||
Field notes the UI depends on:
|
||||
|
||||
* `pipeline.state` ∈ `up` · `down` · `error` · `starting` · `stopping`.
|
||||
`pipeline.health` ∈ `ok` · `unknown` · `fail`. `up`+`ok` is genuinely healthy; `up`+`error`
|
||||
means the process is there but its health probe fails — say that, do not show a green light.
|
||||
* `component.state` may be `up` with `detail: "external"` — a process the cockpit did **not**
|
||||
start (ComfyUI started by the Pictures page). The cockpit will not kill it; stopping the
|
||||
pipeline is best-effort. Surface this, it explains "why didn't it stop".
|
||||
* `component.detail` is `"pid <n>"` for owned components, `"external"` otherwise.
|
||||
* `card.idle_floor_mib` is what the desktop itself uses; `usable_mib` = total − floor. Use
|
||||
`pct_of_usable` for the gauge, not `used/total`.
|
||||
* `card.processes[].pipeline` is `null` for processes that belong to no pipeline (desktop, kwin).
|
||||
Show those greyed; they are why the card is never "0 MiB".
|
||||
* `job` (top level) and `pipeline.job` are `null` when nothing is running. A job keeps running on
|
||||
the **server** if the app closes — on reconnect, re-read the state, never resume a client-side
|
||||
job model.
|
||||
|
||||
## Job lifecycle (the part that makes the app honest)
|
||||
|
||||
1. POST the action → `202 {"job_id": …}`
|
||||
2. Poll `GET /api/jobs/<job_id>` (≈1 s) or just re-read `/api/state`; both carry
|
||||
`state` (`running` → `done` / `error`) and the latest `step`.
|
||||
|
||||
The job record is:
|
||||
|
||||
```json
|
||||
{"id":"j-…","pipeline":"image","action":"start","state":"running","started":"<iso8601>",
|
||||
"error":null,
|
||||
"steps":[{"ts":"<iso8601>","text":"stopping Realtime voice (releasing the card)…","ok":true}]}
|
||||
```
|
||||
|
||||
**`steps` elements are objects, not strings** — `{"ts","text","ok"}` — oldest first, so `step`
|
||||
is simply the last one's `text`, and `ok:false` marks a step that reported a problem. `error`
|
||||
carries the failure message when `state` is `error`, and is `null` otherwise. Both fields are
|
||||
named exactly as written here; do not invent alternates.
|
||||
3. Show the real step text verbatim — e.g. `stopping Realtime voice (releasing the card)…`,
|
||||
`waiting for VRAM (used 5100 of 2000 MiB)…`, `starting ComfyUI server (:8188)…`.
|
||||
A switch takes ~15–45 s and sometimes reports an honest failure with the log tail
|
||||
(e.g. the voice stack needs >20 s to load Kokoro). **Do not show a success state the server
|
||||
did not report.** If the job ends in `error`, show the last step and the server's message.
|
||||
4. `force: true` is required for a `protected` pipeline stop/restart; without it the server
|
||||
answers 409 and the app should offer the force path explicitly (the LoRA training pipeline is
|
||||
protected). Never send `force: true` silently.
|
||||
|
||||
## Captured fixtures
|
||||
|
||||
* `android/fixtures/state-sample.json` — a real `/api/state` capture (history trimmed), for unit
|
||||
tests of the parsing layer. Parse it in a JVM test; if a field is renamed, that test should fail.
|
||||
@@ -0,0 +1,46 @@
|
||||
plugins {
|
||||
id("com.android.application")
|
||||
id("org.jetbrains.kotlin.android")
|
||||
id("org.jetbrains.kotlin.plugin.compose")
|
||||
}
|
||||
|
||||
android {
|
||||
namespace = "dev.gpucockpit"
|
||||
compileSdk = 34
|
||||
|
||||
defaultConfig {
|
||||
applicationId = "dev.gpucockpit"
|
||||
minSdk = 26
|
||||
targetSdk = 34
|
||||
versionCode = 1
|
||||
versionName = "1.0"
|
||||
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
|
||||
}
|
||||
|
||||
compileOptions {
|
||||
sourceCompatibility = JavaVersion.VERSION_17
|
||||
targetCompatibility = JavaVersion.VERSION_17
|
||||
}
|
||||
kotlinOptions { jvmTarget = "17" }
|
||||
buildFeatures { compose = true }
|
||||
testOptions { unitTests.isIncludeAndroidResources = false }
|
||||
}
|
||||
|
||||
dependencies {
|
||||
implementation(platform("androidx.compose:compose-bom:2024.06.00"))
|
||||
implementation("androidx.activity:activity-compose:1.9.0")
|
||||
implementation("androidx.compose.ui:ui")
|
||||
implementation("androidx.compose.ui:ui-tooling-preview")
|
||||
implementation("androidx.compose.material3:material3")
|
||||
implementation("androidx.compose.material:material-icons-extended")
|
||||
implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.8.3")
|
||||
implementation("androidx.lifecycle:lifecycle-runtime-compose:2.8.3")
|
||||
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.8.1")
|
||||
implementation("com.google.code.gson:gson:2.10.1")
|
||||
debugImplementation("androidx.compose.ui:ui-tooling")
|
||||
testImplementation("junit:junit:4.13.2")
|
||||
}
|
||||
|
||||
tasks.matching { it.name == "assembleDebug" }.configureEach {
|
||||
doLast { println("APK: ${layout.buildDirectory.file("outputs/apk/debug/app-debug.apk").get().asFile.absolutePath}") }
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
|
||||
<uses-permission android:name="android.permission.INTERNET" />
|
||||
<application android:allowBackup="false" android:label="GPU Cockpit" android:theme="@android:style/Theme.Material.NoActionBar" android:usesCleartextTraffic="true">
|
||||
<activity android:name=".MainActivity" android:exported="true">
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.MAIN" />
|
||||
<category android:name="android.intent.category.LAUNCHER" />
|
||||
</intent-filter>
|
||||
</activity>
|
||||
</application>
|
||||
</manifest>
|
||||
@@ -0,0 +1,365 @@
|
||||
package dev.gpucockpit
|
||||
|
||||
import android.content.Context
|
||||
import android.os.Bundle
|
||||
import androidx.activity.ComponentActivity
|
||||
import androidx.activity.compose.setContent
|
||||
import androidx.compose.foundation.*
|
||||
import androidx.compose.foundation.gestures.detectVerticalDragGestures
|
||||
import androidx.compose.foundation.layout.*
|
||||
import androidx.compose.foundation.lazy.LazyColumn
|
||||
import androidx.compose.foundation.lazy.items
|
||||
import androidx.compose.foundation.shape.CircleShape
|
||||
import androidx.compose.foundation.shape.RoundedCornerShape
|
||||
import androidx.compose.foundation.text.KeyboardOptions
|
||||
import androidx.compose.material3.*
|
||||
import androidx.compose.runtime.*
|
||||
import androidx.compose.ui.*
|
||||
import androidx.compose.ui.draw.clip
|
||||
import androidx.compose.ui.graphics.Color
|
||||
import androidx.compose.ui.input.pointer.pointerInput
|
||||
import androidx.compose.ui.text.font.FontWeight
|
||||
import androidx.compose.ui.text.input.KeyboardType
|
||||
import androidx.compose.ui.text.input.PasswordVisualTransformation
|
||||
import androidx.compose.ui.unit.dp
|
||||
import androidx.compose.ui.unit.sp
|
||||
import androidx.lifecycle.Lifecycle
|
||||
import androidx.lifecycle.LifecycleEventObserver
|
||||
import androidx.lifecycle.compose.LocalLifecycleOwner
|
||||
import dev.gpucockpit.data.ApiException
|
||||
import dev.gpucockpit.data.CockpitApi
|
||||
import dev.gpucockpit.model.JobStep
|
||||
import dev.gpucockpit.model.Pipeline
|
||||
import dev.gpucockpit.model.Snapshot
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.launch
|
||||
import java.text.SimpleDateFormat
|
||||
import java.util.Date
|
||||
import java.util.Locale
|
||||
|
||||
private val Bg = Color(0xFF0c1118)
|
||||
private val Panel = Color(0xFF141d27)
|
||||
private val Line = Color(0xFF293747)
|
||||
private val Ink = Color(0xFFe7edf4)
|
||||
private val Muted = Color(0xFF91a2b5)
|
||||
private val Green = Color(0xFF47d7a0)
|
||||
private val Amber = Color(0xFFf3b95f)
|
||||
private val Red = Color(0xFFff6c72)
|
||||
private val Blue = Color(0xFF71b7ff)
|
||||
private const val DefaultUrl = "http://10.10.10.1:8770"
|
||||
|
||||
class MainActivity : ComponentActivity() {
|
||||
override fun onCreate(savedInstanceState: Bundle?) {
|
||||
super.onCreate(savedInstanceState)
|
||||
window.statusBarColor = android.graphics.Color.rgb(12, 17, 24)
|
||||
window.navigationBarColor = android.graphics.Color.rgb(12, 17, 24)
|
||||
val prefs = getSharedPreferences("cockpit", Context.MODE_PRIVATE)
|
||||
setContent { CockpitApp(prefs) }
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun CockpitApp(prefs: android.content.SharedPreferences) {
|
||||
var baseUrl by remember { mutableStateOf(prefs.getString("base_url", DefaultUrl) ?: DefaultUrl) }
|
||||
var token by remember { mutableStateOf(prefs.getString("token", "") ?: "") }
|
||||
var showSettings by remember { mutableStateOf(token.isBlank()) }
|
||||
var snapshot by remember { mutableStateOf<Snapshot?>(null) }
|
||||
var connectionError by remember { mutableStateOf<String?>(null) }
|
||||
var pingMessage by remember { mutableStateOf<String?>(null) }
|
||||
var pingBusy by remember { mutableStateOf(false) }
|
||||
var busy by remember { mutableStateOf(false) }
|
||||
var jobView by remember { mutableStateOf<JobView?>(null) }
|
||||
var pendingConfirm by remember { mutableStateOf<PendingAction?>(null) }
|
||||
var protectedPrompt by remember { mutableStateOf<ProtectedPrompt?>(null) }
|
||||
var expanded by remember { mutableStateOf(setOf<String>()) }
|
||||
var foreground by remember { mutableStateOf(true) }
|
||||
val optionsByPipeline = remember { mutableStateMapOf<String, MutableMap<String, String>>() }
|
||||
val scope = rememberCoroutineScope()
|
||||
val api = remember(baseUrl, token) { CockpitApi(baseUrl, token) }
|
||||
val lifecycleOwner = LocalLifecycleOwner.current
|
||||
|
||||
suspend fun refresh() {
|
||||
if (token.isBlank()) return
|
||||
try {
|
||||
snapshot = api.state(); connectionError = null
|
||||
val currentJob = jobView
|
||||
if (currentJob?.state == "running") {
|
||||
try {
|
||||
val latest = api.job(currentJob.id)
|
||||
jobView = JobView(latest.id, latest.step, latest.state, currentJob.started, latest.error, latest.steps)
|
||||
} catch (_: Exception) { /* The next foreground refresh will re-read the server job. */ }
|
||||
}
|
||||
}
|
||||
catch (e: ApiException) { connectionError = if (e.code == 401) "token rejected" else e.message }
|
||||
catch (_: Exception) { connectionError = "is WireGuard connected? this only works over the tunnel" }
|
||||
}
|
||||
LaunchedEffect(api, token) {
|
||||
if (token.isNotBlank()) refresh()
|
||||
while (true) { delay(3000); if (foreground && !showSettings && token.isNotBlank()) refresh() }
|
||||
}
|
||||
DisposableEffect(lifecycleOwner, api, token) {
|
||||
val observer = LifecycleEventObserver { _, event ->
|
||||
when (event) {
|
||||
Lifecycle.Event.ON_RESUME -> { foreground = true; scope.launch { refresh() } }
|
||||
Lifecycle.Event.ON_PAUSE -> foreground = false
|
||||
else -> {}
|
||||
}
|
||||
}
|
||||
lifecycleOwner.lifecycle.addObserver(observer)
|
||||
onDispose { lifecycleOwner.lifecycle.removeObserver(observer) }
|
||||
}
|
||||
|
||||
fun perform(id: String?, action: String, force: Boolean = false) {
|
||||
scope.launch {
|
||||
busy = true
|
||||
try {
|
||||
val options = id?.let { optionsByPipeline[it]?.toMap().orEmpty() }.orEmpty()
|
||||
if (id != null && options.isNotEmpty()) api.rememberOptions(id, options)
|
||||
val jobId = if (id == null) api.freeCard() else api.action(id, action, options, force)
|
||||
val started = System.currentTimeMillis()
|
||||
var job: dev.gpucockpit.model.Job? = null
|
||||
jobView = JobView(jobId, "", "running", started, null)
|
||||
while (true) {
|
||||
job = api.job(jobId)
|
||||
jobView = JobView(jobId, job.step, job.state, started, job.error, job.steps)
|
||||
refresh()
|
||||
if (job.state == "done" || job.state == "error") break
|
||||
delay(1000)
|
||||
}
|
||||
refresh()
|
||||
} catch (e: ApiException) {
|
||||
if (e.code == 409 && id != null && !force) protectedPrompt = ProtectedPrompt(id, action, e.message)
|
||||
else connectionError = if (e.code == 401) "token rejected" else e.message
|
||||
} catch (_: Exception) { connectionError = "is WireGuard connected? this only works over the tunnel" }
|
||||
finally { busy = false }
|
||||
}
|
||||
}
|
||||
|
||||
MaterialTheme(colorScheme = darkColorScheme(primary = Green, background = Bg, surface = Panel, onSurface = Ink)) {
|
||||
if (showSettings) {
|
||||
SettingsScreen(baseUrl, { baseUrl = it }, token, { token = it }, pingBusy, pingMessage,
|
||||
onTest = {
|
||||
scope.launch {
|
||||
pingBusy = true; pingMessage = null
|
||||
try {
|
||||
val result = api.ping()
|
||||
pingMessage = "Connected · ${result["host"] ?: "unknown host"} · v${result["version"] ?: "?"}"
|
||||
} catch (e: ApiException) { pingMessage = if (e.code == 401) "token rejected" else e.message }
|
||||
catch (_: Exception) { pingMessage = "Cannot reach the tunnel. Is WireGuard connected? This only works over the tunnel." }
|
||||
finally { pingBusy = false }
|
||||
}
|
||||
}, onSave = {
|
||||
prefs.edit().putString("base_url", baseUrl.trim()).putString("token", token).apply()
|
||||
showSettings = false; snapshot = null; connectionError = null
|
||||
scope.launch { refresh() }
|
||||
})
|
||||
} else {
|
||||
Column(Modifier.fillMaxSize().background(Bg)) {
|
||||
Row(Modifier.fillMaxWidth().padding(horizontal = 20.dp, vertical = 14.dp), verticalAlignment = Alignment.CenterVertically) {
|
||||
Column(Modifier.weight(1f)) {
|
||||
Text("GPU COCKPIT", color = Green, fontSize = 12.sp, fontWeight = FontWeight.Bold, letterSpacing = 2.sp)
|
||||
Text(snapshot?.host?.uppercase() ?: "TUNNEL CONSOLE", color = Muted, fontSize = 12.sp)
|
||||
}
|
||||
TextButton(onClick = { showSettings = true }) { Text("SETTINGS", color = Blue) }
|
||||
}
|
||||
if (connectionError != null) ErrorBanner(connectionError!!, onSettings = { showSettings = true })
|
||||
jobView?.let { JobPanel(it) { jobView = null } }
|
||||
LazyColumn(Modifier.fillMaxSize().pointerInput(Unit) {
|
||||
var drag = 0f
|
||||
detectVerticalDragGestures(onVerticalDrag = { _, amount -> drag += amount }, onDragEnd = {
|
||||
if (drag > 70) scope.launch { refresh() }; drag = 0f
|
||||
})
|
||||
}, contentPadding = PaddingValues(start = 16.dp, end = 16.dp, top = 4.dp, bottom = 28.dp), verticalArrangement = Arrangement.spacedBy(12.dp)) {
|
||||
val s = snapshot
|
||||
if (s == null) item { LoadingCard(connectionError == null) }
|
||||
else {
|
||||
item { CardGauge(s, onFree = { pendingConfirm = PendingAction(null, "free") }, enabled = !busy) }
|
||||
item { Text("PIPELINES · ${s.pipelines.size}", color = Muted, fontSize = 11.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.5.sp, modifier = Modifier.padding(start = 4.dp, top = 6.dp)) }
|
||||
items(s.pipelines, key = { it.id }) { pipeline ->
|
||||
PipelineCard(pipeline, expanded.contains(pipeline.id), onExpand = {
|
||||
expanded = if (expanded.contains(pipeline.id)) expanded - pipeline.id else expanded + pipeline.id
|
||||
val current = optionsByPipeline[pipeline.id] ?: pipeline.options.associate { option -> option.id to (pipeline.selected[option.id] ?: option.selected ?: option.choices.firstOrNull { it.is_default }?.id.orEmpty()) }.toMutableMap()
|
||||
optionsByPipeline[pipeline.id] = current
|
||||
}, selected = optionsByPipeline[pipeline.id].orEmpty(), onSelect = { optionId, choice ->
|
||||
val current = optionsByPipeline[pipeline.id]?.toMutableMap() ?: mutableMapOf()
|
||||
current[optionId] = choice; optionsByPipeline[pipeline.id] = current
|
||||
}, enabled = !busy, onAction = { action -> pendingConfirm = PendingAction(pipeline, action) })
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pendingConfirm?.let { pending ->
|
||||
val p = pending.pipeline
|
||||
AlertDialog(onDismissRequest = { pendingConfirm = null }, containerColor = Panel,
|
||||
title = { Text(if (pending.action == "free") "Free GPU card?" else "${pending.action.replaceFirstChar { it.uppercase() }} ${p?.label}?", color = Ink) },
|
||||
text = { Text(if (pending.action == "free") "Run the server's configured card reclaim recipe." else "The server will manage the pipeline transition and report each step.", color = Muted) },
|
||||
confirmButton = { TextButton(onClick = { pendingConfirm = null; perform(p?.id, if (pending.action == "free") "free" else pending.action) }) { Text("CONTINUE", color = Green) } },
|
||||
dismissButton = { TextButton(onClick = { pendingConfirm = null }) { Text("CANCEL", color = Muted) } })
|
||||
}
|
||||
protectedPrompt?.let { prompt ->
|
||||
AlertDialog(onDismissRequest = { protectedPrompt = null }, containerColor = Panel,
|
||||
title = { Text("Protected pipeline", color = Amber) },
|
||||
text = { Text("${prompt.serverMessage}\n\nForce ${prompt.action} this pipeline?", color = Ink) },
|
||||
confirmButton = { TextButton(onClick = { protectedPrompt = null; perform(prompt.id, prompt.action, true) }) { Text("FORCE ${prompt.action.uppercase()}", color = Red) } },
|
||||
dismissButton = { TextButton(onClick = { protectedPrompt = null }) { Text("CANCEL", color = Muted) } })
|
||||
}
|
||||
}
|
||||
|
||||
private data class PendingAction(val pipeline: Pipeline?, val action: String)
|
||||
private data class ProtectedPrompt(val id: String, val action: String, val serverMessage: String)
|
||||
private data class JobView(val id: String, val step: String, val state: String, val started: Long, val error: String?, val steps: List<JobStep> = emptyList())
|
||||
|
||||
@Composable private fun SettingsScreen(url: String, onUrl: (String) -> Unit, token: String, onToken: (String) -> Unit,
|
||||
busy: Boolean, result: String?, onTest: () -> Unit, onSave: () -> Unit) {
|
||||
Column(Modifier.fillMaxSize().background(Bg).verticalScroll(rememberScrollState()).padding(20.dp), verticalArrangement = Arrangement.spacedBy(16.dp)) {
|
||||
Text("CONNECTION", color = Green, fontSize = 12.sp, fontWeight = FontWeight.Bold, letterSpacing = 2.sp)
|
||||
Text("WireGuard link", color = Ink, fontSize = 28.sp, fontWeight = FontWeight.Bold)
|
||||
Text("GPU Cockpit is reachable only through the WireGuard tunnel.", color = Muted)
|
||||
OutlinedTextField(url, onUrl, label = { Text("Base URL") }, singleLine = true, modifier = Modifier.fillMaxWidth(), colors = fieldColors())
|
||||
OutlinedTextField(token, onToken, label = { Text("Bearer token") }, singleLine = true, visualTransformation = PasswordVisualTransformation(), modifier = Modifier.fillMaxWidth(), colors = fieldColors())
|
||||
Button(onClick = onTest, enabled = !busy, modifier = Modifier.fillMaxWidth(), colors = ButtonDefaults.buttonColors(containerColor = Line)) { Text(if (busy) "TESTING…" else "TEST CONNECTION", color = Ink) }
|
||||
result?.let { Text(it, color = if (it.startsWith("Connected")) Green else Red) }
|
||||
Button(onClick = onSave, modifier = Modifier.fillMaxWidth()) { Text("SAVE AND OPEN DASHBOARD") }
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun CardGauge(snapshot: Snapshot, onFree: () -> Unit, enabled: Boolean) {
|
||||
val card = snapshot.card
|
||||
Column(Modifier.fillMaxWidth().clip(RoundedCornerShape(20.dp)).background(Panel).border(1.dp, Line, RoundedCornerShape(20.dp)).padding(18.dp)) {
|
||||
Row(verticalAlignment = Alignment.CenterVertically) {
|
||||
Column(Modifier.weight(1f)) {
|
||||
Text("SHARED GPU", color = Muted, fontSize = 11.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.5.sp)
|
||||
Text(card.name.ifBlank { "Card unavailable" }, color = Ink, fontSize = 19.sp, fontWeight = FontWeight.SemiBold)
|
||||
}
|
||||
Text("${card.pct_of_usable}%", color = if (card.pct_of_usable >= 90) Amber else Green, fontSize = 30.sp, fontWeight = FontWeight.Bold)
|
||||
}
|
||||
Spacer(Modifier.height(16.dp))
|
||||
Box(Modifier.fillMaxWidth().height(10.dp).clip(CircleShape).background(Line)) {
|
||||
Box(Modifier.fillMaxWidth((card.pct_of_usable.coerceIn(0, 100) / 100f)).fillMaxHeight().clip(CircleShape).background(if (card.pct_of_usable >= 90) Amber else Green))
|
||||
}
|
||||
Spacer(Modifier.height(10.dp))
|
||||
Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.SpaceBetween) {
|
||||
Text("${card.used_mib} / ${card.usable_mib} MiB usable", color = Ink, fontSize = 13.sp)
|
||||
Text("${card.free_mib} MiB free", color = Muted, fontSize = 13.sp)
|
||||
}
|
||||
Spacer(Modifier.height(14.dp))
|
||||
Row(Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
Metric("TEMP", card.temp_c?.let { "$it° C" } ?: "—", Amber, Modifier.weight(1f))
|
||||
Metric("UTIL", "${card.util_pct}%", Blue, Modifier.weight(1f))
|
||||
Metric("CARD", if (card.available) "ONLINE" else "OFFLINE", if (card.available) Green else Red, Modifier.weight(1f))
|
||||
}
|
||||
if (card.processes.isNotEmpty()) {
|
||||
Spacer(Modifier.height(12.dp)); Text("PROCESSES", color = Muted, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
card.processes.forEach { proc ->
|
||||
Row(Modifier.fillMaxWidth().padding(top = 5.dp), horizontalArrangement = Arrangement.SpaceBetween) {
|
||||
Text(proc.name + (proc.pipeline?.let { " · $it" } ?: " · unassigned"), color = if (proc.pipeline == null) Muted else Ink, fontSize = 12.sp)
|
||||
Text("${proc.used_mib} MiB", color = Muted, fontSize = 12.sp)
|
||||
}
|
||||
}
|
||||
}
|
||||
Spacer(Modifier.height(12.dp))
|
||||
OutlinedButton(onClick = onFree, enabled = enabled, modifier = Modifier.fillMaxWidth(), border = BorderStroke(1.dp, Line)) { Text("FREE CARD", color = Blue, fontWeight = FontWeight.Bold) }
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun Metric(label: String, value: String, color: Color, modifier: Modifier = Modifier) {
|
||||
Column(modifier.clip(RoundedCornerShape(12.dp)).background(Bg).padding(10.dp)) {
|
||||
Text(label, color = Muted, fontSize = 9.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
Text(value, color = color, fontSize = 16.sp, fontWeight = FontWeight.Bold)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun PipelineCard(p: Pipeline, expanded: Boolean, onExpand: () -> Unit,
|
||||
selected: Map<String, String>, onSelect: (String, String) -> Unit,
|
||||
enabled: Boolean, onAction: (String) -> Unit) {
|
||||
val unhealthy = p.health == "fail" || (p.state == "up" && p.health != "ok")
|
||||
val stateColor = when { unhealthy || p.state == "error" -> Red; p.state == "up" -> Green; p.state == "starting" || p.state == "stopping" -> Amber; else -> Muted }
|
||||
Column(Modifier.fillMaxWidth().clip(RoundedCornerShape(18.dp)).background(Panel).border(1.dp, Line, RoundedCornerShape(18.dp)).clickable { onExpand() }.padding(16.dp)) {
|
||||
Row(verticalAlignment = Alignment.CenterVertically) {
|
||||
Column(Modifier.weight(1f)) {
|
||||
Text(p.label, color = Ink, fontSize = 18.sp, fontWeight = FontWeight.SemiBold)
|
||||
Text(if (unhealthy) "${p.state.uppercase()} · HEALTH FAIL" else "${p.state.uppercase()} · ${p.health.uppercase()}", color = stateColor, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
}
|
||||
if (p.requires_card) Text("${p.vram_mib} MiB", color = Blue, fontSize = 12.sp)
|
||||
}
|
||||
Spacer(Modifier.height(8.dp))
|
||||
Text(p.description, color = Muted, fontSize = 13.sp, lineHeight = 18.sp)
|
||||
Spacer(Modifier.height(10.dp))
|
||||
Text("SERVING", color = Muted, fontSize = 9.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
Text(p.serving, color = Ink, fontSize = 13.sp)
|
||||
p.last_change?.let { Text("Changed · $it", color = Muted, fontSize = 11.sp, modifier = Modifier.padding(top = 7.dp)) }
|
||||
if (expanded) {
|
||||
p.note?.let { Text(it, color = Muted, fontSize = 12.sp, modifier = Modifier.padding(top = 10.dp)) }
|
||||
if (p.components.isNotEmpty()) {
|
||||
Spacer(Modifier.height(12.dp)); Text("COMPONENTS", color = Muted, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
p.components.forEach { c ->
|
||||
Column(Modifier.fillMaxWidth().padding(top = 7.dp).clip(RoundedCornerShape(10.dp)).background(Bg).padding(10.dp)) {
|
||||
Text("${c.label} · ${c.state.uppercase()}", color = if (c.state == "up" && c.health?.ok == true) Green else if (c.state == "up") Amber else Muted, fontSize = 12.sp, fontWeight = FontWeight.SemiBold)
|
||||
Text(c.detail, color = Muted, fontSize = 11.sp)
|
||||
if (c.detail == "external") Text("Started outside Cockpit. Cockpit will not kill a process it did not start.", color = Amber, fontSize = 11.sp)
|
||||
}
|
||||
}
|
||||
}
|
||||
p.options.forEach { option ->
|
||||
OptionPicker(option, selected[option.id] ?: option.selected ?: option.choices.firstOrNull { it.is_default }?.id.orEmpty(), { onSelect(option.id, it) })
|
||||
}
|
||||
Row(Modifier.fillMaxWidth().padding(top = 14.dp), horizontalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
Button(onClick = { onAction("start") }, enabled = enabled, modifier = Modifier.weight(1f)) { Text("SWITCH TO THIS", fontSize = 11.sp) }
|
||||
OutlinedButton(onClick = { onAction("stop") }, enabled = enabled, modifier = Modifier.weight(.7f), border = BorderStroke(1.dp, Line)) { Text("STOP", color = Ink, fontSize = 11.sp) }
|
||||
OutlinedButton(onClick = { onAction("restart") }, enabled = enabled, modifier = Modifier.weight(.8f), border = BorderStroke(1.dp, Line)) { Text("RESTART", color = Ink, fontSize = 11.sp) }
|
||||
}
|
||||
} else Text("TAP FOR COMPONENTS & CONTROLS", color = Blue, fontSize = 10.sp, fontWeight = FontWeight.Bold, modifier = Modifier.padding(top = 11.dp))
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun OptionPicker(option: dev.gpucockpit.model.PipelineOption, chosen: String, onSelect: (String) -> Unit) {
|
||||
var open by remember { mutableStateOf(false) }
|
||||
val choice = option.choices.firstOrNull { it.id == chosen }
|
||||
Column(Modifier.fillMaxWidth().padding(top = 14.dp)) {
|
||||
Text(option.label.uppercase(), color = Muted, fontSize = 10.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
Box {
|
||||
OutlinedButton(onClick = { open = true }, modifier = Modifier.fillMaxWidth(), border = BorderStroke(1.dp, Line)) {
|
||||
Text(choice?.label ?: "Choose ${option.label}", color = Ink, fontSize = 12.sp)
|
||||
}
|
||||
DropdownMenu(expanded = open, onDismissRequest = { open = false }, modifier = Modifier.background(Panel)) {
|
||||
option.choices.forEach { c -> DropdownMenuItem(text = { Text(c.label + if (c.is_default) " · DEFAULT" else "", color = Ink, fontSize = 12.sp) }, onClick = { onSelect(c.id); open = false }) }
|
||||
}
|
||||
}
|
||||
choice?.note?.let { Text(it, color = Muted, fontSize = 11.sp) }
|
||||
if (choice?.verified != null) Text("Verified ${choice.verified}", color = Green, fontSize = 10.sp)
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun JobPanel(job: JobView, onDismiss: () -> Unit) {
|
||||
val elapsed = ((System.currentTimeMillis() - job.started) / 1000).toInt()
|
||||
Column(Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 4.dp).clip(RoundedCornerShape(14.dp)).background(Color(0xFF1B2937)).padding(14.dp)) {
|
||||
Text(if (job.state == "done") "SERVER REPORTED SUCCESS" else if (job.state == "error") "JOB FAILED" else "SERVER JOB · ${elapsed}s", color = if (job.state == "done") Green else if (job.state == "error") Red else Amber, fontSize = 11.sp, fontWeight = FontWeight.Bold, letterSpacing = 1.sp)
|
||||
Text(job.step.ifBlank { "Waiting for first server step…" }, color = Ink, fontSize = 13.sp, modifier = Modifier.padding(top = 5.dp))
|
||||
job.steps.takeLast(5).forEach { s ->
|
||||
Text((if (s.ok) "· " else "! ") + s.text, color = if (s.ok) Muted else Amber, fontSize = 11.sp, modifier = Modifier.padding(top = 3.dp))
|
||||
}
|
||||
job.error?.let { Text(it, color = Red, fontSize = 12.sp, modifier = Modifier.padding(top = 4.dp)) }
|
||||
if (job.state == "done" || job.state == "error") TextButton(onClick = onDismiss, modifier = Modifier.align(Alignment.End)) { Text("DISMISS", color = Muted) }
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun ErrorBanner(message: String, onSettings: () -> Unit) {
|
||||
Row(Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 4.dp).clip(RoundedCornerShape(12.dp)).background(Color(0xFF392126)).padding(12.dp), verticalAlignment = Alignment.CenterVertically) {
|
||||
Text(message, color = Red, fontSize = 12.sp, modifier = Modifier.weight(1f))
|
||||
if (message == "token rejected") TextButton(onClick = onSettings) { Text("SETTINGS", color = Blue, fontSize = 10.sp) }
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun LoadingCard(loading: Boolean) {
|
||||
Column(Modifier.fillMaxWidth().clip(RoundedCornerShape(18.dp)).background(Panel).padding(20.dp)) {
|
||||
Text(if (loading) "Connecting to Utumno…" else "Dashboard unavailable", color = Ink, fontWeight = FontWeight.SemiBold)
|
||||
Text(if (loading) "Checking the WireGuard route" else "Check the connection and pull down to retry.", color = Muted, fontSize = 13.sp, modifier = Modifier.padding(top = 6.dp))
|
||||
}
|
||||
}
|
||||
|
||||
@Composable private fun fieldColors() = OutlinedTextFieldDefaults.colors(
|
||||
focusedTextColor = Ink, unfocusedTextColor = Ink, focusedBorderColor = Green, unfocusedBorderColor = Line,
|
||||
focusedLabelColor = Green, unfocusedLabelColor = Muted, cursorColor = Green
|
||||
)
|
||||
@@ -0,0 +1,56 @@
|
||||
package dev.gpucockpit.data
|
||||
|
||||
import com.google.gson.Gson
|
||||
import com.google.gson.reflect.TypeToken
|
||||
import dev.gpucockpit.model.Job
|
||||
import dev.gpucockpit.model.Snapshot
|
||||
import dev.gpucockpit.model.SnapshotParser
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import java.net.HttpURLConnection
|
||||
import java.net.URL
|
||||
|
||||
class ApiException(val code: Int, override val message: String) : Exception(message)
|
||||
class CockpitApi(private val baseUrl: String, private val token: String) {
|
||||
private val gson = Gson()
|
||||
private fun request(path: String, method: String = "GET", body: Any? = null): String {
|
||||
val connection = (URL(baseUrl.trimEnd('/') + path).openConnection() as HttpURLConnection)
|
||||
try {
|
||||
connection.requestMethod = method
|
||||
connection.connectTimeout = 5000
|
||||
connection.readTimeout = 8000
|
||||
connection.setRequestProperty("Authorization", "Bearer $token")
|
||||
connection.setRequestProperty("Accept", "application/json")
|
||||
if (body != null) {
|
||||
connection.doOutput = true
|
||||
connection.setRequestProperty("Content-Type", "application/json")
|
||||
connection.outputStream.use { it.write(gson.toJson(body).toByteArray(Charsets.UTF_8)) }
|
||||
}
|
||||
val code = connection.responseCode
|
||||
val stream = if (code in 200..299) connection.inputStream else connection.errorStream
|
||||
val text = stream?.bufferedReader()?.use { it.readText() }.orEmpty()
|
||||
if (code !in 200..299) {
|
||||
val error = try { gson.fromJson(text, Map::class.java)?.get("error")?.toString() } catch (_: Exception) { null }
|
||||
throw ApiException(code, error ?: "HTTP $code")
|
||||
}
|
||||
return text
|
||||
} finally { connection.disconnect() }
|
||||
}
|
||||
suspend fun ping(): Map<String, Any?> = withContext(Dispatchers.IO) {
|
||||
val type = object : TypeToken<Map<String, Any?>>() {}.type
|
||||
gson.fromJson(request("/api/ping"), type)
|
||||
}
|
||||
suspend fun state(): Snapshot = withContext(Dispatchers.IO) { SnapshotParser.parse(request("/api/state")) }
|
||||
suspend fun action(id: String, action: String, options: Map<String, String> = emptyMap(), force: Boolean = false): String = withContext(Dispatchers.IO) {
|
||||
val response = request("/api/pipelines/$id/action", "POST", mapOf("action" to action, "options" to options, "force" to force))
|
||||
gson.fromJson(response, Map::class.java)["job_id"].toString()
|
||||
}
|
||||
suspend fun rememberOptions(id: String, options: Map<String, String>) = withContext(Dispatchers.IO) {
|
||||
request("/api/pipelines/$id/options", "POST", mapOf("options" to options))
|
||||
}
|
||||
suspend fun freeCard(): String = withContext(Dispatchers.IO) {
|
||||
val response = request("/api/card/free", "POST", emptyMap<String, Any>())
|
||||
gson.fromJson(response, Map::class.java)["job_id"].toString()
|
||||
}
|
||||
suspend fun job(id: String): Job = withContext(Dispatchers.IO) { gson.fromJson(request("/api/jobs/$id"), Job::class.java) }
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package dev.gpucockpit.model
|
||||
|
||||
import com.google.gson.Gson
|
||||
import com.google.gson.annotations.SerializedName
|
||||
|
||||
data class Snapshot(
|
||||
val host: String = "", val now: String = "", val version: String = "",
|
||||
val card: Card = Card(), val pipelines: List<Pipeline> = emptyList(), val job: Job? = null,
|
||||
val registry: Registry? = null, val listeners: List<String> = emptyList()
|
||||
)
|
||||
data class Card(
|
||||
val available: Boolean = false, val name: String = "", val used_mib: Int = 0,
|
||||
val total_mib: Int = 0, val util_pct: Int = 0, val temp_c: Int? = null,
|
||||
val free_mib: Int = 0, val idle_floor_mib: Int = 0, val usable_mib: Int = 0,
|
||||
val pct_of_usable: Int = 0, val processes: List<GpuProcess> = emptyList(),
|
||||
val history: List<HistoryPoint> = emptyList()
|
||||
)
|
||||
data class GpuProcess(val pid: Long = 0, val name: String = "", val used_mib: Int = 0, val pipeline: String? = null)
|
||||
data class HistoryPoint(val t: String = "", val used_mib: Int = 0)
|
||||
data class Pipeline(
|
||||
val id: String = "", val label: String = "", val description: String = "", val state: String = "unknown",
|
||||
val health: String = "unknown", val requires_card: Boolean = false, val vram_mib: Int = 0,
|
||||
val protected: Boolean = false, val serving: String = "", val open_url: String? = null,
|
||||
val note: String? = null, val last_change: String? = null,
|
||||
val components: List<Component> = emptyList(), val selected: Map<String, String> = emptyMap(),
|
||||
val options: List<PipelineOption> = emptyList(), val job: Job? = null
|
||||
)
|
||||
data class Component(
|
||||
val id: String = "", val kind: String = "", val label: String = "", val state: String = "unknown",
|
||||
val detail: String = "", val health: ComponentHealth? = null, val port: Int? = null, val unit: String? = null
|
||||
)
|
||||
data class ComponentHealth(val ok: Boolean = false, val detail: String = "", val latency_ms: Int? = null)
|
||||
data class PipelineOption(
|
||||
val id: String = "", val label: String = "", val kind: String = "", val selected: String? = null,
|
||||
val choices: List<Choice> = emptyList()
|
||||
)
|
||||
data class Choice(
|
||||
val id: String = "", val label: String = "", val note: String? = null, val verified: String? = null,
|
||||
val is_default: Boolean = false
|
||||
)
|
||||
data class Job(
|
||||
@SerializedName("id") val id: String = "", val pipeline: String? = null, val state: String = "",
|
||||
val step: String = "", val started: String? = null, val error: String? = null,
|
||||
// The server appends {"ts","text","ok"} objects here. Typing this as List<String> made Gson
|
||||
// throw on any job that had taken a step, which the UI then reported as a tunnel failure.
|
||||
val steps: List<JobStep> = emptyList()
|
||||
)
|
||||
data class JobStep(val ts: String = "", val text: String = "", val ok: Boolean = true)
|
||||
data class Registry(val path: String = "", val pipeline_count: Int = 0, val errors: List<String> = emptyList(), val warnings: List<String> = emptyList())
|
||||
|
||||
object SnapshotParser {
|
||||
private val gson = Gson()
|
||||
fun parse(json: String): Snapshot = gson.fromJson(json, Snapshot::class.java)
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package dev.gpucockpit.model
|
||||
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertNotNull
|
||||
import org.junit.Test
|
||||
import java.io.File
|
||||
|
||||
class SnapshotParserTest {
|
||||
@Test fun parsesCapturedSnapshotAndKeyDashboardValues() {
|
||||
val json = File("../fixtures/state-sample.json").readText()
|
||||
val state = SnapshotParser.parse(json)
|
||||
assertEquals("utumno", state.host)
|
||||
assertEquals(6, state.pipelines.size)
|
||||
val voice = state.pipelines.first { it.id == "voice" }
|
||||
assertEquals(2, voice.components.size)
|
||||
assertEquals(6488, state.card.usable_mib)
|
||||
assertEquals(95, state.card.pct_of_usable)
|
||||
assertNotNull(state.pipelines.first { it.id == "llama" }.options.first().choices.first().note)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
plugins {
|
||||
id("com.android.application") version "8.8.2" apply false
|
||||
id("org.jetbrains.kotlin.android") version "2.0.0" apply false
|
||||
id("org.jetbrains.kotlin.plugin.compose") version "2.0.0" apply false
|
||||
}
|
||||
@@ -0,0 +1,340 @@
|
||||
{
|
||||
"host": "utumno",
|
||||
"now": "2026-09-27T16:22:49+00:00",
|
||||
"version": "0.1.0",
|
||||
"card": {
|
||||
"available": true,
|
||||
"name": "NVIDIA GeForce RTX 4060",
|
||||
"used_mib": 6149,
|
||||
"total_mib": 8188,
|
||||
"util_pct": 0,
|
||||
"temp_c": 47,
|
||||
"processes": [
|
||||
{
|
||||
"pid": 7632,
|
||||
"name": "kwin_wayland",
|
||||
"used_mib": 22,
|
||||
"pipeline": null
|
||||
},
|
||||
{
|
||||
"pid": 2037014,
|
||||
"name": "python",
|
||||
"used_mib": 178,
|
||||
"pipeline": null
|
||||
},
|
||||
{
|
||||
"pid": 3024502,
|
||||
"name": "llama-server",
|
||||
"used_mib": 2874,
|
||||
"pipeline": "voice"
|
||||
},
|
||||
{
|
||||
"pid": 3024507,
|
||||
"name": "python",
|
||||
"used_mib": 2014,
|
||||
"pipeline": "voice"
|
||||
}
|
||||
],
|
||||
"free_mib": 2039,
|
||||
"idle_floor_mib": 1700,
|
||||
"usable_mib": 6488,
|
||||
"pct_of_usable": 95,
|
||||
"history": [
|
||||
{
|
||||
"t": "2026-09-27T16:19:32+00:00",
|
||||
"used_mib": 1151
|
||||
},
|
||||
{
|
||||
"t": "2026-09-27T16:19:34+00:00",
|
||||
"used_mib": 1151
|
||||
},
|
||||
{
|
||||
"t": "2026-09-27T16:19:36+00:00",
|
||||
"used_mib": 1151
|
||||
}
|
||||
]
|
||||
},
|
||||
"pipelines": [
|
||||
{
|
||||
"id": "voice",
|
||||
"label": "Realtime voice",
|
||||
"description": "Talk to the local voice stack: 4B verbalizer, Parakeet STT, Kokoro TTS.",
|
||||
"state": "up",
|
||||
"health": "ok",
|
||||
"requires_card": true,
|
||||
"vram_mib": 4900,
|
||||
"protected": false,
|
||||
"serving": "Qwen3-4B-Instruct-2507 Q4_K_M (verbalizer) + Parakeet-TDT (STT) + Kokoro af_bella (TTS)",
|
||||
"open_url": "http://127.0.0.1:8788",
|
||||
"note": "Boot default - both units are enabled, so this is what comes back after a reboot. Stopping it is not disabling it; one Start brings it all back.",
|
||||
"last_change": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "verbalizer",
|
||||
"kind": "unit",
|
||||
"label": "Verbalizer LLM (:8080)",
|
||||
"state": "up",
|
||||
"detail": "pid 3024502",
|
||||
"health": {
|
||||
"ok": true,
|
||||
"detail": "HTTP 200",
|
||||
"latency_ms": 2
|
||||
},
|
||||
"port": null,
|
||||
"unit": "llama-server-local.service"
|
||||
},
|
||||
{
|
||||
"id": "s2s",
|
||||
"kind": "unit",
|
||||
"label": "Speech-to-speech (:8765)",
|
||||
"state": "up",
|
||||
"detail": "pid 3024507",
|
||||
"health": {
|
||||
"ok": true,
|
||||
"detail": "TCP connected",
|
||||
"latency_ms": 0
|
||||
},
|
||||
"port": 8765,
|
||||
"unit": "speech-to-speech.service"
|
||||
}
|
||||
],
|
||||
"selected": {},
|
||||
"options": [],
|
||||
"job": null
|
||||
},
|
||||
{
|
||||
"id": "image",
|
||||
"label": "Image generation",
|
||||
"description": "Z-Image Turbo behind ComfyUI. The Pictures page is the friendly front door.",
|
||||
"state": "up",
|
||||
"health": "ok",
|
||||
"requires_card": true,
|
||||
"vram_mib": 7019,
|
||||
"protected": false,
|
||||
"serving": "Z-Image-Turbo GGUF Q4_K_M + Qwen3-4B text encoder + nyra ladder LoRAs",
|
||||
"open_url": "http://127.0.0.1:8899",
|
||||
"note": "Verified peak 7019 MiB of 8188, ~25-30 s a picture at 768. The Pictures page can start ComfyUI by itself - if it did, this panel shows the server as external and will not kill it.",
|
||||
"last_change": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "comfy",
|
||||
"kind": "proc",
|
||||
"label": "ComfyUI server (:8188)",
|
||||
"state": "up",
|
||||
"detail": "external",
|
||||
"health": {
|
||||
"ok": true,
|
||||
"detail": "HTTP 200",
|
||||
"latency_ms": 6
|
||||
},
|
||||
"port": null,
|
||||
"unit": null
|
||||
}
|
||||
],
|
||||
"selected": {},
|
||||
"options": [],
|
||||
"job": null
|
||||
},
|
||||
{
|
||||
"id": "llama",
|
||||
"label": "LLM playground",
|
||||
"description": "Load any model we have proved onto the card, with its known-good flags.",
|
||||
"state": "down",
|
||||
"health": "unknown",
|
||||
"requires_card": true,
|
||||
"vram_mib": 6500,
|
||||
"protected": false,
|
||||
"serving": "whatever you pick below (OpenAI-compatible on :11435)",
|
||||
"open_url": null,
|
||||
"note": "Each entry carries the flags measured on this card. The 13 GB+ MoE models are not here - they live in the Agent and Roleplay pipelines, which leave the card free.",
|
||||
"last_change": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "server",
|
||||
"kind": "proc",
|
||||
"label": "llama-server (:11435)",
|
||||
"state": "down",
|
||||
"detail": "not running",
|
||||
"health": {
|
||||
"ok": false,
|
||||
"detail": "connection refused",
|
||||
"latency_ms": 1
|
||||
},
|
||||
"port": null,
|
||||
"unit": null
|
||||
}
|
||||
],
|
||||
"selected": {
|
||||
"model": "qwen3-4b-q4km"
|
||||
},
|
||||
"options": [
|
||||
{
|
||||
"id": "model",
|
||||
"label": "Model",
|
||||
"kind": "argv",
|
||||
"selected": "qwen3-4b-q4km",
|
||||
"choices": [
|
||||
{
|
||||
"id": "qwen3-4b-q4km",
|
||||
"label": "Qwen3-4B-Instruct-2507 (Q4_K_M, 2.4 GB)",
|
||||
"note": "The voice stack's verbalizer. Fits whole, fastest thing here.",
|
||||
"verified": "2026-09-25",
|
||||
"is_default": true
|
||||
},
|
||||
{
|
||||
"id": "qwen3-4b-q5km",
|
||||
"label": "Qwen3-4B-Instruct-2507 (Q5_K_M, 2.9 GB)",
|
||||
"note": "Same model, one quant up. Small quality gain, still fits.",
|
||||
"verified": null,
|
||||
"is_default": false
|
||||
},
|
||||
{
|
||||
"id": "gemma4-12b-heretic-iq3xs",
|
||||
"label": "Gemma 4 12B Heretic uncensored (IQ3_XS, 5.3 GB)",
|
||||
"note": "Uncensored 12B. Weights fit with headroom; keep the context modest.",
|
||||
"verified": null,
|
||||
"is_default": false
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"job": null
|
||||
},
|
||||
{
|
||||
"id": "agent",
|
||||
"label": "Agent LLM (Hermes heavy)",
|
||||
"description": "The big local model Hermes talks to through its llama-local provider.",
|
||||
"state": "down",
|
||||
"health": "unknown",
|
||||
"requires_card": true,
|
||||
"vram_mib": 1500,
|
||||
"protected": false,
|
||||
"serving": "Qwen3.6-35B-A3B UD-IQ3_S, 128K context, q8_0 KV",
|
||||
"open_url": null,
|
||||
"note": "Mostly CPU-resident (a 3B-active MoE), so only a small slice of the card. systemd Conflicts= stops the voice pair automatically when this starts. Hermes reaches it as local-qwen35 on :11434.",
|
||||
"last_change": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "server",
|
||||
"kind": "unit",
|
||||
"label": "llama-server-35b (:11434)",
|
||||
"state": "down",
|
||||
"detail": "inactive",
|
||||
"health": null,
|
||||
"port": null,
|
||||
"unit": "llama-server-35b.service"
|
||||
}
|
||||
],
|
||||
"selected": {},
|
||||
"options": [],
|
||||
"job": null
|
||||
},
|
||||
{
|
||||
"id": "train",
|
||||
"label": "LoRA training",
|
||||
"description": "ai-toolkit Z-Image LoRA runs. Hours, not seconds - this is the one to leave alone.",
|
||||
"state": "down",
|
||||
"health": "unknown",
|
||||
"requires_card": true,
|
||||
"vram_mib": 3200,
|
||||
"protected": true,
|
||||
"serving": "the config you pick (warm-cache two-phase run)",
|
||||
"open_url": null,
|
||||
"note": "PROTECTED: nothing here will stop it to free the card. A run is hours of work, so stopping it is a deliberate, confirmed action. Progress is in the log tail (~3.4 s/step measured).",
|
||||
"last_change": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "run",
|
||||
"kind": "proc",
|
||||
"label": "run-lora.sh",
|
||||
"state": "down",
|
||||
"detail": "not running",
|
||||
"health": {
|
||||
"ok": false,
|
||||
"detail": "no health check",
|
||||
"latency_ms": 0
|
||||
},
|
||||
"port": null,
|
||||
"unit": null
|
||||
}
|
||||
],
|
||||
"selected": {},
|
||||
"options": [
|
||||
{
|
||||
"id": "config",
|
||||
"label": "Config",
|
||||
"kind": "argv",
|
||||
"selected": "config-emo",
|
||||
"choices": [
|
||||
{
|
||||
"id": "config-emo",
|
||||
"label": "config-emo.yaml (emo outfit set)",
|
||||
"note": "146 + 11 barefoot hero images. Completed 2026-09-26.",
|
||||
"verified": "2026-09-26",
|
||||
"is_default": true
|
||||
},
|
||||
{
|
||||
"id": "config-sundress",
|
||||
"label": "config-sundress.yaml (sundress set)",
|
||||
"note": "92 + 5 hero images. Completed 2026-09-26.",
|
||||
"verified": "2026-09-26",
|
||||
"is_default": false
|
||||
},
|
||||
{
|
||||
"id": "config-canon",
|
||||
"label": "config-canon.yaml (canon set)",
|
||||
"note": "The first canon run - the one the pet atlas came from.",
|
||||
"verified": null,
|
||||
"is_default": false
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"job": null
|
||||
},
|
||||
{
|
||||
"id": "rp",
|
||||
"label": "Roleplay guest (CPU)",
|
||||
"description": "Uncensored Gemma 4 26B served to the lab guest. CPU only, card untouched.",
|
||||
"state": "up",
|
||||
"health": "ok",
|
||||
"requires_card": false,
|
||||
"vram_mib": 0,
|
||||
"protected": false,
|
||||
"serving": "Gemma4-26B-A4B QAT Uncensored Q4_K_M + vision projector, --device none",
|
||||
"open_url": null,
|
||||
"note": "Runs entirely on CPU (32 cores, 93 GB RAM) and is exposed on the lab bridge, not loopback. Starting it never disturbs whatever owns the card.",
|
||||
"last_change": null,
|
||||
"components": [
|
||||
{
|
||||
"id": "server",
|
||||
"kind": "unit",
|
||||
"label": "llama-server-rp (:11434 on the lab bridge)",
|
||||
"state": "up",
|
||||
"detail": "pid 1034519",
|
||||
"health": {
|
||||
"ok": true,
|
||||
"detail": "HTTP 200",
|
||||
"latency_ms": 1
|
||||
},
|
||||
"port": null,
|
||||
"unit": "llama-server-rp.service"
|
||||
}
|
||||
],
|
||||
"selected": {},
|
||||
"options": [],
|
||||
"job": null
|
||||
}
|
||||
],
|
||||
"job": null,
|
||||
"registry": {
|
||||
"path": "/home/harley/.config/gpu-cockpit/registry.toml",
|
||||
"pipeline_count": 6,
|
||||
"errors": [],
|
||||
"warnings": []
|
||||
},
|
||||
"listeners": [
|
||||
"127.0.0.1:8770",
|
||||
"10.10.10.1:8770"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
|
||||
android.useAndroidX=true
|
||||
kotlin.code.style=official
|
||||
BIN
Binary file not shown.
@@ -0,0 +1,7 @@
|
||||
distributionBase=GRADLE_USER_HOME
|
||||
distributionPath=wrapper/dists
|
||||
distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip
|
||||
networkTimeout=10000
|
||||
validateDistributionUrl=true
|
||||
zipStoreBase=GRADLE_USER_HOME
|
||||
zipStorePath=wrapper/dists
|
||||
+252
@@ -0,0 +1,252 @@
|
||||
#!/bin/sh
|
||||
|
||||
#
|
||||
# Copyright © 2015-2021 the original authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# https://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
|
||||
##############################################################################
|
||||
#
|
||||
# Gradle start up script for POSIX generated by Gradle.
|
||||
#
|
||||
# Important for running:
|
||||
#
|
||||
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
|
||||
# noncompliant, but you have some other compliant shell such as ksh or
|
||||
# bash, then to run this script, type that shell name before the whole
|
||||
# command line, like:
|
||||
#
|
||||
# ksh Gradle
|
||||
#
|
||||
# Busybox and similar reduced shells will NOT work, because this script
|
||||
# requires all of these POSIX shell features:
|
||||
# * functions;
|
||||
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
|
||||
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
|
||||
# * compound commands having a testable exit status, especially «case»;
|
||||
# * various built-in commands including «command», «set», and «ulimit».
|
||||
#
|
||||
# Important for patching:
|
||||
#
|
||||
# (2) This script targets any POSIX shell, so it avoids extensions provided
|
||||
# by Bash, Ksh, etc; in particular arrays are avoided.
|
||||
#
|
||||
# The "traditional" practice of packing multiple parameters into a
|
||||
# space-separated string is a well documented source of bugs and security
|
||||
# problems, so this is (mostly) avoided, by progressively accumulating
|
||||
# options in "$@", and eventually passing that to Java.
|
||||
#
|
||||
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
|
||||
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
|
||||
# see the in-line comments for details.
|
||||
#
|
||||
# There are tweaks for specific operating systems such as AIX, CygWin,
|
||||
# Darwin, MinGW, and NonStop.
|
||||
#
|
||||
# (3) This script is generated from the Groovy template
|
||||
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
|
||||
# within the Gradle project.
|
||||
#
|
||||
# You can find Gradle at https://github.com/gradle/gradle/.
|
||||
#
|
||||
##############################################################################
|
||||
|
||||
# Attempt to set APP_HOME
|
||||
|
||||
# Resolve links: $0 may be a link
|
||||
app_path=$0
|
||||
|
||||
# Need this for daisy-chained symlinks.
|
||||
while
|
||||
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
|
||||
[ -h "$app_path" ]
|
||||
do
|
||||
ls=$( ls -ld "$app_path" )
|
||||
link=${ls#*' -> '}
|
||||
case $link in #(
|
||||
/*) app_path=$link ;; #(
|
||||
*) app_path=$APP_HOME$link ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# This is normally unused
|
||||
# shellcheck disable=SC2034
|
||||
APP_BASE_NAME=${0##*/}
|
||||
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
|
||||
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s
|
||||
' "$PWD" ) || exit
|
||||
|
||||
# Use the maximum available, or set MAX_FD != -1 to use that value.
|
||||
MAX_FD=maximum
|
||||
|
||||
warn () {
|
||||
echo "$*"
|
||||
} >&2
|
||||
|
||||
die () {
|
||||
echo
|
||||
echo "$*"
|
||||
echo
|
||||
exit 1
|
||||
} >&2
|
||||
|
||||
# OS specific support (must be 'true' or 'false').
|
||||
cygwin=false
|
||||
msys=false
|
||||
darwin=false
|
||||
nonstop=false
|
||||
case "$( uname )" in #(
|
||||
CYGWIN* ) cygwin=true ;; #(
|
||||
Darwin* ) darwin=true ;; #(
|
||||
MSYS* | MINGW* ) msys=true ;; #(
|
||||
NONSTOP* ) nonstop=true ;;
|
||||
esac
|
||||
|
||||
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
|
||||
|
||||
|
||||
# Determine the Java command to use to start the JVM.
|
||||
if [ -n "$JAVA_HOME" ] ; then
|
||||
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
|
||||
# IBM's JDK on AIX uses strange locations for the executables
|
||||
JAVACMD=$JAVA_HOME/jre/sh/java
|
||||
else
|
||||
JAVACMD=$JAVA_HOME/bin/java
|
||||
fi
|
||||
if [ ! -x "$JAVACMD" ] ; then
|
||||
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
|
||||
|
||||
Please set the JAVA_HOME variable in your environment to match the
|
||||
location of your Java installation."
|
||||
fi
|
||||
else
|
||||
JAVACMD=java
|
||||
if ! command -v java >/dev/null 2>&1
|
||||
then
|
||||
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
|
||||
|
||||
Please set the JAVA_HOME variable in your environment to match the
|
||||
location of your Java installation."
|
||||
fi
|
||||
fi
|
||||
|
||||
# Increase the maximum file descriptors if we can.
|
||||
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
|
||||
case $MAX_FD in #(
|
||||
max*)
|
||||
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
|
||||
# shellcheck disable=SC2039,SC3045
|
||||
MAX_FD=$( ulimit -H -n ) ||
|
||||
warn "Could not query maximum file descriptor limit"
|
||||
esac
|
||||
case $MAX_FD in #(
|
||||
'' | soft) :;; #(
|
||||
*)
|
||||
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
|
||||
# shellcheck disable=SC2039,SC3045
|
||||
ulimit -n "$MAX_FD" ||
|
||||
warn "Could not set maximum file descriptor limit to $MAX_FD"
|
||||
esac
|
||||
fi
|
||||
|
||||
# Collect all arguments for the java command, stacking in reverse order:
|
||||
# * args from the command line
|
||||
# * the main class name
|
||||
# * -classpath
|
||||
# * -D...appname settings
|
||||
# * --module-path (only if needed)
|
||||
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
|
||||
|
||||
# For Cygwin or MSYS, switch paths to Windows format before running java
|
||||
if "$cygwin" || "$msys" ; then
|
||||
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
|
||||
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
|
||||
|
||||
JAVACMD=$( cygpath --unix "$JAVACMD" )
|
||||
|
||||
# Now convert the arguments - kludge to limit ourselves to /bin/sh
|
||||
for arg do
|
||||
if
|
||||
case $arg in #(
|
||||
-*) false ;; # don't mess with options #(
|
||||
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
|
||||
[ -e "$t" ] ;; #(
|
||||
*) false ;;
|
||||
esac
|
||||
then
|
||||
arg=$( cygpath --path --ignore --mixed "$arg" )
|
||||
fi
|
||||
# Roll the args list around exactly as many times as the number of
|
||||
# args, so each arg winds up back in the position where it started, but
|
||||
# possibly modified.
|
||||
#
|
||||
# NB: a `for` loop captures its iteration list before it begins, so
|
||||
# changing the positional parameters here affects neither the number of
|
||||
# iterations, nor the values presented in `arg`.
|
||||
shift # remove old arg
|
||||
set -- "$@" "$arg" # push replacement arg
|
||||
done
|
||||
fi
|
||||
|
||||
|
||||
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
||||
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
|
||||
|
||||
# Collect all arguments for the java command:
|
||||
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
|
||||
# and any embedded shellness will be escaped.
|
||||
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
|
||||
# treated as '${Hostname}' itself on the command line.
|
||||
|
||||
set -- \
|
||||
"-Dorg.gradle.appname=$APP_BASE_NAME" \
|
||||
-classpath "$CLASSPATH" \
|
||||
org.gradle.wrapper.GradleWrapperMain \
|
||||
"$@"
|
||||
|
||||
# Stop when "xargs" is not available.
|
||||
if ! command -v xargs >/dev/null 2>&1
|
||||
then
|
||||
die "xargs is not available"
|
||||
fi
|
||||
|
||||
# Use "xargs" to parse quoted args.
|
||||
#
|
||||
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
|
||||
#
|
||||
# In Bash we could simply go:
|
||||
#
|
||||
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
|
||||
# set -- "${ARGS[@]}" "$@"
|
||||
#
|
||||
# but POSIX shell has neither arrays nor command substitution, so instead we
|
||||
# post-process each arg (as a line of input to sed) to backslash-escape any
|
||||
# character that might be a shell metacharacter, then use eval to reverse
|
||||
# that process (while maintaining the separation between arguments), and wrap
|
||||
# the whole thing up as a single "set" statement.
|
||||
#
|
||||
# This will of course break if any of these variables contains a newline or
|
||||
# an unmatched quote.
|
||||
#
|
||||
|
||||
eval "set -- $(
|
||||
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
|
||||
xargs -n1 |
|
||||
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
|
||||
tr '\n' ' '
|
||||
)" '"$@"'
|
||||
|
||||
exec "$JAVACMD" "$@"
|
||||
Vendored
+94
@@ -0,0 +1,94 @@
|
||||
@rem
|
||||
@rem Copyright 2015 the original author or authors.
|
||||
@rem
|
||||
@rem Licensed under the Apache License, Version 2.0 (the "License");
|
||||
@rem you may not use this file except in compliance with the License.
|
||||
@rem You may obtain a copy of the License at
|
||||
@rem
|
||||
@rem https://www.apache.org/licenses/LICENSE-2.0
|
||||
@rem
|
||||
@rem Unless required by applicable law or agreed to in writing, software
|
||||
@rem distributed under the License is distributed on an "AS IS" BASIS,
|
||||
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
@rem See the License for the specific language governing permissions and
|
||||
@rem limitations under the License.
|
||||
@rem
|
||||
@rem SPDX-License-Identifier: Apache-2.0
|
||||
@rem
|
||||
|
||||
@if "%DEBUG%"=="" @echo off
|
||||
@rem ##########################################################################
|
||||
@rem
|
||||
@rem Gradle startup script for Windows
|
||||
@rem
|
||||
@rem ##########################################################################
|
||||
|
||||
@rem Set local scope for the variables with windows NT shell
|
||||
if "%OS%"=="Windows_NT" setlocal
|
||||
|
||||
set DIRNAME=%~dp0
|
||||
if "%DIRNAME%"=="" set DIRNAME=.
|
||||
@rem This is normally unused
|
||||
set APP_BASE_NAME=%~n0
|
||||
set APP_HOME=%DIRNAME%
|
||||
|
||||
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
|
||||
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
|
||||
|
||||
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
|
||||
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
|
||||
|
||||
@rem Find java.exe
|
||||
if defined JAVA_HOME goto findJavaFromJavaHome
|
||||
|
||||
set JAVA_EXE=java.exe
|
||||
%JAVA_EXE% -version >NUL 2>&1
|
||||
if %ERRORLEVEL% equ 0 goto execute
|
||||
|
||||
echo. 1>&2
|
||||
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
|
||||
echo. 1>&2
|
||||
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
|
||||
echo location of your Java installation. 1>&2
|
||||
|
||||
goto fail
|
||||
|
||||
:findJavaFromJavaHome
|
||||
set JAVA_HOME=%JAVA_HOME:"=%
|
||||
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
|
||||
|
||||
if exist "%JAVA_EXE%" goto execute
|
||||
|
||||
echo. 1>&2
|
||||
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
|
||||
echo. 1>&2
|
||||
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
|
||||
echo location of your Java installation. 1>&2
|
||||
|
||||
goto fail
|
||||
|
||||
:execute
|
||||
@rem Setup the command line
|
||||
|
||||
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
|
||||
|
||||
|
||||
@rem Execute Gradle
|
||||
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
|
||||
|
||||
:end
|
||||
@rem End local scope for the variables with windows NT shell
|
||||
if %ERRORLEVEL% equ 0 goto mainEnd
|
||||
|
||||
:fail
|
||||
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
|
||||
rem the _cmd.exe /c_ return code!
|
||||
set EXIT_CODE=%ERRORLEVEL%
|
||||
if %EXIT_CODE% equ 0 set EXIT_CODE=1
|
||||
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
|
||||
exit /b %EXIT_CODE%
|
||||
|
||||
:mainEnd
|
||||
if "%OS%"=="Windows_NT" endlocal
|
||||
|
||||
:omega
|
||||
@@ -0,0 +1,7 @@
|
||||
pluginManagement { repositories { google(); mavenCentral(); gradlePluginPortal() } }
|
||||
dependencyResolutionManagement {
|
||||
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
|
||||
repositories { google(); mavenCentral() }
|
||||
}
|
||||
rootProject.name = "GpuCockpit"
|
||||
include(":app")
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Registry-driven single-GPU pipeline console."""
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,4 @@
|
||||
from .server import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,28 @@
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>GPU Cockpit</title><link rel="icon" href="/static/logo.png">
|
||||
<style>
|
||||
:root{color-scheme:dark;--bg:#0c1118;--panel:#141d27;--line:#293747;--text:#e7edf4;--muted:#91a2b5;--green:#47d7a0;--amber:#f3b95f;--red:#ff6c72;--blue:#71b7ff}*{box-sizing:border-box}body{margin:0;background:radial-gradient(ellipse at 40% -10%,#1a2a3c,var(--bg) 55%);color:var(--text);font:15px/1.45 system-ui,sans-serif}header{padding:18px max(20px,calc((100vw - 1440px)/2));display:flex;gap:14px;align-items:center;border-bottom:1px solid var(--line);position:sticky;top:0;background:#0c1118eF;z-index:4}header .brand{width:38px;height:38px;flex:none;border-radius:11px;overflow:hidden;border:1px solid #2b3a4d;background:#14161f;box-shadow:0 0 12px #71b7ff22}header .brand img{width:180%;height:180%;margin:-40% 0 0 -40%;display:block}.unattributed td{color:var(--muted);font-style:italic}h1{font-size:19px;margin:0}header small,.muted{color:var(--muted)}.right{margin-left:auto;display:flex;align-items:center;gap:14px}.wrap{max-width:1440px;margin:auto;padding:22px}.top{display:grid;grid-template-columns:minmax(300px,1fr) minmax(0,2fr);gap:18px}.panel,.card{background:linear-gradient(145deg,#17222e,#121a23);border:1px solid var(--line);border-radius:15px;padding:18px;box-shadow:0 12px 32px #0002}.panel h2{margin:0 0 14px;font-size:16px}.gauge{display:flex;gap:18px;align-items:center}.ring{width:120px;height:120px;flex:none}.ring text{fill:var(--text);font-size:17px;font-weight:700}.metrics{display:grid;grid-template-columns:repeat(3,1fr);gap:10px;flex:1}.metric{border-left:1px solid var(--line);padding-left:12px}.metric b{display:block;font-size:20px}.metric span{font-size:12px;color:var(--muted)}.spark{width:100%;height:40px;margin-top:10px}.process{width:100%;border-collapse:collapse;font-size:13px}.process td,.process th{text-align:left;padding:7px;border-bottom:1px solid #29374788}.external{color:var(--amber)}.section{display:flex;align-items:end;justify-content:space-between;margin:25px 0 12px}.section h2{margin:0}.grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:14px}.card h3{margin:0;font-size:17px}.title{display:flex;align-items:center;gap:9px}.dot{width:10px;height:10px;border-radius:50%;background:#778493}.dot.up{background:var(--green);box-shadow:0 0 10px #47d7a088}.dot.partial,.dot.starting,.dot.stopping{background:var(--amber)}.dot.error{background:var(--red)}.desc{min-height:42px;color:var(--muted);font-size:13px;margin:7px 0 12px}.tags{display:flex;gap:7px;flex-wrap:wrap;color:var(--muted);font-size:12px}.serving{margin:10px 0 4px;font-weight:600}.note{color:var(--amber);font-size:12px;min-height:18px}.components{border-top:1px solid var(--line);margin-top:12px;padding-top:9px}.comp{display:flex;justify-content:space-between;gap:8px;font-size:12px;padding:3px 0}.selects{margin-top:10px;display:grid;gap:8px}.choice{display:grid;grid-template-columns:auto 1fr;align-items:center;gap:8px;font-size:12px}select,button{font:inherit;color:var(--text);background:#202e3d;border:1px solid #3a4e63;border-radius:8px;padding:8px}select{min-width:0;width:100%}button{cursor:pointer}button:hover{border-color:var(--blue)}button:focus-visible,a:focus-visible,select:focus-visible{outline:2px solid var(--blue);outline-offset:2px}.primary{background:#216b58;border-color:#399b7e;font-weight:650}.danger{background:#45272c;border-color:#704047}.buttons{display:flex;gap:7px;flex-wrap:wrap;margin-top:13px}.buttons button{font-size:13px}.job{color:var(--blue);font-size:12px;min-height:18px;margin-top:8px}.log{margin-top:8px;color:var(--muted);font-size:12px}details summary{cursor:pointer;color:var(--muted)}.banner{display:none;padding:14px 17px;margin-top:16px;border:1px solid #a77e38;background:#382d1c;border-radius:12px}.banner.show{display:flex;align-items:center;gap:12px;flex-wrap:wrap}.banner span{flex:1}.activity{margin-top:20px}.activity table{width:100%;font-size:12px}#activityContent{max-height:320px;overflow:auto}.status{display:inline-flex;align-items:center;gap:7px}.live{color:var(--green)}.offline{color:var(--red)}.notice{color:var(--amber);font-size:13px;max-width:34ch;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.notice.bad{color:var(--red)}a{color:var(--blue)}@media(max-width:1000px){.grid{grid-template-columns:repeat(2,minmax(0,1fr))}.top{grid-template-columns:1fr}}@media(max-width:600px){header{padding:12px 14px}.wrap{padding:14px}.grid{grid-template-columns:1fr}.right{gap:7px}header small{display:none}.gauge{align-items:flex-start}.ring{width:95px;height:95px}.metrics{grid-template-columns:repeat(2,1fr)}.metric b{font-size:17px}}
|
||||
</style></head><body><header><span class="brand"><img src="/static/logo.png" onerror="this.parentNode.hidden=true" alt=""></span><div><h1>GPU Cockpit</h1><small id="host">Connecting…</small></div><div class="right"><span id="notice" class="notice"></span><span id="connection" class="status offline">● offline</span><span id="stream" class="muted">polling</span><button class="primary" onclick="freeCard()">Free the card</button></div></header>
|
||||
<main class="wrap"><section class="top"><div class="panel"><h2>Card memory</h2><div class="gauge"><svg class="ring" viewBox="0 0 120 120" aria-label="VRAM usage"><circle cx="60" cy="60" r="48" fill="none" stroke="#293747" stroke-width="10"/><circle id="arc" cx="60" cy="60" r="48" fill="none" stroke="#47d7a0" stroke-width="10" stroke-linecap="round" transform="rotate(-90 60 60)" stroke-dasharray="301.6" stroke-dashoffset="301.6"/><text x="60" y="65" text-anchor="middle" id="pct">—</text></svg><div class="metrics"><div class="metric"><b id="used">—</b><span>used / total MiB</span></div><div class="metric"><b id="util">—</b><span>GPU utilization</span></div><div class="metric"><b id="temp">—</b><span>temperature</span></div></div></div><small class="muted" id="floor"></small><svg class="spark" viewBox="0 0 300 40" preserveAspectRatio="none"><polyline id="spark" fill="none" stroke="#71b7ff" stroke-width="2" points=""/></svg></div><div class="panel"><h2>GPU processes</h2><div id="processes" class="muted">No GPU probe</div></div></section>
|
||||
<div id="banner" class="banner"><span id="bannerText"></span><button id="switchBtn" class="primary">Switch to this</button><button id="forceBtn" class="danger" hidden>Force switch</button></div><div class="section"><h2>Pipelines</h2><span class="muted" id="regline"></span></div><section id="pipes" class="grid"></section>
|
||||
<details class="panel activity"><summary>Activity · command audit and registry report</summary><div id="activityContent"><h3>Registry validation</h3><div id="validation"></div><h3>Command audit</h3><div id="audit"></div></div></details></main>
|
||||
<script>
|
||||
const $=id=>document.getElementById(id);let state=null,busy=false,evt=null;
|
||||
async function api(path,opts){let r=await fetch(path,opts);let j=await r.json().catch(()=>({}));if(!r.ok)throw Error(j.error||r.statusText);return j}
|
||||
function esc(x){return String(x??'').replace(/[&<>"']/g,c=>({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c]))}
|
||||
function render(s){state=s;$('host').textContent=`${s.host} · ${s.now}`;$('connection').className='status live';$('connection').textContent='● live';let c=s.card;if(!c.available){$('used').textContent='No GPU probe';$('util').textContent='—';$('temp').textContent='—';$('pct').textContent='—';$('arc').style.strokeDashoffset=301.6;$('floor').textContent=c.detail||'nvidia-smi unavailable';$('processes').textContent='No GPU probe';}else{$('used').textContent=`${c.used_mib} / ${c.total_mib}`;$('util').textContent=`${c.util_pct}%`;$('temp').textContent=`${c.temp_c}°C`;$('pct').textContent=`${c.pct_of_usable}%`;$('arc').style.strokeDashoffset=301.6*(1-c.pct_of_usable/100);$('floor').textContent=`Idle floor ${c.idle_floor_mib} MiB · gauge uses ${c.usable_mib} MiB usable`;$('processes').innerHTML=c.processes.length?`<table class="process"><thead><tr><th>PID</th><th>Process</th><th>MiB</th><th>Owner</th></tr></thead><tbody>${c.processes.map(p=>`<tr class="${p.pipeline?'':'external'}"><td>${p.pid}</td><td>${esc(p.name)}</td><td>${p.used_mib}</td><td>${p.pipeline?esc(p.pipeline):'External holder'}</td></tr>`).join('')+((c.used_mib-c.processes.reduce((a,p)=>a+p.used_mib,0))>150?`<tr class="unattributed"><td>—</td><td>desktop & graphics surfaces</td><td>${c.used_mib-c.processes.reduce((a,p)=>a+p.used_mib,0)}</td><td>—</td></tr>`:'')}</tbody></table>`:'No GPU processes reported';let vals=c.history.map(x=>x.used_mib),max=Math.max(1,...vals);$('spark').setAttribute('points',vals.map((v,i)=>`${i*300/Math.max(1,vals.length-1)},${38-v/max*34}`).join(' '));}
|
||||
$('regline').textContent=`${s.registry.pipeline_count} pipelines · ${s.registry.errors.length} errors · ${s.registry.warnings.length} warnings`;
|
||||
$('pipes').innerHTML=s.pipelines.map(p=>`<article class="card"><div class="title"><span class="dot ${p.state}"></span><h3>${esc(p.label)}</h3><span class="right muted">${esc(p.state)}</span></div><div class="desc">${esc(p.description)}</div><div class="tags"><span>${p.vram_mib??'—'} MiB expected</span><span>${p.requires_card?'shared card':'CPU / independent'}</span>${p.protected?'<span>protected</span>':''}</div><div class="serving">${esc(p.serving||'')}</div><div class="note">${esc(p.note||'')}</div><div class="components">${p.components.map(c=>`<div class="comp"><span>${esc(c.label)} · ${esc(c.state)}${c.port?` · :${c.port}`:''}</span><span>${esc(c.health?.detail||c.detail||'')}</span></div>`).join('')}</div><div class="selects">${p.options.map(o=>`<label class="choice">${esc(o.label)}<select aria-label="${esc(o.label)}" onchange="choose('${p.id}','${o.id}',this.value)">${o.choices.map(ch=>`<option value="${esc(ch.id)}" ${ch.id===o.selected?'selected':''}>${esc(ch.label)}${ch.verified?' · verified':''}</option>`).join('')}</select>${o.choices.find(x=>x.id===o.selected)?.note?`<small class="muted">${esc(o.choices.find(x=>x.id===o.selected).note)}</small>`:''}</label>`).join('')}</div><div class="buttons"><button class="primary" onclick="action('${p.id}','start')">Start</button><button onclick="stopPipeline('${p.id}')">Stop</button><button onclick="action('${p.id}','restart')">Restart</button><button onclick="switchTo('${p.id}')">Switch to this</button>${p.open_url?`<a href="${esc(p.open_url)}" target="_blank" rel="noopener">Open ↗</a>`:''}</div><div class="job">${p.job?`${esc(p.job.id)} · ${esc(p.job.step)}`:''}</div><details class="log"><summary>Log tail</summary><pre id="log-${p.id}">Open to load</pre></details></article>`).join('');
|
||||
let active=s.pipelines.find(p=>p.job);busy=!!s.job;$('pipes').querySelectorAll('button,select').forEach(x=>x.disabled=busy);if(active){$('banner').classList.add('show');$('bannerText').textContent=`${active.label} is changing: ${active.job.step}`;$('switchBtn').hidden=true;$('forceBtn').hidden=true;}else{$('banner').classList.remove('show');}
|
||||
$('validation').innerHTML=[...s.registry.errors.map(x=>`<div class="external">Error: ${esc(x)}</div>`),...s.registry.warnings.map(x=>`<div class="muted">Warning: ${esc(x)}</div>`)].join('')||'<span class="live">No registry problems</span>';api('/api/audit?n=30').then(a=>$('audit').innerHTML=`<table class="process"><tbody>${a.slice().reverse().map(x=>`<tr><td>${esc(x.ts)}</td><td>${esc(x.pipeline)}</td><td>${esc(x.argv.join(' '))}</td><td>${x.rc}</td><td>${x.duration_ms} ms</td></tr>`).join('')}</tbody></table>`).catch(()=>{});
|
||||
}
|
||||
async function refresh(){try{render(await api('/api/state'))}catch(e){$('connection').className='status offline';$('connection').textContent='● offline';$('host').textContent='Waiting for server';}}
|
||||
let noticeTimer=null;
|
||||
function notice(msg,bad){let n=$('notice');n.textContent=msg;n.className=bad?'notice bad':'notice';clearTimeout(noticeTimer);noticeTimer=setTimeout(()=>{n.textContent=''},12000)}
|
||||
async function action(id,a,force=false){if(busy)return;let options={};state.pipelines.find(p=>p.id===id)?.options.forEach(o=>options[o.id]=o.selected);try{let j=await api(`/api/pipelines/${id}/action`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({action:a,options,force})});notice(`job ${j.job_id} accepted`);busy=true;refresh()}catch(e){notice(e.message,true);}}
|
||||
async function choose(pid,oid,value){let p=state.pipelines.find(p=>p.id===pid),options={};p.options.forEach(o=>options[o.id]=o.selected);options[oid]=value;try{await api(`/api/pipelines/${pid}/options`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({options})});refresh()}catch(e){notice(e.message,true)}}
|
||||
async function switchTo(id){let p=state.pipelines.find(x=>x.id===id),owners=state.pipelines.filter(x=>x.id!==id&&x.requires_card&&x.state!=='down');let blocked=owners.find(x=>x.protected);if(blocked){$('banner').classList.add('show');$('bannerText').textContent=`${blocked.label} is protected and would keep ${p.label} from using the card. Force stops it and loses its current work.`;$('switchBtn').hidden=true;$('forceBtn').hidden=false;$('forceBtn').onclick=()=>action(id,'start',true);return;}if(owners.length){$('banner').classList.add('show');$('bannerText').textContent=`Starting ${p.label} will stop ${owners.map(x=>x.label).join(', ')} and release the card.`;$('switchBtn').hidden=false;$('switchBtn').textContent='Stop them and switch';$('switchBtn').onclick=()=>action(id,'start');}else action(id,'start');}
|
||||
async function stopPipeline(id){let p=state.pipelines.find(x=>x.id===id);if(p&&p.protected&&p.state!=='down'){$('banner').classList.add('show');$('bannerText').textContent=`${p.label} is protected \u2014 stopping it now would lose work in progress.`;$('switchBtn').hidden=true;$('forceBtn').hidden=false;$('forceBtn').textContent='Force stop';$('forceBtn').onclick=()=>action(id,'stop',true);return;}action(id,'stop');}
|
||||
async function freeCard(){try{await api('/api/card/free',{method:'POST'});busy=true;refresh()}catch(e){notice(e.message,true)}}
|
||||
document.addEventListener('click',e=>{let d=e.target.closest('details.log');if(d&&d.open){let id=d.querySelector('pre').id.slice(4);api(`/api/logs/${id}?n=40`).then(x=>$(d.querySelector('pre').id).textContent=x.join('\n')).catch(()=>{})}});
|
||||
function connect(){if(!('EventSource'in window))return;$('stream').textContent='live stream';evt=new EventSource('/api/events');evt.addEventListener('state',e=>{try{render(JSON.parse(e.data))}catch{}});evt.onerror=()=>{$('stream').textContent='polling fallback';evt.close();evt=null;setTimeout(connect,5000)}}refresh();connect();setInterval(()=>{if(!evt||evt.readyState!==1)refresh()},3000);
|
||||
</script></body></html>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
@@ -0,0 +1,63 @@
|
||||
"""Read-only host, process and health probes."""
|
||||
import json, os, re, socket, subprocess, time, urllib.request, urllib.error
|
||||
|
||||
def friendly_error(exc,timeout=None):
|
||||
"""A status line has no room for '<urlopen error [Errno 111] Connection refused>'.
|
||||
Map the usual socket failures to something a person reads at a glance."""
|
||||
reason=getattr(exc,"reason",None); errno=getattr(reason,"errno",None) or getattr(exc,"errno",None)
|
||||
if errno==111: return "connection refused"
|
||||
if errno==113: return "no route to host"
|
||||
if errno==98: return "address already in use"
|
||||
if errno==110 or isinstance(exc,(TimeoutError,socket.timeout)) or "timed out" in str(exc).lower():
|
||||
return f"timed out after {timeout}s" if timeout else "timed out"
|
||||
if isinstance(exc,urllib.error.URLError): exc=reason or exc
|
||||
return str(exc).strip(" <>")[:120]
|
||||
|
||||
def health(spec, runner=None, pipeline=None):
|
||||
if not spec or spec.get("kind")=="none": return {"ok":True,"detail":"no health check","latency_ms":0}
|
||||
start=time.monotonic(); kind=spec.get("kind"); timeout=min(30,max(.1,float(spec.get("timeout_s",3))))
|
||||
try:
|
||||
if kind=="http":
|
||||
req=urllib.request.Request(spec["url"],method="GET")
|
||||
with urllib.request.urlopen(req,timeout=timeout) as r: ok=200<=r.status<400; detail=f"HTTP {r.status}"
|
||||
elif kind=="tcp":
|
||||
with socket.create_connection((spec["host"],int(spec["port"])),timeout=timeout): pass
|
||||
ok=True; detail="TCP connected"
|
||||
elif kind=="command":
|
||||
if runner:
|
||||
rc,out,_=runner.run(spec["argv"],pipeline,"health",timeout=timeout,readonly=True)
|
||||
else:
|
||||
return {"ok":False,"detail":"command health probe needs runner","latency_ms":0}
|
||||
ok=rc==0; detail=f"exit {rc}"
|
||||
else: return {"ok":False,"detail":f"unknown health kind: {kind}","latency_ms":0}
|
||||
return {"ok":ok,"detail":detail,"latency_ms":round((time.monotonic()-start)*1000)}
|
||||
except Exception as exc: return {"ok":False,"detail":friendly_error(exc,timeout),"latency_ms":round((time.monotonic()-start)*1000)}
|
||||
|
||||
def gpu(runner=None):
|
||||
try:
|
||||
if runner:
|
||||
rc,out,_=runner.run(["nvidia-smi","--query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu","--format=csv,noheader,nounits"],None,"gpu-probe",timeout=3,readonly=True)
|
||||
else:
|
||||
p=subprocess.run(["nvidia-smi","--query-gpu=name,memory.used,memory.total,utilization.gpu,temperature.gpu","--format=csv,noheader,nounits"],timeout=3,text=True,capture_output=True); rc,out=p.returncode,p.stdout if p.returncode==0 else p.stderr
|
||||
if rc: raise RuntimeError(out.strip() or f"exit {rc}")
|
||||
row=out.strip().splitlines()[0].split(",")
|
||||
name,used,total,util,temp=[x.strip() for x in row]
|
||||
procs=[]
|
||||
if runner: prc,pout,_=runner.run(["nvidia-smi","--query-compute-apps=pid,process_name,used_memory","--format=csv,noheader,nounits"],None,"gpu-process-probe",timeout=3,readonly=True)
|
||||
else:
|
||||
pp=subprocess.run(["nvidia-smi","--query-compute-apps=pid,process_name,used_memory","--format=csv,noheader,nounits"],timeout=3,text=True,capture_output=True); prc,pout=pp.returncode,pp.stdout
|
||||
if prc==0:
|
||||
for line in pout.splitlines():
|
||||
x=[z.strip() for z in line.split(",")]
|
||||
if len(x)>=3:
|
||||
try: procs.append({"pid":int(x[0]),"name":os.path.basename(x[1]),"used_mib":int(float(x[2])),"pipeline":None})
|
||||
except ValueError: pass
|
||||
return {"available":True,"name":name,"used_mib":int(float(used)),"total_mib":int(float(total)),"util_pct":int(float(util)),"temp_c":int(float(temp)),"processes":procs}
|
||||
except Exception as exc: return {"available":False,"name":None,"used_mib":None,"total_mib":None,"util_pct":None,"temp_c":None,"processes":[],"detail":friendly_error(exc)}
|
||||
|
||||
def proc_start(pid):
|
||||
try:
|
||||
raw=open(f"/proc/{pid}/stat").read(); fields=raw[raw.rfind(")")+2:].split()
|
||||
if fields[0] in ("Z","X"): return None
|
||||
return fields[19]
|
||||
except Exception: return None
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Registry loading and validation. Invalid pipelines are retained for reporting."""
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
ID = re.compile(r"^[a-z0-9-]+$")
|
||||
PLACEHOLDER = re.compile(r"\{([A-Za-z_][A-Za-z0-9_]*)\}")
|
||||
|
||||
|
||||
def load(path):
|
||||
errors, warnings = [], []
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
data = tomllib.load(f)
|
||||
except Exception as exc:
|
||||
return {"card": {}, "pipeline": []}, [f"registry: {exc}"], warnings
|
||||
card = data.setdefault("card", {})
|
||||
if not isinstance(card, dict):
|
||||
errors.append("card must be a table")
|
||||
card = data["card"] = {}
|
||||
if "reclaim" in card and not isinstance(card["reclaim"], dict):
|
||||
errors.append("card.reclaim must be a table")
|
||||
card.pop("reclaim", None)
|
||||
for key, default, minimum in (("total_mib", None, 1), ("idle_floor_mib", 0, 0), ("poll_seconds", 2, .1)):
|
||||
if key not in card:
|
||||
if default is not None: card[key] = default
|
||||
continue
|
||||
try:
|
||||
value=float(card[key])
|
||||
if value < minimum: raise ValueError()
|
||||
card[key]=int(value) if key != "poll_seconds" else value
|
||||
except (TypeError, ValueError):
|
||||
errors.append(f"card.{key} must be a number >= {minimum}")
|
||||
if default is not None: card[key]=default
|
||||
else: card.pop(key,None)
|
||||
rec=card.get("reclaim")
|
||||
if rec and rec.get("kind") not in ("http", "command"):
|
||||
errors.append("card.reclaim.kind must be http or command")
|
||||
if rec:
|
||||
try:
|
||||
if int(rec.get("wait_timeout_s",45)) < 1: raise ValueError()
|
||||
except (TypeError,ValueError): errors.append("card.reclaim.wait_timeout_s must be a positive integer")
|
||||
if rec.get("kind")=="http" and not isinstance(rec.get("url"),str): errors.append("card.reclaim.url is required for http reclaim")
|
||||
if rec.get("kind")=="command" and (not isinstance(rec.get("argv"),list) or not rec.get("argv")):
|
||||
errors.append("card.reclaim.argv is required for command reclaim")
|
||||
pipes = data.setdefault("pipeline", [])
|
||||
if not isinstance(pipes, list):
|
||||
errors.append("pipeline must be an array of tables")
|
||||
pipes = []
|
||||
data["pipeline"] = pipes
|
||||
ids = set()
|
||||
for i, p in enumerate(pipes):
|
||||
prefix = f"pipeline[{i}]"
|
||||
if not isinstance(p, dict):
|
||||
errors.append(f"{prefix}: pipeline entry must be a table")
|
||||
continue
|
||||
pid = p.get("id")
|
||||
if not isinstance(pid, str) or not ID.fullmatch(pid):
|
||||
errors.append(f"{prefix}: invalid id")
|
||||
elif pid in ids:
|
||||
errors.append(f"{prefix}: duplicate id {pid}")
|
||||
if isinstance(pid, str):
|
||||
ids.add(pid)
|
||||
comps = p.get("component", [])
|
||||
if not isinstance(comps, list):
|
||||
errors.append(f"{prefix} {pid}: component must be an array of tables")
|
||||
comps = []
|
||||
p["component"] = comps
|
||||
if not comps:
|
||||
errors.append(f"{prefix} {pid}: at least one component is required")
|
||||
opts = p.get("option", [])
|
||||
if not isinstance(opts, list):
|
||||
errors.append(f"{prefix} {pid}: option must be an array of tables")
|
||||
opts = []
|
||||
p["option"] = opts
|
||||
clean_opts = [o for o in opts if isinstance(o, dict) and isinstance(o.get("id"),str)]
|
||||
if len(clean_opts) != len(opts):
|
||||
errors.append(f"{prefix} {pid}: option entries must be tables with string ids")
|
||||
p["option"] = clean_opts
|
||||
for o in clean_opts:
|
||||
if not isinstance(o.get("id"),str): errors.append(f"{prefix} {pid}: option id must be a string")
|
||||
if o.get("kind") not in ("argv", "note"):
|
||||
errors.append(f"{prefix} {pid}: option {o.get('id')} kind must be argv or note")
|
||||
choices = o.get("choice", [])
|
||||
if not isinstance(choices, list):
|
||||
errors.append(f"{prefix} {pid}: option {o.get('id')} choices must be an array")
|
||||
choices = []
|
||||
choices = [ch for ch in choices if isinstance(ch, dict) and isinstance(ch.get("id"),str)]
|
||||
o["choice"] = choices
|
||||
if any(not isinstance(ch.get("id"),str) for ch in choices): errors.append(f"{prefix} {pid}: choice ids must be strings")
|
||||
if o.get("kind") == "argv":
|
||||
names = set()
|
||||
for ch in choices:
|
||||
vars_ = ch.get("vars", {})
|
||||
if not isinstance(vars_, dict):
|
||||
errors.append(f"{prefix} {pid}: choice {ch.get('id')} vars must be a table")
|
||||
vars_ = ch["vars"] = {}
|
||||
names.update(vars_)
|
||||
for ch in choices:
|
||||
missing = names - set(ch.get("vars", {}))
|
||||
if missing:
|
||||
errors.append(f"{prefix} {pid}: choice {ch.get('id')} missing vars: {', '.join(sorted(missing))}")
|
||||
if o.get("default") not in [ch.get("id") for ch in choices]:
|
||||
errors.append(f"{prefix} {pid}: option {o.get('id')} has invalid default")
|
||||
compids = set()
|
||||
for c in comps:
|
||||
if not isinstance(c, dict):
|
||||
errors.append(f"{prefix} {pid}: component entries must be tables")
|
||||
continue
|
||||
if c.get("kind") not in ("proc", "unit"):
|
||||
errors.append(f"{prefix} {pid}: component kind must be proc or unit")
|
||||
cid = c.get("id")
|
||||
if isinstance(cid,str) and cid in compids:
|
||||
errors.append(f"{prefix} {pid}: duplicate component id {cid}")
|
||||
if isinstance(cid, str):
|
||||
compids.add(cid)
|
||||
if c.get("kind")=="unit" and not isinstance(c.get("unit"),str):
|
||||
errors.append(f"{prefix} {pid}/{cid}: unit name is required")
|
||||
if c.get("kind") == "proc":
|
||||
argv = c.get("argv", [])
|
||||
if not isinstance(argv, list) or not argv or any(not isinstance(x, str) for x in argv):
|
||||
errors.append(f"{prefix} {pid}/{cid}: proc argv must be a non-empty string array")
|
||||
argv = []
|
||||
elif argv[0].startswith("/") and not Path(argv[0]).exists():
|
||||
warnings.append(f"{pid}/{cid}: executable path does not exist: {argv[0]}")
|
||||
providers = {name for o in clean_opts if o.get("kind") == "argv" for ch in o.get("choice", []) for name in ch.get("vars", {})}
|
||||
for arg in argv:
|
||||
for name in PLACEHOLDER.findall(arg):
|
||||
if name not in providers:
|
||||
errors.append(f"{prefix} {pid}: placeholder {{{name}}} has no option value")
|
||||
if "env" in c and (not isinstance(c["env"], dict) or any(not isinstance(k, str) or not isinstance(v, str) for k,v in c["env"].items())):
|
||||
errors.append(f"{prefix} {pid}/{cid}: env must map strings to strings")
|
||||
if "health" in c and not isinstance(c["health"], dict):
|
||||
errors.append(f"{prefix} {pid}/{cid}: health must be a table")
|
||||
c["health"] = {"kind": "none"}
|
||||
health=c.get("health",{"kind":"none"})
|
||||
if isinstance(health,dict):
|
||||
if health.get("kind") not in ("http","tcp","command","none"):
|
||||
errors.append(f"{prefix} {pid}/{cid}: invalid health kind")
|
||||
if health.get("kind")=="command" and (not isinstance(health.get("argv"),list) or not health.get("argv")):
|
||||
errors.append(f"{prefix} {pid}/{cid}: command health requires argv")
|
||||
cwd = c.get("cwd")
|
||||
if cwd and not Path(cwd).exists():
|
||||
warnings.append(f"{pid}/{cid}: configured cwd does not exist: {cwd}")
|
||||
return data, errors, warnings
|
||||
|
||||
|
||||
def problem_count(reg):
|
||||
return len(reg.get("pipeline", []))
|
||||
@@ -0,0 +1,66 @@
|
||||
"""The sole subprocess boundary and command audit trail."""
|
||||
import json, os, subprocess, threading, time
|
||||
import signal
|
||||
from datetime import datetime, timezone
|
||||
|
||||
def stamp(): return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
class Runner:
|
||||
def __init__(self, state_dir, dry_run=False, systemctl="systemctl", sudo="sudo"):
|
||||
self.state_dir, self.dry_run, self.systemctl, self.sudo = state_dir, dry_run, systemctl, sudo
|
||||
self.lock = threading.Lock(); self.audit_path = os.path.join(state_dir, "audit.jsonl")
|
||||
os.makedirs(state_dir, exist_ok=True)
|
||||
def run(self, argv, pipeline=None, action="probe", timeout=15, env=None, cwd=None, readonly=False):
|
||||
start=time.monotonic(); rc=0; out=""
|
||||
try:
|
||||
if self.dry_run and not readonly:
|
||||
out="dry-run: command not executed"
|
||||
else:
|
||||
p=subprocess.run(list(map(str,argv)), cwd=cwd, env=env, text=True, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, timeout=timeout, check=False)
|
||||
rc=p.returncode; out=p.stdout or ""
|
||||
except Exception as exc:
|
||||
rc=127; out=f"{type(exc).__name__}: {exc}"
|
||||
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":list(argv),"rc":rc,
|
||||
"duration_ms":round((time.monotonic()-start)*1000),"dry_run":self.dry_run}
|
||||
with self.lock:
|
||||
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
|
||||
return rc,out,row
|
||||
def note(self,argv,action,pipeline=None,rc=0,duration_ms=0):
|
||||
"""Audit a side effect that has no subprocess of its own - the HTTP reclaim
|
||||
frees VRAM on the card like any command does, so the trail must carry it."""
|
||||
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":list(map(str,argv)),"rc":rc,
|
||||
"duration_ms":round(duration_ms),"dry_run":self.dry_run}
|
||||
with self.lock:
|
||||
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
|
||||
return row
|
||||
def audit(self,n=100):
|
||||
try:
|
||||
with open(self.audit_path,encoding="utf-8") as f: rows=[json.loads(x) for x in f if x.strip()]
|
||||
return rows[-max(1,min(1000,n)):]
|
||||
except OSError: return []
|
||||
def spawn(self, argv, *, cwd=None, env=None, stdout=None, stderr=None, pipeline=None, action="start"):
|
||||
"""Start an owned long-lived child; launch intent is audited by run()."""
|
||||
start=time.monotonic()
|
||||
try:
|
||||
child=subprocess.Popen(list(map(str,argv)), cwd=cwd, env=env, stdin=subprocess.DEVNULL,
|
||||
stdout=stdout, stderr=stderr, start_new_session=True)
|
||||
except Exception as exc:
|
||||
self.record_intent(argv,pipeline,action,cwd,rc=127,duration_ms=round((time.monotonic()-start)*1000),error=str(exc))
|
||||
raise
|
||||
self.record_intent(argv,pipeline,action,cwd,rc=0,duration_ms=round((time.monotonic()-start)*1000))
|
||||
return child
|
||||
def record_intent(self, argv, pipeline=None, action="start", cwd=None, *, rc=0, duration_ms=0, error=None):
|
||||
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":list(argv),"rc":rc,
|
||||
"duration_ms":duration_ms,"dry_run":False,"cwd":cwd}
|
||||
if error: row["error"]=error
|
||||
with self.lock:
|
||||
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
|
||||
def signal(self, pid, sig, pipeline=None, action="stop"):
|
||||
start=time.monotonic(); rc=0; out=""
|
||||
try: os.kill(pid,sig)
|
||||
except ProcessLookupError: rc=1; out="process no longer exists"
|
||||
row={"ts":stamp(),"pipeline":pipeline,"action":action,"argv":["signal",signal.Signals(sig).name,str(pid)],"rc":rc,"duration_ms":round((time.monotonic()-start)*1000),"dry_run":False}
|
||||
with self.lock:
|
||||
with open(self.audit_path,"a",encoding="utf-8") as f: f.write(json.dumps(row)+"\n")
|
||||
return rc,out,row
|
||||
@@ -0,0 +1,423 @@
|
||||
"""HTTP API, asynchronous action jobs and snapshot assembly."""
|
||||
import argparse, hmac, json, os, re, secrets, signal, socket, sys, threading, time, urllib.request
|
||||
from collections import deque
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse, parse_qs, unquote
|
||||
from . import __version__
|
||||
from .registry import load, PLACEHOLDER
|
||||
from .runner import Runner, stamp
|
||||
from .probes import health, gpu, proc_start, friendly_error
|
||||
|
||||
def iso(): return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
|
||||
def is_loopback(addr):
|
||||
"""Loopback is the machine itself: the desktop GUI and the TUI. Everything else has
|
||||
crossed a network boundary and has to prove it may talk to us."""
|
||||
a=str(addr or "")
|
||||
return a=="::1" or a.startswith("127.") or a.startswith("::ffff:127.")
|
||||
|
||||
def bearer(headers,query):
|
||||
got=(headers.get("Authorization") or "").strip()
|
||||
if got[:7].lower()=="bearer ": return got[7:].strip()
|
||||
return (query or {}).get("token",[""])[0]
|
||||
|
||||
def token_matches(expected,headers,query,client_addr):
|
||||
"""No token configured == loopback-only: main() refuses to open a front-door listener
|
||||
without one, so this can never be the thing that leaves a port wide open."""
|
||||
if not expected: return True
|
||||
if is_loopback(client_addr): return True
|
||||
return hmac.compare_digest(bearer(headers,query),expected)
|
||||
|
||||
def load_token(path):
|
||||
if not path: return None
|
||||
try: return (Path(path).read_text(encoding="utf-8").strip() or None)
|
||||
except FileNotFoundError: return None
|
||||
except OSError as exc:
|
||||
print(f"cannot read token file {path}: {exc}",file=sys.stderr); raise SystemExit(1)
|
||||
|
||||
class App:
|
||||
def __init__(self,args):
|
||||
self.args=args; self.registry,self.errors,self.warnings=load(args.registry)
|
||||
self.pipelines={p.get("id"):p for p in self.registry["pipeline"] if isinstance(p,dict) and isinstance(p.get("id"),str) and re.fullmatch(r"[a-z0-9-]+",p.get("id"))}
|
||||
self.runner=Runner(args.state_dir,args.dry_run,args.systemctl,args.sudo)
|
||||
self.lock=threading.Lock(); self.action_lock=threading.Lock(); self.job_guard=threading.Lock(); self.jobs={}; self.active_job=None
|
||||
self.history=deque(maxlen=90); self.last_gpu_sample=0; self.last_change={}; self.selected=self.read_selected()
|
||||
def read_selected(self):
|
||||
try:
|
||||
import tomllib
|
||||
with open(os.path.join(self.args.state_dir,"selected.toml"),"rb") as f: return tomllib.load(f)
|
||||
except Exception: return {}
|
||||
def save_selected(self):
|
||||
path=os.path.join(self.args.state_dir,"selected.toml"); lines=[]
|
||||
for pid,opts in self.selected.items():
|
||||
if not isinstance(opts,dict): continue
|
||||
lines.append(f'[{json.dumps(pid)}]')
|
||||
lines.extend(f'{k} = {json.dumps(v)}' for k,v in opts.items())
|
||||
Path(path).write_text("\n".join(lines)+("\n" if lines else ""),encoding="utf-8")
|
||||
def component(self,p,c):
|
||||
result={"id":c.get("id"),"kind":c.get("kind"),"label":c.get("label",c.get("id")),"state":"unknown","detail":"state unavailable","health":None,"port":None,"unit":None}
|
||||
spec=c.get("health",{}); result["port"]=spec.get("port") if spec.get("kind")=="tcp" else None
|
||||
if c.get("kind")=="unit":
|
||||
unit=c.get("unit",""); result["unit"]=unit
|
||||
rc,out,_=self.runner.run([self.args.systemctl,"show",unit,"--property=LoadState,ActiveState,SubState,ExecMainPID","--no-pager"],p.get("id"),"probe",timeout=4,readonly=True)
|
||||
vals=dict(x.split("=",1) for x in out.splitlines() if "=" in x)
|
||||
if rc or not vals: result["state"]="unknown"; result["detail"]=out.strip() or "systemd unit not found"; return result
|
||||
if vals.get("LoadState")=="not-found": result["state"]="unknown"; result["detail"]=f"systemd unit not found: {unit}"; return result
|
||||
if vals.get("ActiveState")=="active":
|
||||
h=health(spec,self.runner,p.get("id")); result.update(state="up" if h["ok"] else "error",detail=f"pid {vals.get('ExecMainPID','?')}",health=h)
|
||||
else: result.update(state="down",detail=vals.get("ActiveState","inactive"))
|
||||
return result
|
||||
pidfile=Path(self.args.state_dir)/"pids"/f"{p.get('id')}-{c.get('id')}.json"
|
||||
recorded=None
|
||||
try: recorded=json.loads(pidfile.read_text())
|
||||
except Exception: pass
|
||||
if recorded:
|
||||
pid=recorded.get("pid"); start=proc_start(pid) if isinstance(pid,int) else None
|
||||
if start is not None and str(start)==str(recorded.get("start")):
|
||||
h=health(spec,self.runner,p.get("id")); result.update(state="up" if h["ok"] else "error",detail=f"pid {pid}",health=h); return result
|
||||
h=health(spec,self.runner,p.get("id"))
|
||||
if spec.get("kind")=="none": h={"ok":False,"detail":"no health check","latency_ms":0}
|
||||
if h["ok"]: result.update(state="up",detail="external",health=h)
|
||||
elif recorded: result.update(state="down",detail="stale pidfile; process identity changed",health=h)
|
||||
else: result.update(state="down",detail="not running",health=h)
|
||||
return result
|
||||
def snapshot(self):
|
||||
g=gpu(self.runner); cardcfg=self.registry.get("card",{}); total=cardcfg.get("total_mib") or g.get("total_mib"); floor=int(cardcfg.get("idle_floor_mib",0)); usable=max(0,(total or 0)-floor)
|
||||
if g.get("available") and time.monotonic()-self.last_gpu_sample>=max(.1,float(cardcfg.get("poll_seconds",2))):
|
||||
self.history.append({"t":iso(),"used_mib":g["used_mib"]})
|
||||
self.last_gpu_sample=time.monotonic()
|
||||
g.update(total_mib=total,free_mib=max(0,total-g["used_mib"]) if g.get("available") and total else None,idle_floor_mib=floor,usable_mib=usable,pct_of_usable=min(100,round(100*g["used_mib"]/usable)) if g.get("available") and usable else None,history=list(self.history))
|
||||
pipes=[]; activejob=self.jobs.get(self.active_job) if self.active_job else None
|
||||
for pid,p in self.pipelines.items():
|
||||
comps=[self.component(p,c) for c in p.get("component",[])]; ups=sum(c["state"]=="up" for c in comps)
|
||||
errors=sum(c["state"]=="error" for c in comps)
|
||||
state="up" if comps and ups==len(comps) else "error" if errors and ups==0 else "partial" if ups else "down"
|
||||
job=next((j for j in self.jobs.values() if j.get("pipeline")==pid and j.get("state")=="running"),None)
|
||||
if job: state="stopping" if job.get("action")=="stop" else "starting"
|
||||
options=[]; selected=self.selected.get(pid,{})
|
||||
for o in p.get("option",[]):
|
||||
choice=selected.get(o.get("id"),o.get("default"))
|
||||
options.append({"id":o.get("id"),"label":o.get("label"),"kind":o.get("kind"),"selected":choice,"choices":[{"id":c.get("id"),"label":c.get("label"),"note":c.get("note"),"verified":c.get("verified"),"is_default":c.get("id")==o.get("default")} for c in o.get("choice",[])]})
|
||||
pipes.append({"id":pid,"label":p.get("label",pid),"description":p.get("description",""),"state":state,"health":"ok" if state=="up" else "unknown" if state=="down" else "fail","requires_card":p.get("requires_card",False),"vram_mib":p.get("vram_mib"),"protected":p.get("protected",False),"serving":p.get("serving"),"open_url":p.get("open_url"),"note":p.get("note"),"last_change":self.last_change.get(pid),"components":comps,"selected":selected,"options":options,"job":({"id":job["id"],"state":job["state"],"step":job["steps"][-1]["text"] if job["steps"] else "working"} if job else None)})
|
||||
if g.get("available"):
|
||||
for pipe in pipes:
|
||||
for comp in pipe["components"]:
|
||||
detail=comp.get("detail","")
|
||||
if comp.get("state")=="up" and detail.startswith("pid "):
|
||||
try: pidnum=int(detail.split()[1])
|
||||
except ValueError: continue
|
||||
for pr in g["processes"]:
|
||||
if pr["pid"]==pidnum: pr["pipeline"]=pipe["id"]
|
||||
return {"host":socket.gethostname(),"now":iso(),"version":__version__,"card":g,"pipelines":pipes,"job":({"id":activejob["id"],"pipeline":activejob.get("pipeline"),"state":activejob["state"],"step":activejob["steps"][-1]["text"] if activejob["steps"] else "working","started":activejob["started"]} if activejob else None),"registry":{"path":self.args.registry,"pipeline_count":len(self.pipelines),"errors":self.errors,"warnings":self.warnings},"listeners":list(getattr(self,"listeners",[]))}
|
||||
def start_job(self,pid,action,options=None,force=False):
|
||||
with self.job_guard:
|
||||
if self.active_job:
|
||||
raise ValueError(f"another action is running ({self.active_job})")
|
||||
jid=f"j-{int(time.time()*1000)}-{os.getpid()}"; job={"id":jid,"pipeline":pid,"action":action,"state":"running","started":iso(),"steps":[],"error":None}
|
||||
self.jobs[jid]=job; self.active_job=jid
|
||||
threading.Thread(target=self.work,args=(job,options or {},force),daemon=True).start(); return jid
|
||||
def preflight(self,p,action,options,force):
|
||||
if p.get("protected") and action in ("stop","restart") and not force:
|
||||
live=[self.component(p,c)["state"] for c in p.get("component",[])]
|
||||
if any(x in ("up","error","partial") for x in live):
|
||||
raise PermissionError(f"protected pipeline {p['id']} is active; force is required to stop it")
|
||||
if action in ("start","restart") and p.get("requires_card") and not force:
|
||||
for other in self.pipelines.values():
|
||||
if other["id"] != p["id"] and other.get("requires_card") and other.get("protected"):
|
||||
if any(self.component(other,c)["state"] in ("up","error","partial") for c in other.get("component",[])):
|
||||
raise PermissionError(f"protected pipeline {other['id']} is active; force is required")
|
||||
if action in ("start","restart"):
|
||||
for opt in p.get("option",[]):
|
||||
oid=opt.get("id"); value=options.get(oid,self.selected.get(p["id"],{}).get(oid,opt.get("default")))
|
||||
choice=next((x for x in opt.get("choice",[]) if x.get("id")==value),None)
|
||||
if choice is None: raise LookupError(f"unknown option value {oid}={value}")
|
||||
required=set().union(*(set(x.get("vars",{})) for x in opt.get("choice",[]))) if opt.get("kind")=="argv" else set()
|
||||
if required-set(choice.get("vars",{})): raise LookupError(f"choice {value} is missing required vars")
|
||||
def step(self,j,text,ok=True): j["steps"].append({"ts":iso(),"text":text,"ok":ok})
|
||||
def work(self,j,options,force):
|
||||
try:
|
||||
if j["action"]=="free":
|
||||
self.step(j,"asking the configured reclaim recipe to free the card")
|
||||
self.reclaim(j)
|
||||
else:
|
||||
p=self.pipelines[j["pipeline"]]
|
||||
if j["action"] in ("stop","restart"): self.stop_pipeline(j,p,force)
|
||||
if j["action"] in ("start","restart"):
|
||||
self.start_pipeline(j,p,options,force)
|
||||
j["state"]="done"; self.last_change[j.get("pipeline")]=iso() if j.get("pipeline") else None
|
||||
except Exception as exc:
|
||||
j["error"]=str(exc); self.step(j,f"failed: {exc}",False); j["state"]="failed"
|
||||
finally:
|
||||
if self.active_job==j["id"]: self.active_job=None
|
||||
def validate_options(self,p,requested):
|
||||
out={}; psel=self.selected.setdefault(p["id"],{})
|
||||
for o in p.get("option",[]):
|
||||
oid=o.get("id"); choice=requested.get(oid,psel.get(oid,o.get("default")))
|
||||
found=next((c for c in o.get("choice",[]) if c.get("id")==choice),None)
|
||||
if not found: raise ValueError(f"unknown option value {oid}={choice}")
|
||||
allvars=set().union(*(set(c.get("vars",{})) for c in o.get("choice",[]))) if o.get("kind")=="argv" else set()
|
||||
if allvars-set(found.get("vars",{})): raise ValueError(f"choice {choice} is missing required vars")
|
||||
psel[oid]=choice
|
||||
if o.get("kind")=="argv": out.update(found.get("vars",{}))
|
||||
self.save_selected(); return out
|
||||
def start_pipeline(self,j,p,requested,force):
|
||||
vars=self.validate_options(p,requested)
|
||||
current=[self.component(p,c)["state"] for c in p.get("component",[])]
|
||||
if current and all(x=="up" for x in current):
|
||||
self.step(j,f"{p.get('label',p['id'])} is already healthy; no command was run")
|
||||
return
|
||||
if p.get("requires_card"):
|
||||
for other in self.pipelines.values():
|
||||
if other["id"]==p["id"] or not other.get("requires_card"): continue
|
||||
state=[self.component(other,c)["state"] for c in other.get("component",[])]
|
||||
if any(x in ("up","error","partial") for x in state):
|
||||
if other.get("protected") and not force: raise RuntimeError(f"protected pipeline {other['id']} is active; force is required")
|
||||
self.step(j,f"stopping {other.get('label',other['id'])} (releasing the card)…")
|
||||
self.stop_pipeline(j,other,force,implicit=True)
|
||||
reclaim=self.registry.get("card",{}).get("reclaim")
|
||||
if reclaim: self.reclaim(j)
|
||||
threshold=p.get("start_below_mib")
|
||||
if threshold is not None:
|
||||
g=gpu(self.runner); used=g.get("used_mib") if g.get("available") else None
|
||||
deadline=time.monotonic()+min(120,int(reclaim.get("wait_timeout_s",45)) if reclaim else 45)
|
||||
if used is None: self.step(j,"GPU probe unavailable; cannot verify VRAM before start",False)
|
||||
else:
|
||||
while used>=int(threshold) and time.monotonic()<deadline:
|
||||
self.step(j,f"waiting for used VRAM < {threshold} MiB (now {used})…")
|
||||
time.sleep(min(max(.1,float(self.registry.get("card",{}).get("poll_seconds",2))),max(0,deadline-time.monotonic())))
|
||||
g=gpu(self.runner); used=g.get("used_mib") if g.get("available") else None
|
||||
if used is None: break
|
||||
if used is None or used>=int(threshold): self.step(j,f"VRAM wait timed out/unavailable (observed {used}); continuing with start",False)
|
||||
else: self.step(j,f"used VRAM is {used} MiB; below {threshold} MiB")
|
||||
for c in p.get("component",[]):
|
||||
self.step(j,f"starting {c.get('label',c.get('id'))}…")
|
||||
self.control(p,c,"start",vars)
|
||||
if self.args.dry_run:
|
||||
self.step(j,f"dry-run: would start {c.get('label',c.get('id'))}; health was not asserted")
|
||||
continue
|
||||
timeout=max(1,int(c.get("ready_timeout_s",20))); deadline=time.monotonic()+min(timeout,300); h=health(c.get("health",{}),self.runner,p["id"]); elapsed=0
|
||||
while not h["ok"] and time.monotonic()<deadline:
|
||||
time.sleep(min(.5,max(0,deadline-time.monotonic()))); elapsed+=.5; h=health(c.get("health",{}),self.runner,p["id"])
|
||||
if elapsed and int(elapsed)%5==0: self.step(j,f"waiting for health ({int(elapsed)}s)…")
|
||||
if not h["ok"]:
|
||||
tail=self.logs(p["id"],20); raise RuntimeError(f"{c.get('label',c.get('id'))} health failed after {timeout}s: {h['detail']}; last log lines: {' | '.join(tail[-8:]) or 'log file does not exist'}")
|
||||
self.step(j,f"{c.get('label',c.get('id'))} healthy ({h['detail']})")
|
||||
def stop_pipeline(self,j,p,force,implicit=False):
|
||||
if p.get("protected") and implicit and not force: raise RuntimeError(f"protected pipeline {p['id']} is active; force is required")
|
||||
for c in reversed(p.get("component",[])):
|
||||
state=self.component(p,c)
|
||||
if state["state"]=="down": continue
|
||||
if c.get("kind")=="proc" and state["detail"]=="external":
|
||||
msg=f"cannot stop {c.get('label')}: not started by the cockpit (no owned pid)"
|
||||
if implicit:
|
||||
self.step(j,msg+" - leaving it running; the card must be freed another way",False); continue
|
||||
raise RuntimeError(msg)
|
||||
self.step(j,f"stopping {c.get('label',c.get('id'))}…")
|
||||
self.control(p,c,"stop",{})
|
||||
states=[self.component(p,c)["state"] for c in p.get("component",[])]
|
||||
self.step(j,f"stop verification: {', '.join(states)}; GPU probe reports {gpu(self.runner).get('used_mib')} MiB used")
|
||||
def control(self,p,c,action,vars):
|
||||
pid=p["id"]; cid=c.get("id"); kind=c.get("kind")
|
||||
if kind=="unit":
|
||||
argv=[self.args.systemctl,action,c["unit"]]
|
||||
if self.args.sudo: argv=[self.args.sudo,"-n",*argv]
|
||||
rc,out,_=self.runner.run(argv,pid,action,timeout=30)
|
||||
if rc: raise RuntimeError(f"systemctl {action} {c['unit']} failed: {out.strip()}")
|
||||
return
|
||||
pf=Path(self.args.state_dir)/"pids"/f"{pid}-{cid}.json"; pf.parent.mkdir(parents=True,exist_ok=True)
|
||||
old=None
|
||||
try: old=json.loads(pf.read_text())
|
||||
except Exception: pass
|
||||
if action=="stop":
|
||||
if self.args.dry_run: self.runner.run(["kill","-TERM",str((old or {}).get("pid","unknown"))],pid,action); return
|
||||
if not old or not isinstance(old.get("pid"),int) or proc_start(old["pid"]) is None or str(proc_start(old["pid"]))!=str(old.get("start")):
|
||||
if health(c.get("health",{}),self.runner,pid)["ok"]: raise RuntimeError(f"refusing to stop {c.get('label')}: external process has no verified pid")
|
||||
return
|
||||
self.runner.signal(old["pid"],signal.SIGTERM,pid,action)
|
||||
deadline=time.monotonic()+min(60,int(c.get("stop_timeout_s",20)))
|
||||
while proc_start(old["pid"]) is not None and time.monotonic()<deadline: time.sleep(.2)
|
||||
if proc_start(old["pid"]) is not None:
|
||||
self.runner.signal(old["pid"],signal.SIGKILL,pid,action)
|
||||
try: pf.unlink()
|
||||
except OSError: pass
|
||||
return
|
||||
argv=[]
|
||||
for arg in c.get("argv",[]):
|
||||
v=PLACEHOLDER.sub(lambda m:str(vars.get(m.group(1),"")),str(arg))
|
||||
if v: argv.append(v)
|
||||
if self.args.dry_run:
|
||||
self.runner.run(argv,pid,action,cwd=c.get("cwd")); return
|
||||
# Start processes through the runner-owned subprocess API, then detach into their logfile.
|
||||
log=Path(self.args.state_dir)/"logs"/f"{pid}.log"; log.parent.mkdir(parents=True,exist_ok=True)
|
||||
env=os.environ.copy(); env.update({str(k):str(v) for k,v in c.get("env",{}).items()})
|
||||
proc=self.runner.spawn(argv,cwd=c.get("cwd"),env=env,stdout=open(log,"ab"),stderr=__import__("subprocess").STDOUT,pipeline=pid,action=action)
|
||||
time.sleep(.03); start=proc_start(proc.pid)
|
||||
if start is None: raise RuntimeError(f"process exited during launch; see {log}")
|
||||
pf.write_text(json.dumps({"pid":proc.pid,"start":start}),encoding="utf-8")
|
||||
def reclaim(self,j):
|
||||
rec=self.registry.get("card",{}).get("reclaim")
|
||||
if not rec: self.step(j,"no card.reclaim recipe is configured",False); return
|
||||
if rec.get("kind")=="http":
|
||||
method,url=rec.get("method","POST"),rec.get("url","")
|
||||
if self.args.dry_run: self.runner.run(["HTTP",method,url],None,"reclaim"); return
|
||||
started=time.monotonic()
|
||||
try:
|
||||
req=urllib.request.Request(url,data=rec.get("body","").encode() if rec.get("body") else None,method=method,headers={"Content-Type":"application/json"})
|
||||
with urllib.request.urlopen(req,timeout=10) as res:
|
||||
self.runner.note(["HTTP",method,url],"reclaim",rc=res.status,duration_ms=(time.monotonic()-started)*1000)
|
||||
self.step(j,f"reclaim endpoint returned HTTP {res.status}")
|
||||
except Exception as exc:
|
||||
self.runner.note(["HTTP",method,url],"reclaim",rc=127,duration_ms=(time.monotonic()-started)*1000)
|
||||
self.step(j,f"reclaim endpoint failed: {friendly_error(exc,10)}",False)
|
||||
elif rec.get("kind")=="command":
|
||||
rc,out,_=self.runner.run(rec.get("argv",[]),None,"reclaim",timeout=30)
|
||||
self.step(j,f"reclaim command exit {rc}: {out.strip()}",rc==0)
|
||||
else: self.step(j,f"unknown reclaim kind {rec.get('kind')}",False)
|
||||
if rec.get("wait_below_mib") is not None and not self.args.dry_run:
|
||||
deadline=time.monotonic()+min(120,int(rec.get("wait_timeout_s",45))); observed=None
|
||||
while time.monotonic()<deadline:
|
||||
g=gpu(self.runner); observed=g.get("used_mib") if g.get("available") else None
|
||||
if observed is not None and observed<int(rec["wait_below_mib"]):
|
||||
self.step(j,f"used VRAM is {observed} MiB; below reclaim threshold {rec['wait_below_mib']} MiB"); return
|
||||
if observed is None: break
|
||||
time.sleep(min(max(.1,float(self.registry.get("card",{}).get("poll_seconds",2))),max(0,deadline-time.monotonic())))
|
||||
self.step(j,f"reclaim VRAM wait timed out/unavailable (observed {observed} MiB)",False)
|
||||
def logs(self,pid,n):
|
||||
p=self.pipelines.get(pid,{})
|
||||
units=[c.get("unit") for c in p.get("component",[]) if c.get("kind")=="unit" and c.get("unit")]
|
||||
if units:
|
||||
result=[]
|
||||
for unit in units:
|
||||
rc,out,_=self.runner.run(["journalctl","-u",unit,"-n",str(max(1,min(1000,n))),"--no-pager"],pid,"logs",timeout=8,readonly=True)
|
||||
if rc: result.append(f"{unit}: journal unavailable: {out.strip()}")
|
||||
else: result.extend(out.splitlines())
|
||||
return result[-n:]
|
||||
path=Path(self.args.state_dir)/"logs"/f"{pid}.log"
|
||||
try: return path.read_text(errors="replace").splitlines()[-n:]
|
||||
except OSError: return [f"log file does not exist: {path}"]
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
server_version="gpu-cockpit"
|
||||
def log_message(self,*a): pass
|
||||
@property
|
||||
def app(self): return self.server.app
|
||||
def sendj(self,obj,status=200):
|
||||
b=json.dumps(obj).encode(); self.send_response(status); self.send_header("Content-Type","application/json"); self.send_header("Content-Length",str(len(b))); self.end_headers(); self.wfile.write(b)
|
||||
def do_GET(self):
|
||||
u=urlparse(self.path); path=u.path
|
||||
if not token_matches(self.app.auth_token,self.headers,parse_qs(u.query),self.client_address[0]): return self.sendj({"error":"unauthorized"},401)
|
||||
if path=="/": return self.file(Path(__file__).parent/"gui"/"index.html","text/html; charset=utf-8")
|
||||
if path.startswith("/static/"):
|
||||
name=unquote(path[len("/static/"):]); ext=Path(name).suffix.lower()
|
||||
if "/" in name or ".." in name or ext not in (".png",".svg",".css",".js",".ico"): return self.send_error(404)
|
||||
return self.file(Path(__file__).parent/"gui"/"static"/name, {".png":"image/png",".svg":"image/svg+xml",".css":"text/css",".js":"text/javascript",".ico":"image/x-icon"}[ext])
|
||||
if path=="/api/ping": return self.sendj({"ok":True,"host":socket.gethostname(),"version":__version__,"now":iso()})
|
||||
if path=="/api/state": return self.sendj(self.app.snapshot())
|
||||
if path=="/api/registry": return self.sendj({"registry":self.app.registry,"errors":self.app.errors,"warnings":self.app.warnings})
|
||||
if path=="/api/audit": return self.sendj(self.app.runner.audit(int(parse_qs(u.query).get("n",[100])[0])))
|
||||
if path.startswith("/api/logs/"):
|
||||
pid=path.rsplit("/",1)[-1]
|
||||
if pid not in self.app.pipelines: return self.sendj({"error":"unknown pipeline"},404)
|
||||
return self.sendj(self.app.logs(pid,int(parse_qs(u.query).get("n",[200])[0])))
|
||||
if path.startswith("/api/jobs/"):
|
||||
jid=path.rsplit("/",1)[-1]; job=self.app.jobs.get(jid)
|
||||
return self.sendj(job or {"error":"job not found"},200 if job else 404)
|
||||
if path=="/api/events":
|
||||
q=parse_qs(u.query); once=q.get("once")==["1"]; data=json.dumps(self.app.snapshot())
|
||||
self.send_response(200); self.send_header("Content-Type","text/event-stream"); self.send_header("Cache-Control","no-cache"); self.end_headers()
|
||||
self.wfile.write(f"event: state\ndata: {data}\n\n".encode()); self.wfile.flush()
|
||||
if not once:
|
||||
try:
|
||||
last=data; heartbeat=time.monotonic()
|
||||
while True:
|
||||
time.sleep(1); current=json.dumps(self.app.snapshot())
|
||||
old_state=json.loads(last); new_state=json.loads(current)
|
||||
old_state.pop("now",None); new_state.pop("now",None)
|
||||
if old_state != new_state:
|
||||
self.wfile.write(f"event: state\ndata: {current}\n\n".encode()); self.wfile.flush(); last=current
|
||||
elif time.monotonic()-heartbeat>=15:
|
||||
self.wfile.write(b": heartbeat\n\n"); self.wfile.flush(); heartbeat=time.monotonic()
|
||||
except (BrokenPipeError,ConnectionResetError): pass
|
||||
return
|
||||
self.send_error(404)
|
||||
def file(self,path,mime):
|
||||
try: b=path.read_bytes()
|
||||
except OSError: return self.send_error(404)
|
||||
self.send_response(200); self.send_header("Content-Type",mime); self.send_header("Content-Length",str(len(b))); self.end_headers(); self.wfile.write(b)
|
||||
def do_POST(self):
|
||||
u=urlparse(self.path); path=u.path
|
||||
if not token_matches(self.app.auth_token,self.headers,parse_qs(u.query),self.client_address[0]): return self.sendj({"error":"unauthorized"},401)
|
||||
try: body=json.loads(self.rfile.read(int(self.headers.get("Content-Length",0))) or b"{}")
|
||||
except Exception: return self.sendj({"error":"invalid JSON"},400)
|
||||
try:
|
||||
if path=="/api/card/free": jid=self.app.start_job(None,"free")
|
||||
else:
|
||||
sm=re.fullmatch(r"/api/pipelines/([a-z0-9-]+)/options",path)
|
||||
if sm:
|
||||
pid=sm.group(1); p=self.app.pipelines.get(pid)
|
||||
if not p: return self.sendj({"error":"unknown pipeline"},404)
|
||||
opts=body.get("options",{})
|
||||
if not isinstance(opts,dict): return self.sendj({"error":"options must be an object"},400)
|
||||
self.app.preflight(p,"start",opts,True)
|
||||
self.app.validate_options(p,opts)
|
||||
return self.sendj({"selected":self.app.selected.get(pid,{})})
|
||||
m=re.fullmatch(r"/api/pipelines/([a-z0-9-]+)/action",path)
|
||||
if not m: return self.send_error(404)
|
||||
pid=m.group(1); p=self.app.pipelines.get(pid)
|
||||
if not p: return self.sendj({"error":"unknown pipeline"},404)
|
||||
action=body.get("action")
|
||||
if action not in ("start","stop","restart"): return self.sendj({"error":"invalid action"},400)
|
||||
options=body.get("options",{})
|
||||
if not isinstance(options,dict): return self.sendj({"error":"options must be an object"},400)
|
||||
for opt in p.get("option",[]):
|
||||
if opt.get("id") in options and options[opt["id"]] not in [x.get("id") for x in opt.get("choice",[])]: return self.sendj({"error":f"unknown option value {opt['id']}={options[opt['id']] }"},400)
|
||||
self.app.preflight(p,action,options,bool(body.get("force",False)))
|
||||
jid=self.app.start_job(pid,action,options,bool(body.get("force",False)))
|
||||
return self.sendj({"job_id":jid},202)
|
||||
except PermissionError as exc: return self.sendj({"error":str(exc)},409)
|
||||
except LookupError as exc: return self.sendj({"error":str(exc)},400)
|
||||
except ValueError as exc: return self.sendj({"error":str(exc)},409)
|
||||
except Exception as exc: return self.sendj({"error":str(exc)},500)
|
||||
|
||||
def serve_extras(app,hosts,port):
|
||||
"""Bind the front-door listeners, retrying until they exist. A WireGuard address only
|
||||
appears once wg0 is up, which can happen after this service starts - without the retry a
|
||||
boot race would close the door permanently and look like an auth problem from the phone."""
|
||||
pending=list(hosts)
|
||||
while pending:
|
||||
for h in list(pending):
|
||||
try: srv=ThreadingHTTPServer((h,port),Handler)
|
||||
except OSError as exc:
|
||||
print(f"front door {h}:{port} not available yet ({exc}) - retrying in 5s",flush=True); continue
|
||||
srv.app=app; app.listeners.append(f"{h}:{port}")
|
||||
print(f"listening on http://{h}:{port} (front door)",flush=True)
|
||||
threading.Thread(target=srv.serve_forever,daemon=True).start()
|
||||
pending.remove(h)
|
||||
if pending: time.sleep(5)
|
||||
|
||||
def main(argv=None):
|
||||
home=str(Path.home()); envreg=os.environ.get("COCKPIT_REGISTRY")
|
||||
ap=argparse.ArgumentParser(); ap.add_argument("--registry",default=envreg or os.path.join(home,".config/gpu-cockpit/registry.toml")); ap.add_argument("--host",default="127.0.0.1",help="comma-separated; anything non-loopback needs a token"); ap.add_argument("--token-file",default=os.path.join(home,".config/gpu-cockpit/token")); ap.add_argument("--port",type=int,default=8770); ap.add_argument("--state-dir",default=os.path.join(home,".local/state/gpu-cockpit")); ap.add_argument("--systemctl",default="systemctl"); ap.add_argument("--sudo",default=None); ap.add_argument("--dry-run",action="store_true"); ap.add_argument("--check",action="store_true"); args=ap.parse_args(argv)
|
||||
reg,errors,warnings=load(args.registry)
|
||||
if args.check:
|
||||
print(f"registry OK: {len(reg.get('pipeline',[]))} pipelines, {len(errors)} errors, {len(warnings)} warnings")
|
||||
for x in errors: print(f"ERROR: {x}")
|
||||
for x in warnings: print(f"WARNING: {x}")
|
||||
return 1 if errors else 0
|
||||
app=App(args)
|
||||
app.auth_token=load_token(args.token_file); app.listeners=[]
|
||||
hosts=[h.strip() for h in str(args.host).split(",") if h.strip()] or ["127.0.0.1"]
|
||||
extras=[h for h in hosts if not is_loopback(h)]
|
||||
if extras and not app.auth_token:
|
||||
print(f"refusing to listen on {', '.join(extras)} with no token ({args.token_file}) - that is an open front door",file=sys.stderr); return 1
|
||||
try: server=ThreadingHTTPServer((hosts[0],args.port),Handler)
|
||||
except OSError as exc: print(f"cannot listen on http://{hosts[0]}:{args.port}: {exc}",file=sys.stderr); return 1
|
||||
server.app=app; app.listeners.append(f"{hosts[0]}:{args.port}")
|
||||
print(f"listening on http://{hosts[0]}:{args.port}",flush=True)
|
||||
if extras: threading.Thread(target=serve_extras,args=(app,extras,args.port),daemon=True).start()
|
||||
try: server.serve_forever()
|
||||
except KeyboardInterrupt: pass
|
||||
finally: server.server_close()
|
||||
return 0
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=GPU Cockpit single-card pipeline console
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/python3 -m cockpit --registry %h/.config/gpu-cockpit/registry.toml --state-dir %h/.local/state/gpu-cockpit --host 127.0.0.1 --port 8770
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,119 @@
|
||||
[card]
|
||||
label = "RTX 4060"
|
||||
total_mib = 8188
|
||||
idle_floor_mib = 1700
|
||||
poll_seconds = 2
|
||||
|
||||
[card.reclaim]
|
||||
label = "ask the image server to unload models"
|
||||
kind = "http"
|
||||
url = "http://127.0.0.1:8188/free"
|
||||
method = "POST"
|
||||
body = '{"unload_models": true, "free_memory": true}'
|
||||
wait_below_mib = 2000
|
||||
wait_timeout_s = 45
|
||||
|
||||
[[pipeline]]
|
||||
id = "image"
|
||||
label = "Image generation"
|
||||
description = "An image generation service on the shared card"
|
||||
requires_card = true
|
||||
vram_mib = 7019
|
||||
protected = false
|
||||
start_below_mib = 2600
|
||||
serving = "Z-Image-Turbo Q4_K_M + LoRA ladder"
|
||||
open_url = "http://127.0.0.1:8188"
|
||||
note = "Peaks near 7 GB; leave the card alone while it renders."
|
||||
|
||||
[[pipeline.component]]
|
||||
kind = "proc"
|
||||
id = "image-server"
|
||||
label = "Image server"
|
||||
argv = ["/usr/bin/python3", "-m", "http.server", "8188", "--bind", "127.0.0.1", "{extra}"]
|
||||
cwd = "/tmp"
|
||||
env = { PYTHONUNBUFFERED = "1" }
|
||||
ready_timeout_s = 15
|
||||
stop_timeout_s = 5
|
||||
health = { kind = "http", url = "http://127.0.0.1:8188/", timeout_s = 2 }
|
||||
|
||||
[[pipeline.option]]
|
||||
id = "model"
|
||||
label = "Model"
|
||||
kind = "argv"
|
||||
default = "turbo"
|
||||
[[pipeline.option.choice]]
|
||||
id = "turbo"
|
||||
label = "Z-Image Turbo"
|
||||
note = "Fast image generation"
|
||||
verified = "2026-09-26"
|
||||
vars = { model = "/models/z-image.gguf", alias = "image-turbo", extra = "" }
|
||||
[[pipeline.option.choice]]
|
||||
id = "quality"
|
||||
label = "Image quality model"
|
||||
vars = { model = "/models/image-quality.gguf", alias = "image-quality", extra = "--directory ." }
|
||||
|
||||
[[pipeline.option]]
|
||||
id = "profile"
|
||||
label = "Profile note"
|
||||
kind = "note"
|
||||
default = "desktop"
|
||||
[[pipeline.option.choice]]
|
||||
id = "desktop"
|
||||
label = "Desktop preset"
|
||||
note = "Leaves memory available for the desktop."
|
||||
[[pipeline.option.choice]]
|
||||
id = "render"
|
||||
label = "Render preset"
|
||||
note = "Use for dedicated render sessions."
|
||||
|
||||
[[pipeline]]
|
||||
id = "voice"
|
||||
label = "Voice stack"
|
||||
description = "A systemd managed voice pipeline"
|
||||
requires_card = true
|
||||
vram_mib = 3200
|
||||
protected = false
|
||||
serving = "Speech and verbalization"
|
||||
|
||||
[[pipeline.component]]
|
||||
kind = "unit"
|
||||
id = "voice-server"
|
||||
label = "Voice server"
|
||||
unit = "gpu-cockpit-example-voice.service"
|
||||
ready_timeout_s = 20
|
||||
health = { kind = "command", argv = ["/usr/bin/test", "-e", "/dev/null"], timeout_s = 2 }
|
||||
|
||||
[[pipeline]]
|
||||
id = "trainer"
|
||||
label = "Trainer"
|
||||
description = "Explicitly protected card workload"
|
||||
requires_card = true
|
||||
protected = true
|
||||
vram_mib = 7000
|
||||
serving = "Training job"
|
||||
|
||||
[[pipeline.component]]
|
||||
kind = "proc"
|
||||
id = "trainer-proc"
|
||||
label = "Training process"
|
||||
argv = ["/bin/sleep", "300"]
|
||||
ready_timeout_s = 2
|
||||
stop_timeout_s = 2
|
||||
health = { kind = "none" }
|
||||
|
||||
[[pipeline]]
|
||||
id = "cpu-llm"
|
||||
label = "CPU model"
|
||||
description = "Runs without claiming the GPU"
|
||||
requires_card = false
|
||||
vram_mib = 0
|
||||
serving = "CPU inference"
|
||||
|
||||
[[pipeline.component]]
|
||||
kind = "proc"
|
||||
id = "cpu"
|
||||
label = "CPU service"
|
||||
argv = ["/bin/sleep", "300"]
|
||||
ready_timeout_s = 2
|
||||
stop_timeout_s = 2
|
||||
health = { kind = "none" }
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
root="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
exec python3 "$root/tests/selftest.py" "$root"
|
||||
@@ -0,0 +1,119 @@
|
||||
import json, os, pathlib, socket, subprocess, sys, tempfile, threading, time, urllib.error, urllib.request
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
root=pathlib.Path(sys.argv[1]); tmp=pathlib.Path(tempfile.mkdtemp(prefix='cockpit-test-')); cases=0; proc=None
|
||||
|
||||
def check(name, test):
|
||||
global cases
|
||||
if not test: print(f'FAIL: {name}: assertion failed',flush=True); raise SystemExit(1)
|
||||
cases+=1; print(f'PASS: {name}',flush=True)
|
||||
class Reclaim(BaseHTTPRequestHandler):
|
||||
hits=0
|
||||
def do_POST(self): Reclaim.hits+=1; self.send_response(200); self.end_headers(); self.wfile.write(b'ok')
|
||||
def log_message(self,*a): pass
|
||||
reclaim=HTTPServer(('127.0.0.1',0),Reclaim); threading.Thread(target=reclaim.serve_forever,daemon=True).start()
|
||||
stub=tmp/'systemctl'; stub.write_text('''#!/usr/bin/env sh
|
||||
s="$COCKPIT_STUB_STATE"; c="$1"; u="$2"
|
||||
case "$c" in
|
||||
show) if grep -qx "$u" "$s" 2>/dev/null; then echo ActiveState=active; echo SubState=running; echo ExecMainPID=1; else echo ActiveState=inactive; echo SubState=dead; echo ExecMainPID=0; fi;;
|
||||
start|restart) grep -vx "$u" "$s" 2>/dev/null > "$s.tmp" || true; echo "$u" >> "$s.tmp"; mv "$s.tmp" "$s";;
|
||||
stop) grep -vx "$u" "$s" 2>/dev/null > "$s.tmp" || true; mv "$s.tmp" "$s";;
|
||||
is-active) grep -qx "$u" "$s" && exit 0 || exit 3;; *) exit 0;; esac
|
||||
'''); stub.chmod(0o755)
|
||||
port=socket.socket(); port.bind(('127.0.0.1',0)); pnum=port.getsockname()[1]; port.close()
|
||||
reg=tmp/'registry.toml'
|
||||
reg.write_text(f'''[card]\ntotal_mib=8188\nidle_floor_mib=1700\npoll_seconds=0.05\n[card.reclaim]\nkind="http"\nurl="http://127.0.0.1:{reclaim.server_port}/free"\nmethod="POST"\nwait_timeout_s=1\n[[pipeline]]\nid="a"\nlabel="A"\nrequires_card=true\nstart_below_mib=2600\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="A process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline]]\nid="b"\nlabel="B"\nrequires_card=true\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="B process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline]]\nid="protected"\nlabel="Protected"\nrequires_card=true\nprotected=true\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Protected process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline]]\nid="bad-health"\nlabel="Unreachable"\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Bad process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=1\nstop_timeout_s=1\nhealth={{kind="tcp",host="127.0.0.1",port=1,timeout_s=0.1}}\n[[pipeline]]\nid="options"\nlabel="Options"\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Option process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)","{{extra}}"]\nready_timeout_s=2\nstop_timeout_s=1\nhealth={{kind="none"}}\n[[pipeline.option]]\nid="model"\nlabel="Model"\nkind="argv"\ndefault="one"\n[[pipeline.option.choice]]\nid="one"\nlabel="One"\nvars={{extra=""}}\n[[pipeline.option.choice]]\nid="two"\nlabel="Two"\nvars={{extra="-v"}}\n[[pipeline.option]]\nid="required"\nlabel="Required var"\nkind="argv"\ndefault="present"\n[[pipeline.option.choice]]\nid="present"\nlabel="Present"\nvars={{required="yes"}}\n[[pipeline.option.choice]]\nid="absent"\nlabel="Missing required value"\nvars={{}}\n[[pipeline]]\nid="broken"\nlabel="Broken"\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="Broken"\nargv=["{sys.executable}","{{missing}}"]\nhealth={{kind="none"}}\n[[pipeline]]\nid="external"\nlabel="External"\nrequires_card=true\n[[pipeline.component]]\nid="p"\nkind="proc"\nlabel="External process"\nargv=["{sys.executable}","-c","import time;time.sleep(60)"]\nready_timeout_s=2\nhealth={{kind="tcp",host="127.0.0.1",port={reclaim.server_port},timeout_s=1}}\n''')
|
||||
state=tmp/'state'; env=os.environ.copy(); env['COCKPIT_STUB_STATE']=str(tmp/'units'); log=open(tmp/'server.log','w+')
|
||||
proc=subprocess.Popen([sys.executable,'-m','cockpit','--registry',str(reg),'--port',str(pnum),'--state-dir',str(state),'--systemctl',str(stub)],cwd=root,stdout=log,stderr=log,env=env)
|
||||
base=f'http://127.0.0.1:{pnum}'
|
||||
def get(path):
|
||||
with urllib.request.urlopen(base+path,timeout=5) as r:return r.status,r.read().decode(),r.headers
|
||||
def post(path,obj):
|
||||
req=urllib.request.Request(base+path,data=json.dumps(obj).encode(),headers={'Content-Type':'application/json'},method='POST')
|
||||
try:
|
||||
with urllib.request.urlopen(req,timeout=5) as r:return r.status,json.loads(r.read())
|
||||
except urllib.error.HTTPError as e:return e.code,json.loads(e.read())
|
||||
def snapshot():return json.loads(get('/api/state')[1])
|
||||
def job(jid):
|
||||
end=time.time()+5
|
||||
while time.time()<end:
|
||||
j=json.loads(get('/api/jobs/'+jid)[1])
|
||||
if j['state']!='running':return j
|
||||
time.sleep(.05)
|
||||
raise AssertionError('job timed out')
|
||||
def action(pid,act='start',**kw):
|
||||
code,r=post(f'/api/pipelines/{pid}/action',{'action':act,**kw})
|
||||
return job(r['job_id']) if code==202 else (code,r)
|
||||
try:
|
||||
deadline=time.time()+5
|
||||
while time.time()<deadline:
|
||||
try: status,raw,_=get('/api/state'); break
|
||||
except Exception: time.sleep(.05)
|
||||
check('server starts, listens, and state shape',status==200 and all(k in json.loads(raw) for k in ('host','card','pipelines','registry')))
|
||||
check('fixture pipeline count',len(json.loads(raw)['pipelines'])==7)
|
||||
check('down state for not-started proc',next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
|
||||
check('unreachable health state is down',next(x for x in snapshot()['pipelines'] if x['id']=='bad-health')['state']=='down')
|
||||
j=action('a'); steps=' '.join(x['text'] for x in j['steps'])
|
||||
vram_verdict=('GPU probe unavailable' in steps) or ('VRAM wait' in steps) or ('below 2600 MiB' in steps)
|
||||
check('start verifies health/PID with an honest VRAM verdict',j['state']=='done' and vram_verdict and 'healthy' in steps and next(x for x in snapshot()['pipelines'] if x['id']=='a')['components'][0]['detail'].startswith('pid '))
|
||||
j=action('a','stop'); check('stop verifies pipeline down',j['state']=='done' and next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
|
||||
action('a'); j=action('b'); check('arbitration stops A before B',j['state']=='done' and any('stopping A' in x['text'] for x in j['steps']) and next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
|
||||
action('protected'); status,body=post('/api/pipelines/b/action',{'action':'start'}); check('protected switch returns 409 without change',status==409 and 'protected' in body['error'] and next(x for x in snapshot()['pipelines'] if x['id']=='protected')['state']=='up')
|
||||
status,body=post('/api/pipelines/protected/action',{'action':'stop'}); check('protected stop requires force',status==409 and 'protected' in body['error'] and next(x for x in snapshot()['pipelines'] if x['id']=='protected')['state']=='up')
|
||||
j=action('protected','stop',force=True); check('force stop of a protected pipeline works',j['state']=='done' and next(x for x in snapshot()['pipelines'] if x['id']=='protected')['state']=='down')
|
||||
j=action('b','start',force=True); check('force allows protected switch',j['state']=='done')
|
||||
check('reclaim HTTP recipe called',Reclaim.hits>=2)
|
||||
action('b')
|
||||
ext=next(x for x in snapshot()['pipelines'] if x['id']=='external')
|
||||
check('unowned but answering component reads up/external',ext['state']=='up' and ext['components'][0]['detail']=='external')
|
||||
j=action('external','stop'); check('explicit stop of an unowned process fails loudly',j['state']=='failed' and 'not started by the cockpit' in j['error'])
|
||||
j=action('a','start'); check('a switch leaves an unowned holder running and still proceeds',j['state']=='done' and any('leaving it running' in x['text'] for x in j['steps']))
|
||||
status,_=post('/api/pipelines/options/action',{'action':'start','options':{'model':'invalid'}}); check('unknown option value returns 400',status==400)
|
||||
status,_=post('/api/pipelines/options/action',{'action':'start','options':{'required':'absent'}}); check('choice missing a declared var returns 400',status==400)
|
||||
status,body=post('/api/pipelines/options/options',{'options':{'model':'two'}}); check('option selection persists',status==200 and next(x for x in snapshot()['pipelines'] if x['id']=='options')['selected'].get('model')=='two')
|
||||
j=action('bad-health'); check('ready timeout fails with log tail',j['state']=='failed' and 'last log lines' in j['error'])
|
||||
pf=state/'pids'/'a-p.json'; pf.parent.mkdir(parents=True,exist_ok=True); pf.write_text('not json'); check('garbage pidfile reads down',next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
|
||||
pf.write_text(json.dumps({'pid':os.getpid(),'start':'wrong'})); check('reused pid mismatch reads down',next(x for x in snapshot()['pipelines'] if x['id']=='a')['state']=='down')
|
||||
status,raw,_=get('/api/events?once=1'); check('SSE once emits state event',status==200 and 'event: state' in raw)
|
||||
status,html,_=get('/'); check('GUI markup served',status==200 and 'Free the card' in html)
|
||||
status,raw,_=get('/api/registry'); check('broken registry validation reported',status==200 and bool(json.loads(raw)['errors']))
|
||||
# Verify dry-run records intent without creating proc children.
|
||||
dryport=socket.socket(); dryport.bind(('127.0.0.1',0)); dp=dryport.getsockname()[1]; dryport.close()
|
||||
dry=subprocess.Popen([sys.executable,'-m','cockpit','--registry',str(reg),'--port',str(dp),'--state-dir',str(tmp/'dry'),'--dry-run'],cwd=root,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True,env=env)
|
||||
try:
|
||||
line=dry.stdout.readline(); check('dry-run server starts',line.startswith('listening on'))
|
||||
req=urllib.request.Request(f'http://127.0.0.1:{dp}/api/pipelines/a/action',data=b'{"action":"start"}',headers={'Content-Type':'application/json'},method='POST')
|
||||
jid=json.loads(urllib.request.urlopen(req).read())['job_id']; time.sleep(.2)
|
||||
req=urllib.request.Request(f'http://127.0.0.1:{dp}/api/pipelines/options/action',data=b'{"action":"start","options":{"model":"two"}}',headers={'Content-Type':'application/json'},method='POST')
|
||||
urllib.request.urlopen(req).read(); time.sleep(.2); rows=[json.loads(x) for x in (tmp/'dry'/'audit.jsonl').read_text().splitlines()]
|
||||
check('dry-run audits and launches no process',any(x['dry_run'] for x in rows) and not (tmp/'dry'/'pids'/'a-p.json').exists())
|
||||
check('selected option changes recorded argv',any(x['pipeline']=='options' and '-v' in x['argv'] for x in rows))
|
||||
finally: dry.terminate(); dry.wait(timeout=3)
|
||||
# --- the front door: the tunnel listener is token-gated, and cannot be opened without one ---
|
||||
if str(root) not in sys.path: sys.path.insert(0,str(root))
|
||||
from cockpit import server as srvmod
|
||||
check('loopback needs no token (the desktop GUI and the TUI live there)',srvmod.token_matches('sekret',{},{},'127.0.0.1'))
|
||||
check('ipv6 loopback counts as loopback',srvmod.token_matches('sekret',{},{},'::1'))
|
||||
check('a tunnel caller with no token is refused',not srvmod.token_matches('sekret',{},{},'192.168.2.1'))
|
||||
check('a tunnel caller with the right bearer token gets in',srvmod.token_matches('sekret',{'Authorization':'Bearer sekret'},{},'192.168.2.1'))
|
||||
check('a wrong bearer token is refused',not srvmod.token_matches('sekret',{'Authorization':'Bearer nope'},{},'192.168.2.1'))
|
||||
check('a query token serves clients that cannot set headers',srvmod.token_matches('sekret',{}, {'token':['sekret']},'192.168.2.1'))
|
||||
(tmp/'tok').write_text('sekret\n')
|
||||
check('a token file is read back trimmed',srvmod.load_token(str(tmp/'tok'))=='sekret')
|
||||
check('a missing token file means no token, not an empty one',srvmod.load_token(str(tmp/'absent')) is None)
|
||||
spare=socket.socket(); spare.bind(('127.0.0.1',0)); sport=spare.getsockname()[1]; spare.close()
|
||||
guard=subprocess.run([sys.executable,'-m','cockpit','--registry',str(reg),'--port',str(sport),'--state-dir',str(tmp/'guard'),
|
||||
'--host','127.0.0.1,10.10.10.1','--token-file',str(tmp/'absent'),'--dry-run'],
|
||||
cwd=root,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True,env=env,timeout=30)
|
||||
check('a front door with no token is refused outright, not silently left open',guard.returncode==1 and 'open front door' in guard.stdout)
|
||||
print(f'ALL TESTS PASSED ({cases} cases)',flush=True)
|
||||
finally:
|
||||
if proc:
|
||||
proc.terminate()
|
||||
try:proc.wait(timeout=3)
|
||||
except subprocess.TimeoutExpired:proc.kill();proc.wait()
|
||||
for f in (state/'pids').glob('*.json') if (state/'pids').exists() else []:
|
||||
try:
|
||||
pid=json.loads(f.read_text())['pid']
|
||||
if pid != os.getpid(): os.kill(pid,15)
|
||||
except Exception:pass
|
||||
reclaim.shutdown(); log.close()
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Textual HTTP client for gpu-cockpit; --print needs only the standard library."""
|
||||
import argparse, json, sys, time, urllib.error, urllib.request
|
||||
|
||||
def get(url):
|
||||
with urllib.request.urlopen(url,timeout=3) as r: return json.loads(r.read())
|
||||
|
||||
def plain(url):
|
||||
try:
|
||||
s=get(url.rstrip('/')+'/api/state')
|
||||
except Exception as exc:
|
||||
print(f"Cannot reach gpu-cockpit at {url}: {exc}. Start it with: python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml")
|
||||
return 2
|
||||
c=s['card'];
|
||||
if c['available']: print(f"{s['host']} · GPU {c['used_mib']}/{c['usable_mib']} MiB usable · {c['util_pct']}% · {c['temp_c']}°C")
|
||||
else: print(f"{s['host']} · no GPU probe ({c.get('detail','unavailable')})")
|
||||
for p in s['pipelines']: print(f"{p['state']:9} {p['id']:<16} {p['label']} · {p.get('vram_mib') or 0} MiB")
|
||||
return 0
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser(); ap.add_argument('--url',default='http://127.0.0.1:8770'); ap.add_argument('--refresh',type=float,default=2); ap.add_argument('--print',dest='plain',action='store_true'); a=ap.parse_args()
|
||||
if a.plain: return plain(a.url)
|
||||
try:
|
||||
from textual.app import App, ComposeResult
|
||||
from textual.containers import Horizontal, Vertical
|
||||
from textual.widgets import Button, DataTable, Footer, Header, Select, Static
|
||||
except ImportError:
|
||||
print('Textual is required for the interactive TUI. Install it with: pip install textual',file=sys.stderr); return 2
|
||||
class Cockpit(App):
|
||||
TITLE='GPU Cockpit'; BINDINGS=[('q','quit','Quit'),('r','reload','Refresh'),('s','start','Start'),('x','stop','Stop'),('R','restart','Restart'),('f','free','Free card'),('l','logs','Logs'),('m','choose','Choose option')]
|
||||
CSS='Screen{layout:vertical} #content{height:1fr} #pipes{width:40%} #detail{width:60%;border:round $primary;padding:1} #step{height:3;color:$accent}'
|
||||
def compose(self)->ComposeResult:
|
||||
yield Header(); yield Static('Connecting…',id='step')
|
||||
with Horizontal(id='content'):
|
||||
yield DataTable(id='pipes'); yield Static('Select a pipeline',id='detail')
|
||||
yield Footer()
|
||||
def on_mount(self):
|
||||
self.query_one('#pipes',DataTable).add_columns('State','ID','Pipeline','VRAM MiB'); self.set_interval(a.refresh,self.reload_state)
|
||||
def reload_state(self):
|
||||
try: self.snap=get(a.url.rstrip('/')+'/api/state'); table=self.query_one('#pipes',DataTable); table.clear()
|
||||
except Exception as exc:
|
||||
self.snap=None; self.query_one('#step',Static).update(f"Server unavailable at {a.url}. Start: python3 -m cockpit --registry ~/.config/gpu-cockpit/registry.toml ({exc})"); return
|
||||
c=self.snap['card']; head=f"{self.snap['host']} · "+(f"GPU {c['used_mib']}/{c['usable_mib']} MiB · {c['util_pct']}% · {c['temp_c']}°C" if c['available'] else 'no GPU probe')
|
||||
if self.snap.get('job'): head+=f" · {self.snap['job']['step']}"
|
||||
self.query_one('#step',Static).update(head)
|
||||
for p in self.snap['pipelines']: table.add_row({'up':'●','down':'○','partial':'◐','starting':'◒','stopping':'◓'}.get(p['state'],'?'),p['id'],p['label'],str(p.get('vram_mib') or 0),key=p['id'])
|
||||
def on_data_table_row_highlighted(self,event):
|
||||
if not self.snap:return
|
||||
p=next((x for x in self.snap['pipelines'] if x['id']==str(event.row_key.value)),None)
|
||||
if p:
|
||||
try: logs='\n'.join(get(a.url.rstrip('/')+f"/api/logs/{p['id']}?n=20"))
|
||||
except Exception as exc: logs=f"Logs unavailable: {exc}"
|
||||
opts='\n'.join(f"{o['label']}: {o['selected']}" for o in p['options']) or 'No options'
|
||||
comps='\n'.join(f"{c['label']}: {c['state']} · {c.get('health',{}).get('detail') if c.get('health') else c['detail']}" for c in p['components'])
|
||||
detail=f"{p['label']} · {p['state']}\n{p['description']}\nServing: {p.get('serving') or '—'}\nVRAM: {p.get('vram_mib') or 0} MiB\n\nComponents\n{comps}\n\nOptions\n{opts}\n\nLast 20 log lines\n{logs}"
|
||||
if p.get('job'): detail+=f"\n\nJob {p['job']['id']}: {p['job']['step']}"
|
||||
self.query_one('#detail',Static).update(detail)
|
||||
def selected(self):
|
||||
if not self.snap:return None
|
||||
key=self.query_one('#pipes',DataTable).cursor_row
|
||||
try: pid=self.query_one('#pipes',DataTable).get_row_at(key)[1]
|
||||
except Exception:return None
|
||||
return next((x for x in self.snap['pipelines'] if x['id']==pid),None)
|
||||
def post(self,path,data):
|
||||
req=urllib.request.Request(a.url.rstrip('/')+path,data=json.dumps(data).encode(),headers={'Content-Type':'application/json'},method='POST')
|
||||
try: urllib.request.urlopen(req,timeout=3).read()
|
||||
except Exception as exc:self.query_one('#step',Static).update(str(exc))
|
||||
self.set_timer(.2,self.reload_state)
|
||||
def action(self,name):
|
||||
p=self.selected()
|
||||
if p:self.post(f"/api/pipelines/{p['id']}/action",{'action':name,'options':p['selected']})
|
||||
def action_reload(self):self.reload_state()
|
||||
def action_start(self):self.action('start')
|
||||
def action_stop(self):self.action('stop')
|
||||
def action_restart(self):self.action('restart')
|
||||
def action_free(self):self.post('/api/card/free',{})
|
||||
def action_logs(self):
|
||||
p=self.selected()
|
||||
if p:
|
||||
try:self.push_screen(LogsScreen('\n'.join(get(a.url.rstrip('/')+f"/api/logs/{p['id']}?n=20"))))
|
||||
except Exception as exc:self.query_one('#step',Static).update(str(exc))
|
||||
def action_choose(self):
|
||||
p=self.selected()
|
||||
if p:self.push_screen(OptionScreen(p))
|
||||
from textual.screen import ModalScreen
|
||||
class LogsScreen(ModalScreen):
|
||||
BINDINGS=[('escape','dismiss','Close')]
|
||||
def __init__(self,text):super().__init__();self.text=text
|
||||
def compose(self):yield Static(self.text)
|
||||
class OptionScreen(ModalScreen):
|
||||
BINDINGS=[('escape','dismiss','Close')]
|
||||
CSS='OptionScreen{align: center middle} #box{width:70%;height:auto;max-height:80%;border:round $primary;background:$surface;padding:1 2} Select{margin:1 0} Button{margin-top:1}'
|
||||
def __init__(self,pipeline):super().__init__();self.pipeline=pipeline;self.values=dict(pipeline['selected'])
|
||||
def compose(self):
|
||||
with Vertical(id='box'):
|
||||
yield Static(f"Choose options · {self.pipeline['label']}")
|
||||
for option in self.pipeline['options']:
|
||||
values=[(c['label'],c['id']) for c in option['choices']]
|
||||
if values: yield Select(values,value=option['selected'],prompt=option['label'],id='option-'+option['id'])
|
||||
yield Button('Save selection',variant='primary',id='save')
|
||||
def on_select_changed(self,event):
|
||||
if event.select.id and event.select.id.startswith('option-') and event.value is not Select.BLANK:
|
||||
self.values[event.select.id[7:]]=event.value
|
||||
def on_button_pressed(self,event):
|
||||
if event.button.id=='save':
|
||||
path=f"/api/pipelines/{self.pipeline['id']}/options"
|
||||
req=urllib.request.Request(a.url.rstrip('/')+path,data=json.dumps({'options':self.values}).encode(),headers={'Content-Type':'application/json'},method='POST')
|
||||
try: urllib.request.urlopen(req,timeout=3).read(); self.dismiss(self.values)
|
||||
except Exception as exc: self.query_one(Static).update(str(exc))
|
||||
Cockpit().run(); return 0
|
||||
if __name__=='__main__': raise SystemExit(main())
|
||||
Reference in New Issue
Block a user