genai-server

NixOS flake for a local GenAI serving stack, sized for a 32GB NVIDIA GPU + 128GB RAM (built for logistikon: RTX 5090 / Ryzen 9700X).

The portal — start here

http://<host>:8897 (replace <host> with e.g. logistikon.lan) is the one address worth bookmarking. It is the stack's front door and its monitoring surface: what's wrong, what the hardware is doing, what you can open, and what models are on disk — on one page, with a shared nav across every page it hosts.

Portal page	What
/	Overview — problems strip, live GPU/CPU/RAM gauges, the service directory (grouped, with up/down + health dots), and a fleet summary
/viz	3D system view — live hardware and data-flow visualization
/studio	Studio — make and change pictures, video and sound; one form over image-server, the media tools and published ComfyUI workflows
/rag	Knowledge — document collections: search, ingest by URL or paste, inspect and delete
/code	Code sandbox — run Python / JavaScript / shell in a disposable VM, with input and output files
/memory	Memory — the durable facts models carry between conversations: search, add, forget, and the reconciliation queue
/models	Model fleet — every LLM with its capabilities, context, KV quant, offload, group and TTL; filter, download, enable, remove

It also fronts the APIs, so a client needs one host:port for the whole box:

Route	What
/v1/*	OpenAI API → llama-swap, filtered to ready-and-enabled models
/svc/<name>/*	this flake's own tool servers: image, media, search, tts, segment, rag, memory, code, mcp — e.g. /svc/media/openapi.json
/api/health	one call: {ok, problems[], services, gpus, cpu, mem, models} (503 when unhealthy, so curl -f works)
/api/portal	the portal surface — hosted pages + grouped service directory
/api/activity	one line: what the box is working on right now ({activity, model, kind, detail})
/metrics	Prometheus: GPU/CPU/RAM, per-service up + health, per-model ready/enabled/size

Hosts extend it through portal.extraPages, portal.extraServices and portal.proxy.extraUpstreams — see Usage. Anything this flake adds in future registers here rather than becoming another bookmark (ROADMAP.md Stage 2a, "one front door").

Web UIs

The third-party interfaces the portal links to (all reachable directly, and all probed on the portal's Overview page):

URL	What
http://<host>:3000	Open WebUI — main chat: LLMs, vision, web search, mic/read-aloud, image generation, agentic media tools
http://<host>:8188	ComfyUI — node-graph image / video / music generation (FLUX.2, Wan 2.2, LTX 2.3, ACE-Step)
http://<host>:8895	MagenticLite — human-in-the-loop web agent (Fara drives a QEMU-sandboxed browser; approve/steer/take over at any point)
http://<host>:8888	SearXNG — private metasearch

Minor built-in pages that ship with their tools: :8080 llama-swap model manager (its Transcription tab uploads audio to the asr model), :4000 LiteLLM admin. The remaining ports (:8892/8893/8894/8896, and the search/tool servers) are JSON APIs with no browser UI — reach them through /svc/<name>/ rather than by port.

Services (all ports)

Port	Service	What
8080	llama-swap	OpenAI-compatible LLM API + web UI; hot-swaps models on demand
8897	portal	the front door: pages (/, /viz, /studio, /models, /rag, /memory, /code, /evals), service directory + health (/api/health), model admin, and the API proxies (/v1/* → llama-swap, /svc/<name>/* → the tool servers below)
3000	Open WebUI	chat frontend wired to llama-swap
8188	ComfyUI	image/video/music generation (systemd service, genai user; version pinned by comfyui.rev; the bulk model sets come from comfyui.modelSets)
8888	SearXNG	local metasearch; JSON API via /search?q=...&format=json
8891	search tool	OpenAPI web-search tool server (SearXNG backend)
4000	LiteLLM	Anthropic-protocol bridge (/v1/messages) for Claude Code etc.
8892	tts-server	OpenAI-compatible /v1/audio/speech; routes Piper / Kokoro / Chatterbox by voice
8893	image-server	image gen (/generations, engines from imageServer.engines: Z-Image default, plus "model": "krea-2" / "flux-dev" / "hidream-o1" (in-image text; "hidream-o1-dev" for the distilled draft build) / "pony" / "epicrealism" / "juggernaut") + edit (/edits, engines from imageServer.editEngines: Kontext default, "model": "qwen-edit" for text/identity that has to survive repeated passes), reimagine (/reimagine schnell), inpaint (/inpaint, masked region only — source composited back outside the mask; engines from imageServer.inpaintEngines: Fill default, "model": "pony" stylized SDXL-inpainting); optional "seed" on every sampling path (omit for random; the seed used is always resolved server-side, returned as data[i].seed and recorded in the genai chunk, so any render can be repeated after the fact — a batch walks it, seed, seed+1, …); LoRAs via <lora:name:0.9> prompt tags on every path (on /generations a registered LoRA's base auto-selects the engine; the source-image paths each run one fixed model, so a LoRA from another architecture is refused there instead of rerouted); the three source-image paths render at the source's resolution, so a larger one is downscaled to imageServer.editMaxPixels (2MP) first — per-request "max_pixels" overrides that up to editMaxPixelsCeiling (4MP), and /inpaint still returns the full-resolution original outside the mask; a render is killed when its requester disconnects (Open WebUI's stop button) or via POST /cancel
8894	media-tools	OpenAPI tool server: generate_image (with installed subject LoRAs), generate_image_with_face (PuLID via ComfyUI), swap_face / swap_face_full (PuLID onto an existing photo — masked face region, or the whole frame), swap_face_fast (ReActor/inswapper pixel transplant, seconds), edit_image, reimagine_image, inpaint_image, smart_edit (auto-mask: SAM 3 segment + inpaint), create_mask (segmentation preview: red-tint overlay + reusable mask), list_regions (which phrases the segmenter actually recognizes in this picture — the words smart_edit can act on), transcribe_audio, text_to_speech; an optional seed on all eight sampling tools, always reported back in the result so a chat model can answer "same picture but …"; lora/loras/lora_strength on every tool that runs a diffusion model — generate_image, the four edit tools and the two PuLID face tools (generate_image_with_face, swap_face_full), each offering only the LoRAs its model can bind, and the face tools splicing ComfyUI loader nodes rather than prompt tags; a photo that gets re-sampled as a latent (the edit tools' source, the two PuLID swaps' target) is downscaled to imageServer.editMaxPixels first — the identity/reference photos are not — with max_pixels to spend more on one render and swap_face's full_resolution to composite the face back over the untouched original; artifacts under /files/
8895	MagenticLite	web-agent UI (magentic-ui 0.2.x): Fara browser-use + orchestrator, browser sandboxed in a Quicksand QEMU micro-VM (KVM-accelerated, no Docker). The live browser view (noVNC, per-session password) uses ports 8860-8879, firewalled alongside the service ports
8896	tts-hq	internal HQ TTS backend (Kokoro narration, Chatterbox cloning)
8903	code-sandbox	run Python / JavaScript / shell in a throwaway QEMU microVM (exec-sandbox on KVM): no host filesystem, no state between runs, no network by default. openapi.json so chat and MCP both get it (codeSandbox.enable)
8902	memory-server	durable cross-session memory: remember / recall / forget, reinforcement and near-duplicate consolidation. Namespaced by a trusted identity header (memory.enable)
8900	rag-server	knowledge collections: hybrid BM25 + semantic search, ingest by text/URL/path, openapi.json so Open WebUI and the MCP gateway pick it up. State in /var/lib/genai-rag (rag.enable)
8899	mcp-gateway	MCP over Streamable HTTP at /mcp — every tool server's OpenAPI operation as an MCP tool (web search, image gen/edit, segmentation, speech). GET /tools is the resolved inventory. Also at :8897/svc/mcp/mcp (mcp.enable)
11434	ollama-shim	Ollama API dialect — /api/tags, /api/show, /api/chat, /api/generate, /api/embed, /api/ps, /api/pull, NDJSON streaming. Translates to the portal, so Ollama-only clients (Home Assistant, JetBrains, mobile apps) use the same fleet. Models stay declarative: pulling a name that isn't in llmModels errors, and create/copy/push/delete answer 501 (ollama.enable)
10300	wyoming-openai	Wyoming protocol bridge for Home Assistant (STT + TTS). Off by default; adds no models — forwards to the asr model and tts-server. No authentication, so wyoming.openFirewall is separate from the global one
8901	voice-server	Realtime speech-to-speech over WebSocket (/ws), plus /health. Off by default. The chat model runs on the CPU, so this service uses no VRAM
8898	segment-server	SAM 3 text-prompted mask generation (/segment); the auto-mask half of smart_edit, CPU-resident. A phrase can also be scoped to a container (within, which crops and re-encodes so small regions are big enough to find) or replaced by example boxes (boxes, no vocabulary involved), and /probe sweeps a vocabulary to report which phrases this image answers to. Weights are license-gated: accept at hf.co/facebook/sam3 and set hfTokenFile

STT is served OpenAI-style at :8080/v1/audio/transcriptions — llama-swap's asr model (Qwen3-ASR-1.7B, ggml-org GGUF; also answers to the whisper-1 alias), so it swaps and unloads like any other model instead of running as a second daemon. response_format must be json: llama-server does not implement the text/srt/vtt variants whisper.cpp had, so subtitle files need a separate conversion step. Open WebUI is wired to all three (mic button = asr, read-aloud = Piper, image toggle in the chat input = FLUX.1-dev at ~20 steps — the quality default, ~40-60s/image; ask the chat for a Z-Image generation when you want the ~5s fast path); the seeding service enforces the engine config. The image server unloads llama-swap models before generating (full GPU), so expect the next LLM request to re-load for a few seconds.

Ports are opened only on firewallInterfaces (default tailscale0); loopback always works.

Models (select by name via the API / UI)

Name	Model	Footprint / speed	Reach for it when
deepseek4-284b-a13b	DeepSeek-V4-Flash-0731 284B-A13B	19194MiB VRAM + ~84GB RAM, 14 tok/s generation and 513 tok/s prefill measured with the resident set up (13.4-19.1 across loads), 128k ctx	The agentic escalation tier: a 284B model with a 128k window that costs almost no KV, tuned for tool use and long agent loops rather than for single answers. Reach for it when qwen3-coder-80b-a3b has lost the thread on a long session — not for a quick question, because it generates at roughly 1/20th of qwen36-35b-a3b.
fara-7b	Fara1.5-27B (computer-use agent)	~24GB VRAM (~17.5GB weights + ~1GB mmproj), ~45 tok/s, 128k ctx	Browser automation from screenshots. NOT a chat or coding model — drive it with fara-cli or Magentic-UI pointed at this endpoint. Picking it in a chat window can only fail.
fast-cpu	Qwen3-4B-Instruct-2507 with tools (on the CPU)	zero VRAM, ~2.5GB host RAM, 22 tok/s generate / 223 tok/s prefill measured on 8 threads, 16k ctx	A background model for coding harnesses: the slot that writes conversation titles and does small classification calls, moved off the GPU so it stops queueing behind the conversation it is describing.
glm47-30b-a3b	GLM-4.7-Flash 30B-A3B	~24GB VRAM, entirely on card (~17.5GB weights + ~3.8GB KV), 128k ctx	Snappy agentic coding entirely in VRAM — no experts streaming from host RAM, so no CPU-offload latency. The iteration tier: cycle against it, escalate when it stalls.
glm53-320b-a18b	GLM-5.3-Flash 320B-A18B — the best paper scores, the worst configuration	15.2 tok/s generation at 29700MiB warm through llama-swap (4.09 cold, same config — this entry is the control that measured what a cold page cache costs), 66.8 tok/s prefill measured cold, plus 93.1GB of weights in host RAM, 128k ctx	The highest paper scores in this catalog — Terminal Bench 2.1 84.3, DeepSWE 63.4, HLE 55.3, MIT, natively multimodal — running in the most compromised configuration in it. Measured here it is a DEAD HEAT with qwen38-125b-a6b-max: 4.09 tok/s against 4.1, 66.8 prefill against 71. What it gives up for that parity is half the window (131072 against 262144) and its quant — 1-bit, the only one of seven that fits, against that entry's 92.3%. So the question this entry cannot answer from a benchmark table is whether a 320B at 1-bit still beats a 125B at 92.3%, and nothing here has driven either on real work.
gptoss-120b	gpt-oss-120b (reasoning: high)	~26GB VRAM + ~40GB RAM, ~30 tok/s, 64k ctx	Knowledge-heavy queries and a second opinion: 117B total params carry broad world knowledge the 27-35B models do not have. No longer the reasoning escalation — the Qwen3.6 generation took that.
minimax27-229b-a10b	MiniMax-M2.7 229B-A10B	~19GB VRAM + ~65GB RAM, ~17 tok/s generation, 728 tok/s prefill, 64k ctx	A second opinion, not a better coder. It is the biggest model in the fleet and reads like the strongest one, but against qwen-dense it is a PEER — reach for it when qwen36-27b-128k is confidently stuck and you want a genuinely independent take, not when you want more power.
ornith1-35b-a3b	Ornith-1.0-35B-A3B (thinking)	22548MiB measured at 128k (~1.70GB of it q8 KV), 288 tok/s generation, 9387 tok/s prefill, entirely on card, 128k ctx	qwen3-coder-80b-a3b's benchmarks without qwen3-coder-80b-a3b's host RAM, and the fastest model on the box: a 35B MoE with 3B active that fits on the card whole, measured at 288 tok/s against qwen36-35b-a3b's 270 and qwen3-coder-80b-a3b's 20-35. Coexists with the resident set. Still to be A/B'd against qwen3-coder-80b-a3b on real agent work.
qwen-dense	Qwen3.6-27B MTP (thinking) — off by default	25744MiB measured, ~80 tok/s (MTP spec decode), f16 KV, 80k ctx	The best coder on the box by benchmark, and the most expensive to keep warm: it cannot share a 32GB card with transcription, RAG and memory. Off by default for capacity, not for quality.
qwen3-coder-80b-a3b	Qwen3-Coder-Next 80B-A3B	19752MiB VRAM + 49.6GB RAM at nCpuMoe 36, leaving 5679MiB free with the resident set up; ~55 tok/s generation and ~1420 tok/s prefill, MEASURED 2026-08-19 — the old "~20-35 tok/s" here carried no date or conditions and was low by a factor of two, 256k ctx	The Claude Code / OpenCode model. Purpose-trained for long-horizon agent loops and recovery from failed steps, and the only model here whose window clears Claude Code's assumed ~200k — so long sessions auto-compact instead of overflowing.
qwen35-122b-a10b	Qwen3.5-122B-A10B (thinking) — the wide-window agent tier	27448MiB total with the resident set up (20272MiB of its own) leaving 4703MiB free, plus 57GB of weights in host RAM; 24.6 tok/s generation and ~1071 tok/s prefill measured, but the same command re-loaded reads 20.6-24.6 (19.4% spread) while its VRAM repeats to 26MiB, 256k ctx	The model to escalate to for a long agentic session that runs subagents. 10B active of 122B, and the widest window here shared across every request that is live at once — which is the number that actually runs out when Claude Code puts a background agent beside your main loop. ~23 tok/s measured against qwen3-coder-80b-a3b's ~55, which is the real price and it is steep. What it buys is Terminal-Bench 49.4 against 36.2, a reasoning trace, and vision — pick it when the task is worth 2.4x the wait, not by default.
qwen36-27b-128k	Qwen3.6-27B (thinking), no MTP — the 3.6 default 27B	23518MiB measured (~17GB weights + ~4.3GB KV), ~47 tok/s, q8 KV, 128k ctx	The 27B this box ran on until qwen38-27b-96k took the dense alias, and the only one of them with real sessions behind it. qwen-dense's weights and answers with 128k instead of MTP speed.
qwen36-35b-a3b	Qwen3.6-35B-A3B (thinking)	~20GB VRAM, ~270 tok/s, entirely on card, 256k ctx	The daily driver, coding included — the reflex choice unless you specifically need the best code, the missing refusals, or a long agent loop. Entirely in VRAM at ~270 tok/s, the only model here fast enough that you stop noticing latency.
qwen36-35b-a3b-uc	Qwen3.6-35B-A3B abliterated (thinking)	~21GB VRAM, ~260 tok/s, entirely on card, 256k ctx	qwen36-35b-a3b with refusal-direction removal — same architecture, same speed, same 262k window. For when a refusal is blocking legitimate work, not as a general upgrade.
qwen38-125b-a6b	Qwen3.8-Flash-Next 125B-A6B Q2_K_XL — the Qwen4 preview, interactive	41.7 tok/s generation and 489-729 tok/s prefill at 29670MiB (29922 with 28107 tokens in cache, 2685 free), measured warm through llama-swap on the b10760 pin at nCpuMoe 38, -ub 1024 and the MTP drafter, greedy, with the weights resident. 38.7 tok/s at the shipped preset's temp 1.0, where the drafter is accepted less often. 22.4GB RESIDENT since --lazy-mode on-direct, down from 65.9GB, against a 78.9GB file — 24GB less than the -max default and the reason to reach for this one, 256k ctx	The Qwen4 architecture, a flagship early, and now the one to reach for among the big CPU-offloaded tiers. It does deepseek4-284b-a13b's job — one hard question, a long agentic session — at 41.7 tok/s with twice its window, because 6B of its 125B is active per token against their 13B and 10B. The catch is the engine: qwen4exp merged upstream at b10660, but this host still pins a branch by hand for the MTP draft head, which is not in any release. It runs 2-bit of a seven-rung ladder, so it is the fastest of the three and the least certain of its answers.
qwen38-125b-a6b-max	Qwen3.8-Flash-Next 125B-A6B Q4_K_XL — the best answer this box can give	31.2 tok/s generation and 457 tok/s prefill at 30200MiB (1949 free), measured warm through llama-swap on an idle card at nCpuMoe 38, -ub 1024, q8_0 KV, greedy, on a 31538-token prompt with the weights resident; 23.7 tok/s and 359 tok/s with 253344 tokens in cache, i.e. at 96.6% of the window. Page cache is still the largest lever on this entry — the same config read 4.66 tok/s with only half the weights resident — so no number here means anything without its regime. 46.4GB resident since --lazy-mode on-direct, down from 83.0GB, against a 111.3GB file — the PLE table is read rather than held, 256k ctx	THE DEFAULT since 2026-08-28, and the best quant this card and this much RAM can hold: 92.3% top-1 against the full weights where the sibling is 82.7%, at 31.2 tok/s and 457 tok/s prefill. It was written as a one-hard-question tier on cold measurements that read 4.1 and 71; warm it is simply the better model at conversational speed, and the fidelity it buys back is measured — the two quants disagree on the top token one sampled token in six. It used to cost 83GB of resident host RAM out of 123, so anything else large evicted it; --lazy-mode on-direct cut that to 46.4GB for no measured speed, which is what makes the warm number survive company. qwen38-125b-a6b is still the same weights at 1.6x the speed, for when ingest matters more than the answer.
qwen38-125b-a6b-max-uc	Qwen3.8-Flash-Next 125B-A6B Q4_K_XL abliterated — the best answer, without the refusals	30200MiB predicted (not yet measured) — the qwen38-125b-a6b-max configuration on a file with the same tensor shapes; ~31 tok/s warm expected, 256k ctx	qwen38-125b-a6b-max with huihui-ai's abliteration applied, at the same UD-Q4_K_XL quant: the strongest model here for a question the official weights refuse.
qwen38-125b-a6b-q5	Qwen3.8-Flash-Next 125B-A6B Q5_K_XL — the rung above the default, at 18% of the speed	25.30 tok/s generation and 332.7 tok/s prefill at 29186MiB (2965 free), measured warm on an idle card at nCpuMoe 40, -ub 1024, q8_0 KV, greedy, on a 31538-token prompt. A 147.41GiB file, of which 50.66GiB is the PLE table read on demand rather than held, 256k ctx	THE HIGHEST-FIDELITY QUANT THIS BOX CAN HOLD, and it only became loadable on 2026-09-18 — the file is 147.41GiB against 123GB of RAM, and what makes it fit is reading the 50.66GiB PLE table from disk instead of keeping it resident. unsloth puts it at 93.68% top-1 against the full weights where the shipped default is 92.26%, which closes about three quarters of the remaining gap to Q8_0. It costs 18% of generation and 27% of ingest. Whether that fidelity is visible in work is the open question this entry cannot answer for you.
qwen38-27b-128k	Qwen3.8-27B (thinking), 128k — the wide half of the pair	22378-22792MiB measured, 74 tok/s generation and ~3770 tok/s prefill, 128k q8 KV; needs llama.cpp b10434+, 128k ctx	qwen38-27b-96k's weights with the full 128k window and no speculative decoding. Come here when the prompt is bigger than qwen38-27b-96k's 96k — which sessions do automatically, so naming it explicitly is for when you know that up front.
qwen38-27b-224k	Qwen3.8-27B (thinking), 224k — unloads speech-to-text	27016MiB predicted (not yet measured), 224k q8 KV, no drafter; unloads asr while warm, 224k ctx	qwen38-27b-96k's weights with the widest window this card can hold, bought by unloading speech-to-text. For sessions that run background subagents: llama-server's slots share one KV pool, so concurrent streams need window rather than a second model.
qwen38-27b-256k	Qwen3.8-27B (thinking), 256k — the default, card to itself	28424MiB predicted (not yet measured), 256k q8 KV, no drafter; needs the card to itself, 256k ctx	The dense 27B at the full 262144 the weights support, with nothing else on the card. The default, and the widest window this box can serve. Retrieval, durable memory and transcription all swap it out while it runs.
qwen38-27b-96k	Qwen3.8-27B (thinking) — the default 27B	30316MiB measured with the drafter, 134 tok/s generation and ~3460 tok/s prefill, 96k q8 KV; needs llama.cpp b10434+, 96k ctx	The 27B you get unless you name another one, and the 3.8 generation of qwen36-27b-128k's weights: by its own card a large jump on every coding and agent benchmark. The fast half of the 3.8 pair — 134 tok/s with its own in-file drafter, at 96k — and a session that outgrows that window reroutes to qwen38-27b-128k on its own.
qwen38-27b-uc-128k	Qwen3.8-27B abliterated (thinking) — what a new chat opens on	23746MiB measured, 69 tok/s generation and ~3570 tok/s prefill, q8 KV; needs llama.cpp b10434+, 128k ctx	The uncensored dense 27B, and what a new chat opens on — a refusal is the one wrong answer a chat frontend cannot route around. It is a lossy edit of exactly qwen38-27b-96k's weights, so name qwen38-27b-96k instead when refusals are not the problem. Answers to qwen-dense-uc, the 3.6 abliteration it replaced on 2026-08-21.
qwen38-27b-uc-208k	Qwen3.8-27B abliterated (thinking), 208k — unloads speech-to-text	27266MiB predicted (not yet measured), 208k q8 KV, no drafter; unloads asr while warm, 208k ctx	qwen38-27b-uc-128k's weights at the widest window this card can hold, bought by unloading speech-to-text. The wide entry for sessions where a refusal is the actual problem.
qwen38-27b-uc-256k	Qwen3.8-27B abliterated (thinking), 256k — card to itself	29378MiB predicted (not yet measured), 256k q8 KV, no drafter; needs the card to itself, 256k ctx	The abliterated dense 27B at the full 262144 the weights support, with nothing else on the card. The default, and the widest window this box can serve. Retrieval, durable memory and transcription all swap it out while it runs.
voice	Qwen3-4B-Instruct-2507 (on the CPU)	zero VRAM (runs on the CPU), 0.15s to first token, 12-14 tok/s, 8k ctx	The realtime voice model, and it runs on the CPU — which is the whole reason a spoken turn never has to fight qwen3-coder-80b-a3b for the card.
asr	Qwen3-ASR-1.7B	3708MiB measured, always resident, ~23 tok/s, 4k ctx	Speech-to-text behind Open WebUI's mic button and media-tools' transcribe_audio. Resident, so a transcription never evicts a warm coding session.
embed	Qwen3-Embedding-0.6B	1698MiB measured, always resident, 2k ctx	RAG embeddings. Resident by design, so a chat-model swap never evicts it and retrieval keeps working while the big models come and go.
rerank	Qwen3-Reranker-0.6B	1698MiB measured, always resident (the old "~0.7GB" was weights only), 2k ctx	Cross-encoder reranking for hybrid RAG — it scores the candidates that BM25 and embeddings surfaced. Open WebUI's hybrid search reranks through it.

deepseek4-284b-a13b — DeepSeek-V4-Flash-0731 284B-A13B

The agentic escalation tier: a 284B model with a 128k window that costs almost no KV, tuned for tool use and long agent loops rather than for single answers. Reach for it when qwen3-coder-80b-a3b has lost the thread on a long session — not for a quick question, because it generates at roughly 1/20th of qwen36-35b-a3b.

Outgrowing its window reroutes to qwen3-coder-80b-a3b rather than hard-erroring.

Best at

• Purpose-built for agentic work: the 0731 release is a large tool-use and coding upgrade over the V4 preview, and beats DeepSeek-V4-Pro (Preview) on benchmarks at a fraction of the active parameters.
• 128k window for 3873MiB of f16 KV, measured — hybrid CSA+HCA attention with 1 KV head and a K-only cache, so a token of context costs 0.026MB here against minimax's 0.129MB, and the full window costs less than minimax's 64k does.
• A third lineage on the box: not a Qwen and not MiniMax, so it fails differently from both.
• Only 13B active of 284B, so it generates at roughly MiniMax speed despite being 55B bigger.
• Reasoning-effort levels are native to its chat template rather than bolted on by a prompt.

Costs and limits

• UD-IQ2_M — 2-bit, the lowest-precision quant in the fleet. Forced by 277B of experts against 123GB of RAM, not chosen.
• 14 tok/s measured, and the figure moves: identical commands re-measured hours apart differ by 16-18% while VRAM repeats to the MiB. Nothing under ~20% is a real tuning result on this model.
• ~84GB of host RAM while loaded — the largest footprint here, and the box cannot cache it and minimax's 75GB at once. Whichever ran second re-reads its weights from NVMe on every token: minimax measured 2.4-2.5 tok/s straight after this model, against its own recorded ~17, and did not recover over three consecutive loads. It reads as a broken model rather than a cold one.
• Needs llama-cpp >= b10254. On anything older it answers, re-prefills every agentic turn, and looks slow rather than broken.
• q8 KV corrupts it silently, so it cannot buy VRAM back the way every other model here can.
• 1M native context is not reachable: the window is capped at 131072 by VRAM, not by the model.

Compared with

• gptoss-120b — Both non-Qwen second opinions. research is a knowledge cross-check that fits on the card; this is an agentic coder that does not.
• minimax27-229b-a10b — The same job — one hard question off an independent lineage — with 2x the window and 55B more total parameters, against minimax's higher-fidelity 3-bit quant. Both are measured now and they are peers on speed: 14 tok/s here against minimax's ~17. What actually decides is that they cannot both be page-cached — 84GB and 75GB on a 123GB box — so whichever runs second re-reads its weights from NVMe on every token and collapses to ~2.5 tok/s. Pick one for a session; do not alternate.
• qwen-dense — Not a comparison worth making on speed. This is a far bigger model at 2-bit; qwen-dense answers ~20x faster and wins most questions that fit its window.
• qwen3-coder-80b-a3b — qwen3-coder-80b-a3b drives sessions at ~10x the speed with a 256k window and is where agent loops belong. Come here when it is stuck, not to replace it.

fara-7b — Fara1.5-27B (computer-use agent)

Browser automation from screenshots. NOT a chat or coding model — drive it with fara-cli or Magentic-UI pointed at this endpoint. Picking it in a chat window can only fail.

Best at

• 72.3 Online-Mind2Web: form filling, shopping, bookings, information gathering.
• 128k, because screenshots are token-hungry and computer-use traces run long.
• MIT licensed.

Costs and limits

• Vision-only perception and a click/type/scroll action space — no files, no shell, no code.
• Its model card scopes it to web tasks; it is not a general agent.

Compared with

• qwen36-35b-a3b — Unrelated jobs that both involve images: qwen36-35b-a3b understands a picture you attach, fara acts on a browser it is looking at.

fast-cpu — Qwen3-4B-Instruct-2507 with tools (on the CPU)

A background model for coding harnesses: the slot that writes conversation titles and does small classification calls, moved off the GPU so it stops queueing behind the conversation it is describing.

Best at

• Zero VRAM, so it never competes with a chat model and never triggers an eviction.
• Tool calls work, which is what separates it from voice and what a harness's background slot needs.
• Reuses a GGUF the store already has, so enabling it downloads nothing.

Costs and limits

• PREFILL is the constraint: ~600 tokens in 3.8s, but 4000 tokens takes 26s. A client that sends long prompts to its background slot will feel that.
• A 4B instruct model. It is a helper, not a second opinion — never point a SUBAGENT slot at it, because subagents do real work.

Compared with

• voice — The same weights and the same CPU. voice is the spoken-reply path (8k, no tools); this one has tools and a bigger window for a harness's background traffic.

glm47-30b-a3b — GLM-4.7-Flash 30B-A3B

Snappy agentic coding entirely in VRAM — no experts streaming from host RAM, so no CPU-offload latency. The iteration tier: cycle against it, escalate when it stalls.

Outgrowing its window reroutes to qwen3-coder-80b-a3b rather than hard-erroring.

Best at

• Fully resident in VRAM (~17.5GB of weights), so latency is GPU-bound rather than RAM-bandwidth-bound.
• MLA attention makes context nearly free: 47 layers of 576-elem latent ≈ 28KB/tok, so 128k is only ~3.8GB.
• coder-pro-class agentic behaviour at in-VRAM speed.
• MIT licensed.

Costs and limits

• A capacity tier below qwen3-coder-80b-a3b — this is the model you escalate FROM.
• ~24GB loaded leaves little room beside the resident set.
• Native context is 202752; 128k is what fits here, not what the model can do.

Compared with

• qwen3-coder-80b-a3b — qwen3-coder-80b-a3b has 2x the window and long-horizon agent training. glm47-30b-a3b is quicker to iterate against; escalate when it gets stuck rather than starting there.
• qwen36-35b-a3b — qwen36-35b-a3b is faster still with a 2x window, but glm47-30b-a3b is the more agentic of the two — better at tool loops.

glm53-320b-a18b — GLM-5.3-Flash 320B-A18B — the best paper scores, the worst configuration

The highest paper scores in this catalog — Terminal Bench 2.1 84.3, DeepSWE 63.4, HLE 55.3, MIT, natively multimodal — running in the most compromised configuration in it. Measured here it is a DEAD HEAT with qwen38-125b-a6b-max: 4.09 tok/s against 4.1, 66.8 prefill against 71. What it gives up for that parity is half the window (131072 against 262144) and its quant — 1-bit, the only one of seven that fits, against that entry's 92.3%. So the question this entry cannot answer from a benchmark table is whether a 320B at 1-bit still beats a 125B at 92.3%, and nothing here has driven either on real work.

Outgrowing its window reroutes to qwen38-125b-a6b rather than hard-erroring.

Best at

• Terminal Bench 2.1 84.3 on its own card, against qwen38-27b-96k's 73.0 — the highest agentic score any entry here quotes, and on the same harness generation. Full-precision, like every number on this card.
• DeepSWE 63.4 and HLE 55.3, against qwen38-125b-a6b's 58.7 and 35.9. The HLE gap is the widest quality claim in this catalog.
• MIT, where qwen38-125b-a6b is Qwen Community 1.0 — the more permissive licence of the two frontier-class entries here.
• Natively multimodal AND it ships an mmproj. So does the Flash-Next repo now, so that is no longer the distinction: vision here is a question of engine support rather than a missing file.
• KV is cheap for its size: 36.4KB/token at f16 measured, so 131072 costs ~2.4GB at q8 — the linear-attention half of the hybrid earning its keep, and the reason the window is capped by the weights rather than by the cache.

Costs and limits

• No ENGINE, rather than no room: this box has already served both 93.7GB and 111.3GB of weights, which brackets this model's two reachable quants. What is missing is a llama.cpp that knows glm5next, and building one is a THIRD hand-pinned engine — unsloth's glm5next branch carries no qwen4exp and the qwen4exp branch carries no glm5next, so it cannot share the one built for the Flash-Next pair.
• That branch is mid-flight. It landed eight commits on 2026-08-27 including 'do not inherit deepseek4's n_embd_out' and a rework of indexer pooling — numerics and KV correctness, not polish. A rev pinned now is a snapshot of unfinished work.
• ONE of seven quants is reachable, and the second one was TRIED: UD-Q2_K_XL (108.7GB) was downloaded and measured on 2026-08-28 at 1.9-2.3 tok/s across three passes on a quiet box, disk-bound, with page-cache residency plateauing at 48GB of its 108.7 and I/O pressure at 5.7. It does not run here. UD-IQ1_S (93.1GB) at 15.2 tok/s is the configuration, and the fidelity this box can have is the bottom of this model's ladder rather than the top.
• FILE SIZE DOES NOT PREDICT WHETHER A BIG MoE FITS — ACTIVE PARAMETERS DO, and this pair is the demonstration. qwen38-125b-a6b-max is a LARGER file (111.3GB against 108.7) on the same box and runs at 26.8 tok/s, because 6B activate per token against this model's 18B. Three times the active weight is three times the per-token read from the CPU side, so the hot working set stops fitting in page cache while the file still would. Comparing two MoEs by gigabytes on disk is the mistake that made UD-Q2_K_XL look obviously affordable here.
• UD-IQ1_S IS THE TIER THAT COLLAPSED IN THE ONE PUBLISHED TEST OF IT. quesma.com benchmarked Qwen3.8-27B quant by quant and found the damage nonlinear: flat from BF16 through 4-bit, a few points at UD-Q2_K_XL, and UD-IQ1 scoring ~50% on GPQA Diamond — random guessing — with longer reasoning making it worse. That was a 27B dense and this is a 320B MoE with dynamic quants, which should be more robust, so it is a red flag rather than a verdict. But it is the only tier this box can run this model at, and it is the tier that has been seen to fail, which is worth knowing before trusting an answer from it.
• 4.09 tok/s and 66.8 prefill MEASURED — which lands it on qwen38-125b-a6b-max's numbers exactly while carrying half the window and a far lower quant. The estimate this entry shipped with said 7-8 tok/s, reasoned from bytes-per-token; it was ~2x optimistic because 18B active is compute the CPU has to do, not just bytes it has to read.
• 131072 is now measured rather than provisional, and it is the ceiling: 262144 would want ~2.4GB more KV than the 2923MiB this config leaves free. The weights claim 1048576 and the card evaluates to 300000; neither is reachable on a 32GB card.

Compared with

• deepseek4-284b-a13b — The nearest thing here in shape — a big CPU-offloaded MoE off a third lineage at a low-bit quant, 131072, and the same 112GB RAM floor. That one is 284B against 320B, measured at 14 tok/s, and needs no pinned engine. This is what it would look like a generation on, if the engine existed.
• glm47-30b-a3b — The GLM already here, and not a small version of this: 30B-A3B, entirely on the card, no host RAM, no pinned engine. Two generations and a different architecture apart — they share a vendor and nothing else.
• qwen38-125b-a6b-max — The comparison that decides whether this is worth a third engine. That one is measured, runs today, and holds 262144 at 92.3% quant fidelity; this one scores higher on every published benchmark and would run at 1-bit through an engine nobody has built yet. Until it is measured, the paper gap and the fidelity gap point in opposite directions and neither number settles it.

⚠ Measured 2026-08-27 on a hand-pinned engine: 4.09 tok/s and 66.8 tok/s prefill at 131072 — a dead heat with qwen38-125b-a6b-max on both, at half the window and a 1-bit quant against that entry's 92.3%. Needs its own llama.cpp (glm5next is PR ggml-org/llama.cpp#27752, open), a third engine on this box.

gptoss-120b — gpt-oss-120b (reasoning: high)

Knowledge-heavy queries and a second opinion: 117B total params carry broad world knowledge the 27-35B models do not have. No longer the reasoning escalation — the Qwen3.6 generation took that.

Outgrowing its window reroutes to qwen36-35b-a3b rather than hard-erroring.

Best at

• 117B total params: breadth of factual coverage is what it is actually for.
• f16 KV per the official gpt-oss guide, which never quantizes it — a report there measured KV quant halving throughput.
• Sliding-window attention halves KV cost, which is how 64k fits at all.

Costs and limits

• AA reasoning index 24 against qwen-dense's 37 — passed by a model a quarter its size.
• 64k window; 128k would OOM next to the resident services.
• ~61GB MXFP4 with experts in RAM, for a model that is now a specialist rather than a default.

Compared with

• minimax27-229b-a10b — Both are second-opinion models on a different lineage from the Qwens. minimax is the stronger coder; research has the broader factual base.
• qwen36-27b-128k — That one out-reasons this at a fraction of the size. Come here for breadth of world knowledge, not for reasoning power.

minimax27-229b-a10b — MiniMax-M2.7 229B-A10B

A second opinion, not a better coder. It is the biggest model in the fleet and reads like the strongest one, but against qwen-dense it is a PEER — reach for it when qwen36-27b-128k is confidently stuck and you want a genuinely independent take, not when you want more power.

Outgrowing its window reroutes to qwen3-coder-80b-a3b rather than hard-erroring.

Best at

• A different lineage from every Qwen here, so it fails differently. That independence is the actual reason to run it.
• Wins SWE-bench Multilingual (76.5), SWE-Bench Pro (56.2), NL2Repo and GDPval-AA.
• Good on multi-language repos and long-horizon spec->repo work.
• 728 tok/s prefill: it reads a big prompt quickly even though it generates slowly.

Costs and limits

• Not an upgrade over qwen-dense: it loses Terminal-Bench 2.0 (57.0 vs 59.3), and its wins are full-precision numbers this deployment does not run at.
• IQ3_XXS — roughly 3 points of Aider pass rate below 4-bit on MoE models, and the quant is forced by architecture, not chosen.
• ~17 tok/s, about 1/4.7 of qwen-dense. Below the interactive bar on purpose.
• 64k, the smallest window of the coding-capable models: minimax-m2 is plain full attention (62 layers x 8 KV heads), so KV costs a measured 8432MiB at 64k and the native 196608 would want ~25GB of card.
• ~65GB of host RAM while loaded.

Compared with

• deepseek4-284b-a13b — Peers on speed once warm — ~17 tok/s here against its 14 — and it has 2x the window. But 75GB and 84GB will not both stay in page cache on a 123GB box: whichever runs second re-reads its weights from NVMe every token and drops to ~2.5 tok/s, measured, without recovering over three loads. Alternating between the two costs far more than either choice does.
• gptoss-120b — Both second opinions off a non-Qwen lineage. This one codes better; research knows more.
• qwen-dense — A peer, not an upgrade. It wins SWE-bench Multilingual and SWE-Bench Pro, loses Terminal-Bench 2.0, and runs 3-bit at ~1/4.7 the speed. Its value is independence, not capability.
• qwen3-coder-80b-a3b — qwen3-coder-80b-a3b has 4x the window and is the model that drives agent loops. This answers one hard question; it does not run a session.
• qwen36-27b-128k — Start there. Come here only when that one is stuck — the same conclusion in a quarter the time beats a different one slowly.

ornith1-35b-a3b — Ornith-1.0-35B-A3B (thinking)

qwen3-coder-80b-a3b's benchmarks without qwen3-coder-80b-a3b's host RAM, and the fastest model on the box: a 35B MoE with 3B active that fits on the card whole, measured at 288 tok/s against qwen36-35b-a3b's 270 and qwen3-coder-80b-a3b's 20-35. Coexists with the resident set. Still to be A/B'd against qwen3-coder-80b-a3b on real agent work.

Outgrowing its window reroutes to qwen3-coder-80b-a3b rather than hard-erroring.

Best at

• 288 tok/s generation measured — the fastest here, ahead of qwen36-35b-a3b's 270, and ~10x qwen3-coder-80b-a3b for the same class of coding work.
• 9387 tok/s prefill over a 28k-token prompt (3.0s): nothing streams from host RAM, where qwen3-coder-80b-a3b manages 2298 tok/s at 99k.
• 75.6 SWE-bench Verified — above qwen3-coder-80b-a3b's 70.6, from a model that needs no CPU offload to run.
• 22548MiB at 128k, so it coexists with asr + embed + rerank with ~2.9GB to spare — verified live, unlike qwen-dense which locks them out.
• Self-scaffolding RL post-training: it learns the task harness alongside the solution, which is the part single-shot benchmarks understate for agentic work.
• MIT licensed, with no regional restrictions.

Costs and limits

• Terminal-Bench 2.1 (64.2) is a different harness from the 2.0 numbers quoted for qwen36-27b-128k and qwen3-coder-80b-a3b. It cannot be ranked against them as published, and no real-task A/B has been run here yet.
• Every published Ornith score is the vendor's own and independent verification was still pending as of 2026-08. Reviewers who skipped the benchmarks for held-out tasks found it genuinely strong rather than benchmaxxed — reassuring, but not a reproduced number.
• Over-gates. Reviewers report it stalling on straightforward, fully-disclosed requests by demanding access or prerequisites it has already been given. In an unattended loop that reads as a hang, and it is the opposite of what qwen3-coder-80b-a3b's failed-step recovery training buys.
• Reported to hit a ceiling on genuinely long jobs — one tester had it botch a ~100-iteration kernel implementation that larger models completed. Strong on short and mid-length agent chains is the consistent finding; long-horizon is where it stops.
• Half of qwen3-coder-80b-a3b's window (128k vs 256k), which is below Claude Code's assumed ~200k — long sessions reroute rather than compact.
• 256k does not fit: the measured KV slope (13.6KB/tok at q8) puts it ~1.2GB from the ceiling with the residents loaded, under the 1.5GB margin.
• A thinking model, so there is reasoning latency before the first tool call. qwen3-coder-80b-a3b has none.
• No MTP wired, so no spec-decode speedup: the head plus the f16 KV it needs would overrun the card next to the resident set.
• Generation falls to ~245 tok/s by 28k of context — still the fastest here, but the headline number is a short-context one.

Compared with

• glm47-30b-a3b — The other agentic coder that fits on the card whole. glm47-30b-a3b is smaller and thinking-free; this one is bigger, reasons first, benchmarks higher and measures faster.
• qwen3-coder-80b-a3b — The comparison this entry exists for, and the measurements favour this one hard: ~10x the generation speed, ~4x the prefill, better published SWE-bench (75.6 vs 70.6), and on the card instead of streaming 80B of experts from host RAM. It stays the challenger anyway, because NOBODY HAS RUN THIS COMPARISON — the published Ornith write-ups are all against Qwen3.6-35B-A3B, which is qwen36-35b-a3b here, not against a purpose-built agent driver. qwen3-coder-80b-a3b keeps 2x the window (the part that clears Claude Code's assumed ~200k) and the failed-step recovery training, and the two weaknesses reviewers do report — over-gating and a ceiling on very long jobs — land exactly on that axis. Default stays qwen3-coder-80b-a3b until a real-task A/B says otherwise.
• qwen36-27b-128k — Still the higher SWE-bench score (77.2 vs 75.6), but dense and ~47 tok/s. This is the MoE bet, and measurement settled it: ~6x the throughput for 1.6 points.
• qwen36-35b-a3b — Same architecture and size class; this one is measured slightly FASTER (288 vs 270) and coding-tuned. qwen36-35b-a3b keeps 2x the window and native vision.

qwen-dense — Qwen3.6-27B MTP (thinking) — off by default

The best coder on the box by benchmark, and the most expensive to keep warm: it cannot share a 32GB card with transcription, RAG and memory. Off by default for capacity, not for quality.

Outgrowing its window reroutes to qwen36-35b-a3b rather than hard-erroring.

Best at

• 77.2 SWE-bench Verified and 59.3 Terminal-Bench 2.0 — the highest coding scores here, above qwen3-coder-80b-a3b's 70.6/36.2.
• MTP speculative decoding: ~80 tok/s, ~1.7x the same weights without it, and lossless by construction.
• Official Qwen3.6 "precise coding" sampling (temp 0.6).
• Native vision.
• Claude Code can drive it: since 2026-08-02 the Anthropic bridge turns reasoning_content into real thinking blocks, so being a thinking model no longer rules it out.

Costs and limits

• 25744MiB measured — overruns the card next to the 7104MiB resident set. Enabling it trades transcription, voice, RAG and memory for the speed.
• 80k, the smallest window of the Qwen3.6 variants: f16 KV is what keeps MTP draft acceptance at ~90%, and f16 is what costs the context.
• Half of qwen3-coder-80b-a3b's window (80k vs 256k), which is below Claude Code's assumed ~200k — long sessions overflow rather than compacting.

Compared with

• qwen3-coder-80b-a3b — Better benchmarks (77.2 vs 70.6 SWE-bench Verified) and faster, but a third of the window. qwen3-coder-80b-a3b's 256k is what clears Claude Code's assumed ~200k so sessions compact instead of overflowing.
• qwen36-27b-128k — Identical weights, identical answers — speculative decoding verifies every token. This one is ~1.7x faster; that one has 128k instead of 80k and coexists with the resident set. You are trading speed against everything else on the box.

⚠ 25744MiB measured — cannot be loaded alongside the resident set (asr + embed + rerank, 7104MiB) on a 32GB card. Whichever loads second exits with "upstream command exited prematurely", so enabling this trades transcription, voice, RAG and memory for MTP's ~1.7x speed. qwen36-27b-128k is the same weights at 128k and coexists with everything.

qwen3-coder-80b-a3b — Qwen3-Coder-Next 80B-A3B

The Claude Code / OpenCode model. Purpose-trained for long-horizon agent loops and recovery from failed steps, and the only model here whose window clears Claude Code's assumed ~200k — so long sessions auto-compact instead of overflowing.

Best at

• 256k context: the practical reason to pick it, not a spec-sheet number.
• Trained for tool use and multi-step recovery, which single-shot benchmarks understate.
• Not a thinking model, so no reasoning latency before the first tool call — the Qwen3.6 thinking variants now survive the Anthropic bridge too, but they make you wait for the reasoning.
• ttl 7200 keeps a warm prompt cache across long idles; anything needing the whole card calls /unload explicitly.

Costs and limits

• 70.6 SWE-bench Verified and 36.2 Terminal-Bench 2.0 — below qwen36-27b-128k on both.
• By far the fastest model here that is not fully on the card: ~55 tok/s and ~1420 tok/s prefill, roughly 2.4x qwen35-122b-a10b and 3.8x deepseek4-284b-a13b on the same prompt. The figure this entry shipped with (~20-35) was an estimate nobody had taken, and it was low by a factor of two.
• It shipped for months in a state where a single transcription made it unloadable: at the old nCpuMoe 30 it wanted 25376MiB, which does not fit beside the 7176MiB resident set on a 32607MiB card, and llama-swap reported only "upstream command exited prematurely". Fixed by the retune above; the reason it went unnoticed is that asr only becomes resident once somebody speaks to the box.

Compared with

• glm47-30b-a3b — glm47-30b-a3b is entirely in VRAM and snappier to iterate against; qwen3-coder-80b-a3b has 2x the window and is what you escalate to when it stalls.
• minimax27-229b-a10b — 4x the window and the model that actually drives agents. minimax is for one hard question, not a session.
• qwen35-122b-a10b — Same window, and the two comparable card numbers both favour that one (SWE-bench Verified 72.0 vs 70.6, Terminal-Bench 49.4 vs 36.2). This one answers roughly TWICE as fast — 47.7 tok/s against 24.6, and 1858 prefill against 1071, both measured 2026-08-19 — does not think before its first tool call, and wants 7.4GB less RAM. The one thing it cannot do is share the card with a warm asr, which that one can. Speed and latency here; benchmarks, vision and coexistence there.
• qwen36-27b-128k — That one scores higher on coding (77.2/59.3 vs 70.6/36.2) but has half the window and no agent-loop training. Drive agents with qwen3-coder-80b-a3b; ask hard coding questions of qwen36-27b-128k.

qwen35-122b-a10b — Qwen3.5-122B-A10B (thinking) — the wide-window agent tier

The model to escalate to for a long agentic session that runs subagents. 10B active of 122B, and the widest window here shared across every request that is live at once — which is the number that actually runs out when Claude Code puts a background agent beside your main loop. ~23 tok/s measured against qwen3-coder-80b-a3b's ~55, which is the real price and it is steep. What it buys is Terminal-Bench 49.4 against 36.2, a reasoning trace, and vision — pick it when the task is worth 2.4x the wait, not by default.

Best at

• 72.0 SWE-bench Verified and 49.4 Terminal-Bench 2 on its own card — above qwen3-coder-80b-a3b's 70.6 and 36.2 at the same window, which is the only directly comparable pair in this fleet.
• 10B active of 122B: the highest activation ratio here, against 3B for qwen3-coder-80b-a3b and qwen36-35b-a3b. Depth is what this entry is buying.
• Agentic scores the single-shot coding benchmarks miss: BFCL-V4 72.2 and BrowseComp 63.8.
• A hybrid Gated DeltaNet + sparse MoE over 48 layers, so 262144 costs KV on qwen36-35b-a3b's scale rather than a dense model's — the reason the window is affordable at all.
• Thinking and vision, neither of which qwen3-coder-80b-a3b has, and Apache 2.0.
• ~23 tok/s and ~976 tok/s prefill against deepseek4-284b-a13b's 14.4 and 467, measured the same afternoon on the same prompt at 400 decoded tokens — the faster of the two big CPU-offloaded tiers, and the one with twice the window.
• Coexists with the full resident set at 5133MiB free — though so does qwen3-coder-80b-a3b since its 2026-08-19 retune, so this is table stakes here rather than a distinguishing feature.
• An MTP repo exists (unsloth/Qwen3.5-122B-A10B-MTP-GGUF) if the generation rate ever needs buying back — a second download, not a flag.

Costs and limits

• Its throughput is not a stable number. Three loads of the identical command measured 20.61, 23.45 and 24.61 tok/s — 19.4% apart — while VRAM repeated to within 26MiB. Nothing under ~20% is a real tuning result here, which is the same caveat deepseek4-284b-a13b carries and the reason nCpuMoe was chosen on headroom rather than on speed.
• Measured but not DRIVEN. Nothing has run a real agent session through it: the numbers above are one 3137-token prefill and a 220-token generation, not a day of Claude Code. Every quality claim below is still the vendor's card.
• The dense 27Bs out-score it on paper: qwen36-27b-128k reports 77.2 SWE-bench Verified and 59.3 Terminal-Bench 2.0 on a quarter of the parameters. What it has over them is 2x the window, which is what a subagent workload runs out of first.
• 57GB resident in host RAM. Loading it and deepseek4-284b-a13b in the same stretch puts both outside the page cache, and a model re-reading weights from NVMe per token reads as broken rather than cold.
• The 3.5 generation, two behind the dense slot's 3.8 weights. Qwen has shipped no 3.6 or 3.8 model in this size class, so this is the newest large MoE available rather than the newest Qwen.
• -n 16384 caps generation, where the card asks for 32768 on ordinary queries and 81920 on hard ones. A long thinking trace can meet that ceiling.

Compared with

• deepseek4-284b-a13b — The other big CPU-offloaded escalation tier, and the deeper model of the two at 284B against 122B. This one is faster on both counts (24.6 vs 14.4 tok/s, 1071 vs 467 prefill), 28GB lighter in host RAM, and twice the window — and 131072 is the window that failed under a background subagent on 2026-08-18. So that one is for a hard question and this one is for a long session.
• qwen3-coder-80b-a3b — Same 262144 and the same job, and that one wins most of the axes: 2.4x the generation (~55 vs ~23 tok/s), 1.5x the prefill, 7.4GB less RAM, no reasoning trace before the first tool call, and — since its 2026-08-19 retune — more free VRAM too. Come here for Terminal-Bench 49.4 against 36.2, for a reasoning trace on a hard problem, or for vision. Not by default.
• qwen36-35b-a3b — Same window and ~11x the speed (270 against 24.6) on a third of the active parameters. qwen36-35b-a3b stays the reflex choice; this is for the sessions where it loses the thread.
• qwen38-27b-128k — The dense 27B scores higher on both coding benchmarks and generates ~3x faster (74 against 24.6). Come here only when the work will not fit in 131072 — which, with a subagent live, happens well before a 131072-token prompt does.

⚠ 57GB of weights in host RAM. The box cannot page-cache this and deepseek4-284b-a13b's 84GB at once — whichever loads second re-reads from NVMe on every token and reads as broken rather than cold.

qwen36-27b-128k — Qwen3.6-27B (thinking), no MTP — the 3.6 default 27B

The 27B this box ran on until qwen38-27b-96k took the dense alias, and the only one of them with real sessions behind it. qwen-dense's weights and answers with 128k instead of MTP speed.

Outgrowing its window reroutes to qwen36-35b-a3b rather than hard-erroring.

Best at

• Identical output to qwen-dense — dropping speculative decoding costs throughput and nothing else.
• 77.2 SWE-bench Verified / 59.3 Terminal-Bench 2.0, the top coding scores here.
• 128k at q8 KV costs ~4.3GB — LESS than the ~5.4GB qwen-dense spends on 80k at f16.
• 23518MiB leaves ~1.9GB beside the full resident set, so transcription, RAG and memory keep working.
• Months of real Claude Code and OpenCode sessions behind it, which is exactly what the 3.8 pair that replaced it does not have yet.

Costs and limits

• ~47 tok/s against qwen-dense's ~80. That is the entire cost of the trade.
• Half of qwen3-coder-80b-a3b's window, and no long-horizon agent training.
• It thinks, so first-token latency is higher than a non-thinking coder's — the reasoning arrives intact through the Anthropic bridge, but you wait for it.

Compared with

• minimax27-229b-a10b — This one is ~4.7x faster and a peer on benchmarks. Go to minimax only when this is confidently stuck and you want an independent lineage.
• qwen-dense — Same weights, same answers. That one is ~1.7x faster and locks out transcription, voice, RAG and memory; this one is 128k and shares the card.
• qwen3-coder-80b-a3b — Better raw coding scores here, but qwen3-coder-80b-a3b has 2x the window and is trained for long agent loops. Use qwen3-coder-80b-a3b to drive an agent, this to answer a hard coding question.
• qwen36-35b-a3b — qwen36-35b-a3b is ~5.7x faster with a 2x window; this is the better coder (77.2 vs 73.4 SWE-bench Verified). Ask qwen36-35b-a3b first, escalate here when the code has to be right.
• qwen38-27b-96k — The same weights one generation on, and the entry that took dense and the harness default on 2026-08-17. Its card beats this on everything it reports and its measured throughput is far higher, but the two have never been compared on real work here — this is the one with sessions behind it, so it is where to go back to if the newer pair disappoints.

qwen36-35b-a3b — Qwen3.6-35B-A3B (thinking)

The daily driver, coding included — the reflex choice unless you specifically need the best code, the missing refusals, or a long agent loop. Entirely in VRAM at ~270 tok/s, the only model here fast enough that you stop noticing latency.

Best at

• ~270 tok/s fully in VRAM: ~5.7x qwen36-27b-128k, ~16x minimax.
• Full native 262k window for only ~2.7GB of KV — a GDN hybrid, so just 10 of 40 layers carry KV.
• 73.4 SWE-bench Verified — genuinely strong agentic coding, not a consolation prize for picking the fast model.
• Native vision — attach images directly, no separate model.
• Strong Japanese<->English translation.

Costs and limits

• Out-coded by qwen36-27b-128k (77.2 vs 73.4 SWE-bench Verified) and out-driven by qwen3-coder-80b-a3b on long agent loops.
• 35B total but only ~3B active per token, so it has less depth on hard single-shot reasoning than the dense 27B.

Compared with

• qwen3-coder-80b-a3b — Same 256k window, ~10x the speed. qwen3-coder-80b-a3b is purpose-trained to DRIVE agent loops and recover from failed steps; qwen36-35b-a3b is the generalist you ask directly.
• qwen36-27b-128k — qwen36-35b-a3b is ~5.7x faster with a 2x window; qwen36-27b-128k is the better coder. Speed versus code quality, and for most questions speed wins.
• qwen36-35b-a3b-uc — Same architecture, same speed, refusals removed at a small fidelity cost. Stay here unless a refusal is the actual problem.

qwen36-35b-a3b-uc — Qwen3.6-35B-A3B abliterated (thinking)

qwen36-35b-a3b with refusal-direction removal — same architecture, same speed, same 262k window. For when a refusal is blocking legitimate work, not as a general upgrade.

Best at

• The cleanest abliteration of the field: KL 0.0074 against the base model, smallest capability deltas in the Abliterlitics benchmarks.
• ~260 tok/s and 256k ctx — it costs essentially nothing against qwen36-35b-a3b.
• Fiction, red-teaming, and prompts that trip false refusals.

Costs and limits

• Every abliteration is a lossy edit of the base weights. qwen36-35b-a3b is the higher-fidelity model when refusals are not the problem.
• That KL number ranks it against other abliterations, not against the base model — it is the best of a lossy field, not lossless.

Compared with

• qwen36-35b-a3b — Same model and speed with refusals removed, at a small fidelity cost. Default to qwen36-35b-a3b and come here when it refuses something it should not.
• qwen38-27b-uc-128k — This is the fast MoE (~260 tok/s, 256k); that is the dense 27B — deeper on hard reasoning, ~4x slower, half the window. It answers to qwen-dense-uc too, which was the 3.6 dense abliteration this pairing used to name.

qwen38-125b-a6b — Qwen3.8-Flash-Next 125B-A6B Q2_K_XL — the Qwen4 preview, interactive

The Qwen4 architecture, a flagship early, and now the one to reach for among the big CPU-offloaded tiers. It does deepseek4-284b-a13b's job — one hard question, a long agentic session — at 41.7 tok/s with twice its window, because 6B of its 125B is active per token against their 13B and 10B. The catch is the engine: qwen4exp merged upstream at b10660, but this host still pins a branch by hand for the MTP draft head, which is not in any release. It runs 2-bit of a seven-rung ladder, so it is the fastest of the three and the least certain of its answers.

Best at

• 6B active of 125B — the lowest activation ratio in the fleet, against deepseek4-284b-a13b's 13B and qwen35-122b-a10b's 10B. On a CPU-offloaded MoE that ratio is what generation speed is made of.
• 41.7 tok/s generation and 489-729 tok/s prefill, measured warm through llama-swap at nCpuMoe 38 with the MTP drafter on the b10760 pin, weights resident. READ THE REGIME BEFORE COMPARING: the deepseek4-284b-a13b and qwen35-122b-a10b figures elsewhere in this catalog were taken on a different engine and a colder page cache, and have NOT been re-measured against this. Comparing them to this number is the mistake this entry has already made once, in the paragraph this one replaces.
• 262144 native, ties the widest window here, and it holds it with the card to itself. KV measured 38.0KB/token at f16 — ~9.5GB for the full window — and the cache is q8_0 since 2026-09-09, so the window is now paid for partly out of the cache rather than entirely out of expert offload. See kvQuant, which records a q8_0 attempt that loaded cleanly and then aborted mid-decode for days.
• Beats Qwen3.8-27B on all TWELVE benchmarks Qwen's own card runs head to head, same harnesses: DeepSWE 1.1 58.7 vs 42.2, JobBench 55.7 vs 33.4, SWE-bench Multilingual 81.0 vs 73.8, Agents' Last Exam 51.2 vs 42.9, Toolathlon Verified 73.5 vs 67.1, HLE 35.9 vs 30.8, NL2Repo-Bench 48.1 vs 42.3, CoWorkBench 73.9 vs 70.7, GPQA Diamond 91.7 vs 89.2, IFBench 81.3 vs 79.5, LiveCodeBench v6 91.9 vs 90.3, SWE-bench Pro 62.5 vs 61.7. Those are full-precision figures for BOTH models — see the qwen38-27b-96k comparison for which of them a 2-bit quant can be expected to keep.
• The widest margins are agentic — DeepSWE 1.1 +16.5 and JobBench +22.3 — which is the axis a long tool-driven session actually runs on, and the axis single-shot coding scores miss.
• A fourth architecture on the box, and the newest one anywhere — hyper-connections, per-layer n-gram embeddings and Qwen Sparse Attention are not in any other entry here, so it fails differently from all of them.
• Reasoning effort is native to its template (low / medium / xhigh), the same interface the Qwen3.8 dense pair already uses here.
• Multimodal weights with a vision encoder, routed through the existing Qwen3-VL ViT path, and the projector now exists — but it is wired on the -max half only, because on this one it does not fit. Send images there; see the mmprojUrl comment for the VRAM growth that decided it.

Costs and limits

• It runs on an engine pinned by hand, and the REASON changed under it. qwen4exp merged upstream on 2026-08-27 (ggml-org/llama.cpp#27742), first carried by b10660 — so the architecture is no longer what the pin is for. The MTP draft head is: a branch of the engine, carried by three competing pull requests as of 2026-09-09 and merged by none of them. A host that does not pin gets this model on any build from b10660 and gets it WITHOUT the drafter, which is reported at rebuild rather than left to be noticed in the tok/s. The pin still expires on its own, but it now takes both strings — serve.engineArch and serve.engineFeature — before the check calls it redundant.
• MULTI-SEGMENT PROMPTS CAN DEGRADE TO REPEATED '/' — ggml-org/llama.cpp#27797, open, no fix. Two or more chat message segments can make the first generated token '/' and never stop, deterministic and temperature-independent where it reproduces. Reported on gfx1151 running a quantized cache with LLAMA_ATTN_ROT_DISABLE=1 — the configuration this entry no longer ships — and unconfirmed on CUDA at f16. This box saw it once, on q8_0, and not since the revert. If it comes back it is this and not a sampler problem: the reporter's workaround is a chat template folding system text and tools into the first user turn.
• q8_0 KV SINCE 2026-09-09, AND IT USED TO CRASH THE MODEL. The abort was never the Hadamard rotation an earlier version of this line blamed: it is qwen4exp.cpp:284, the indexer cache failing to track the attention cache cell for cell, which also fired at f16 with four slots and no quantized cache at all. ggml-org#27941 (merged 2026-09-01) fixed both halves, and a re-run of the multi-slot repro on the pinned engine went 140 requests clean where the old build aborted at 32. The cache is worth 3956MiB and, on this engine, +29% prefill at a 41k prompt against f16 — the reverse of what it cost before, because it is f16 whose prefill falls away as the window fills. Generation gives up ~4% there.
• UD-Q2_K_XL is 2-bit, 82.7% top-1 against the full model. It is the widest quant that stays interactive on this box, not the best one available — qwen38-125b-a6b-max carries UD-Q4_K_XL at 92.3%, and pays 4.5x in generation and 4.3x in prefill for it.
• The quality ladder is steep in SPEED and shallow in ACCURACY: 77.3 / 82.7 / 89.6 / 92.3 percent across quants whose generation goes 34.7 / 18.3 / 7.9 / 4.1 tok/s. Nothing about that curve is a page-cache problem, so no amount of RAM moves it.
• WHAT UD-Q2_K_XL COSTS ON THESE WEIGHTS, measured here rather than borrowed: llama-perplexity's KL-divergence mode against UD-Q4_K_XL as reference, 100 chunks of wikitext-2 test at -c 512. PPL 3.7796 against 3.3333, a ratio of 1.134 — 13.4% worse. Mean KLD 0.265, median 0.072. And the number that reads most directly: SAME TOP TOKEN 82.98%, so the 2-bit quant picks a different most-likely token than the 4-bit one on about one token in six.
• THAT INDEPENDENTLY CORROBORATES UNSLOTH'S PUBLISHED FIDELITY FIGURES, which is worth more than either number alone. They quote 82.7% top-1 for this quant and 92.3% for UD-Q4_K_XL, both against full precision; two quants that agree with BF16 at those rates should agree with EACH OTHER in the low-to-mid 80s, and 82.98% is where it landed. The vendor's ladder can be trusted for the tiers this box has not measured.
• WHAT UD-Q2_K_XL COSTS, from the only third-party measurement of these tiers found so far — quesma.com benchmarked Qwen3.8-27B quant by quant, and the damage is NONLINEAR: BF16, Q8_0 and Q4_K_M are indistinguishable (~95% GPQA Diamond, ~75-77% Terminal-Bench 2.1), UD-Q2_K_XL slips to ~93% and ~70%, and UD-IQ1 falls to ~50% on GPQA, which is random guessing. That is the 27B and not these weights — a 320B-class MoE with dynamic quants may be more robust — but it is the closest evidence available, and it says this entry's tier costs a few points rather than a capability.
• A 78.9GB file but 22.4GB RESIDENT, and this is the half where that matters most. unsloth keeps the n-gram table at IQ4_NL whatever the main quant is, so the SAME 26.8GiB table sits in this 73.4GiB file as in the -max half's 103.7 — 36.5% of this model against 25.9% of that one. --lazy-mode on-direct reads it instead of holding it: 65.9GB resident before, 22.4GB after, measured. The old note here said this joined deepseek4-284b-a13b and minimax27-229b-a10b in the set this box cannot page-cache two of at once; at 22.4GB it no longer does, which is the whole point of the entry the catalog sends you to when the box is busy.
• Its card's numbers are comparable to the 3.8 dense pair and to NOTHING ELSE here. Qwen benchmarks it against Qwen3.8-27B directly, so that comparison is exact; the SWE-bench Verified and Terminal-Bench 2.0 figures on the 3.5/3.6-era entries are different harnesses, and reading down the column would rank it against them anyway.
• MTP IS ON, and it is worth less than the vendor number. unsloth publishes 1.3-1.7x; measured here it is +15-20% greedy and +7% at temp 1.0, because their figure is for a model resident on one card where verifying k+1 tokens is nearly free, and here a verification batch streams close to k+1 times the expert weights out of host RAM. Acceptance is 84.5% at a 28k context greedy and 66-71% at temp 1.0, so the gain follows the temperature and a long agentic session gets the top of the range. NO VISION ON THIS HALF, and it was loaded to find out rather than assumed. The projector WORKS here — same generated test image, same correct answer, 5.4s against the -max half's 29.6 — but it does not fit: with mmproj at nCpuMoe 32 and -ub 2048 the card reads 30878MiB at load (1729 free), 31290 at 16470 tokens and 31796 at 41063, i.e. ~20MiB per 1k tokens, which exhausts 32607 somewhere near 80k of a 262144 window. It would load clean, answer about images, and then die deep into a long session, which is this pair's signature failure and not a trade worth making twice. Buying the room costs about five layers of offload (~5 tok/s of 18.3) or -ub 1024 (the ingest speed that is this entry's whole reason to exist). Send images to qwen38-125b-a6b-max.
• Measured on an idle card. The ~3.8GB left beside the resident set is arithmetic on a measured slope, not a load with embed, rerank and asr actually warm.
• Qwen Community 1.0, not Apache-2.0 — unlike qwen35-122b-a10b and the abliterations, this one carries a vendor licence worth reading before it is put in front of anybody.

Compared with

• deepseek4-284b-a13b — The same slot — a big CPU-offloaded escalation tier off an independent lineage — and this one still wins it, though not as widely as the 1-bit quant suggested. THOSE NUMBERS ARE NOW STALE ON ONE SIDE: this entry reads 41.7 tok/s on the b10760 pin with a drafter, deepseek4's 14 and 467-513 were taken on the old engine and a colder cache, and re-measuring it is owed before the comparison means anything again. What does not depend on the numbers: twice the window, and 5GB less host RAM. Both are 2-bit now, which was the fair comparison all along. What deepseek4 keeps is an engine from a channel.
• qwen35-122b-a10b — Nearly the same size and the same geometry one generation on, and the model this entry's estimates were copied from before anything was measured — they came out 37% low on KV, which is the caution rather than the comparison. The measured gap is OWED A RE-RUN: this entry now reads 41.7 tok/s and 489-729 prefill on a newer engine with a drafter, against that one's 24.6 and ~1071 from the old engine, so the direction that used to hold on both counts is no longer established. What this one has is the newer architecture and, on Qwen's own card, the better answers. That one also keeps working vision and an engine nobody has to pin by hand.
• qwen38-27b-96k — Qwen runs these two against each other on its own card and this model wins all twelve — but READ THE QUANTS BEFORE READING THAT ACROSS. The card compares full-precision weights; this box runs UD-Q2_K_XL here against UD-Q4_K_XL there, 82.7% top-1 against a 27B dense at a quant gentle enough to be near its own ceiling. So the twelve wins split. The wide agentic ones (DeepSWE 1.1 +16.5, JobBench +22.3, Agents' Last Exam +8.3, SWE-bench Multilingual +7.2) are far outside what 2-bit plausibly costs and should survive. The narrow ones (SWE-bench Pro +0.8, LiveCodeBench v6 +1.6, IFBench +1.8, GPQA Diamond +2.5) are inside it, and nobody here has measured which way they land. RUN HEAD TO HEAD on this box's own champion/challenger suite 2026-08-28, both scored 6/6 — every case, including the exact-output contract and finding a bug in code it did not write — so at that difficulty the 2-bit quant has broken nothing, and the only separation was speed: 23.7s against 70.2s for the six. That is a CEILING rather than a verdict; six deterministic coding cases both models ace cannot rank them, and the margins in question are agentic. Though quesma.com's quant-by-quant run on the 27B puts UD-Q2_K_XL's cost at roughly 2 points of GPQA and 5-7 of Terminal-Bench, which would erase every one of those four and leave the wide ones standing. qwen38-125b-a6b-max is the configuration that carries the paper result, which is most of the argument for it existing. Meanwhile the 27B is ~7x faster (134 tok/s with its in-file drafter against 18.3), has vision wired where this does not, and needs no hand-pinned engine. Reach here for long tool-driven work; stay there for anything interactive, anything with an image, and anything the narrow columns describe.

⚠ Runs only on an engine the host pins itself: qwen4exp is PR ggml-org/llama.cpp#27742, merged upstream 2026-08-27 but not yet carried by this box's channel (b10472), so no llama.cpp it offers can load it. Without serve.enginePackage the entry is dropped by serve.minLlamaCpp and nothing downloads its 78.9GB. q8_0 KV since 2026-09-09: the abort that forced f16 here was the engine's, fixed by ggml-org/llama.cpp#27941, and the repro was re-run before the cache was changed back.

qwen38-125b-a6b-max — Qwen3.8-Flash-Next 125B-A6B Q4_K_XL — the best answer this box can give

THE DEFAULT since 2026-08-28, and the best quant this card and this much RAM can hold: 92.3% top-1 against the full weights where the sibling is 82.7%, at 31.2 tok/s and 457 tok/s prefill. It was written as a one-hard-question tier on cold measurements that read 4.1 and 71; warm it is simply the better model at conversational speed, and the fidelity it buys back is measured — the two quants disagree on the top token one sampled token in six. It used to cost 83GB of resident host RAM out of 123, so anything else large evicted it; --lazy-mode on-direct cut that to 46.4GB for no measured speed, which is what makes the warm number survive company. qwen38-125b-a6b is still the same weights at 1.6x the speed, for when ingest matters more than the answer.

Best at

• 92.3% top-1 agreement with the full model, the highest of any quant that fits here — against 82.7% for the interactive entry and 77.3% for the 1-bit.
• The same 262144 window and the same Qwen4 architecture as qwen38-125b-a6b: this is a fidelity choice, not a different model, so nothing about its answers' shape changes.
• Beats Qwen3.8-27B on all twelve of Qwen's own head-to-head benchmarks, and — unlike the interactive half — does it at a quant whose fidelity is in the same class as the 27B's own UD-Q4_K_XL. The paper result is a full-precision comparison, so this is the configuration on this box that can actually be expected to carry it.

Costs and limits

• PAGE CACHE IS THE LARGEST LEVER ON THIS ENTRY, still, and by a wide margin: the same configuration reads 31.2 tok/s with all 104GB of weights resident and 4.66 with only half of them, so a number quoted without saying which regime it came from is not worth much. Anything large sharing this box — a render, a media job — moves you down that curve, and the first answer after an eviction is the slow one.
• GENERATION FALLS WITH THE WINDOW BY ABOUT A QUARTER: 31.2 tok/s at 31538 tokens against 23.7 with 253344 in cache, measured 2026-09-17 at nCpuMoe 38. This line said HALF for three weeks (18.6 against 9.9) and that is what a pre-gather engine did — those figures were taken at nCpuMoe 44 on b10760, before ggml-org/llama.cpp#28213 landed here on 2026-09-09, and nobody re-measured until the offload sweep. Budget a long agentic session at 23.7, not at the headline.
• IT IS BOUND BY CPU EXPERT COMPUTE, NOT BY THE CARD, which is why engine patches aimed at the GPU do nothing here. Sampled during pure generation at 253344 context: the RTX 5090 sits at 42.5% utilisation and 23.3% memory bus while this box's eight cores run at 92% saturation, because nCpuMoe 38 puts thirty-eight of forty-eight layers' expert FFN on the CPU. Raising -t to use the other eight SMT threads makes it 20x WORSE (1.52 tok/s against 31.11) since llama.cpp spins at every layer barrier. The only lever that has ever moved this entry is moving layers onto the card, and VRAM is what limits that.
• Prefill holds as the window fills far better than generation does: 457 tok/s at 31538 tokens, 359 at 253344, so a full-window ingest is ~12 minutes before the first token. Both are above the 302-318 this entry used to record, and the ingest time rather than the generation rate is what makes a full window expensive here.
• The full window is bought with -ub 1024 rather than the shared 2048, and that has not been re-tested since the engine moved. It cost ~3% of generation when it was measured (24.39 tok/s against 25.10) and the compute graph it was buying room from no longer grows with context, so it may now be free or may be unnecessary — but raising it is what reintroduced a crash above ~200k before, so the window and the micro-batch stay one decision until somebody re-runs it.
• A 111.3GB file, but 46.4GB RESIDENT since --lazy-mode on-direct — the 26.8GiB PLE table is read on demand instead of held, measured at 83.0GB before and 46.4GB after with prefill, generation and VRAM all unchanged. The file is still the largest in this catalog and still wants its weights in page cache; what changed is that it no longer holds two thirds of the box to get them.
• NEVER --load-mode none ON THIS ENTRY, however loudly llama.cpp asks. Loading with any --n-cpu-moe or -ot override prints 'tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance', and taking that advice here wedges the machine: it reads the weights into ANONYMOUS memory instead of mmap'ing them, and 104GB of unreclaimable pages against 123GB of RAM and a 32GB swapfile is terminal reclaim thrash. Measured 2026-08-31 — 37 minutes stuck in load, then ping answering while sshd, the console and journald were all blocked; recovery was a hard reboot, and the journal ends mid-line with no oom-kill recorded because journald could not flush. The hint is correct for a model that leaves headroom and fatal for one that is 85% of RAM. mmap is not a tuning choice at this size, it is what makes the weights loadable.
• MULTI-SEGMENT PROMPTS CAN DEGRADE TO REPEATED '/' — ggml-org/llama.cpp#27797, open, no fix. Two or more chat message segments can make the first generated token '/' and never stop, deterministic and temperature-independent where it reproduces. Reported on gfx1151 running a quantized cache with LLAMA_ATTN_ROT_DISABLE=1 — the configuration this entry no longer ships — and unconfirmed on CUDA at f16. This box saw it once, on q8_0, and not since the revert. If it comes back it is this and not a sampler problem: the reporter's workaround is a chat template folding system text and tools into the first user turn.
• nCpuMoe is 38, down from 44 then 42, and what moved it is that the compute graph stopped growing with context rather than that the card got bigger. On the old engine the transient climbed the whole way and 37 died mid-prefill at 98025 tokens; at 38 on this one it grows 320MiB, all of it under 16k tokens, then 2MiB across the next 233k — so a 253344-token prefill peaks at 30200MiB with 1949MiB free. That is +9.1% generation over 42. 37 is NOT next: a layer is 1500MiB here, which would leave ~449MiB, and every failure this entry has ever had was headroom.
• Needs a host-pinned engine, and NO LONGER THE SAME ONE AS THE SIBLING. Both halves load on any build from b10660, so what a pin still buys differs: the sibling needs the unreleased MTP draft head, and this half declares no drafter because MTP measures as a 10% loss on it — so its pin is for the QSA gather (ggml-org/llama.cpp#28213, open) and rides current master instead of the draft-head branch. A host that wires serve.enginePackage for only one of the two gets the other dropped without noticing, which is unchanged; what changed is that the two lines now name different builds on purpose.

Compared with

• deepseek4-284b-a13b — The other model here for one hard question, and it is 3.4x faster to generate and 6.6x faster to prefill. What this one has is Qwen's own head-to-head against the 3.8 dense line and a far higher fidelity quant. Reach here when the answer matters more than the afternoon.
• qwen38-125b-a6b — The same weights, and the only question is what the answer is worth. That one is 4.5x faster to generate and 4.3x faster to ingest at 82.7% fidelity; this one is 92.3%. Nearly ten points of quant agreement is not eight points of quality — it is the ceiling on how much the quantization could have cost — so the honest reading is that this removes a doubt rather than adds a capability. Default to the sibling.

⚠ 31.2 tok/s at a 31k prompt and 457 tok/s prefill at nCpuMoe 38 with the weights resident, and 23.7 tok/s with 253344 tokens in cache — measured 2026-09-17, so the fall across the window is now 24% rather than the half this line used to quote. Filling all 262144 is ~12 minutes before the first token. Use qwen38-125b-a6b when the ingest time matters more than the fidelity.

qwen38-125b-a6b-max-uc — Qwen3.8-Flash-Next 125B-A6B Q4_K_XL abliterated — the best answer, without the refusals

qwen38-125b-a6b-max with huihui-ai's abliteration applied, at the same UD-Q4_K_XL quant: the strongest model here for a question the official weights refuse.

Best at

• Same quant, same imatrix, same serving config as qwen38-125b-a6b-max — the uncensored option that does not trade down to a smaller model or a lower quant.
• Same huihui lineage as the other abliterations served here.

Costs and limits

• Abliteration is lossy and this one is unmeasured: no published KL or capability delta, and -max's 92.3% top-1 figure is about the quant, not about these weights.
• Does not share page cache with qwen38-125b-a6b-max. Alternating between the two keeps both cold, and cold is the regime where this architecture reads 4.66 tok/s instead of 31.
• Everything else — the CPU-bound expert compute, the -ub 1024 window, NEVER --load-mode none, the host-pinned engine — is qwen38-125b-a6b-max's, and so are its limits.

Compared with

• qwen38-125b-a6b-max — The same weights, not abliterated, same quant. Prefer it unless refusals are the problem.
• qwen38-27b-uc-128k — The other uncensored default: a smaller dense model, faster, and able to sit beside the resident set. Reach here when the answer matters more than the speed.

⚠ The same quant, shapes and serving config as qwen38-125b-a6b-max, so the same footprint and speed are expected; not yet measured on these weights. Switching between the two evicts the other's 111GB from page cache.

qwen38-125b-a6b-q5 — Qwen3.8-Flash-Next 125B-A6B Q5_K_XL — the rung above the default, at 18% of the speed

THE HIGHEST-FIDELITY QUANT THIS BOX CAN HOLD, and it only became loadable on 2026-09-18 — the file is 147.41GiB against 123GB of RAM, and what makes it fit is reading the 50.66GiB PLE table from disk instead of keeping it resident. unsloth puts it at 93.68% top-1 against the full weights where the shipped default is 92.26%, which closes about three quarters of the remaining gap to Q8_0. It costs 18% of generation and 27% of ingest. Whether that fidelity is visible in work is the open question this entry cannot answer for you.

Best at

• 93.68% top-1 against the full weights, the highest of any quant that fits here — against 92.26% for the shipped default and 82.72% for the interactive half. Q8_0 is 94.12%, so this recovers roughly three quarters of what is left.
• unsloth's mean KL divergence falls 35% against the default (0.030415 against 0.046893), and the 99.9th-percentile KLD — the tail, which is where quantization damage actually lives rather than in the average — falls from 1.5468 to 1.0036.
• Still conversational: 25.3 tok/s generation and a 262144 window, on the same architecture and the same sampling as the rest of the pair. Nothing about the shape of its answers differs from the default; this is a fidelity choice, not a different model.

Costs and limits

• THE FIDELITY MAY NOT BE VISIBLE, AND THAT IS THE HONEST STATE OF IT. The metrics that favour this entry are KL divergence and top-1 agreement. The only task-level quant study anyone has run on this family — quesma's, on the 27B dense — found 4-bit matching BF16 on GPQA Diamond, IFBench and Terminal-Bench 2.1, and never tested 5-bit at all. So a 35% reduction in KLD is real and may still buy nothing you can see. Nobody has run this box's own champion/challenger suite across the two.
• 18% slower to generate and 27% slower to ingest than qwen38-125b-a6b-max: 25.30 against 30.96 tok/s, 332.7 against 453.3. Part of that is the quant and part is that Q5 experts do not fit at nCpuMoe 38, so two more layers run on the CPU.
• IT EXISTS ONLY WHILE ggml-org/llama.cpp#29030 IS PATCHED IN. 96.75GiB of non-PLE weight fits in 114.6GiB of usable RAM; the full 147.41GiB does not. --lazy-mode on-direct is what makes the difference, and it is an unmerged patch carried on the pinned engine. If that patch comes off, this entry stops being loadable rather than getting slower.
• NO VISION, and unlike the -max half that is untested rather than impossible. The projector costs 1172MiB and this entry has 2965MiB free, so it would probably fit — but nothing here is declared on probably, and nobody has shown this half an image.
• 2965MiB of headroom at nCpuMoe 40, which is more than the -max half runs at. 39 has not been tried, so there is probably a rung of speed left on the table.

Compared with

• qwen38-125b-a6b — Three rungs apart: 82.72% top-1 against 93.68%, and roughly 1.8x the generation speed in the other direction. If you are choosing between these two you are choosing between ingest speed and fidelity with nothing in between, which is what the -max half exists to sit in.
• qwen38-125b-a6b-max — The same weights one quant up, and the question is whether 1.4 points of top-1 is worth 18% of your generation and 27% of your ingest. The default is the default because that trade has never been shown to pay on real work — reach here when an answer is worth waiting for and you want the doubt removed, not as a habit.

⚠ 25.3 tok/s at a 31k prompt and 333 tok/s prefill at nCpuMoe 40 with the weights resident — about 18% slower to generate and 27% slower to ingest than qwen38-125b-a6b-max, for unsloth's 93.68% top-1 against that entry's 92.26%. Whether that shows up in work is NOT established; reach for the -max half unless you have a reason.

qwen38-27b-128k — Qwen3.8-27B (thinking), 128k — the wide half of the pair

qwen38-27b-96k's weights with the full 128k window and no speculative decoding. Come here when the prompt is bigger than qwen38-27b-96k's 96k — which sessions do automatically, so naming it explicitly is for when you know that up front.

Outgrowing its window reroutes to qwen38-27b-224k rather than hard-erroring.

Best at

• The full 131072 window on the 3.8 weights, where qwen38-27b-96k sold 25% of it to fit its drafter.
• Same file as qwen38-27b-96k, so having both costs one download and one 17GB of disk, not two.
• Every benchmark on qwen38-27b-96k applies unchanged — identical weights, identical sampling, only the window and the drafter differ.
• Native vision and video, mmproj shipped in the same repo.

Costs and limits

• 74 tok/s against qwen38-27b-96k's 134 — the drafter is worth 1.81x, measured back to back, and this is what declining it costs.
• Needs llama.cpp b10434 or newer and is dropped below it. Older builds run it happily and then kill llama-server SILENTLY past ~90-100k of prompt (llama.cpp #27090) — which is INSIDE this entry's window specifically.
• Carries qwen38-27b-96k's MTP tensors without using them: one artifact, so they load either way.
• Leaves 2653-3067MiB free with the resident set up — clears the margin, does not clear a warm ComfyUI's 3900MiB.
• Measured but not driven: nothing here has run it through Claude Code or OpenCode, so the vendor's benchmark claims are still the vendor's.

Compared with

• qwen36-27b-128k — Exactly this role one generation back, and the same 128k q8 window. That one held dense until 2026-08-17 and is the only 27B here with real sessions behind it; this one has the better card and the newer engine.
• qwen36-35b-a3b — qwen36-35b-a3b is far faster with a 2x window and stays the reflex choice; this is the deep 27B for when the answer has to be right and the prompt is long.
• qwen38-27b-96k — The same file, 96k, with the drafter on and 1.81x the generation. That is the default of the pair; this one is for prompts it cannot hold.

qwen38-27b-224k — Qwen3.8-27B (thinking), 224k — unloads speech-to-text

qwen38-27b-96k's weights with the widest window this card can hold, bought by unloading speech-to-text. For sessions that run background subagents: llama-server's slots share one KV pool, so concurrent streams need window rather than a second model.

Best at

• 224k of shared KV pool — the room a conversation plus a background subagent actually needs, on the 3.8 dense weights rather than a generation back.
• Same file as qwen38-27b-96k and qwen38-27b-128k, so it costs no second download and no second 17GB of disk.
• Every benchmark on qwen38-27b-96k applies unchanged: identical weights, identical sampling.
• RAG and durable memory keep working — embed and rerank are protected from the eviction, and only asr yields.

Costs and limits

• Speech-to-text is down for EVERYBODY on the box while this is warm, and does not come back on demand — llama-swap exclusivity is one-directional. genai-transcribe (CPU) still works.
• No drafter, so ~74 tok/s rather than qwen38-27b-96k's 134. The window is what the 2184MiB bought.
• 27016MiB is EXTRAPOLATED from the 704MiB/16384 KV linearity, ~1.75x past the range it was measured over. Nothing has loaded this entry yet.
• Needs llama.cpp b10434 or newer, and is further past #27090's cliff than any other entry here.

Compared with

• qwen36-35b-a3b — The other way to buy pool: 262144 for ~2.7GB because only 10 of 40 layers carry KV, costing nothing on the card but a generation of coding depth. Prefer qwen36-35b-a3b when somebody else may need the microphone.
• qwen38-27b-128k — The same file at 128k, also coexisting with everything. This one buys 96k more pool by taking transcription off the box — a cost that lands on other people, so prefer that one unless the pool is the problem.
• qwen38-27b-96k — The same file at 96k with the drafter and 1.81x the generation, coexisting with everything. That is the default; come here only when a second stream is live.

qwen38-27b-256k — Qwen3.8-27B (thinking), 256k — the default, card to itself

The dense 27B at the full 262144 the weights support, with nothing else on the card. The default, and the widest window this box can serve. Retrieval, durable memory and transcription all swap it out while it runs.

Best at

• The model's whole native window — 262144, where every other variant here is capped by what had to fit beside it.
• 256k of shared KV pool, so a conversation and background subagents have room without a second model.
• Same file as the narrower variants: no second download and no second 17GB of disk.
• Every benchmark on the 3.8 dense weights applies unchanged — identical weights, identical sampling.

Costs and limits

• Needs the card to itself: 28424MiB of 32607 leaves no room for embed (1698), rerank (1756) or asr (3708).
• So retrieval, durable memory and speech-to-text each cost a model swap, and swapping back is a FULL re-prefill — a hybrid cannot rewind recurrent state. Use a narrower variant for sessions that lean on those.
• No drafter, so ~74 tok/s rather than the 96k variant's 134.
• 28424MiB is EXTRAPOLATED from the 704MiB/16384 KV linearity, ~2x past the range it was measured over. Nothing has loaded this entry yet.
• Needs llama.cpp b10434 or newer, and is further past #27090's cliff than any other entry here.

Compared with

• qwen38-27b-128k — The same file at 128k. Narrower again, and the point of it is speed and company on the card rather than window.
• qwen38-27b-224k — The same file 32768 tokens narrower, which is exactly what embed and rerank cost. Go there when a session leans on retrieval or durable memory.
• qwen38-27b-96k — The same file at 96k WITH the drafter: 1.81x the generation, a third of the window.

qwen38-27b-96k — Qwen3.8-27B (thinking) — the default 27B

The 27B you get unless you name another one, and the 3.8 generation of qwen36-27b-128k's weights: by its own card a large jump on every coding and agent benchmark. The fast half of the 3.8 pair — 134 tok/s with its own in-file drafter, at 96k — and a session that outgrows that window reroutes to qwen38-27b-128k on its own.

Outgrowing its window reroutes to qwen38-27b-128k then qwen38-27b-224k rather than hard-erroring.

Best at

• Beats Qwen3.6-27B on every benchmark its card reports, head to head on the same harness: Terminal Bench 2.1 73.0 vs 63.4, SWE-bench Pro 61.7 vs 53.5, QwenSWEBench 79.0 vs 49.3, LiveCodeBench v6 90.3 vs 83.9, IFBench 79.5 vs 69.1, HLE 30.8 vs 24.0.
• The agent gap is the widest one: DeepSWE 1.1 42.2 vs 13.3, CoWorkBench 70.7 vs 61.0, Agents' Last Exam 20.4 pass@1 vs 10.6.
• Identical architecture to Qwen3.6-27B, so it inherits this box's GDN-hybrid tuning rather than needing its own: same qwen35 arch, same 16-of-64 KV layers, 128k for ~4.3GB. A 24GB 4090 running 170k at q8 KV with vision and MTP resident corroborates that KV cost from the other direction.
• Speculative decoding needs no second model and no second download — the MTP head is in this same file, and community configs report roughly 2x decode from it.
• Native vision and video, mmproj shipped in the same repo.
• reasoning_effort (xhigh/medium/low) is a real per-request knob this generation added, not just a sampling temperature.
• Its sampling is the box default rather than a special case: Qwen's thinking-mode recipe is temp 1.0 / top-p 0.95 / top-k 20 / min-p 0 with presence and repetition penalties at 0 and 1.0, and stripping client-sent penalties is already what stripPenalties does.

Costs and limits

• Needs llama.cpp b10434 or newer and is dropped below it. Older builds run it happily and then kill llama-server SILENTLY somewhere past ~90-100k of prompt (llama.cpp #27090) — inside this model's own 128k window.
• Footprint and throughput are measured here (134 tok/s generation, ~3460 prefill, 6.0s for an 8k-prompt turn, reproducing to 0.15% across loads); the MODEL is not. It became the default 27B by decision on 2026-08-17, not by beating qwen36-27b-128k on real work — that head-to-head has never been run — so every quality claim below is the vendor's and not this box's.
• 96k, not the 128k the weights support: the drafter costs 2184MiB and at 131072 that leaves 883MiB, under the 1500 margin and under the 414MiB by which this model's own footprint varies between loads. qwen38-27b-128k is the same weights with the window back and the drafter off.
• Leaves 2291MiB free with the resident set up — enough for the margin, not enough for a warm ComfyUI's 3900MiB beside it. Either can have the card; not both.
• Speculative decoding is lossless by construction for greedy decode, but llama.cpp #25618 reports draft-mtp diverging from vanilla on QUANTIZED targets, and this is a Q4_K_XL target. qwen-dense has run draft-mtp on the same quant for months, so this is a known-shape risk here rather than a new one.
• Thinking defaults to reasoning_effort: xhigh, the most deliberative setting there is, and the consistent early complaint is over-thinking. Turn it down per-request with --chat-template-kwargs '{"preserve_thinking":true,"reasoning_effort":"medium"}' — but reports agree the long reasoning is carrying much of the quality, so it is a real trade rather than free speed. Use xhigh, medium or low only: the vendor also documents none, and on the template this box serves it silently means xhigh.
• Qwen asks for 262k of reasoning tokens and 131k of final answer on agentic work; this box caps generation at 8192 for every model, after a runaway stream once wedged llama-swap. An xhigh trace can meet that ceiling on a hard question.
• Its card reports SWE-bench Pro and Terminal Bench 2.1; the 77.2 / 59.3 quoted on qwen36-27b-128k are SWE-bench Verified and Terminal-Bench 2.0. Those pairs are NOT comparable — the head-to-head above is the only honest comparison, and it is the vendor's own.
• Day-one model: the official chat template needed a community fix on release day (it crashes on OpenAI-style string tool arguments), one report has unsloth's UD-Q8_K_XL looping in its thinking block, and quants may be re-uploaded as bugs surface.

Compared with

• qwen-dense — The same idea one generation back: 3.6 weights with MTP on, off by default because the f16 KV its drafter needed put it at 80k and over the card. 3.8 did not inherit that trade — measured here, q8 KV holds 71.5% draft acceptance, so this pays 96k rather than 80k and keeps the cheaper cache. Same artifact as qwen38-27b-128k either way; the 3.6-era second download does not exist in this generation.
• qwen36-27b-128k — The same weights one generation on, at the same q8 KV, and its card puts it ahead on everything it measures — the agent benchmarks by a lot. Both are measured now, but NOT against each other: 134 tok/s here was read on b10472 with the drafter on, and that entry's ~47 on an older engine without one, so the gap is engine and speculation as much as model until somebody re-runs the pair back to back. That one held the dense alias until 2026-08-17 and is still served, unchanged — going back is one line.
• qwen36-35b-a3b — qwen36-35b-a3b is far faster (~270 tok/s) with a 2x window and is still the reflex choice; this is the deep 27B for when the answer has to be right. Same split as qwen36-35b-a3b vs qwen36-27b-128k, with the 27B side upgraded.
• qwen38-125b-a6b — Measured against each other here on 2026-08-28 and they tied: 6/6 apiece on the champion/challenger suite, with this one three times faster over the six cases (23.7s against 70.2s). The suite has a ceiling — both ace it — so that is evidence the bigger model's 2-bit quant costs nothing at this difficulty, and no evidence about which is better. On Qwen's own card this one loses all twelve — the agentic ones badly (DeepSWE 1.1 42.2 against 58.7, JobBench 33.4 against 55.7). What this one has is 4x the generation (134 tok/s against 34.7), weights that fit on the card with no host RAM at all, and an engine from the channel rather than a hand-pinned PR build. Stay here for anything interactive; go there when the answer is worth waiting for.
• qwen38-27b-128k — The same file at 128k with the drafter off. This one is 1.81x faster (134 vs 74 tok/s, measured back to back on the same afternoon); that one is the 25% of window this one sold to buy the speed. Sessions overflow this way automatically, so pick that one only when you know the prompt is over 96k.

qwen38-27b-uc-128k — Qwen3.8-27B abliterated (thinking) — what a new chat opens on

The uncensored dense 27B, and what a new chat opens on — a refusal is the one wrong answer a chat frontend cannot route around. It is a lossy edit of exactly qwen38-27b-96k's weights, so name qwen38-27b-96k instead when refusals are not the problem. Answers to qwen-dense-uc, the 3.6 abliteration it replaced on 2026-08-21.

Outgrowing its window reroutes to qwen38-27b-uc-208k rather than hard-erroring.

Best at

• The 3.8 generation's gains without the refusal behaviour — the base model beats Qwen3.6-27B on every benchmark its card reports.
• Same huihui lineage and method as the 3.6 abliterations here, which is why it is this one rather than one of the dozen same-day community uploads; qwen36-35b-a3b-uc is still served on that lineage and still carries its KL 0.0074.
• Q5_K_M, a higher quant than the 4-bit variants here, at 128k for ~4.3GB of KV.
• Vision untouched by the ablation and the mmproj ships in the same mirror.

Costs and limits

• Needs llama.cpp b10434 or newer and is dropped below it, for the same silent-crash reason as qwen38-27b-96k.
• Footprint and throughput are measured (69 tok/s generation, ~3570 prefill, 23746MiB); the MODEL is not. It took webui.defaultModel and the caption model from the 3.6 abliteration on 2026-08-17 and retired it outright on 2026-08-21, by the same decision that moved dense onto qwen38-27b-96k — newer weights, no head-to-head against the model it replaced. It is also confirmed the heaviest of the 27Bs at Q5_K_M and has the least headroom beside the resident set: 1699MiB free, which is 199MiB over the margin and cannot host a warm ComfyUI.
• Abliteration is lossy: qwen38-27b-96k is the higher-fidelity version of the same weights at the same 128k.
• No published KL or capability delta for this ablation, where the 3.6 one had KL 0.0074 against its base. Same author and method is a reason to trust it provisionally, not evidence about these weights — and since 2026-08-21 there is no measured-KL uncensored 27B here to fall back to, only qwen36-35b-a3b-uc at 35B-A3B.

Compared with

• qwen36-27b-128k — The 3.6 dense 27B, not abliterated, at the same 128k. Since 2026-08-21 it is the only 3.6 27B left here — its abliteration was this entry's predecessor and is retired — so the choice against it is generation and refusals at once rather than refusals alone.
• qwen36-35b-a3b-uc — This is the dense 27B (deeper, half the window); that is the 35B-A3B MoE (~260 tok/s, 256k) and is still the only fast uncensored option, because 3.8 has no MoE.
• qwen38-27b-96k — Same weights, not abliterated, same 128k window. Prefer that one unless refusals are why you are here — the context was never what made this one worth picking.

qwen38-27b-uc-208k — Qwen3.8-27B abliterated (thinking), 208k — unloads speech-to-text

qwen38-27b-uc-128k's weights at the widest window this card can hold, bought by unloading speech-to-text. The wide entry for sessions where a refusal is the actual problem.

Best at

• 208k of shared KV pool on the uncensored 27B — the only uncensored dense option here that is not stuck at 128k.
• Same file as qwen38-27b-uc-128k: no second download.
• RAG and durable memory keep working; only asr yields.

Costs and limits

• Speech-to-text is down for everybody on the box while this is warm, and does not return on demand.
• 16384 tokens narrower than qwen38-27b-224k for the heavier quant, and abliteration is lossy either way — qwen38-27b-224k is the higher-fidelity version of the same weights.
• 27266MiB is EXTRAPOLATED, not measured.
• No published KL or capability delta for this ablation, inherited from qwen38-27b-uc-128k.

Compared with

• qwen36-35b-a3b-uc — 262144 and ~260 tok/s on the 35B-A3B MoE, with no eviction cost to anybody. Faster and wider; a generation back and not dense.
• qwen38-27b-224k — The same trade on the un-abliterated weights, 16384 tokens wider and higher fidelity. Name that one unless refusals are why you are here.
• qwen38-27b-uc-128k — The same file at 128k, coexisting with everything. Prefer it unless a second stream is live.

qwen38-27b-uc-256k — Qwen3.8-27B abliterated (thinking), 256k — card to itself

The abliterated dense 27B at the full 262144 the weights support, with nothing else on the card. The default, and the widest window this box can serve. Retrieval, durable memory and transcription all swap it out while it runs.

Best at

• The model's whole native window — 262144, where every other variant here is capped by what had to fit beside it.
• 256k of shared KV pool, so a conversation and background subagents have room without a second model.
• Same file as the narrower variants: no second download and no second 17GB of disk.
• Every benchmark on the 3.8 dense weights applies unchanged — identical weights, identical sampling.

Costs and limits

• Needs the card to itself: 29378MiB of 32607 leaves no room for embed (1698), rerank (1756) or asr (3708).
• So retrieval, durable memory and speech-to-text each cost a model swap, and swapping back is a FULL re-prefill — a hybrid cannot rewind recurrent state. Use a narrower variant for sessions that lean on those.
• No drafter, so ~74 tok/s rather than the 96k variant's 134.
• 29378MiB is EXTRAPOLATED from the 704MiB/16384 KV linearity, ~2x past the range it was measured over. Nothing has loaded this entry yet.
• Needs llama.cpp b10434 or newer, and is further past #27090's cliff than any other entry here.

Compared with

• qwen38-27b-128k — The same file at 128k. Narrower again, and the point of it is speed and company on the card rather than window.
• qwen38-27b-96k — The same file at 96k WITH the drafter: 1.81x the generation, a third of the window.
• qwen38-27b-uc-208k — The same file 32768 tokens narrower, which is exactly what embed and rerank cost. Go there when a session leans on retrieval or durable memory.

voice — Qwen3-4B-Instruct-2507 (on the CPU)

The realtime voice model, and it runs on the CPU — which is the whole reason a spoken turn never has to fight qwen3-coder-80b-a3b for the card.

Best at

• Zero VRAM. Always available, whatever is loaded on the GPU.
• 0.15s to first token and 12-14 tok/s measured: speech is spoken at ~3 words/second, so it stays three times ahead of the speaker.
• Serves a GGUF the store already has (the Z-Image text encoder) rather than downloading a second copy.

Costs and limits

• A 4B instruct model — it handles voice turns, not hard questions.
• Completion only: no tools, no vision, no thinking.

Compared with

• qwen36-35b-a3b — qwen36-35b-a3b is vastly more capable and needs the GPU. This exists so that answering "turn on the lights" out loud never evicts somebody's warm session.

asr — Qwen3-ASR-1.7B

Speech-to-text behind Open WebUI's mic button and media-tools' transcribe_audio. Resident, so a transcription never evicts a warm coding session.

Also answers to whisper-1.

Best at

• Serves OpenAI's /v1/audio/transcriptions natively, and answers to the whisper-1 alias, so stock clients work unchanged.
• Its audio encoder ships as an mmproj in the same repo, so it is just another llama-swap model rather than a second STT daemon.

Costs and limits

• 3708MiB resident at 4k ctx — the single largest permanent claim on the card, and the reason the context has been cut twice.
• 4k is ~3 minutes of audio. media-tools' transcribe_audio sends whole files unchunked, so longer clips fail there — that is genai-transcribe's job.
• response_format must be json (no srt/vtt), and it cannot emit timestamps at all, so subtitles go through genai-transcribe.

Compared with

• voice — asr is the ear, voice is the mouth: this transcribes on the GPU, voice answers on the CPU.

embed — Qwen3-Embedding-0.6B

RAG embeddings. Resident by design, so a chat-model swap never evicts it and retrieval keeps working while the big models come and go.

Also answers to text-embedding-3-small.

Best at

• Multilingual, and small enough (1698MiB) to stay loaded permanently.
• -c 2048 caps the KV and compute buffers; uncapped it allocates for the model's full context and balloons to ~5GB.

Costs and limits

• Not a chat model — hidden from model pickers because selecting it can only fail.
• Resident means never evicted, not VRAM reserved for free: its 1698MiB is permanently unavailable to the biggest chat model.

Compared with

• rerank — The two halves of hybrid retrieval: embed finds candidates by meaning, rerank puts them in order.

rerank — Qwen3-Reranker-0.6B

Cross-encoder reranking for hybrid RAG — it scores the candidates that BM25 and embeddings surfaced. Open WebUI's hybrid search reranks through it.

Best at

• Jina-compatible /v1/rerank: {model, query, documents[, top_n]} -> relevance-scored list.
• A cross-encoder reads query and document together, so it catches relevance that independent embeddings miss.

Costs and limits

• 1698MiB measured, not the ~0.7GB the weights suggest — and it is resident, so that is card the big models never get.
• Not a chat model; hidden from pickers.

Compared with

• embed — embed retrieves, rerank orders. Hybrid RAG runs both, which is why they share the resident group.

asr, embed and rerank serve non-chat endpoints (they sort last in the table above for that reason), so webui.utilityModels hides them from the dashboard proxy's /v1/models — otherwise they show up as selectable models in Open WebUI's chat picker (and LiteLLM's model list), where picking one can only fail. Hiding is listing-only: they stay fully routable, which is what lets Open WebUI's reranker keep POSTing {"model": "rerank"} to that same proxy. webui.utilityModels = [ ] shows everything again; adding "fara" also drops it from the picker, at the cost of breaking MagenticLite, which reaches it through this proxy.

That section is generated, and this is the only copy of it. Everything above between the model-fleet markers is rendered by docs.nix from each catalog entry's serve.guide in options.nix; the same fields are published by /api/status and render as the expandable panels on the portal's /models page. Advice about which model to pick used to live here and in the module, which is exactly the kind of duplication that comes apart quietly — the interesting entries are the ones that changed after a measurement (qwen-dense went off by default when the resident set grew; rerank turned out to cost 1698MiB rather than the ~0.7GB of its weights), and an edit in one place was invisible from the other.

So: edit serve.guide, never this section. Then

nix run .#update-readme          # re-render the block
nix build .#checks.x86_64-linux.docs   # or just `nix flake check`

The check fails the build if the committed text and the catalog disagree, in either direction, and prints the diff plus the command that fixes it. Two smaller guards ride along: versus keys must name real catalog entries (a module assertion, so a renamed model cannot leave dangling advice), and every shipped model must carry a summary (or it would silently vanish from the table). Context windows are not written into guide.footprint — renderers append them from serve.context, which module.nix already pins to the -c in the shipped llama-swap command.

Rule of thumb: dense is the 27B an unnamed session gets, and since 2026-08-17 it resolves to qwen38 — the 3.8 weights with their own in-file drafter, rerouting to qwen38-long when a prompt outgrows the window. Thinking models are safe on the Anthropic protocol as of 2026-08-02: /v1/messages routes through chat-completions and the streaming adapter is patched, so reasoning_content arrives as a real thinking block — the older advice to keep Claude Code on a non-thinking model is obsolete, and the anthropic-bridge eval suite is what keeps it that way. coder-pro is the pick when the window is the constraint (256k, agent-RL-trained, non-thinking), which is why OpenCode defaults to it here. glm-flash when iteration speed matters more than depth. qwen is the fast default for everything else (73.4 SWE-V at ~6× dense speed); research for knowledge-heavy queries. qwen-dense-long is the 3.6 model this box ran on until that date and the thing to fall back to if the newer pair disappoints — the two generations have never been A/B'd here, so the promotion is a decision, not a result. All chat models have web-search + media tools attached and use native function calling. (The former coder Qwen3-Coder-30B slot was retired: obsoleted by qwen-dense/glm-flash, and its two 64k slots silently truncated long agent prompts.)

fara is the odd one out: a computer-use agent, not a chat model. The intended frontend is MagenticLite on :8895 (the magentic-ui service): it runs Fara as the browser-use model and qwen as the orchestrator (tunable via the magenticUi.* options), with the agent's browser inside a Quicksand QEMU micro-VM. The two roles swap on llama-swap at agent-round boundaries — a few seconds each from page cache; set magenticUi.agentMode = "websurfer_only" to eliminate swapping entirely. Ad-hoc alternative: uvx fara-cli --base_url http://<host>:8080/v1 --api_key none --model fara "book a table for two". fara has no LiteLLM context fallback on purpose — a CUA session degrading to a chat model would emit garbage browser actions.

Beyond /v1/chat/completions, /v1/embeddings and /v1/rerank, llama-server also exposes legacy /v1/completions and /tokenize — all routed by model name through llama-swap (:8080) and the dashboard filter proxy (:8897).

Models land in /var/lib/genai-models/llm (part of the unified model store — see Storage below). On activation/boot the genai-models-prefetch service downloads any missing model blobs with aria2 (16 parallel connections — llama-server's own downloader is single-stream and much slower), so a fresh box warms itself automatically. genai-prefetch <repo[:tag]> runs the same tool manually. Small mmproj sidecars are still fetched by llama-server on first use.

Quant authors (esp. unsloth) sometimes re-upload fixed GGUFs after llama.cpp correctness/tool-parsing fixes — a stale file silently degrades coding. To force-refresh one model: sudo rm -rf /var/lib/genai-models/llm/models--<org>--<repo> then re-run genai-prefetch <repo:tag> (or wait for the boot prefetch).

Qwen thinking models run with preserve_thinking=true so agentic loops keep prior reasoning in context. MoE models bigger than VRAM use --n-cpu-moe N (lower N until CUDA OOM, then back off). Sampling flags follow vendor recommendations — see comments in module.nix.

All Qwen 3.5/3.6 chat models run with a fixed chat template (vendored in templates/, from froggeric/Qwen-Fixed-Chat-Templates): the stock template makes the model occasionally emit an empty tool call, which agent clients (Claude Code, OpenCode) read as "task complete" and silently stop mid-session. Requires llama.cpp ≥ b9925 (llama-server --version; a NixOS assertion enforces this at eval time).

The resident set is a standing VRAM tax, and it does not fit with everything

embed, rerank and asr are in the resident group, which means never evicted — so their cost is subtracted from every chat model, permanently. Measured 2026-08-03 on the 32607MiB card:

	measured
embed (-c 2048)	1698MiB
rerank (-c 2048)	1698MiB
asr (4k ctx)	3708MiB
resident total	7104MiB
leaves for a chat model	25503MiB

Against that budget, qwen-dense needs 25784MiB (80k f16 KV, which f16 is there to keep MTP draft acceptance high) and qwen-dense-long needs 23526MiB (measured alone on an empty card). So:

• qwen-dense and asr cannot both be loaded. Whichever arrives second dies with upstream command exited prematurely — llama.cpp fast-failing a CUDA allocation, not a llama-swap fault. resident means "never evicted", not "VRAM reserved", so this is order-dependent: transcribe first and the 27B is locked out; load the 27B first and transcription is. qwen-dense therefore ships disabled. Turn it on with:

services.genai-server.llmModels.qwen-dense.serve.enable = true;

accepting that transcription, voice, RAG and memory lock it out (and it locks them out) for as long as either side is resident. asr's share of that budget is transcription.maxAudioMinutes — ~152MiB of VRAM per minute of audio, defaulting to 3.

• qwen-dense-long fits with ~1977MiB spare, inside the 1.5GB margin this repo keeps. It was 537MiB before asr dropped from 16k to 4k.
• Everything smaller (qwen, glm-flash, coder-pro, minimax) coexists fine, because their weights or their CPU offload leave more room.

Dropping embed and rerank from -c 8192 to -c 2048 recovered 1356MiB of this and is why qwen-dense-long fits at all. There is no similar free win left: qwen-dense's weights alone are ~20GB, so no context setting makes it coexist without also giving up MTP. That is a deliberate open tradeoff, not an oversight — the speech-under-load eval exists to keep it visible.

Virtual model IDs (selectors)

A selector is a name clients ask for that resolves to a real model per request (needs llamaSwap.useNewerBuild until nixpkgs ships >= v242):

services.genai-server.llamaSwap.selectors.coder = {
  strategy = "warm";                       # warm | pin | spillover
  targets = [ "coder-pro" "qwen-dense" ];
  description = "Whichever coding model is already loaded";
};

Two uses. A/B without touching clients: reorder targets to promote a challenger and Claude Code, OpenCode and Open WebUI all follow. Not evicting a warm model: strategy = "warm" picks a target already in VRAM, which on a single shared card is usually worth more than the difference between two good models.

Selectors show up in /v1/models (tagged type: selector) and are added to the portal's allowlist automatically — they are not catalog entries, so otherwise the availability filter would hide them.

Adding an LLM

A catalog entry, not a code change. Set serve.enable = true and the llama-swap block is generated from serve.*:

services.genai-server.llmModels.my-model = {
  repo = "unsloth/Some-Model-GGUF";
  tag  = "Q4_K_M";
  serve = {
    enable   = true;
    context  = 65536;          # -c; the VRAM lever (KV scales linearly)
    preset   = "qwen-thinking";# vendor sampling preset
    kvQuant  = "q8_0";         # halves KV VRAM, near-lossless
    nCpuMoe  = 24;             # MoE experts kept in host RAM
    aliases  = [ "mine" ];
    ttl      = 900;
    capabilities = [ "completion" "tools" "vision" ];
  };
};

It joins serve.group (main swaps, resident stays loaded), gets prefetched with everything else, and shows up on /models with its capabilities.

A catalog entry is not the whole job for a chat model. Open WebUI keeps tool attachments and vision flags in its own config database, seeded from two lists that serve.capabilities does not drive:

declared	also add it to	or it will
"tools"	webui.toolModels	get no tool servers — no image, video, or web search
"vision"	webui.visionModels	fail on attached images instead of seeing them

These are separate lists on purpose — fara is tool-capable and excluded because the media tools collide with its browser-action space — but the cost is that nothing warns you. A model missing from toolModels does not error; it answers "I don't have image-to-image or video tools," which is true of what it was handed and false about the box. Both lists take effect on the next rebuild, when the seeder runs.

Other fields: type (chat / embedding / rerank / transcription — picks the server mode), mmprojUrl (only for repos whose HF manifest hides their projector), shardFile (split GGUFs — see Big models), minVramGB / minRamGB (hardware floors below which the entry is omitted rather than left to fail), draft (MTP or a draft repo for speculative decoding), stripPenalties (for models whose tool-calling breaks under client-injected penalty samplers), extraFlags (appended last, so it wins) and rawEntry (llama-swap keys the submodule doesn't model).

The shipped models keep hand-written llama-swap entries in module.nix because their flags encode measured VRAM math that a submodule can't carry — they still fill in serve.* as the metadata /models, /api/status and Ollama-dialect clients publish. Those two could drift, so a NixOS assertion checks every declared serve.context against the -c in the command that actually runs. If you set serve.enable on a name that is hand-written, the hand-written entry wins and you get a warning saying so.

Big models: sharded GGUFs

Past roughly 50GB a quant is published as a split GGUF, and such a repo has no HF manifest at all — the endpoint answers 400 The specified repository contains sharded GGUF. That is the same endpoint genai-prefetch and llama.cpp's own -hf resolve through, so repo:tag cannot name these models. Declare the first shard instead:

services.genai-server.llmModels.big = {
  repo = "unsloth/MiniMax-M2.7-GGUF";
  tag  = "UD-IQ3_XXS";                 # the quant DIRECTORY, not a tag
  serve.shardFile =
    "UD-IQ3_XXS/MiniMax-M2.7-UD-IQ3_XXS-00001-of-00003.gguf";
};

genai-prefetch enumerates the remaining parts from the tree API into /var/lib/genai-models/llm/sharded/<repo>/<quant>/, and the model is served with -m <first shard> — llama.cpp finds the siblings by name. An assertion checks that shardFile lives in the directory tag names, since those two are one path spelled in two places. The portal reports such a model ready only when every part is present; a partial set reads absent, because llama.cpp opens the first shard and then dies demanding the rest.

Downloads resolve the CDN redirect before handing the URL to aria2, and that detail is load-bearing: HF's Xet-backed CDN signs a redirect with a byte-range condition whenever the request that triggered it carried a Range header, so aria2's 16 parallel ranges against one resolved URL get 403s and eventually stall outright. Resolving with a plain request yields a URL valid for every range. Don't "optimise" that step away.

Sizing is not file size. Two numbers decide whether a big MoE is usable, and the second one is the one people miss:

• Total size → whether it loads. serve.minRamGB is the guard, and it is separate from minVramGB on purpose: --n-cpu-moe makes a model fit on a small card by construction, so it clears the VRAM floor and then wants 60-90GB of host RAM nothing asked about. Below that floor the failure is swap thrash, not a clean OOM — it reads as "the model is slow".
• KV cache architecture → how much card is left for weights. A hybrid or MLA model (coder-pro, glm-flash, the Qwen3.6s) carries almost no KV, so context is nearly free. A full-attention model does not: minimax spends 8.3GB on 64k, which is what forces it to the 3-bit quant. Read the GGUF metadata (block_count, attention.head_count_kv, key_length) before assuming a quant fits.
• Active params → speed. Only active experts are read per token, so throughput is roughly host RAM bandwidth ÷ active bytes. On a single-CCD Zen 5 that bandwidth is ~60-65GB/s regardless of the DDR5 rating: 3B active ≈ 20-35 tok/s, 5B ≈ 30, 10B ≈ 17 measured. Treat that arithmetic as a lower bound — it ignores the layers still resident on the card, which is why minimax measured 17 tok/s against a predicted 10-14.

Derive nCpuMoe from measurement, not arithmetic. The component costs are what matter, and on this box they are: KV as computed from the GGUF header, compute buffers ~1.1GB at -ub 2048, non-expert weights ~3.2GB, and ~1.2GB per layer of experts. A first guess that ignored a warm ComfyUI's ~3.9GB residual put minimax at 51 and it OOM'd on the KV allocation; the real number is 57. Tune against a warm box, not a freshly rebooted one, or you ship a model that loads in the morning and OOMs by evening.

Knowledge collections (RAG)

http://<host>:8897/rag manages them; :8900 is the API. Retrieval is hybrid — BM25 catches exact terms and identifiers, embeddings catch paraphrase — fused with reciprocal-rank fusion and reranked by the resident cross-encoder.

curl :8900/ingest_url  -d '{"collection":"docs","url":"https://example/page"}'
curl :8900/ingest_text -d '{"collection":"docs","text":"...","uri":"note:1"}'
curl :8900/search      -d '{"query":"how do I free the GPU?","limit":5}'

Results carry score and reranked. The score is the reranker's relevance when reranking ran and the fusion score otherwise — it always explains the order — and reranked: false means the cross-encoder was unavailable and you are seeing fusion order, rather than silently pretending.

Re-ingesting the same uri replaces that document instead of duplicating it, so refreshing a source is just running the ingest again. Filesystem ingest is off unless rag.ingestRoots lists directories; paths are resolved with realpath, so .. and symlinks cannot escape them.

It publishes openapi.json, so Open WebUI can register it as a knowledge tool and the MCP gateway exposes every operation automatically — including drop_collection, which is marked destructive and needs confirm: true.

Running code

:8903 executes code in a disposable QEMU microVM — a fresh VM per run, destroyed afterwards. Chat models and MCP clients both get it as a tool, so they can compute an answer instead of guessing one.

curl :8903/run_code -d '{"code":"print(sum(range(101)))"}'
curl :8903/run_code -d '{"code":"...","language":"bash"}'

Pass data in with files ({"in.csv": "..."}) and set return_files to get back what the code wrote. ~45ms per run on a warm-pool hit.

What it cannot do, by construction: see this machine's filesystem, keep state between runs, or reach the network. Network needs two gates — the host permitting it (codeSandbox.allowNetwork, default false) and the request asking. Think before opening that: it is model-written code, from text a user or web page supplied, gaining outbound access from inside your network. codeSandbox.allowedDomains narrows it.

Requires /dev/kvm. Without it the service answers 503 rather than quietly running code unsandboxed.

Memory

:8902 stores short durable facts a model can recall in a later conversation — distinct from the document collections above.

curl :8902/remember -d '{"text":"Prefers metric units and 24-hour time"}'
curl :8902/recall   -d '{"query":"what units should I use?"}'

Better still, hand it a conversation and let it decide what is worth keeping:

curl :8902/observe -d '{"text":"User: I always use NixOS, never Ubuntu..."}'

Corrections replace what they correct. Each new fact is shown to a model alongside the memories nearest to it, which picks ADD / UPDATE / DELETE / NOOP — so "switched to imperial" deletes "prefers metric" instead of sitting next to it. A similarity threshold cannot do this: those two are only 0.73 apart, and the stale one would otherwise be returned first.

It never takes the GPU from you. That reconciliation runs only against a model llama-swap already has resident; otherwise the work queues and a cheap 60s poll picks it up when the card is next warm. Facts are stored before they are reconciled, so a busy GPU delays a correction but never loses a write. forget takes an id or exact text — never a fuzzy match — and is gated behind confirm: true over MCP.

One store, not two. Open WebUI 0.11 ships its own memory, and running both would give the box two divergent sets — chat writing to one while Claude Code reads the other. So rag-style, exactly one is on: this service is registered with Open WebUI as a tool server and ENABLE_MEMORIES=False turns its built-in memory off. Chat models still get remember/recall; Claude Code over MCP sees the same memories. Browse and edit them at :8897/memory. Reverse the choice with services.genai-server.memory.useAsWebuiMemory = false.

Namespacing is only as strong as the identity you can trust. Memories are scoped by a header (memory.identityHeaders, default including Open WebUI's X-OpenWebUI-User-Id), never by a field the model fills in — a model that can name its own namespace can read every other one. Over MCP the gateway carries the name rather than deciding it: it reads mcp.identityHeaders from the caller — believed from loopback and identity.trustedProxies only — and passes it on, so LibreChat's per-user connection and a claude mcp add -H each land in their own namespace. A client that says nothing is nobody and shares memory.defaultNamespace. That is fine for a single-user box; it is not isolation between users who distrust each other.

MCP clients (Claude Code, Claude Desktop, Zed)

The same tools chat models use, over MCP — no second implementation. Point a client at http://<host>:8899/mcp (or http://<host>:8897/svc/mcp/mcp to keep one origin for the whole box):

claude mcp add --transport http genai http://<host>:8899/mcp \
  -H "X-Genai-User: you@example.com"      # who the tools act for
curl http://<host>:8899/tools     # what's bridged right now

The tool list is per person, and that is why this exists beside Open WebUI. A tool server publishes one OpenAPI document for the whole deployment, fetched once with nobody attached, so the menu it produces is a property of the box. MCP's tools/list is a per-session call, so the gateway fetches each document as the caller (mcp.identityHeaders, believed from loopback and identity.trustedProxies only) and passes the name on to the call — which is what scopes a memory to its owner, a render to whoever asked for it, and the engines and LoRAs on offer to what that person is allowed to be shown. One decision, made once, in the tool server: a tool you are not offered has no route here either.

Say nothing and you are nobody — the tool servers apply their default policy, which is a stranger's rather than the owner's. Claim a name from an address that is not trusted and it is ignored for the same reason: every upstream believes what the gateway tells it, so a header from anywhere would make the LAN a way to be anybody.

Tools come from the tool servers' OpenAPI documents — web_search, generate_image, generate_image_with_face, swap_face, swap_face_full, swap_face_fast, edit_image, reimagine_image, inpaint_image, smart_edit, create_mask, list_regions, transcribe_audio, text_to_speech — so adding a tool to a tool server makes it appear here with no MCP-side change.

Destructive operations get MCP's destructiveHint annotation and a required confirm: true argument, so a model can't trigger one by accident. Nothing shipped is destructive; mcp.upstreams.<name>.destructive marks the ones that are when a host adds its own tool server:

services.genai-server.mcp.upstreams.rag = {
  url = "http://127.0.0.1:8900";
  prefix = "rag_";                    # avoids operationId clashes
  destructive = [ "drop_collection" ];
};

Browsers must be same-origin or listed in mcp.allowedOrigins (the spec's DNS-rebinding guard); non-browser clients send no Origin and are unaffected.

Ollama-only clients

Point anything that speaks Ollama at http://<host>:11434 — Home Assistant's conversation agent, the JetBrains AI plugin, mobile apps. It is a translation layer on top of the same llama-swap fleet, not a second model server, so ollama is never installed and the model store is not duplicated.

curl http://<host>:11434/api/tags                       # the ready+enabled catalog
curl http://<host>:11434/api/show -d '{"model":"qwen"}' # capabilities, context
curl http://<host>:11434/api/chat \
  -d '{"model":"qwen","messages":[{"role":"user","content":"hi"}]}'

/api/show reports each model's capabilities (completion, tools, vision, thinking, embedding, rerank) from its serve block, which is what Ollama clients auto-route on — a vision model that didn't advertise vision would simply never be sent an image. Aliases work here too (dense, whisper-1, uncensored).

Ollama's API has no notion of who is asking and its clients send none, so a request that arrives with no Authorization is attributed to the dialect's own service identity — enough to run the fleet, and not a person. A client that does send one (put a key from identity.extraTokenFiles in whatever the app calls its API-key field) is that person instead, and its work is attributed, filtered and arbitrated exactly as a signed-in session is. That makes :11434 the same kind of door as :8893/:8894: anything that can reach it can use the models, so put it only on an interface you would hand the models to.

Models stay declarative. /api/pull starts a download only for a name already in llmModels; anything else returns an error naming the option to add it to. /api/create, /api/copy, /api/push and /api/delete answer 501 — the git-tracked catalog is the gallery. Disable the whole dialect with services.genai-server.ollama.enable = false.

Phone and desktop chat apps

A browser signs in; an app cannot. Every identity on this box otherwise arrives as a header from whatever gate is in front of it, and getting one means following a redirect, answering the provider and coming back with a cookie — which a phone chat client, a desktop app or a script will not do. Pointed at a gated host they are handed a login page where they expected JSON; pointed at the LAN they are nobody, and nobody cannot write, which is every inference call.

So the portal mints keys. Open /keys, name one after the device it is going on, and put it in whatever the app calls its API key field alongside the portal's own /v1 as the base URL:

https://<portal>/v1        # or http://<host>:8897/v1 on the LAN

A key acts as the person who made it and reaches exactly what they reach — their gallery, their availability filter, their GPU leases, their admin standing or lack of it. It is not a second class of access.

The context window comes with the model list. /v1/models carries each model's window as context_length, max_model_len and max_context_length — the same number under the three names vLLM, OpenRouter and LM Studio each invented, because OpenAI's own listing has no field for it. A client that reads any of them sizes its history correctly with nothing typed in; one that reads none assumes a default (commonly 128k) and truncates or overflows against a number nobody here chose. All three are derived from serve.context, which is already pinned to the -c the model is served with.

Pick the "OpenAI-compatible" provider, not "OpenAI". An app that offers both will let you set a custom base URL under either, and the difference is invisible until nothing renders: the "OpenAI" setting may speak the Responses API (POST /v1/responses), which this box answers correctly — reasoning and message arrive as separate output items — but which such a client often mis-parses when the first item is a thinking block. The compatible setting speaks /v1/chat/completions. Measured here with GPTMobile: identical endpoint and key, blank replies under one and working replies under the other.

One per device, not one per person. Revoking is then one line and one device rather than every client that person owns. A key is shown once, when it is made; lose it and mint another.

Minting is always for yourself, revoking is not. Nobody can make a key naming somebody else — an administrator included, because a key naming another person is a way to be them, and that is the one power identity.admins is not. Taking one away is the opposite: your own from anywhere, anybody's if you administer the box, which is the same rule the GPU arbiter applies to cancelling a render. An admin sees who holds a key and when it was last used, never what they called the device.

identity.extraTokenFiles is the declarative sibling, for a key a HOST wants to pin (a <token> <person> file kept out of the nix store). People's own keys do not belong in a config: granting access should not be a rebuild.

Home Assistant voice (Wyoming)

Home Assistant speaks Wyoming, not OpenAI, so it gets a protocol bridge:

services.genai-server.wyoming = {
  enable = true;
  openFirewall = true;          # HA is normally on another machine
};

Then in Home Assistant: Settings → Devices & Services → Add Integration → Wyoming Protocol, host <host>, port 10300. Speech-to-text and text-to-speech both appear, ready to drop into an Assist pipeline.

This adds no models. STT forwards to the asr model llama-swap already serves; TTS to the same tts-server facade behind /v1/audio/speech. nixpkgs' wyoming-faster-whisper and wyoming-piper would each mean a second copy of a model this box already has, competing for the same card.

Voices default to Piper, which runs on the CPU — the right default for a voice assistant on a shared GPU, since answering "what's the weather" should not evict a 46GB coding model. Kokoro voices sound better and do use the GPU; add them deliberately:

services.genai-server.wyoming.ttsVoices = [ "en_US-lessac-medium" "af_heart" ];

Wyoming has no authentication. Anything that can reach the port can transcribe and synthesize, so openFirewall is separate from openFirewallGlobally and this should never be port-forwarded.

STT goes through the portal's /v1 proxy rather than llama-swap directly, because that is where the ASR preamble is stripped — see below.

Transcripts and the ASR preamble

Qwen3-ASR emits a structured header and llama-server passes it into the text field, so a raw transcription reads:

language English<asr_text>The quick brown fox jumps over the lazy dog.

The portal strips it on /v1/audio/transcriptions, which is why anything doing speech-to-text should go through :8897 rather than llama-swap directly — Open WebUI's microphone, the media-tools STT tool and Wyoming all do. Tune or disable it with portal.asrTextCleanup (a regex; "" passes the model's raw output through):

services.genai-server.portal.asrTextCleanup = "^\\s*language\\s+[A-Za-z]+\\s*<asr_text>\\s*";

Realtime voice (:8901)

services.genai-server.voice.enable = true;

Speech in, speech out, over one WebSocket — then open :8897/voice and talk. Distinct from /v1/audio/*, which is request/response: the value here is in the seams — noticing you stopped talking, answering before the reply is finished, and shutting up when you interrupt.

It costs no VRAM. The chat model (voice, a 4B) is served with -ngl 0, so it lives in system RAM and is permanently warm without taking anything from the card. Measured: 0.15s to first token at 12-14 tok/s — about three times faster than speech is spoken, which is the only throughput voice actually needs. It also reuses a GGUF already in the model store (the Z-Image text encoder) rather than downloading the same weights twice.

Measured end to end on this box, warm:

leg
transcription (asr, GPU, resident)	~30ms
chat + first-sentence synthesis	~960ms
detection → first audio	~990ms
plus the VAD hangover you actually wait through	+600ms default

So roughly 1.6s from falling silent to hearing a reply. The page shows both numbers, because the smaller one flatters the experience.

Two behaviours worth knowing:

• Synthesis starts at the first sentence, not the last. Waiting for a complete answer would blow the latency budget on its own.
• Barge-in. Talking over the reply cancels it mid-sentence. An assistant that keeps talking while being interrupted is worse than a slow one.

voice.vadHangoverMs (default 600) is how long the server waits before deciding you finished. Both directions of that trade are bad — too short cuts you off mid-thought, too long adds dead air — and it errs long because real speech pauses more than synthesized speech does.

Microphone access needs a secure context. Browsers refuse getUserMedia over plain HTTP from another machine, so /voice works at http://localhost:8897/voice on the box itself, or behind TLS. The page says so rather than failing mysteriously.

Coding agents (Claude Code / OpenCode)

Claude Code — point it at the LiteLLM bridge:

export ANTHROPIC_BASE_URL=http://logistikon:4000
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_MODEL=coder-pro             # primary; glm-flash for snappier loops.
                                             # NEVER a thinking model (qwen/qwen-dense):
                                             # the bridge drops reasoning_content
export ANTHROPIC_DEFAULT_HAIKU_MODEL=coder-pro  # haiku-tier background calls → same warm
                                             # model (otherwise they hit llama-swap as
                                             # claude-haiku-* -> 404 / pointless swaps)
export ANTHROPIC_SMALL_FAST_MODEL=coder-pro  # older name of the same knob, kept for
                                             # back-compat with older Claude Code
export CLAUDE_CODE_SUBAGENT_MODEL=coder-pro  # subagents stay on the warm model too
export CLAUDE_CODE_ATTRIBUTION_HEADER=0      # the attribution block mutates the prompt
                                             # prefix and silently defeats llama-server's
                                             # prefix cache (full re-prefill every turn)
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1  # cut parallel background calls:
                                             # they serialize on the single slot and
                                             # evict the prompt cache (minutes/turn)
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=16384   # smaller output reservation = more
                                             # prompt headroom before overflow
claude

Caveat: Claude Code estimates context with Anthropic's tokenizer, which undercounts Qwen tokens on code — a session it believes is at ~185k can really be at ~230k+, past even coder-pro's 256k. When that happens the LiteLLM fallbacks entry converts the overflow into a properly-typed context error so Claude Code compacts and continues (instead of a retry loop that pins the GPU — see the litellm comments in module.nix).

Claude Code assumes a ~200k context window and will not self-limit. coder-pro (256k) clears that, so sessions auto-compact normally. On the smaller-window models the LiteLLM context_window_fallbacks ladder is the safety net: an overflowing session silently continues on a larger-window model (qwen38 → qwen38-long → qwen, qwen38-uc → qwen-uc, qwen-dense → qwen, glm-flash → coder-pro, research → qwen).

OpenCode — declare the true per-model windows so it compacts before overflowing instead of dying; put this in ~/.config/opencode/opencode.json (routes through LiteLLM to keep the fallback net):

Per-model options matter: opencode sends NO temperature for custom models (the server-side --temp would apply) but force-sends top_p: 1.0 for any model id containing "qwen" — the explicit options below pin the vendor sampling and neutralize that. autoupdate off (nix manages the binary), compaction.prune trims old tool outputs before compacting, and disabling the title agent removes a concurrent request that evicts the single slot's prefix cache at session start. Do NOT set small_model — titles already run on the session's model; pinning one would create llama-swap churn.

{
  "$schema": "https://opencode.ai/config.json",
  "autoupdate": false,
  "compaction": { "prune": true },
  "agent": { "title": { "disable": true } },
  "provider": {
    "logistikon": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "logistikon (local)",
      "options": { "baseURL": "http://logistikon:4000/v1" },
      "models": {
        "coder-pro":     { "name": "Qwen3-Coder-Next 80B (256k)",     "limit": { "context": 262144, "output": 32768 }, "options": { "temperature": 1.0, "top_p": 0.95 } },
        "qwen-dense":    { "name": "Qwen3.6-27B MTP (80k, top coder)", "limit": { "context": 81920,  "output": 32768 }, "options": { "temperature": 0.6, "top_p": 0.95 } },
        "glm-flash":     { "name": "GLM-4.7-Flash (128k)",            "limit": { "context": 131072, "output": 32768 }, "options": { "temperature": 0.7, "top_p": 1.0 } },
        "qwen":          { "name": "Qwen3.6-35B-A3B (256k)",          "limit": { "context": 262144, "output": 32768 }, "options": { "temperature": 0.6, "top_p": 0.95 } },
        "qwen-uc":       { "name": "Qwen3.6-35B huihui (256k)",       "limit": { "context": 262144, "output": 32768 }, "options": { "temperature": 0.6, "top_p": 0.95 } },
        "qwen38-uc":     { "name": "Qwen3.8-27B huihui (128k)",       "limit": { "context": 131072, "output": 32768 }, "options": { "temperature": 0.6, "top_p": 0.95 } },
        "research":      { "name": "gpt-oss-120b (64k)",              "limit": { "context": 65536,  "output": 16384 }, "options": { "temperature": 1.0, "top_p": 1.0 } }
      }
    }
  }
}

Speech (TTS)

One OpenAI endpoint (:8892/v1/audio/speech), three engines picked by voice:

• en_US-lessac-medium (default) — Piper: CPU, faster-than-realtime; for UI read-aloud and anything latency-sensitive.
• af_heart, bm_george, jf_alpha, ... — Kokoro-82M: narration quality; use for audiobook-style output.
• clone:<name> — Chatterbox zero-shot voice cloning (MIT; beat ElevenLabs 63.75% in blind tests). Register a reference first: curl -T sample.wav http://<host>:8896/voices/<name> (10-30s of clean speech), then request voice clone:<name>.

Video subtitles (genai-transcribe)

genai-transcribe talk.mkv                 # -> talk.srt
genai-transcribe -l ja ~/Videos           # recursive; language forced
genai-transcribe -V noisy-recording.mp4   # add Silero VAD (rarely needed)

Directories are searched recursively for video files. A video is skipped if a .srt with the same basename already exists, and the .srt is only written after a fully successful run — an interrupted run leaves a .whisper.partial and resumes from the last decoded segment.

Why this uses whisper and not the asr model. Subtitles need timestamps, and asr has none: llama-server returns {"text": ...} and rejects verbose_json/srt/vtt with "Only 'json' response_format is supported". Everything that does not need timing — the mic button, transcribe_audio, Wyoming — still uses asr, which is better at it and always resident. This is a second engine for a capability the first cannot express, not a second way to do the same job.

It runs on the CPU, deliberately: a subtitle job lasts hours and the GPU is shared. Nothing here loads a model onto the card, and there is no daemon — it is a command, not a service.

Two behaviours worth knowing, because they are the difference between usable subtitles and subtly wrong ones:

• Audio is split at silences longer than 2.5s and each region transcribed separately. Whisper will otherwise emit a single cue whose text spans a long pause, so a line appears tens of seconds before it is spoken. It cannot do that if it never sees across the pause.
• -mc 0 disables context carry-over, which is what stops whisper falling into repetition loops over music and silence.

The model (large-v3-turbo, 1.5GB) is a mediaModels entry in its own whisper set, so a host that never transcribes a video never downloads it; the command fetches the set on first use via genai-fetch-media.

Audiobooks (genai-narrate)

genai-narrate book.epub              # -> book.m4b, chaptered
genai-narrate -v bm_george novel.md  # British male; .txt/.md converted first
genai-narrate -L                     # list voices

Kokoro-82M via audiblez: e-book in, chaptered .m4b out, next to the input. Chapter WAVs accumulate in a .narration.tmp/ work dir, kept on failure so a long run is not lost, removed on success.

On the CPU by default (--cuda is deliberately not passed). A novel runs an hour or two at ~60-100 characters/sec, and holding the shared card that long is not a fair trade. The environment self-installs on first use — torch and Kokoro, several GB — into /var/lib/genai-narrate, with the model weights going to the shared HF_HOME so Kokoro is fetched once for the box.

There is no "speak this aloud" mode: a shared server has no audio sink, and /v1/audio/speech already synthesizes Kokoro on demand — ask for a Kokoro voice (af_heart, bm_george, ...) and tts-server routes it. This command is for turning a book into a file.

Repairing older audiobooks (-r)

genai-narrate -r ~/Audiobooks        # directory, recursive
genai-narrate -r book.m4b            # or one file

Audiblez writes raw pcm_s16le into the MP4 container, which the ipod muxer refuses — so a .m4b produced before this was fixed is ~385 kbps of PCM that many audiobook players reject, still tagged with the placeholder title .. -r re-encodes to 64k AAC and restores the title from the filename, taking a minute per book instead of re-narrating for hours.

It only touches files that are actually broken (already-correct AAC with a real title is left alone, so it is safe to re-run), writes to a temporary file and only replaces the original on success, and refuses to replace anything if the chapter count came out lower than it went in — losing chapters turns an audiobook back into one long track. It needs only ffmpeg, so it never triggers the multi-GB environment install.

Non-epub input is converted with pandoc first, with a placeholder title: audiblez always narrates "{title} – {author}." before chapter 1, and an empty title is spoken aloud as the literal word "None". The real title is restored in the .m4b tags afterwards.

Media generation (ComfyUI)

ComfyUI runs as a service on :8188 (self-installs to /var/lib/genai-comfyui as the genai user on first start; model weights come from the shared store via extra-model-paths). comfyui.modelSets = [ "comfy" ] declares the recommended node-graph set (~90GB), which the unit fetches before it serves: FLUX.2 dev fp8 (image, best prompt adherence), Wan 2.2 14B t2v+i2v (video), LTX 2.3 distilled (fast video with synchronized audio). Use the built-in workflow templates. Z-Image stays on :8893 for fast API image gen. Music generation also runs in ComfyUI (native ACE-Step nodes) — no separate service.

The ComfyUI version is pinned by comfyui.rev (default v0.30.1), and the unit reconciles the checkout against it on every start — fetch, check the ref out, and reinstall requirements.txt only if the commit actually moved. Before this option the version was pinned by accident: the launcher cloned the default branch on first start and never touched it again, so what a box ran was whatever upstream HEAD was on the day it was first built, and moving it meant a git pull under /var/lib that no rebuild reproduces. Safe because everything the service owns (models/, custom_nodes/, user/, output/, .venv/) is in ComfyUI's .gitignore — only tracked source moves. Node packs pin separately, so raising this can land one on a core it does not support; the symptom is a node that stops loading, logged in journalctl -u comfyui.

Video with native audio: MiniMax H3 (comfyui.modelSets = [ "h3" ], ~63GB, and not to be confused with the minimax LLM — same vendor, unrelated model). A diffusion transformer that emits video and stereo audio — dialogue, effects and music — in one forward pass, driven by the stock video_minimax_h3_{t2v,i2v,r2v} templates. Needs comfyui.rev ≥ v0.30.0. 42.5GB of weights run on a 32GB card because the stages are sequential and --disable-smart-memory frees between them: text encoder (15.7GB, the NVFP4 build — this box is Blackwell), then the DiT (21GB), then the two VAEs. Native canvas is 768px on the short edge (capped 768×1344) at 24fps for ~15s; 2K is an upscale on top of that, not the sampler's working resolution. A clip takes minutes and holds the whole card — queueing it asks llama-swap to unload first, so expect a warm chat model to be evicted.

Two levels of idle reclaim, because they reach different memory. comfyui.idleReleaseMinutes (default 10) calls /free once the queue has been empty that long, which returns what torch allocated. What it cannot return is what torch never allocated: measured here, ComfyUI idle for two days after its last render reported 528MB of torch_vram_total while the process held 3.9GB of VRAM and 13GB of RAM — the CUDA primary context and the cuBLAS/cuDNN/torch kernel images loaded into it. No CUDA call frees those; only process exit does. (--disable-cuda-malloc is already on and is why this is 3.9GB rather than the 6-7GB it once was.)

So comfyui.idleStopMinutes (default 0, off) stops the service outright after that long with no client connected, and a systemd socket starts it again on the next request — measured 7 seconds. Turning it on moves ComfyUI to loopback behind that socket; the portal card links to the public port as before but health-probes the server directly, because probing an activation socket is indistinguishable from using it and would keep the service alive forever. A stopped ComfyUI shows as idle, not as a fault. Idleness is connections, not queue depth — read the option before enabling it, since a render whose client has disconnected can be cut short.

A wake is not a render, and nothing treats it as one: starting the service costs its ~7s and its CUDA context and nothing else. The unload that hands a render the whole card happens when a prompt is queued, inside ComfyUI itself. It used to happen on every start, which meant a health probe or a stray browser tab could end somebody's chat turn mid-stream.

Custom nodes are declarative. comfyui.customNodes is a table of node packs; on every start the unit clones or fast-forwards each one, installs its deps into the venv, links the model categories it reads by bare filename, and fetches its modelSet — so a node pack is a rebuild, not a setup command, and a restart with nothing changed does no pip work. The shipped set is ComfyUI-GGUF, PuLID-FLUX and ReActor (the last two are what the face tools on :8894 drive). Adding one:

services.genai-server.comfyui.customNodes."ComfyUI-Frame-Interpolation" = {
  repo = "https://github.com/Fannovel16/ComfyUI-Frame-Interpolation";
};

Install failures are logged and skipped rather than taking ComfyUI down — journalctl -u comfyui is where a node that did not build shows up.

Single-photo identity (PuLID-FLUX)

The "upload one selfie, get images of that person" technique commercial sites use: zero-shot identity conditioning, no training. InsightFace extracts a face embedding from one reference photo and PuLID injects it during FLUX.1-dev generation. Instant, but it tends to mirror the reference's angle/hairstyle and captures only the face — a trained LoRA (below) still wins on full likeness and pose variety, and the two stack (add a LoraLoaderModelOnly node between the UNET loader and ApplyPulidFlux with a flux-trained LoRA — the shared LoRA store is a ComfyUI loras path, so trained/added LoRAs appear in its loader nodes). Same courtesy applies as for LoRA training: get the subject's okay.

Nothing to set up: PuLID is a shipped comfyui.customNodes entry, so the comfyui unit clones the node into its venv and fetches the pulid model set (~19GB — PuLID needs FLUX.1; the FLUX.2 already in ComfyUI is a different architecture) before it starts serving. Drop it on a host that does not want the download:

services.genai-server.comfyui.customNodes."ComfyUI_PuLID_Flux_ll".enable = false;

Use: open http://<host>:8188, load pulid-flux-workflow.json from this repo (drag it onto the canvas), upload a clear frontal face photo in the "Reference face" node, edit the prompt (photo-caption style), Queue. The first queue auto-downloads EVA-CLIP + facexlib (~1.2GB), so it sits loading for a few minutes once; after that ~30-60s per image. Knobs on ApplyPulidFlux: weight 0.8-1.0 = identity strength; raise start_at toward 0.2 for more prompt freedom at some likeness cost.

From chat (Open WebUI): attach a photo of the person and ask in plain language — "make an image of this person skiing in the Alps". The model calls the generate_image_with_face tool, which drives the same PuLID workflow through ComfyUI's API (reference upload, generation, ~1-2 min; the LLM is unloaded during the job and reloads on your next message). Say "stronger/weaker likeness" to adjust the identity weight. Needs the comfyui service up with its node installed — the tool returns a clear error otherwise. For people you generate often, a trained LoRA (below) still gives better likeness than one-photo transfer.

A style LoRA can ride along: pass lora (or several in loras) and the bridge splices LoraLoaderModelOnly nodes between the graph's UNet loader and ApplyPulidFlux, so the LoRA patches the base weights and the identity adapter goes on top. Only flux LoRAs qualify — every PuLID base is FLUX.1-dev, and anything else is refused with the list of ones that fit. Both bases take it: measured at 20 steps / 1024², ~15s on the fp8 safetensors and ~24s on the Q8 GGUF (ComfyUI-GGUF patches the quantized UNet), against ~18s with no LoRA at all.

Face swapping (putting a face into an existing photo)

PuLID generates a new image of someone. Three tools instead put a face into a photo you already have — same two arguments (url = the photo being edited, source_url = whose face), different trade-offs:

Tool	How	Time	What survives
swap_face	SAM 3 masks the face, FLUX regenerates that region under PuLID identity, composites it back	~1-2 min	everything outside the mask, byte-for-byte; lighting and skin tone adapt
swap_face_full	no mask — the whole frame is re-sampled img2img under PuLID identity	~1-2 min	pose and composition only; clothing, background and detail are regenerated
swap_face_fast	ReActor: inswapper generates a 128px face and pastes it in, then a restorer sharpens it	seconds	the entire photo except the face box — no diffusion model is loaded, so the chat model stays resident (it is evicted only if the ONNX sessions cannot find ~2GB, then the swap retries)

Reach for swap_face_fast first: it is the classic face swap, it is ~100x cheaper, and it cannot drift because nothing is re-rendered. The PuLID paths earn their minutes when the two photos disagree on lighting or angle and a transplant looks pasted on; swap_face_full is the last resort for a face the mask cannot cover cleanly. All three are in chat, the studio form (:8897/studio) and the MCP gateway.

All three come up with the stack: the PuLID and ReActor node packs are shipped comfyui.customNodes entries, and their weights (~19GB and ~2.8GB) are fetched by the comfyui unit before it serves. No setup command.

swap_face_fast is also the most adjustable of the three, because ReActor exposes the parts a transplant is made of:

Argument	What it decides
model	the swap network — inswapper (baseline), reswapper-256 (open reimplementation, 256px), hyperswap-1c (best at an angle)
weight	how much of the swap to keep. The node returns the untouched photo alongside the swapped one and they are identical outside the face, so this crossfades the face alone — 0.5 is half-swapped, 0 is the original photo back
identity_mix	a different thing: averages the two people's face embeddings so inswapper aims at a third identity rather than dissolving between two renders. Measured on this box it is a weak dial — inswapper rebuilds the entire face patch from its own prior either way, so 12.5% and 100% land close together. weight is the knob for "less swapped"
restore_model / restore_strength	which restorer repairs the swapped face, and how much of its output is blended back. Drop the strength when the result looks airbrushed — GFPGAN smooths skin, gpen-512 keeps more of it
restore_fidelity	CodeFormer's quality/fidelity dial; ignored by the ONNX restorers
face_boost	upscale the swapped crop and repair it at full resolution instead of patching 128px in place. The quality knob; ignored by hyperswap
face_index / source_face_index	which face, as a comma list — "0,2" swaps two people in one pass, pairing them with the source faces in order
face_order / source_face_order	what those indices count through. The default is largest face first, so left-right is how you say "the second person from the left"
face_gender / source_gender	only touch faces detected as female/male — picking someone out of a group without counting at all
detector	retinaface (default) or YOLOv5, for a small or angled face that is missed entirely

The swap networks live in faceSwap.models and the restorers in faceSwap.restoreModels — both merge with host entries like every other catalog, and both are fetched by the faceswap set. Which store directory a swapper sits in (insightface/, reswapper/, hyperswap/) is what tells ReActor how to run it, so a new one is a mediaModels entry plus a faceSwap.models entry, never a code change.

Two things to know about ReActor. Its inswapper and buffalo_l weights are InsightFace's, released for non-commercial research use only (ReSwapper is the way around that: same job, trained from scratch). And upstream carries its own check over the input photos that replaces the result with a 512x512 black frame when it fires, saying nothing about why — a GitHub-policy artifact rather than a technical requirement, and one that costs a ~350MB download on first use.

This stack ships that node as its authors do. swap_face_fast recognizes the black placeholder by shape and reports it rather than handing you a black image, and points at swap_face (the PuLID path), which has no such gate. Turning the gate off is not something this repo decides: it is a comfyui.customNodes."ComfyUI-ReActor".patches entry, and whichever module a host imports to make that call owns it. Being declared means it is also reverted — drop the declaration and the next comfyui start puts the file back.

Same courtesy as everywhere else here: these are real people's faces.

Training LoRAs of people (family photos)

lora-train fine-tunes an image model on your own photos so it can generate arbitrary new images of specific people (LoRA subject training, via ostris/ai-toolkit — self-installs to /var/lib/genai-ai-toolkit on first use; jobs live in /var/lib/genai-lora-jobs). Run it as a user in the genai group. Everything runs and stays on this box. Get the subjects' okay first — these are real people.

Three base models, one trainer:

	Train time (5090)	Likeness	Generate with
flux (FLUX.1-dev)	~2-3h	best, most proven	:8893 with "model": "flux-dev", or ComfyUI
zimage (Z-Image-Turbo)	~1h	very good	:8893 default engine = the chat image button
pony (Pony V6 XL / SDXL)	~1-1.5h	stylized/anime — not for photoreal	:8893 with "model": "pony"

LoRAs are architecture-bound: a LoRA only works with the base family it was trained on (the install sidecar records this, and every path — chat tool, image button, raw API — routes to the right engine automatically; the "Generate with" column is where each base lands, not something to type). Community LoRAs from CivitAI etc. are declared, not installed — see Adding a LoRA below. Either way they are usable from chat by name or via <lora:name:0.8> prompt tags. Remove a locally trained one with lora-train uninstall <name>.

One-time for flux only: the base model is gated — accept the license at huggingface.co/black-forest-labs/FLUX.1-dev and export HF_TOKEN=hf_... before lora-train run. (pony downloads its shared-store checkpoint the image server already uses — one download; not gated.)

Walkthrough (one person)

lora-train new dad flux erx_dad        # scaffold /var/lib/genai-lora-jobs/dad/
cp /path/to/photos/*.jpg /var/lib/genai-lora-jobs/dad/dataset/
lora-train caption dad                 # captions via the local qwen-dense vision model
lora-train run dad                     # train; checkpoints + samples in .../dad/output/
lora-train install dad                 # deploy it for chat + API use

Dataset: 15-30 photos, varied — closeups and half/full body, different angles, lighting, expressions, clothing, backgrounds. Crop other people out. The trigger word is what binds the identity; pick something that isn't a real word (erx_dad, not dad). Review the generated .txt captions before training — they should describe scene/pose/clothing, never identity traits (hand-edit freely; re-running caption keeps existing files). During training, sample images land in the output dir every 500 steps — if likeness is good early, you can stop and use the latest checkpoint.

Training wants the whole GPU for hours: run stops llama-swap for the duration (chat and STT are both down while it trains — STT is a llama-swap model) and restarts it when training finishes, crashes, or is interrupted. Tune anything else by editing /var/lib/genai-lora-jobs/<name>/config.yaml before run.

(ai-toolkit also ships a Next.js web UI. It isn't wired up here: its npm install pulls native deps — Prisma engines, sqlite3, sharp — that assume an FHS system and don't resolve on NixOS. The CLI covers the whole workflow.)

Using the result

lora-train install dad     # newest checkpoint -> /var/lib/genai-models/loras

That copies the newest .safetensors from the job's output and writes a dad.json sidecar recording the trigger word and base model. Both the image server and the media tool server read that shared directory. (For ComfyUI, ComfyUI reads the same store — no extra copy needed.)

From chat (easiest). Installed LoRAs are listed in the generate_image tool description, so any tool-enabled model can pick one by name — just ask in plain language:

make a picture of dad jumping on a trampoline in the backyard

The model passes lora: "dad", and the server prepends the <lora:dad:0.9> tag, injects the erx_dad trigger word, and selects the right base model automatically. Say "use a stronger/weaker likeness" to nudge lora_strength (0.1-1.5, default 0.9). Asking for several installed subjects in one scene works too — the model passes loras: ["dad", "mom"] and every tag and trigger word is injected; per-subject weights ride along as loras: ["dad:0.7", "mom"] ("make dad's likeness weaker"), with lora_strength as the default for entries without their own. The LoRAs must share the same base model, and expect some identity bleed between subjects — stacking person LoRAs is a known diffusion weak spot. After installing a new LoRA, restart Open WebUI if the model doesn't seem to know about it — the tool spec is fetched per connection and may be cached.

Direct prompting (Open WebUI's image button, or the API) takes the tag yourself, and there the trigger word is mandatory:

curl -X POST http://<host>:8893/v1/images/generations \
     -H 'Content-Type: application/json' -d '{
       "prompt": "<lora:dad:0.9> photo of erx_dad sailing a boat at sunset"}'

"model" can be omitted: a tag naming a registered LoRA selects the engine its base needs (here flux → flux-dev), overriding whatever model the request carried — a mismatched combo would only make sd-cli skip the weights. Tags naming unregistered files (no .json sidecar) leave the requested model alone.

Write prompts as photo captions (subject, pose, setting, lighting, framing) rather than instructions — that matches the caption style the LoRA was trained on. Strength 0.8-1.0; lower it if outputs get stiff or over-baked.

Grainy LoRA output? Z-Image-Turbo generates in 8 steps, but a LoRA bends its distilled sampling path and 8 steps leave that path under-denoised — the residual is grain. The server auto-raises any LoRA-tagged Z-Image request to LORA_STEPS (16), which clears it while keeping bare generations fast; tune that env in module.nix. Raise it further per request with "steps": N on the API (or "make it more detailed" in chat, which sets the steps tool param). If grain persists only at high strength but low strength looks baked, the LoRA likely overfit — try an earlier checkpoint, or use the FLUX path, which samples at 20 steps and is inherently cleaner for faces.

LoRAs are architecture-bound (a flux LoRA loads on any FLUX.1-dev engine, but not on FLUX.2 — different architecture), but for registered LoRAs the server enforces the match itself: the tag's base picks the engine everywhere — the generate_image tool, the chat image button (which always sends its one configured model — the tag overrides it), and raw API calls alike.

Editing and face tools take LoRAs too. edit_image, reimagine_image, inpaint_image, smart_edit and the two PuLID tools (generate_image_with_face, swap_face_full) all accept the same lora / loras / lora_strength arguments, and the studio's start from an image tab shows a picker for them. One difference matters: these run one fixed model each — Kontext, schnell, the inpaint engine you picked, or the PuLID graph's FLUX.1-dev base — so there is nothing to reroute to and a LoRA from another architecture is refused with the reason rather than switched around. A flux LoRA for the FLUX paths, a pony/sdxl LoRA for an SDXL inpaint engine. Trigger words are still injected for you.

On the two face tools, prefer a style LoRA. A LoRA of a person is a second identity source aimed at one face. Measured on this box the photo won — rita_flux at 0.8 under identity strength 0.9 changed framing, wardrobe and colour grade while the face stayed the reference's, on the fp8 and Q8-GGUF bases alike — but when it goes the other way you get a convincing third face rather than an error, so check the likeness before trusting one.

Managing installed LoRAs

lora-train list              # what's installed, with trigger word + base
lora-list                    # same, plus per-LoRA usage examples: the webui
                             # chat phrasing and a ready-to-send API request
lora-train uninstall dad     # remove it (needs sudo)

Uninstalling deletes only the deployed copy; /var/lib/genai-lora-jobs/dad/ keeps the dataset, config, and every checkpoint, so you can retrain or install again later. Both servers read the directory per request, so removal takes effect immediately with no restart — though Open WebUI may keep advertising the name until it refetches the tool spec, and a stale request naming a removed LoRA just returns an error listing the valid ones. If you also copied the file into ComfyUI's own models dir at some point, remove that copy too.

Not happy with the results? Before discarding it, try dropping the strength (<lora:dad:0.6>, or "weaker likeness" in chat) — over-baked output that ignores your prompt is usually too much LoRA, not a bad LoRA. Pin the seed first, or you are comparing two different rolls and cannot tell a fixed LoRA from a lucky one: take the seed off a render (the studio caption, the gallery's metadata, or the seed a tool returns), put it in the Seed box, and change only the strength. Mangled anatomy that clears up at a lower weight was over-driving; mangled anatomy that survives every weight at the same seed is the LoRA. Failing that, an earlier checkpoint from the job's output/ is often better than the final one, since person LoRAs overfit late in training.

Multiple people in one image: stacking two person-LoRAs blends faces. Either train one LoRA on a joint dataset with a distinct trigger word per person, or generate the scene with one person and fix the other's face via /inpaint (mask the face, prompt with that person's LoRA + trigger).

Portal (:8897)

The single entry point. One nav across every page it hosts, one directory of everything else, one health answer.

The nav is not the only thing that rides with the chrome. A prompt box grows with what is in it, on every page, because a textarea is a fixed box the browser scrolls and anything past the third line is written through a letterbox — worst on a phone, where the box is smallest and the prompt is the whole point of the page. The height follows the content and only then scrolls. The CEILING is shared (min(50dvh, 26rem)) and the FLOOR is each page's own: the chrome is injected after a page's stylesheet, so an opinion about the floor there would silently shrink the code sandbox's editor and the RAG paste box to one line whenever they were empty. So a box never starts smaller than it does today — it only stops being stuck at that size. Dragging the resize handle opts a box out for good, since a drag undone by the next keystroke reads as a handle that does not work.

Which meant the studio's edit and people prompts had to stop being single-line <input>s — a prompt is the longest thing anybody types on that page and it was in the shortest field on the form, which is the complaint this fixes and the one place growing textareas could not reach. The six prompt fields are textareas like the create tab's; the two neighbours that are not prose stay inputs, because a segment prompt is a short noun phrase and the region candidates are a comma list.

Overview (/)

Problems first: anything down, wedged, or enabled-but-not-downloaded appears in a strip at the top (and in the health pill in the nav, on every page). Below that, live GPU / CPU / RAM gauges; the service directory grouped into Chat & agents, Generation, Serving and Tools & APIs, each card carrying a TCP probe and an HTTP health check where the service has one (a green dot means answering, amber means listening but failing its health path); and the model fleet.

Backends with no browser UI (tts-hq, segment-server) render as status chips rather than links — they are monitored, not clickable.

ComfyUI's card is administrators-only (comfyui.adminOnly, on by default, and portal.extraServices.*.admin for a host's own links). Not about the software: it is a full node editor with no identity of its own, where every checkpoint and LoRA on the box is one dropdown away, so it is the one surface here that cannot be filtered per person. That makes it DISCOVERY rather than access — the portal links to ComfyUI rather than proxying it, so anyone holding the URL still reaches it, and a gate that holds belongs in front of the service. Turn it off once ComfyUI runs per user.

Who counts as an administrator is the same question everywhere in this stack, answered once: the per-person setting on /admin first, then the platform's role header, then identity.admins. An override of off beats a platform role of admin, which is the case that makes it an override rather than a hint.

Model fleet (/models)

Every configured model with its state (ready / downloading / absent), size, an enable toggle, and download/remove buttons — plus what it can do (capability chips) and how it is served (context, KV quant, MoE offload, group, TTL). Downloads use the parallel prefetcher. State choices persist in /var/lib/genai-dashboard/enabled.json (runtime, survives restarts; a from-scratch rebuild starts from the flake defaults). The Overview page keeps a two-card summary (swappable / always-on) that links here.

The serving detail is not a description of the config — it is the config: it comes from each catalog entry's serve block (see Adding an LLM), and for the hand-tuned models an assertion checks the declared context against the command that actually runs. Models that came from that declarative path are badged generated.

Evals (/evals)

What was measured on this box, rather than what the model cards claim: the latest genai-eval report with its failures spelled out, any model comparison as a per-model table, and the run history — click a run to load it. Reports are plain JSON in /var/lib/genai-eval, so anything on this page is equally available to a script.

Read-only on purpose. The page never starts a run: suites load models, and the process serving this page is the one answering the health probes. Runs come from genai-eval on the box or a timer.

Model comparison gets a warning banner when no model cleared the suite, because that is as likely to mean the harness is broken as that the models are bad — which is exactly what happened the first time this ran (see ROADMAP Stage 14, and the note under Adding an LLM about thinking models and token budgets).

API surface

• /v1/* — the filter proxy Open WebUI and LiteLLM point at instead of llama-swap directly: only ready+enabled models appear in /v1/models, and a chat request for an unavailable model returns a clean 503 instead of triggering a slow single-stream load. (Open WebUI's model dropdown may still list tool-wired models that aren't downloaded — selecting one 503s.)
• /svc/<name>/* — reverse proxy to this flake's own stdlib services: image, media, search, tts, segment. So curl :8897/svc/media/openapi.json reaches media-tools, and an API client needs one origin for the whole box. Strict allowlist, built from the port map — an unknown name 404s. JSON APIs only; the portal is a stdlib HTTP server and does not proxy websockets, which is why the third-party UIs stay links.
• /api/health — {ok, problems[], services, gpus, cpu, mem, models}, and a 503 status when not ok, so curl -f :8897/api/health is a valid uptime check. This is the thing to poll.
• /api/portal — the hosted pages and the grouped service directory, for anything that wants to render or discover the surface.
• /api/evals — the newest genai-eval report in full plus a summary of the rest; /api/evals/<file> fetches one by name (matched against the directory listing, so the name cannot escape it).
• /svc/mcp/mcp — the MCP endpoint (see below), so an MCP client points at the portal like everything else.

Extending it

services.genai-server.portal = {
  title = "logistikon";
  extraPages = [
    { path = "https://grafana.lan"; name = "Grafana"; desc = "long-term metrics"; }
  ];
  extraServices = [
    { name = "Jupyter"; port = 8899; kind = "ui"; group = "Tools & APIs";
      desc = "notebooks"; health = "/api/status"; }
  ];
  proxy.extraUpstreams.notebooks = "http://127.0.0.1:8899";
};

Health probes, /api/health and the genai_service_* metrics pick up added cards automatically.

Gallery (:8897/gallery)

Everything the box has ever generated, newest first — studio renders, chat tool results and published-workflow clips alike, because all three write one store (mediaStore.dir, below) and this is a listing of it rather than a copy of it. Search — of prompts, engines, LoRAs, seeds and the names you have given faces — plus engine, media type, LoRA, who is in it, date and sort-order filters run server-side against the whole store; it pages itself as you approach the bottom.

Images, video, and what made them are two controls, not one. Images & video narrows to one or the other and is derived from the file itself, so nothing has to be recorded for it to work — which matters, because "anything that moves" must not be a list of engine names somebody keeps up to date as graphs are published. The engine picker is that list, and it now holds the published workflows as well: a clip's engine is its graph (wan22_video, ltx23_animate, …), grouped apart from the image engines in the dropdown. That is derived from the operation the record already carries rather than written into a new field, which is why the clips already in the store have it without being re-rendered. The two compose, like every other filter here: video plus ltx23_animate plus a prompt word is one server-side query.

What went into it is a third question about the same file, and the only evidence this box has about whether a face belongs to anybody: a render made from nothing but a prompt holds a face that is nobody, and one made from a photograph holds somebody. Two answers, and only one of them asserts anything — 568 renders here were made from a prompt alone, and the other 2398 either had a picture go in or carry nothing that says otherwise. A file this box has no record for sits with the photographs, because calling it an invention on no evidence is the error that hides a real person; the scope picker's only unrecorded is how you find those on purpose.

It is derived, like the media type and the engine name — from sources, the operation, the arguments and the chain of renders behind it — so everything already in the store answers without a re-scan. Four cases are worth knowing, because each of them is a whole class of the archive:

• the chain. A clip animated from a text-to-image render is as invented as the frame it came from, and its own record names an input and stops there.
• an operation is itself the record of an input. 317 edit and smart_edit entries predate keeping inputs: they say an edit ran, an edit needs a picture, so a picture went in and only which is missing.
• a carried input. 124 clips look input-less and hold a 2.5MB frame inline in their arguments; sources skips those by design, since it lists what is on disk to be reopened.
• a published workflow that named nothing is the one thing an operation cannot answer, because the graph decides — ltx23_video takes no frame and ltx23_animate animates one, and here their records are the same twelve keys with the picture missing from both. Those 44 fall on the photograph side with everything else that is not evidenced.

An input that is not in the listing — an upload kept under sources/, or something since deleted — is a photograph, because a picture from outside this box is what a photograph looks like from here.

The search box takes a name too, and it has to say so on its face — who is in a picture appears in no prompt, no engine and no seed, so nothing else here would suggest typing one. A substring matches, like the picker's own box: what is being typed is a name being completed, and half of one is what gets typed. It resolves through the same two functions the person filter uses, so a name narrows to exactly what picking that person would have — your own names, your own merges, nothing anybody else called anybody — and the answer is the union with what the words matched, since one box asking two questions must not drop either.

It costs a small read to find out it has nothing to do: the roster takes a fifth of a second to build and the box sends a request per keystroke, so the query is first checked against your own {id: name} settings — the only strings a name search can match. A query matching none of them never builds the roster.

And the listing says when a name is why. A render of somebody whose prompt never mentions them is a hit with nothing on it to explain the match, and a search that widens itself in silence reads as a search that has gone wrong — so the count line names who: plus the pictures Ada is in.

And who came out is the question that one is the evidence for. Faces… on the bar opens the roster's own search — a face, a name if you have given one, and how many of your pictures they are in — and picking somebody narrows the grid to them. The switch inside it includes the groups nobody has named, which on most boxes is all of them: this box holds 3355 groups and no names, so a picker restricted to named people would open on nothing. The button then wears whose it is, the line under the grid says so in words and offers the way back to everyone, and the address bar carries it, so the listing you are looking at is a link you can send. This was the one filter here with no door: it worked from the moment the pane on /people could link to it, and the only way to reach it was to arrive from somewhere else already carrying it — which meant leaving a listing you had narrowed four other ways to go and get it.

Beside the name, also what was made from their pictures widens the listing by what each render's own record names as its input. That is a different claim from the grouping's — a swap, a scene reference or a second face in the frame all leave the same line — so it is offered rather than merged, said out loud in the count when it is on, and it comes off when you choose somebody else, because a different person is a different question. The roster on /people carries the long form of that distinction; this is the same toggle at the width a filter line allows.

Reaching the rest of the archive. Paging alone does not: a few thousand renders is thirty-odd presses from the oldest one and no number of presses from the middle. Two controls answer that. Newest/oldest first flips the whole listing, so the beginning of the store is one click rather than thirty. The date picker offers only months the store actually has, each with its own count — so it doubles as a map of where the renders are — and paging then works inside the month. Both narrow the same server-side query the search box does, which is what makes them compose: July plus flux plus a prompt word is one request, not a page filtered afterwards.

Each month travels with the timestamps that define it and the page sends those back unchanged, rather than turning "2026-03" into a window on its own clock. The browser is not necessarily in the box's timezone, and a month it bucketed itself would disagree at both edges — by a few hours, on exactly the renders nearest the boundary. The windows are half-open ([from, to)), so adjacent months never claim the same midnight and the counts add up to the store.

The date window also reaches Select all matching, which is the half that is easy to miss: a bulk action has to be computed from the query that drew the grid, or it ticks — and Delete is one button along — renders that were filtered away and never shown.

Stars, titles and tags: the ones you thought were good. The viewer has a Starred toggle and, beside it, one button for the words. Starring is the mark; the dialog behind that button takes a title and any number of tags.

A title and a tag are different questions, which is why there are both. A title is what this one render is — "the lighthouse at dusk" — and nothing else has it, so it is read rather than filtered by: it heads the tile above the prompt, it heads the viewer, and the ordinary keyword box finds it. A tag is what a render is one of, and is worth nothing until something else wears it too, so tags are what the filter is built from. One box for both gave a facet full of entries counting one, which is a list of titles pretending to be a filter.

All three are one record, so naming or tagging stars and unstarring forgets both: a title on something unmarked would be one nothing can find. Each field is left alone when the other is saved, which is what lets the toggle be a toggle — pressing star twice must not clear what you typed. A marked tile wears a star in its corner and its tags as chips ahead of the engine.

In the viewer the title and the tags get their own row under the picture rather than a place among the parameter chips: a tag sitting beside seed: 4211 reads as one more thing the machine recorded, when it is the one thing on the entry a person typed. And unlike a parameter it is worth pressing — a tag in the viewer is a button that filters the gallery by it, which is what gathering its renders together means. From the studio's strip, where there is no filter bar to set, it is a link here carrying the tag. The button that opens the editor keeps saying Name and tag… whatever is on the render: it does the same thing every time, and the state is on screen a few lines below.

The tag editor offers the tags already in use, which is most of what makes them shared at all: a tag typed slightly differently is a second tag, and the only defence is showing what is there while you type. One box does both — it narrows the existing ones and takes a new word on Enter — and anything still sitting in it when you press Save counts, because typing a tag and not pressing Enter is the commonest way to lose one. Tags already on the render sit first with a ✕; click one to take it off.

Renaming and deleting a tag everywhere is the operation no per-render editor can do, and it lives behind Edit tags… in the picker — the only place the whole list is on screen with its counts, which is how you notice you have both portrait and portraits. Renaming reaches every render wearing it, and renaming onto a tag you already have merges the two rather than refusing, since that is usually the reason to be there. Deleting takes the word off those renders and leaves the renders alone — still starred, just no longer one of those. Both say how many renders moved, and a tag nothing wears is refused rather than reported as done: an operation over a set nobody is looking at has to say what it touched, or it cannot be told from one that matched nothing.

Two controls on the bar, because they are the common case and the fine one. ★ Starred is a toggle and is always there — including on a gallery where nothing is starred yet, where its tooltip says so and pressing it answers with an instruction rather than "nothing matches". It hid itself until something had been marked, which meant the only way to discover the feature was to have already used it; a control that can briefly show an empty page is a smaller problem than one with no visible door.

Beside it, Tags… opens a picker — a search box over the tags with their counts, which is the only shape that survives three hundred of them. A <select> was right for six and unusable at that: no typing, no counts worth reading, no way to choose two. Several tags mean all of them, not either: two is the narrower question, and "everything wearing either" is what you get by picking one and looking. The button says what the filter currently is — Tags…, #hero, #hero +2 — and applies on close, because a filter has nothing to cancel and reloading per click would be a listing request for a set you are still assembling. A tag already implies starred, so ★ reads pressed while tags are narrowing the grid, and pressing it there widens back to everything starred. The picker itself appears only once something is tagged, drops a tag whose last picture was deleted, and reaches Select all matching for the same reason the date window does.

The search box asks all three — a title and a tag are not in the indexed text and cannot be, since that is built per file and cached with it while both are your words about it, so a query is matched against the prompt, your title and your tags.

They are yours: kept in your own settings beside your face names, never shown to anybody else, administrator included. Two accounts can star the same render and neither learns what the other thought of it. A mark is a name, so a render deleted out from under one contributes to nothing — no tile, no facet, no count — and store names being an epoch and eight hex digits, an old entry can never come to mean a different picture.

Searching by what is in a picture (portal.semantic.enable, off by default). The search box above matches the PROMPT, which is the request rather than the result: silent about everything the model decided, wrong wherever the model ignored it, and absent altogether from the entries that say no parameters were recorded — which on a long-lived box is most of its history. A second box looks at the picture instead. Type a description, press Enter, and the listing comes back closest-first; any entry's viewer also offers Find similar, which is the same ranking taken from that render's own vector.

It runs on a dual encoder — SigLIP 2 by default, portal.semantic.model to change it — on the CPU, and deliberately: the card is shared and an index build is exactly the long hold that must not take it. Each encoder keeps its own index, so trying another one is free and switching back rebuilds nothing.

Staying current (portal.semantic.autoIndex, on by default) has two halves, separately switchable, and the split is the point. A render the PORTAL writes — anything made in the studio — is embedded within seconds of finishing (indexOnWrite). That covers what the portal sees, and is deliberately not trusted to cover the store, because it does not: ComfyUI writes the media directory directly and so does media-tools, neither through this process. A hook on the portal's own write path alone would leave whole categories of file permanently unfindable while looking complete. So a periodic sweep (sweepMinutes) is the backstop, keyed on the directory the way the gallery listing is, which is what makes it catch a file whatever wrote it. Each pass is bounded (sweepBatch) so a first sweep over a full store spreads across passes instead of holding the CPU on a machine somebody is working on; it converges without bookkeeping, since what is left over is simply found again next time.

The sweep is the load-bearing half and indexOnWrite is the one to reach for if either misbehaves — hence its own switch. It is worth being honest about what it earns: it buys freshness for the renders least likely to need a content search (you are looking at the thing you just made), it spends its CPU at the busiest moment, and it is a second code path that can fall silent without reporting anything. What it does buy is a clean "N of M indexed" line, which otherwise reappears on every render until the next sweep. A no-op sweep costs 7ms here against the full store, so a short sweepMinutes is a genuine alternative to switching it on at all.

A full build is also available on demand — Build search index…, administrators only — and is store-wide rather than per-viewer, because the index is machinery like the thumbnail cache: one built from one person's view would leave everybody else's search silently empty. It grants no visibility; results still pass the ownership filter and whatever any optional module keeps back.

Measured on this box (2,396 renders, 485 of them video, SigLIP 2 base): 8.8 minutes to build the whole index at 4.5 files/sec — videos are the slow part at ~1.6/s, since each costs three ffmpeg keyframe seeks. The index is 7.4MB (2,396 × 768 float32). A typed query is ~3.4s, nearly all of it loading the encoder, which is why the box searches on Enter rather than per keystroke; Find similar loads no model and is effectively instant.

Retrieval was checked against an independent labeller rather than against its own model card: a set of per-file scores this store already carried, produced by a different architecture, is a free labelled set. Ranking agreement (AUC, 0.5 being chance): 0.982 for a query matching those labels, 0.259 — correctly inverted — for one that should not. The embeddings encode real picture content on this store.

Describe the picture, not the medium. The encoder sees frames, so it has no notion of motion or of a file "being a video": a video of a person moving is a category error and returns nothing useful, while a person mid-stride works. Clips are not second-class otherwise — measured here, content queries return 58-67% video in their top 100 against a store that is 20% video, so they rank if anything better than stills.

Two things worth knowing about how it behaves. The ranking is computed over the whole index and intersected with what you may see afterwards, so a top-N means N of your renders rather than whatever survived a cutoff taken over somebody else's — and a search can only ever remove entries, never reveal one. And a typed query loads the encoder per call, a second or three, which is why it searches on Enter rather than per keystroke; Find similar loads no model at all and is effectively instant, since the vector it ranks against is already in the index.

If the index does not cover the whole store yet, the gallery says so under the count. That line matters more than it looks: a search over a third of the store returns thin results that read as a bad model rather than as an unfinished build, and nothing else on the page would ever say which.

The same picture in different bytes (portal.duplicates.enable, on by default — it carries no closure, and a store nobody has been scanning accumulates copies silently until the day the disk is full, which is too late for the scan to have been running all along). Two things above already look for copies and neither can reach this case. The gallery reconciles renders whose image data is byte-for-byte equal — no setting involved, because "the same bytes" is not a judgement. The content search ranks by resemblance, and says two unrelated portraits resemble each other, which is a ranking and not a verdict. Between them sits the store's actual duplication: a render re-encoded or re-saved, the same frame at another size, a clip remuxed or transcoded.

genai-dupe-scan finds those. It fingerprints every file with a perceptual hash — a DCT over a 32×32 thumbnail, 64 bits, plus a coarse colour signature that only ever rejects a match the hash proposed — and groups what agrees within threshold. No model and no card: the work is ffmpeg decode and 64-bit popcount, and a store-wide scan is exactly the long hold that must not take a card somebody is working on.

It reports two tiers and they are not the same claim. identical means the two decode to the same pixels — proof, arrived at by hashing the decoded frame (or, for a remux, the encoded video stream) once a cheap hash has already paired the files up. near means the perceptual hash agreed within a threshold: a judgement. The gallery keeps that distinction all the way to the buttons. The near copies filter shows you everything it grouped, and Prune → identical copies selects only the proven tier, so the one thing this deletes in bulk cannot cost a pixel. Nothing is ever written into somebody's render, which is what lets threshold be re-tuned later — a verdict stamped into a file outlives the setting that produced it.

The copy that is kept is the one with the most to lose: most pixels, then longest, then oldest. Size is deliberately not in that list — bigger can be an uncompressed copy of the same pixels and smaller can be a lossy re-encode of them, so preferring either picks the worse file half the time, while age is never backwards.

A sweep (sweepMinutes, hourly by default) keeps it current, keyed on the directory so it sees what ComfyUI and media-tools write without going through the portal, and bounded per pass (sweepBatch) like the semantic one. Fingerprints are keyed on (mtime, size), so a pass over an already-scanned store costs 30ms; a first pass costs about a minute per thousand stills (measured: 100 × 1024² PNGs in 6.2s at the default two workers). Scan for copies… runs one on demand, administrators only, because grouping is store-wide — a copy and the render it copies can belong to two accounts, and scanning one person's half would report a file as unique on the strength of not having looked at the other. The filter is not admin-only: a copy is only ever reported to somebody who can see both ends of it, so a group that straddles two accounts simply is not a group for either of them.

What it will not find, said plainly rather than left to be discovered: a clip inside a longer one. Videos are compared whole and pre-grouped by duration, so a ten-second excerpt of a forty-second render is not a candidate for it at any tolerance — that needs subsequence matching over a frame-hash stream, which is a different tool. nix build .#checks.x86_64-linux.dupes asserts both halves, including that limit, because every way this can be wrong reports "no copies" and exits 0, which is indistinguishable from a clean store.

It also runs against any directory: genai-dupe-scan --dir /some/tree --index-dir /some/where. An index belongs to one tree and refuses to be pointed at a second without --force.

Who is in the store (faces.enable, on by default; the runtime is already here because mediaTools.portrait.enable is, so the cost is a 434MB recogniser and a background first pass rather than any closure). The two indexes above ask what a picture is and which pictures are the same picture. This asks the third question — who is in it — and it is the one that every identity tool on this box needed answered and nobody had: swap_face, swap_face_fast, generate_image_with_face and the LTX identity transfer all take one photograph of a face, and finding it meant scrolling the gallery for a render that happened to be a good frontal shot.

The pipeline is Immich's, because a photo library got there first and got it right: detect → align → embed → cluster. Detection, five-point landmarks and head pose come from insightface's buffalo_l (SCRFD-10G), which is already in the store because the face swappers run it. Every crop is warped onto ArcFace's canonical 112×112 geometry, which is the only reason one crop can feed three different models. Grouping is DBSCAN over cosine distance, at Immich's own minScore (0.7) — that is a detection confidence, and a face is a face on any store.

maxDistance diverges: 0.65, where Immich ships 0.5. 0.5 fits a camera roll — one lighting rig, one lens, one face per person. A render store holds the same person across seeds, checkpoints, LoRA strengths, styles and video compression, and two pictures of one person sit much further apart. Measured against an operator's own labels — hand-merges are ground truth for "same person", distinct named people for "different" — comparing like with like (closest face pair between two groups), on 136 merges and 1698 named pairs:

cutoff	hand-merges it would have caught	different people wrongly joined
0.50	0 of 136	0
0.60	41 (30%)	0
0.65	59 (43%)	0
0.70	75 (55%)	0
0.75	100 (74%)	2

No two different named people had any face pair closer than 0.716, so 0.65 keeps a 0.066 margin under the nearest measured collision while recovering nearly half the work that was being done by hand. It stops well short of 0.75 deliberately: a split is visible and repairable with Merge, a merge destroys the distinction it got wrong. Re-measure before going higher — a store whose people look more alike will have a lower floor.

minFaces is where this diverges: 1, where Immich ships 3. Immich's 3 is right for a photo library, where the same people recur and a face seen once is a passer-by. A store of renders is the opposite — most subjects are generated once — and a floor of 3 measured here left 59% of the faces found in no person at all, so 59% of them appeared nowhere. That would be the wrong trade if it bought merging, since minPts of 1 makes every face a core point and single-link clustering chains: A near B, B near C, and A and C become one person having never been near each other. So it was measured rather than argued — on 2893 faces the largest group is 298 at 1, 2 and 3 alike, holding the same faces. The floor decided only whether the small groups existed. What it costs instead is a long roster (106 people became 1612), sorted most-pictures-first and loaded in batches. A face matching nobody becomes a person of its own, which is the honest reading of "there is a face here", and Merge is how two of those become one.

The recogniser is where it deliberately differs. buffalo_l's own ArcFace-R50 reports IJB-C 97.25 at 1e-4; LVFace-B (ByteDance, ICCV 2025 Highlight, ViT-B on Glint360K, 434MB) reports 97.70 at 1e-4, 97.00 at 1e-5 and 90.06 at 1e-6, and the last column is the reason to care. Clustering compares every face with every other face, so the chances to be wrong grow with the square of the store, and the failure that hurts is a merge — two people who become one person, which browsing cannot undo because the evidence that they were two is exactly what got averaged away. A split is visible and repairable; a merge looks like a working cluster. So the model to pick is the one that holds up where false accepts are rarest. recognizer = "buffalo_l" costs nothing to choose and downloads nothing; lvface-t is not the cheap version of the default but a worse one (96.67, below buffalo_l), and is offered only for a machine that cannot afford a ViT. Each recogniser keeps its own index, so trying a second is free and switching back does not rebuild.

What a person sees is the gallery's answer, not a second one. The index is built store-wide — an index built from one account's view would leave everybody else's people list empty with no way to tell why — and then intersected with _gallery_hits, which has already applied ownership, any optional module's filter and the stricter toggle. So a person appears only if you can see at least one picture they are in, their count is the count of your pictures, and an administrator sees no more than anybody else. A face arrives occluded exactly when the render it was cropped out of does, and a cover is chosen from the unoccluded ones first: a face is judged by the picture it came from, because a crop that escaped its source's standing is the one thing this could get wrong that nothing would report. Names are per-viewer, kept with your other settings rather than in the index — a name you put on a face is your own note, and the index is derived and gets rebuilt.

In Studio, the swap and PuLID forms grow a Find somebody… button. It opens the same search the merge action uses — the people you have named, most pictures first, with a switch for the groups nobody has named — and whoever you pick has their best photograph cropped and attached to whichever slot that operation takes an identity in (source_url for the swappers, url for generate_image_with_face), with the render it came from named in the reply so you can disagree with it. It was a STRIP of every person in the store, which on a store holding three thousand groups is a wall of faces between the form and the button that runs it. One tile stays behind afterwards, for the person you picked and nobody else: attaching them again is a real errand, because which slot takes the identity depends on the operation. That button attaches the best photograph it can measure; the picker's Faces tab below is the same search ending in every picture of them, which is how you disagree with the measurement. Who is in the attached photo? matches whatever is bound against the people this box knows and reports the distance and the line it is judged against, never just the verdict: 0.48 against a threshold of 0.5 is a different thing to be told than "yes". Scan the store for faces… starts the sweep, administrators only.

Four tools go to chat models and through the MCP gateway (faces.tools): list_people, who_is_this, photo_of_person — which is what turns "make a picture of Sam" into a real crop of a real render instead of a request for an attachment — and group_faces_in_folder, which sorts a directory by who is in each image for the job that is otherwise done by eye: pulling one person's shots out of a shoot, or building a LoRA training set. The first three ask the portal as the person who called them, so there is one implementation of who-may-see-whom rather than two that drift. The folder tool touches no store and no identity: nothing it reads is indexed, nobody it finds is added to this box's people, and the answer is the only place the grouping exists. It reads only faces.folderRoots — empty by default, meaning the media store alone — because a service that reads whatever path it is handed is a way to find out what is on a disk, and it names the allowed roots when it refuses.

Clicking somebody opens a pane beside the roster (full-screen on a phone) holding the gallery component itself — the same tiles, viewer, metadata and actions /gallery and /studio mount, filtered to that person. It is not a second gallery: person is a filter on /api/gallery like engine or the date window, resolved through the same ownership and merge rules, and an id naming somebody you cannot see returns nothing rather than everything. Naming is a button in that pane, primary while they are unnamed because that is what the pane is for.

Who is in the picture you are looking at rides in the viewer itself, on every page that mounts the gallery — directly under the render and above its parameters, since recognising a face and reading sampler settings are different errands. Each person there is a crop, the name you gave them (or unnamed), how many of your pictures they are in, and the two things worth doing from a picture: Name them, or Merge with… somebody you have already named — the same search the roster and the pane use, so there is one of it on this box rather than three. A render with nobody in the strip says WHY, because there are four different reasons and they used to share one blank: not looked at yet (a button to change it), being looked at right now (a spinner, no button — it is already happening), looked at, and the faces found are not grouped yet (a spinner and the count, which is seconds at the end of the pass that scanned it), and nobody is in it (said out loud, with Look again to disagree). A picture made four seconds ago is one of the middle two, which is exactly when somebody is looking at it — and it offers Scan for faces, which does that one picture on the spot rather than making you wait for a pass. It needs no card: the scanner clears CUDA_VISIBLE_DEVICES and names the CPU provider, which is what makes it a button at all. While a store-wide scan is running the button hands the picture to that pass instead, and says so.

The announce comes from whichever process wrote the file: media-tools tells the portal that owns its store (/api/media/announce) as soon as it saves a result. On a fleet that is the only process that knows both halves — what was made, and which machine it landed on — because the portal that orchestrated the render is a different box with a face index over a different store. Renders the portal writes itself are scanned as they land (faces.scanOnWrite, on by default) rather than at the next sweep, with faces.settleSeconds to let a burst arrive first — the scanner loads insightface and the recogniser before it reads a pixel, so eight renders answered one at a time is eight of those loads. This is the half faces.autoIndex has described since it was written and did not have: the name was queued and the sweep did announced names first, but only when it next woke, so on a caught-up store that was up to sweepMinutes away. Studio renders only, since that is the one path the portal sees; anything ComfyUI or media-tools writes directly is still found by the sweep.

Two tiles of one person is the mistake clustering makes, and it is the safe one of the two on purpose — a merge averages away the evidence that two people were two and browsing never gets it back, which is why maxDistance sits where it does. So splitting is expected, and Merge on /people is how you say so: tick two or more, and they become one tile. It is stored as your note beside the names, so nobody else's view collapses, and — the part that matters — it lives outside the people file, which every sweep rewrites. A merge recorded in the derived file would look perfect until the next scan and then silently come apart. Separate them again undoes it, and the named person is never the one absorbed. The reverse problem, one tile holding two people, has no button: that needs deciding which faces belong to whom, and the honest lever is maxDistance with a re-cluster, which is admin-only because it moves everybody's groups at once.

Somebody, or nobody. On a render store much of the roster is faces of people who do not exist, and a crop of one looks exactly like a crop of somebody real: of 964 people here, 171 have nothing but prompts behind every picture of them. Every group draws 200 tiles at a time with a button that says how many are left, because a tier is not a bounded thing — "everyone you have named" is most of a roster once somebody has been naming for a while, and each tile is a face crop the browser goes and fetches. How far you have paged survives the refresh that the scan sweep triggers, for the same reason an open fold does. A find box narrows the roster to a name as you type — and a search draws a flat grid rather than the tiers, because a match inside a collapsed fold would be a search that found somebody and showed nothing. It says how many unnamed people it could not answer for, since on a render store those are most of them. That is what makes naming the real ones a thousand tiles of work instead of eight hundred, so /people has a Show filter — everyone, invented, real people — and an invented tile carries a small invented badge.

The roll-up is the gallery's answer applied to a person: any photograph wins. One picture with something real behind it is evidence a person exists and fifty renders beside it do not take that away, so invented needs a positive answer of its own — at least one picture a record says came out of a prompt, and none a record says came out of a picture. Two answers, even though the evidence has three states: somebody every one of whose pictures is unrecorded reads as real, and the third state survives only so the pane can say which of the two reasons it is not invented. It was offered as a third bucket on the roster for a day, and all 323 people in it turned out to be real.

It can be wrong in both directions, which is why the pane carries a decision. A LoRA trained on somebody real answers a prompt with a real person's face and nothing in the file says so; a photograph that was never kept leaves an edit looking like it came from nowhere. So Who this is offers three states — the default one states the evidence ("from their pictures: a photograph went in"), and the other two state a judgement ("invented", "a real person"). It is tri-state: clearing deletes the decision rather than storing today's answer, so somebody who gains a photograph tomorrow moves on their own, and "I decided this" stays distinguishable from "I happen to agree". Like a name and a merge it is your own note, kept in your settings rather than in the derived index, and it survives every re-cluster.

The not decided group is the worklist for it, which is why the filter says what it left out rather than quietly showing fewer people.

A sweep (sweepMinutes, half-hourly once caught up; catchUpSeconds while there is still a backlog) keeps it current, bounded per pass (sweepBatch) like the other two. There is no index-on-write half and that is deliberate: re-clustering is a whole-index operation, and running it per render would repeat the same work for an answer one picture rarely changes. Re-clustering rides on the sweep, costs seconds against the scan's hours, runs by itself when a threshold in the config no longer matches the one the grouping was built with (otherwise changing minFaces on an already-scanned store would change nothing and look broken), and names survive it — a person's id follows their centroid rather than their position in the output, which is what stops three new photographs from silently renaming everybody.

A clip is not a store, and it knows two things a store does not. A clip's faces are deduplicated within the file, each person contributing the frame they look best in — but doing that at the store's own threshold is what made one person in one video arrive as three or four, each with a tile of their own: fifteen seconds of somebody turning their head crosses 0.65 several times. So the grouping inside a file is a tracker:

• two faces in the same frame are two people, whatever they look like, because nobody stands next to themselves. It costs nothing, and it is what makes the rest safe — the reason a store's threshold must be strict is that nothing else stops two people merging, and here something does.
• so the threshold can be looser (faces.linkDistance, 0.70) — still under the 0.716 that is the closest measured distance between two different named people here.
• and where a face was a moment ago carries information: a detection overlapping the box that group had in the previous sampled frame is that face continuing, which rescues the frames an embedding cannot read at all — motion blur, a profile, a hand across the chin. Position without agreement never links, and past 0.85 the agreement is gone: two faces that far apart are as unlike each other as two strangers, and overlap then means somebody else walked into that space.

What it can still get wrong is worth saying: two people crossing in one clip, overlapping between two sampled frames at a distance the recogniser is unsure about, read as one face continuing. That decides which frame of a clip is kept, not who anybody is across the store — which is still DBSCAN at the strict threshold.

The inputs are scanned too (faces.indexInputs), and that is a reversal: sources/ was excluded from both indexes because an input is not a render. Right for the semantic index — a search for a picture must not return the face photo it was made from — and backwards for this one. An input is the picture somebody chose BECAUSE of who is in it: usually clear, frontal and well lit, which makes it the best evidence about a person the store holds, and every render made from it is derived from that face. Without it, a clip's frames had only each other to agree with. They stay invisible everywhere else without being handled specially: what the portal shows about a person is intersected with what the GALLERY can see, and the gallery does not list inputs — so a face found in one contributes to the grouping and to one other thing, deliberately. Change picture offers them, marked in, because the picture somebody was picked OUT of is usually the clearest photograph of them the store holds and that is exactly what a tile wants. They are still not in any count, not on the roster and not in the pane's list of their pictures.

The chooser takes the record's word, and that is the smaller half of the answer to "I see the input picture for some of them, not all". A face used to reach that grid by clustering into the person — a decision about how two pictures look — and the input that produced a clip is exactly the picture the embeddings are worst at placing, because a still is clean in ways a video frame never is. But offering a crop is not a claim: the record already says this person's render was made from that picture, somebody is looking at the result, and picking a wrong one costs a click. So Change picture lists every input this person's renders were made from, whatever the grouping did with them, and the grouping keeps its own standard for the questions that need one. Ownership still gates it — widening what is offered never widens what can be seen.

Including the ones nothing detected a face in, which is 172 of the 795 inputs here — 22%. A face too small or too turned for a detector, a still holding a body and no head, a frame whose extension lied about its format: every one of those is invisible to a grid of face crops however the grouping is tuned, because there is no crop to put in it. Both people reported after the fix above were exactly that. So an input that was scanned and yielded nothing is offered as the picture itself, and a cover pinned to one is served as its thumbnail — the id carries the file (file:<name>) and the crop route resolves the second shape, so no page had to learn that a cover is now two things. An input still queued is not offered whole: its crops are coming, and offering both would leave a duplicate of every input on the grid the moment the sweep caught up.

Whether that face IS them is the other half, and it does need deciding. The grouping is what makes an input count as this person elsewhere: their picture count, who a new render is recognised as, what a merge merges. So after DBSCAN a face is also joined to the person a record names — with a standard the chooser does not need.

Never on the record alone, and the reason is a measurement: over the 1536 provenance pairs in the store here the distances smear from 0 to 1.0 with no gap at all — 41% inside maxDistance, 27% out past 0.9 — because "made from" is often genuinely somebody else (a swap, a scene reference, a second face in the frame). There is no threshold that separates those two populations, which is why this one does not try to be one. What the link turns on is a tie-break: the face must be closer to the person the record names than to every other person it is not already part of. 76 of 651 candidates were, so it joins about 50 people where a bare threshold at the same ceiling would have joined 900. That is what lets faces.sourceDistance (0.80) sit past the 0.716 between two different people — a coincidence has to beat the whole roster to get there. It applies at grouping time, so it reaches an index already built and nothing is re-scanned.

"Not already part of" is the half that was missing, and without it this rule could only ever pick one winner per photograph. A picture gets used over and over — 408 of the inputs here back more than one render — and until a join is applied, each of those renders is a separate person that the input is also near. So the second render counted as "another person who is nearer" than the first, they refused each other, and only the closest one joined: three renders of one person arriving as three people. Measured on 8108 faces, 119 of those 408 pictures had all their renders with them. Applying each join as it is made, and taking the pairs strongest-first, takes that to 169 — with no cluster holding two differently-named people (164 named, 1540 faces) and no merge joining two groups of five faces or more.

And a listing can go further than the grouping will, on request. A person's pane offers Also show what was made from their pictures, which adds the renders whose record names an input this person has a face in — one hop, from their inputs and their own renders alike. It is a separate control and never a default, because it is a claim of a different kind: the record says a picture went in, not who came out, and joining on it alone produced one cluster of 4227 faces holding five differently-named people on the store this was measured against. So the record may widen what a pane shows, labelled as what it is, and never what the index says anybody is.

What it does not do, said plainly. It does not tell people apart in the sense a court would: it reports a distance and a threshold, and 0.5 is a setting. It has no opinion about who anybody is — names come from whoever typed them. It will not find somebody who is in one picture (that is minFaces, which now defaults to 1 — so that is only true where a host has raised it). And it is not a photo library: Immich still owns your photos, and this owns the renders this box made. nix build .#checks.x86_64-linux.faces asserts the clustering, the border-point rule that keeps two people apart, the id carry-over and the index's crash behaviour with no models present — every one of which fails as "0 people" and exit 0, which is what an empty store looks like too.

genai-face-index --people                 # what it thinks it knows
genai-face-index --match photo.jpg        # who is this, and how far off
genai-face-index --cluster --max-distance 0.42   # re-group, seconds
genai-face-index --group-from -           # group arbitrary paths

Tiles are thumbnails, generated once with ffmpeg and cached in the portal's state dir. That is not a nicety: a grid of sixty full-size renders is tens of megabytes to draw postage stamps (31.5MB measured, against 1.0MB now), and a video tile has no frame to show at all until enough of the clip has arrived — which is why clips appeared as black boxes that played perfectly when opened. Every tile is a still now, with a marker on the ones that are clips; the viewer still loads the real thing. If ffmpeg is missing the tile falls back to the original, so the picture is late rather than absent.

A clip in the viewer repeats if you ask it to, and so does part of one. The player is the browser's own — play, scrub, volume, fullscreen and whatever else that browser has — with the two things it does not offer added under it. Loop is sticky, because a run of five-second renders is watched the way a loop is watched and a box that forgets between two clips is one you re-tick on every one; it is remembered per browser and shared with the studio's strip, which mounts the same viewer. Set A and Set B mark the part that repeats at wherever the clip is now, on a track that can also be dragged to pick a section outright, and either mark turns Loop on — there is nothing else the marks do. Clear goes back to the whole clip.

Two things are worth knowing about how that behaves. A whole-clip loop is the browser's own and therefore seamless; a section is wrapped by the page, from an event that reports playback four times a second, so B is honoured to about a frame either side. And the section constrains PLAYING and not seeking — the player's scrubber still reaches the rest of the clip while it is paused, which is how you look at a frame outside the part you are looping. The section belongs to that clip and lasts as long as the page: walking through a run and back finds it still there, and a new tab starts on the whole clip again.

The listing pages itself. The next page is asked for about a screen before the bottom arrives, so scrolling through a store of any size is scrolling and nothing else — where a button was one press per sixty entries, and the deep end of a real store is a dozen of them. It appends below what you are reading and does not move the page: the picture you were looking at stays where it is and the new ones arrive under it. (An earlier version held the button under the pointer instead, by scrolling down past everything it had just added — which put you at the bottom of the page every time you asked for more of it.) The button is still there for a browser with no IntersectionObserver, and comes back — with the reason, and the grid intact — when a page fails to arrive, because a scroll that keeps re-asking a server which is already refusing is a request per frame.

A change made in the viewer does not cost you your place. A star, a tag, a rename, a delete and a render landing all leave the filters exactly as they were, so the listing is read again at the depth it was already at rather than rebuilt as page one — deletes included, because the entry going is not a reason to take the other seven hundred with it. The tiles that did not change are not even redrawn: each carries the markup it was last drawn from, and the grid is patched rather than replaced. That is what keeps the scroll position honest, and it matters more than it sounds — tiles decode synchronously (see below), so rebuilding several hundred of them to change one is a freeze and a flash on the way out of a viewer somebody just closed.

One caveat that took three wrong diagnoses to find: the portal now speaks HTTP/1.1 with keep-alive and listens with a backlog of 128. The stdlib defaults are HTTP/1.0 (a new connection per response) and a backlog of five — so a page asking for sixty thumbnails opened sixty connections and the kernel dropped whatever did not fit, before the process could see them. Blank tiles, and an access log showing nothing but 200s, because a connection that is never accepted is never logged. Tiles also retry once if their picture fails, which covers the same class of hiccup in a proxy.

Tiles are not lazy-loaded, which is deliberate: a page of thumbnails is about a megabyte, less than one of the renders it depicts, and deferring them broke the grid. A page is appended below what you are reading, so with lazy loading those tiles could sit unloaded until scrolled to — a tile showing the background is indistinguishable from a failed render, and it opened perfectly when clicked.

Each entry shows what produced it, read back out of the file itself: the genai record this stack stamps into a PNG (and writes beside anything that cannot hold one), falling back to stable-diffusion.cpp's own parameters chunk. Entries that carry neither are still listed, saying so — most of a long-lived box's history predates any of this, and an entry that says the parameters were not kept is worth more than an entry that does not exist. Where the record is complete, Regenerate re-runs it (same inputs, and the same seed unless the include seed box beside it is unticked — then the same settings roll a new one) and Open in studio hands it to the form to change first. Both go through the studio's form — Regenerate is the second one plus the Run — so a re-run is a thing you can see the settings of before it lands, and there is one path to a render rather than two implementations of it. A re-run that could not be restored faithfully (an engine or LoRA the box no longer has) is loaded but not started, and says which; neither is one fired into somebody else's render while the card is busy. Clicking the picture itself opens it at full size in a new tab — a plain link, so middle-click and "open in new tab" behave normally. A clip is not a link: there, a click belongs to the player.

An edit, a face swap or a workflow also shows what it was made from: a thumbnail per input under the parameters, each a link opening that file at full size in a new tab. Those files are the ones kept in sources/ so the render could be repeated, and the gallery deliberately does not list them — the raw material of a render is not a render — so this is the only place they can be reached from. A reference clip gets a frame like any other video; a voice sample has no frame to take and shows as a link with a mark instead. Nothing is stored for this: the thumbnails are derived from the record every edit already carries, so entries made long before the panel existed have it too. An input whose file has since been deleted by hand says so where its name is, rather than leaving a broken picture to mean it.

An input is named for the picture, so handing the same photograph in again reuses the file rather than storing it twice. Every studio submission used to write a fresh time-random file whatever it held: measured on this box, 1451 redundant copies under sources/, one photograph stored 347 times at 6.3MB — 2.1GB of one picture, and each copy detected, embedded and quality-scored by the face sweep as if it were a different photograph. The bytes are salted by whoever handed them in, because visibility here is ownership and nothing else — two accounts that upload the same picture get one each. A render is never deduplicated: two renders with the same bytes are two renders with two records.

Duplicates from before the stores were merged. Every tool result used to be written twice — media-tools made it, the portal fetched it back and stored its own copy — and only the portal's copy carried a record. Both are in the store now, which is why an entry could sit next to an identical one saying nothing was recorded. The portal reconciles them on start: copies are paired by their image data (PNG text chunks ignored, since the stamped copy differs from the plain one by exactly that), the pair keeps whichever record either half had, and the redundant copy is marked so the listing shows the render once. Nothing is deleted and no name disappears — a chat message linking to the other copy keeps working — and the space is reclaimed, if you want it, through Prune's duplicate copies rule. New renders are written once, so this converges to a no-op.

Nothing ages out. A render is the one artifact here that no rebuild can reproduce, so deleting is always something a person did:

• Select → tick entries → Delete, which names the count and the bytes before it does anything.
• Delete in the viewer, for the one you are looking at — including the studio's strip, which opens the same viewer with the same buttons.
• Prune applies one rule — older than N days, keep only the newest N, everything with no recorded parameters, masks and previews, or duplicate copies — and previews it first: the button that deletes carries the count the server just reported, and changing the rule takes that count away again. Input images kept for repeating an edit are never selected, and neither is the copy of a duplicated render that the gallery lists.

When the disk holding the store drops below mediaStore.diskWarnGB (default 20GB) a banner says so, on this page and in the studio. It never deletes anything to get back above it.

Studio (:8897/studio)

One form over three different backends, grouped by what you are trying to do rather than by which service does it — Create, Start from an image, People. Pick an operation, fill in the form, press the single Run button in the bar at the top. That bar is also where you see what is running and for how long — including work started somewhere else entirely, like Open WebUI or a chat tool.

A busy card does not grey Run out; it turns it into Add to queue. The only thing that disables the button is the FORM — needs: prompt is an incomplete request, and queueing one would move the refusal to a moment when nobody is looking. Everything else waits instead: the item is held by the server, drained one at a time, and asks for the card when its turn comes exactly as a render started by hand does. And it says that it landed, then stops saying it. Adding was silent: the id went on a list, the page refreshed, and the only thing that moved was a number inside a button nobody was looking at — so pressing Run read as having done nothing. The count now flashes, which is where the change is, and a line says the PLACE and goes on its own after a few seconds. The place is read back from the server rather than counted in the page (the drainer may have started it already, and "place 4" over something that is running is its own kind of wrong), and a paused queue is said out loud, because "added to a list that is stopped" is the one outcome nobody would guess from a count going up.

That line is a toast in the bottom-right corner — it started centred under the nav, which on the studio is exactly where the run bar is, so the line saying a press had landed sat on top of the buttons that were pressed. It lifts by --runbar-h, the studio's own measurement of that bar, because on a phone the bar is pinned to the BOTTOM and moves into the corner instead. GenaiGallery.toast, in the shared component beside modal and for the same reason: several pages want to say "that landed", and two spellings of a transient message is two looks. It is not a dialog and not a note left on the page. A note stays until something writes over it, which is right for a form's own state and wrong for the answer to a press — worth reading for a few seconds and clutter afterwards. Messages that tell somebody what they just did are not written at all: opening a render in the studio flashes the form it filled instead of announcing that it filled it. What survives as words is only what could not be inferred — an engine or a LoRA this box no longer has, an operation that opened as a different one, a re-run that was refused.

A count beside Run opens the list, and every row carries the inputs it will run on — thumbnails, with a batch showing the pictures it varies in the order they will be made. Which dozen is not something a label can say, and workflow_ltx23_animate · 1/12 over no pictures at all is the queue's version of a form that previews one picture for twelve. The item in PROGRESS is on the list too, because "what is happening and what is behind it" is one question: it carries the same strip, with Set aside and Stop and remove where the waiting ones have a Remove.

There are three ways to stop a batch, and the labels say which. A batch is one job over a dozen renders, so "this picture is wrong" and "stop the run" are different requests that used to be the same button — Stop on item four ended items five to twelve and nothing said it would. The row now carries Stop this one (the picture on the card; the rest of the batch carries on, and the stopped one is recorded as a failed item with its reason) beside Stop all and remove, and Set aside beside both of those — see below, it is the one that keeps the job. The bar's own button says which of the two whole-job stops it is and points at where the other lives. A single render has nothing to skip past and carries the other two.

And it says that it REMOVES. These read Stop and Stop them all, which is true and is only half of what they do: the job also leaves the list, and once there is a stop that KEEPS it, the removal is the entire difference between the two. On the bar the old label was worse than incomplete — it read Cancel, immediately to the right of Run, and two buttons side by side reading "Add to queue" and "Cancel" are read as the two ends of one decision about the form. They were never that. Underneath it is /api/studio/cancel with scope: "item" against the default "job", and the difference is one line: an item stop reaches the ENGINE and leaves the record alone, so the loop finds a live job and moves on. The failure path was always the skip path — it had just never been reachable on purpose.

Split the rest off breaks a running batch in two, immediately. The job on the card keeps the picture it is making and nothing else; everything it had not started becomes one waiting job on the line below it, which the arrows and the drag then apply to like any other row. That is the whole feature: the list has already changed when the button answers, so something else can be moved above the rest of the batch straight away. POST /api/studio/queue/split.

And it is the ONLY way work comes off a running batch. Run next is this plus a reorder — the promoted item first, what came off the batch behind it — and Pause is this plus a hold. Only a batch can be broken up: it is the one job here with a boundary that costs nothing to stop at, and a render stopped half way is lost. What it made is kept, is already in the store, and comes back as that job's own result, which says how many went and where; the remainder is rebuilt by handing the original submission back through the same builder a fresh Run goes through, with each.values narrowed to what is left. The picture on the card still finishes first, because that one is with the engine — the split changes when the LIST becomes true, not when the card comes free.

There used to be a second mechanism, and it promised instead of moving. A batch could be asked to YIELD: a flag on the record which the run read at its next boundary, at which point the values it had not reached became a queue entry. Same errand, done later — and later was the whole problem. On a batch of video renders the entry was a quarter of an hour away, so for all that time the queue and the row disagreed about what was going to happen; the only way to see any of it was a derived line on the running row explaining a state that need not have existed; and three controls wrote three different sentences about where the rest would land. A button whose only visible effect is its own disappearance is indistinguishable from one that does nothing, which is how it was reported. Splitting is the same handover with the promise taken out, and retiring the yield took a flag, a published field (yieldFor), a row that explained it and a second way for a run to end with it.

So job_split takes the tail off the list the batch loop is walking, there and then, and the run ends after the picture on the card because that list is empty. The order is what makes it safe. The take is atomic with the read, so the loop cannot start one of the values being handed over; the remainder is rebuilt after, and the values go back if that refuses; and the running job's own record is narrowed LAST, once the new job exists, so a portal that dies in the middle restores from a record that still has everything rather than one that has already given it away. The picture on the card is kept out of the half being handed over — _batch_pending's skip_inflight — because the one thing that must never happen is a value in both jobs, which is a render made twice on a card somebody else is waiting for. The check asserts every input is made exactly once, and that a restart in that state would not queue any of them twice.

It does not hold the queue, which is the difference from Pause. The card is given back when the picture on it ends and the list carries on, so this reorders one person's own work and reserves nothing — the property that keeps this queue from being a scheduler.

Set aside stops a job and KEEPS it. There were two answers to "get this off the card" and both ended the job, so wanting the card back for an hour and the work afterwards meant stopping the render, remembering what it was, and setting it up again by hand — the queue failing at the one thing it is for. Setting aside stops it now, puts what it has not made back on the owner's list, and leaves it there not waiting for a turn: the list carries on without it, whatever is behind it runs, and one press of Put it back returns it to the order it was sitting in. POST /api/studio/queue/aside, which the same row uses in both directions.

It must not start again by itself, and that is the whole design. Everything else here that stands work down puts it back as an ordinary waiting item, which the drainer picks up within the tick — so on a list with nothing else on it, "stop this" would mean "start this render over". The entry carries a flag the drainer skips, and skips WITHOUT claiming its owner's turn, because a hold that took the line would make this the Pause button next to it. It is in the saved list too: restoring one as an ordinary waiting item would start, on the next rebuild, exactly the render somebody stopped — the same failure, hours later, from a direction nobody is watching.

What it costs is the picture on the card, and no version of it does not: that one is with the engine, and the only free boundary in a batch is between two items — which is what Pause and Split already use, for the people who would rather wait. So it is ordinary weight rather than danger: nothing about it cannot be taken back, the job is on the list a line below with everything it made still in the store, and the one thing it does lose is named in the title rather than implied by a colour. A batch comes back as what it had not made INCLUDING the picture that was interrupted — a render made twice is recoverable and one silently skipped is not — and the record it came off stops claiming those values in the same breath, because a value in both jobs is the one thing this whole area exists to prevent.

A set-aside item carries no place. A place answers "when does this run", and numbering the one row that is never going to reach its turn pushes every row under it one place away from the truth — so the server publishes it as its own list beside queued, the dialog draws it under the queue in violet (the hue that already means held here), and the count on the button says set aside rather than folding into queued. It is yours alone with no administrator path, the line reordering and splitting already draw: stopping somebody's render is arbitration this box supports, and leaving an entry on their list for later is a different act. checks.set-aside asserts the properties that fail silently — the drainer leaves it and starts what is behind it, a restore brings it back the same way, every input of a batch is accounted for exactly once, and a render whose service has gone away is left RUNNING rather than stopped and then lost.

Nothing on a row promises anything any more, which is the other half of the same lesson. The running row used to carry a line while a batch was yielding — what it was doing, how many would go back and where — because there was a gap between the press and the boundary with nothing else to see. There is no gap: the rows below the running one ARE the answer.

A paused queue says so, in the one colour that already meant it. Violet — the hue the nav pill has always given a held card — on the run bar and on the pill. A pause is neither an error nor a warning, so it takes neither red nor amber; and it must not take the render teal, which was the complaint.

PAUSING and PAUSED are different states, and the difference is the one thing somebody watching wants to know: what is on the card finishes, so while that is still going the list has not stopped — it is stopping. So the bar stays teal and the spinner keeps turning, and the line reads pausing. Only when the work actually ends does it go violet and read paused.

There is no spinner on a paused bar — a ring with its animation stopped is not how anybody draws paused; it reads as a spinner that broke, which is worse than no mark at all. A pause mark takes its place, the same size so the row does not move as one replaces the other.

What "something is running" MEANS is the server's running list, not the page's own tracking and not busy. busy answers "would a render of yours be refused", which is false for your own work by design and false again on the controller when the render is executing on another node — so a batch of yours could be going with both saying nothing, and the bar called that paused (and, unpaused, showed nothing at all).

Pause holds your own list. Nothing new starts; what is on the card finishes and can still be stopped by either Stop. A pause that only stopped the queue would be one you wait an hour for — the commonest thing running when somebody reaches for it is a batch — so it SPLITS a running batch too, and the rest of it goes to the front of the held list. Resuming starts it like any other item.

Pause and Resume are on the run bar as well as in the list, and the second half of that is why: the button that opens the list is hidden when the list is empty, so a queue that was paused and then ran dry had no way back to Resume on the page at all. Inside the dialog they are pinned under the scroller with Done, for the reason the viewer's row is — what a dialog OFFERS must not be reachable only by scrolling past what it SHOWS, and a long queue is exactly the one somebody wants to stop.

The bar says each thing once. It used to read 10 queued on the button and 10 queued on the status line with the spinner between them — one fact twice, with the mark that says still going standing between the two copies of it. The button carries the COUNT and is the doorway to the list (and is not quiet, because a transparent box the same colour as the text beside it does not read as something you can press); the line at the end of the bar says what is on the card right now; the hold button says whether the list is held, by what it offers to do.

The bar is in two halves, and the order is what says which. Everything before the divider acts on WHAT IS ON THE SCREEN — Run / Add to queue / Save changes, Cancel editing, Clear form; everything after it acts on WORK ALREADY SUBMITTED — the count, the hold, Set aside, Stop and remove. They were interleaved, and the pair that did the damage was Run with a Cancel immediately to its right that ended a RENDER, while the button that changes your mind about the form was at the far end of the bar. The editing pair is one button, not three: with something waiting open in the form, Run reads Save changes, so the button beside it reads Cancel editing — it was a separate Stop editing past the queue controls, the same errand in different words in a place that made it look like a fourth thing the bar could do.

The status line is LAST, and that is a layout rule rather than a reading order. It rewrites itself every second — a label, a clock, a reason somebody else has the card — and in a flex row everything after it moves when it does, so the two buttons this bar exists to offer shifted sideways under the pointer while nothing about them had changed. Nothing follows it now, it takes whatever width is left rather than being sized by its text, and the spinner travels with it as one item so a long label cannot leave the mark on one line and the words it belongs to on the next. Below the width where it fits it ellipsizes rather than wrapping, because a sticky bar that changes HEIGHT as the words change is the same jitter one line down — and --runbar-h is what the notices under it are positioned against. The full text is on the element's title.

AND SO IS WHAT JUST HAPPENED. The studio used to keep three localStorage keys — the generate job, the edit job, and a hand-off list of the ones submitted but not yet watched — plus a rejoin() that picked them back up on load, and a pair of TRACKERS that followed one job each and reported it. Everything downstream inherited that scope: Done · 34s · Review these 3 appeared only in the window that pressed Run, a failure line likewise, and a render started at the desktop was invisible on the phone. Reloading the page changed which job was being narrated; a second tab gave a third answer.

There is no tracker now. /api/studio/jobs publishes what is done and what failed for the account alongside what is running and queued, and report() — called by the poll — draws the completion line and the failure line from it. Every page this account has open says the same thing at the same time. generating and editing went with the trackers: "does this page have a job in flight" is not a question anything should be asking, because the answer was different in every tab, and what the callers wanted is mineBusy() — a fact about the account.

The formatting did not move. reportGen and reportEdit are the tails of the two trackers, unchanged except for reading the job off the record instead of off what the tab remembered submitting — which is the tell: the formatting never needed to be the thing that followed the job.

A WINDOW RATHER THAN A LATCH. The box keeps a finished job's record for an hour, which is far too long for Done · 34s to sit at the top of a form; and "have I already announced this" cannot be the test, because a page opened afterwards has announced nothing and would then never show it. So it is the newest one while it is recent (five minutes), which is the same answer everywhere regardless of when the page was opened. Results are NOT in the poll — they can hold inline image data and every page reads it every three seconds — so the one record about to be reported is fetched once through /api/studio/job/<id>, which is already the per-job read scoped to the owner.

THE FORM IS THE ACCOUNT'S, NOT THE BROWSER'S. Where somebody left the studio — every field of every operation they have touched, which pictures are in which slot, which tool is showing — is one document per person on the box (/api/studio/form, kept in their record beside their templates), read once when a page loads and written back on a one-second debounce. It was localStorage, which is a decision about a single-user page this stopped being: the studio looked one way at the desktop and another on the phone, a second tab disagreed with the first, and a private window was a different person. None of that is a fact about a browser.

Read once rather than polled, because it is what the form is SHOWING and re-reading it under somebody's hands would undo what they are typing. Two tabs writing is last one wins, which is what one state per person means and is the honest answer rather than a merge nobody asked for. A failure to read it is an empty form and not a broken one — that is what a first visit is.

And the create form was never being saved at all, which the move uncovered. saveForm is only reached through syncAll(), and the fields of #form and #editform fired their own syncForm/syncEdit instead — so the prompt, the engine, the size, the steps and the seed on the form this page is most used through were written down only when something ELSE called syncAll (choosing an operation, attaching a picture). Type a prompt, reload without touching anything else, and it was gone. Not syncAll on those fields now either: it rebuilds the template list and the model cards, which is a lot to do per keystroke on the most-typed field on the page.

THE STATUS LINE HAS ONE WRITER, AND IT IS THE BOX. It says what is on the card and how long it has been there — workflow_minimax_h3_reference · 3/12 · 184s — drawn from the roster poll with the same label and the same clock the queue dialog's own rows use. How much is behind it is the button beside it. Whether this tab happened to start it does not come into it.

Getting there took three tries, and the failures are the argument. It said queued — place 1: two pieces of jargon and a number with no scale on it, since place is a position in your OWN list and nothing on the bar said there was a list of yours to have a position in. Naming the job made that worse rather than better — workflow_x · waiting its turn — yours is number 12 on your queue — because every part of that sentence is a fact about WHICH BROWSER IS ASKING. All twelve items were the same person's. The one singled out was the oldest job that tab had submitted, handed on by trackNext as each ended and picked back out of localStorage by rejoin() after a reload, so reloading the page changed the answer and a second tab gave a third one.

So the trackers do not write the line at all. They still exist, and what they are for is genuinely per-tab: knowing when the job they are following ENDS, which is where the result goes and where the strip is brought level. MYRUN went with the narration — it existed to tell "this page has a job in flight" from "a picture is being made", and the roster answers that as a fact rather than an inference.

A RENDER FIRST, then anything else of yours: the roster is sorted by how long each job has been going, so a gallery scan running for ten minutes would otherwise headline over the render somebody is actually waiting on — and the render is what the two buttons beside the line are about. What is on the card also OUTRANKS the reason Run is disabled now, which is the reverse of what it did: needs: prompt and somebody else has the card are both answers to "why can I not start one", and neither is news while a render of yours is being made. BLOCK goes on the Run button's own title as well, so a greyed button is never the one thing on the bar with nowhere to say why.

THE TWO STOPS READ THE BOX, NOT THIS BROWSER, and getting that wrong twice is what this paragraph records. They used to act on CURJOB — this TAB's notion of "the job I am narrating", which is the oldest job it submitted that has not finished, kept in localStorage so a reload rejoins it and handed on to the next of yours when one ends. That is exactly right for a progress line and indefensible for a destructive button, and it shipped in its worst form: while the followed job was merely QUEUED the bar offered Remove from the queue, meaning one arbitrary member of your list — chosen by which computer you were sitting at, unrelated to the form in front of you, surviving reloads, and moving to a different job on its own when the one before it finished. Every reading of that button was wrong except the one nobody would guess.

Narrowing it to "only while that job is on the card" fixed the arbitrariness and left the browser-tied half, which is the half that matters on a machine with accounts: a render started at the desktop had no Stop on the phone, and a tab reloaded past the end of its own job was following nothing at all. The card is a fact about the BOX. One render of yours is on it or none is, every device agrees, and the answer does not depend on which tab submitted it — so myRender() reads the roster poll and both buttons are the same two buttons everywhere. CURJOB went with the last thing that read it; the trackers keep generating/editing, which is what they actually needed.

A running job now publishes q — did it come off a queue — because the running list holds gallery scans and semantic index builds too, and offering to stop and remove somebody's scan is a question the page should not be able to put. It is the record's own field (_job_enqueue sets it, _job_start does not), not a judgement made in the view.

A press in flight is remembered as that job's id rather than as a flag, which is what lets the poll redraw these buttons three times a minute without re-enabling one under the finger that just used it — and what makes them come right by themselves when the card takes something else, where a flag needed a tracker to reset it. Removing something that has NOT started stays the queue's own errand: the dialog is one press away on the same bar, listing every item in order with its inputs and its own Remove, which is a thing you can point at before you press it. Putting a set-aside job back went with it, to the row that already has one.

A running batch can be edited, for the part it has not started — it does not have to be paused first. Those values have been sent nowhere, so they are as changeable as a waiting item's. Edit the rest opens the form on exactly what is left; the batch keeps running, and the rest picks up the new settings at its next boundary.

Opening it hands the form a SET, and the head of a set is the picture attached to the slot. Every submission the form makes carries that — args[each.arg] == each.values[0] — and it is how the form recognises a batch it is handed back. Narrowing moves the head, so a narrowing that filtered the value list and left the argument pointing at the old head produced a submission the form could not hold: it showed ONE input (the picture already being made), the bar said one run, and Save sent a single render with every other picture in the set dropped. Silently, which is what made it expensive — the row said twelve, the save sent one. It only ever bit the RUNNING entry, because that is the only one whose request is narrowed at all, so it read as "editing the first item in the queue loses the rest of the batch". The two lists and that one value are narrowed together now, and checks.batch-edit asserts it in both directions.

The race is closed at both ends, and that is the whole design. The form is never offered the picture on the card — it will be finished before an edit can land, so including it would be offering to make it twice. And the run does not adopt a picture it has since started: what comes back is filtered against what the batch has actually attempted, so an item that got going while you were typing keeps the settings it began with and everything after it takes the new ones. Neither end trusts the other's timing, which is why nothing is made twice and nothing already paid for is thrown away.

What a run cannot change about itself is the operation, the machine and which argument it varies — it is already inside those three. An edit changing any of them is refused where you are standing rather than silently ignored at a boundary minutes later, and the message says the alternative: pause it, and the rest becomes a fresh submission that can be anything.

The narrowing is one function, and it works from what the run has ACTUALLY attempted rather than from a count — a batch can take an edit mid-run, so an index into the list it started with stops meaning anything the moment one is adopted. Its one parameter is which side of the picture on the card the caller is on: an edit excludes it (it will be done before the edit lands), a remainder at a boundary has nothing in flight to exclude, and a restart keeps it, because nothing can know whether it landed and re-making a picture is recoverable where silently skipping one is not.

That last part is what keeps none of this from being a scheduler: pausing gives the card back. The prioritised job then asks for it exactly as a render started by hand does, and in the gap anybody else may take it. This reorders one person's own work and reserves nothing — handing the lease across would be a reservation, and the arbiter does not have those. Both are the owner's alone with no administrator path, for the reason reordering is.

A SERVICE THAT IS NOT THERE HAS NOT REFUSED ANYTHING, and until 2026-09-09 those were one thing. Every upstream failure arrived as the same RuntimeError, so the batch loop did to a REFUSED CONNECTION what it does to a render that came back wrong: recorded the reason, counted the item as attempted, moved on. Media-tools restarted under a running batch and thirteen renders were burned in one second, taking the single job somebody had just promoted with them. Nothing errored; the queue simply had fewer things in it, and the only trace was a run of lines in a log nobody reads.

So a request that never landed is its own class. A refused connection, a name that will not resolve, a reset, a connect timeout, a 502/503/504 — nothing was attempted, and the work is still work: a batch stops there and hands back everything it had not made, and a job that had made nothing goes back on the queue under its own id, keeping its place and its request. It comes back with the reason on the row and a wait before the drainer tries again, growing while the outage lasts, so a rebuild costs a pause rather than a queue. It never gives up, because a deadline here means deciding on somebody's behalf that their twenty renders are not worth keeping. A backed-off item HOLDS ITS OWNER'S LINE rather than letting the queue run out of order behind it.

And the handback itself must not need the node either. Rebuilding a remainder asks which machine serves the operation, which is a question the fleet answers by being asked — so during exactly this outage it refuses with 503, and a remainder dropped on that refusal is the same loss one level up, made by the function that was supposed to prevent it. A 503 puts the entry on the list carrying its request and lets it REBUILD ITSELF when its turn comes: still one builder, still every refusal it makes, asked at a moment when the answer can be yes. Anything else is a fact about the entry rather than about right now and is final, exactly as it is at restore.

A read timeout is deliberately not in that class: the service took that request and may be making the picture, so re-running it is a render made twice on a shared card. That is the one direction where being wrong costs the card rather than saving it. checks.queue-loss asserts the property rather than the path — every input of a batch accounted for exactly once, made or handed back, never lost and never in both.

A job that fails is still listed, with the reason. It stops being running and stops being queued, which from the list's point of view is indistinguishable from one that was never there — and "it disappeared" is exactly how that was reported. The error reached whichever page happened to be following that job, and a reload, another device or a glance at the queue saw a list one item shorter and nothing to say why. Failures now sit under the list for as long as the record survives, dimmed, each with its own reason; the count button opens the dialog even when they are the only thing in it; and the server prints a line for every job that ends badly, which it never did for a single render. Cancelled jobs are NOT in there: that one is a press, and reporting it back as a thing that went wrong is the page arguing with the person who made it.

What a running batch shows is what is LEFT of it. The inputs are published in run order, so the first eight of a twelve-item batch nine items in were exactly the ones already made: the strip answered "what did this do" while the label beside it said 9/12, and the two disagreed about the same job — with the nine renders it was showing already sitting in the gallery. The row now drops what has been made, says how many (9 made), rings the one on the card in the same teal every other surface uses for a render, and keeps the constants — a mask, a face — that every item runs with. That last distinction is what makes it correct rather than merely shorter: only the VARYING argument's values get used up.

Settings on any row opens what it will run with — the prompt, the steps, the seed, which LoRAs — because two runs of the same tool over the same picture differ by exactly that, and a row reading smart_edit · 1/12 says nothing about it. Read once, when the row is opened (/api/studio/job/<id>?details=1), and deliberately not carried by either poll: the listing is fetched by every page in the portal every three seconds, and the per-job route is polled every 1.2 seconds by whichever page is following a render. A prompt has no business in either. Between them the strip and the panel answer the question a queue actually raises — not "how many", but "is this one the one I meant, and should it come off".

Nothing on the list has happened yet, so both things you can do to a list are there. An item is dragged by the handle on its left, or moved with the arrows beside it — the same change either way, and both send the whole list (POST /api/studio/queue/order). Two gestures because neither one covers everybody: the arrows are the only version a keyboard or a screen reader can reach, and a drag is what a long list needs, since a dozen items reordered one place at a time is a dozen rounds of press, wait, find the row again. The row itself follows the pointer and the places renumber under it, so what is on the screen during the drag is the list that will be sent; a finger must take the handle, which is the only element that claims the gesture from the browser, or the list could not be scrolled on a phone at all. Edit opens it in the form it came from (POST /api/studio/queue/replace), where every setting, and which pictures it runs over, is changed the way it was chosen in the first place — the same Choose… dialog, the same batch strip, the same Add more… and the same per-chip remove. Run becomes Save changes while that is open, and the item KEEPS ITS PLACE throughout: it is still on the list, still numbered, and still what runs when its turn comes if nothing is saved. That is the difference from removing it and queueing it again, which sends it to the back — the one thing somebody adjusting a prompt on item two is not asking for.

The form is the editor, and that is the design rather than an implementation note. Everything a waiting item could be changed to is something the studio already knows how to ask for, and a second editor drawn inside the dialog would be a second form per operation, out of step with the real one from the day the next argument is added. So an edit goes back through the same load the gallery's Open in studio uses, and it is validated on arrival by the same builder a fresh press of Run goes through — same whitelist, same node resolution, same refusals. An item that STARTED while its settings were open is refused rather than edited, by name, because by then its arguments are with the engine and there is nothing left to decide.

Both are the owner's alone, with no administrator path. Stopping somebody's render is arbitration on a shared card and this box supports it; deciding what order they make their pictures in, or what prompt they run, is not.

What each row shows comes from the server's own copy of the job rather than from anything the browser remembers, so a phone lists what a laptop put on the queue — and reorders it, and edits it.

The list survives a restart, which is a reversal and the reasoning behind it is what changed. "A queue this process can no longer run is one it cannot honestly report on" was right while a queued item was a closure: nothing on disk could rebuild one, so a restored list would have been a screenful of work that could never start. An item carries its submission now — the same one an edit round-trips — so the list is rebuilt through the builder a fresh press of Run goes through, and an entry that cannot be (an operation that stopped being served) is dropped by name in the log rather than shown. The honest property is kept; what it cost has gone. It matters because the commonest reason this process restarts is a rebuild, and a rebuild is something you do deliberately in the middle of a working afternoon.

Saved to studio-queue.json in the portal's state dir by the drainer, which is the one thread that runs whatever moved the list — so no mutation has to remember to save, which is how half of them would not. A hold is saved with it: somebody who paused their queue and then rebuilt the box did not ask for it to start working again while they were not looking. What is RUNNING is not restored, with one exception: a batch has the rest of itself, and those values are exactly a queue entry, so they come back as one.

A restore races the thing it depends on, and that is the one part worth knowing about. Rebuilding an entry asks which node serves its operation, and that answer is a live fetch of each node's openapi.json — so a portal that comes up before media-tools does would get "no node is currently serving" for every entry and drop the whole queue, on precisely the event this exists for. So a refusal is read for which KIND it is: unknown operation is a fact about the entry and is final, no node is serving it is a fact about right now and is retried until the node appears or a few minutes pass. The drainer is held while any of that is outstanding — an entry arriving late must not find its place already run past, and a save mid-restore would write the shortened list over the file the rest is still being read from. The list is per person and holds nothing between two items — anybody else can take the card in the gap — which is what keeps it a work list on top of the arbiter rather than a scheduler inside it. It lives in the portal's memory with the jobs themselves, deliberately: a queue this process can no longer run is one it cannot honestly report on.

One control for every way of attaching something. Each file slot has a single Choose… button — not a file input beside a picker, which is two ways to answer one question with the browser's "No file chosen" as a third thing on the row saying nothing. It opens a dialog with four tabs, in the order somebody reaches for them:

tab	what it offers
This device	Browse (the camera roll on a phone), paste with ⌘V/Ctrl+V or a Paste button where the browser allows reading the clipboard, or drop a file anywhere in the dialog
Used before	everything handed to a tool on this box, which nothing had ever listed — the uploads in sources/ and the renders a second tool was run on
Made here	the gallery
Faces	search somebody by the name you gave them, then every picture they are in — from both of the two tabs above

The two things somebody most often attaches are the thing they attached last time and the thing the last render produced, and both were already in the store — reachable only by finding the file on disk again or by leaving for the gallery and coming back.

Faces is not a fourth place a file comes from — it is the first three ordered by who is in them. That is worth a tab rather than a filter for one reason: it is the only listing on the box that spans both halves of the store. A person's renders are in the gallery; the photograph they were handed to a tool as is in sources/, which the gallery is right never to list — and that photograph is usually the clearest, most frontal picture of them this box holds, which is exactly what an identity tool wants. Searching is the same dialog /people, the gallery and Studio's Find somebody… find people with — one implementation, because the tricky part is not the markup but the five different reasons a list comes back empty — including the switch for the groups nobody has named: a face and a count is all attaching one needs. Picking somebody turns the panel into their pictures — the same tiles, the same multi-select, the same Load more — with the ones handed in marked handed in, because a photograph of somebody and a render of somebody look identical and nothing else in the listing tells them apart. What comes back names them, so the field says a picture of Sam rather than a timestamp and eight hex digits.

Nothing there widens what anybody may see. The renders come from the gallery's own answer — ownership, every optional module's filter, the stricter toggle — and the inputs from the same ownership check that already serves all of sources/ to Used before; intermediates are left out, the way Made here leaves them out, because a mask is the same face one step less finished. An id naming somebody this viewer cannot see is refused exactly as one naming nobody is, or the difference between the two answers would be a way to enumerate people. On a box with no face index the tab says so rather than sending somebody to a page that would tell them the same thing, and an audio slot is not offered the tab at all: who is in a voice sample is not a question this box can answer.

A file from the device is handed BACK rather than uploaded by the dialog, so it goes through the same call the file input's own change event made: the kind check, the size note, the canvas a keyframe sizes, one upload path, none of it duplicated. Picking something already in the store costs no upload at all — what comes back is a URL. The button is injected next to any input[type=file] on the page rather than written into each field, because "everywhere a file can be attached" includes the fields a published workflow generates from its own schema — there is no markup to add a button to there, and a file input added later gets it without being told. An audio slot is offered clips as well as sound, the same rule its own field states: what somebody has of a voice is usually a video of them talking. Both halves are a page of 200, say so (200 newest of 694), and put Load more after the list inside the scroller — where the gallery puts its own, because it is the same control doing the same job. The renders half pages by OFFSET and keeps what it drew, like the gallery, so a store of any size is reachable a page at a time; asking for one ever-larger page instead is what silently stopped it at the listing endpoint's per-response bound, with a Load more still on screen re-fetching the tiles already shown. The renders half also honours the gallery's stricter filter.

Used before is not a directory listing, and the bug that made it one is worth keeping written down. sources/ is where a file from a DEVICE lands. A picture picked off Made here is already in the store, so nothing uploads it and nothing is written there — and a tab meaning "everything handed to a tool on this box" that lists only uploads is one that forgets the render you ran a second tool on ten minutes ago. The answer is DERIVED from the records instead: _persist_edit_inputs already writes what an operation ran on into the result's record, so the store could always answer this and nothing was asking it. Not a second copy of the bytes — the store deduplicates inputs by content precisely because 1451 redundant copies of one picture is what that costs — and nothing to migrate, because every edit ever made here gains its entry the moment the listing looks.

So the two halves are judged differently, and the render half is the stricter one. An upload is scoped by ownership alone: an optional module judges renders by their engine and LoRAs, and a file off somebody's camera has none of that to be judged on. A render offered back as an input has all three, so it passes the same visibility every tile in the gallery passed — ownership, every module's filter, this viewer's own stricter toggle. One function answers that for both tabs, because the second copy is the one that forgets the module, and a picker offering what the gallery beside it is hiding is a hole in the one control somebody set for themselves.

And the clock is "when it was last handed in", not the file's own mtime. An upload's timestamp is when the picture first ARRIVED — re-uploading the same bytes returns the existing name on purpose — so a photograph from March that somebody worked on this morning would sit below three hundred files nobody has touched since, which on a capped listing is indistinguishable from not being there. A file nothing has been run on yet keeps its arrival time, which is the only thing there is to sort it by.

Width and height are two fields, and matching the picture is a BUTTON. Attaching a keyframe still sizes the canvas from it — that is the derivation, and it is right, because a graph that stretches a keyframe onto its canvas comes back squashed when the two disagree and nobody should have to work that out. What is gone is the LINK: typing in one dimension used to rewrite the other from the source ratio, on the reasoning that asking for both is asking the same question twice. That is true of the common case and false as a rule — the deliberate mismatch is a thing people want — and it was not even consistently true. SRCASPECT was set only by an attachment made in the CURRENT session, so the same form with the same picture on it linked the fields before a reload and left them free after one, with nothing on screen saying which state it was in. A control that behaves differently depending on how long the tab has been open is worse than either behaviour alone, because neither one can be learned.

So both are always free, the line under them says whether they match (the picture is landscape 1.78:1; this canvas is 1.00:1), and Match the picture (1376×768) puts them back in one press — the only thing on the form that writes both at once, and a press rather than a side effect of typing. The line names both ratios rather than saying "does not match", because a mismatch is only actionable once you can see what it is; it makes no claim about what the graph will DO with the difference, since some stretch a keyframe and some fit it and the form does not know which — that belongs on the button's title. The aspect itself is remembered per operation, keyed like the attachment it is a fact about, so the line and the button survive a reload exactly as the picture does. Putting the picture back must not rewrite a canvas somebody has since typed, which is why restoreWfPicks re-derives nothing and the ratio is stored instead.

A REFERENCE IS NOT A KEYFRAME, and sizes nothing. Which slot the canvas follows is declared on the parameter (sizesCanvas) rather than worked out by the page, which used to ask "is this a video, or an image on a graph that takes no video" — right about every keyframe, and wrong about every reference slot on a graph with no clip in it. MiniMax H3's <Picture 1>/<Picture 2> slots and LiconMSR's subjects are encoded at their OWN aspect and never laid onto the frame, so the canvas is free of them; taking its shape from a portrait head-shot only ever picked the narrowest frame available, which is where a two-subject prompt loses one of them off the side of the picture. Those tools keep their declared landscape default now, and no line offers to match a picture that decides nothing — including from an aspect a session before the change had already stored. Only the slot the graph puts ON the canvas carries the flag: a keyframe, LiconMSR's scene plate (cover-cropped into the frame, while the subjects are letterboxed and never cropped), or H3's reference CLIP. At most one per workflow, and that is an assertion — two would mean the canvas depended on the order somebody filled the form in.

A worked example goes UNDER its field, not into the paragraph above it. A parameter may declare an example, which is published twice for its two readers: appended to the schema description, which is the only string a chat model reads, and separately as x-example, which the form shows as its own block below the input and takes back out of the text above so it appears once. The description says what a field is FOR and is read while you work that out; an example is reached for at the moment of typing, which is a different moment and a different place on the page. It is where somebody learns that H3's tags are spelled <Picture 1> and that with two of them both belong in the sentence as subjects — a shape a fragment cannot teach.

Picking several is how a batch is made, and the field says so. Hand a slot a dozen pictures and the run happens a dozen times with that argument replaced — one slot at a time, because two varying arguments is a cross product and the server takes one. The head of the set is bound to the slot exactly as a single pick always was, so everything the form already does with an attachment keeps working; beside it the field carries a strip of the rest (+11 more · runs 12×) that opens the same dialog the run bar's count does — thumbnails, each one the control that removes it. The strip is there because a slot can only ever preview ONE thing: without it a dozen pictures in a field looked exactly like one picked into it, and the only thing that said otherwise was a small button at the top of the page, on a tab you may not be looking at.

Choosing again REPLACES what the field holds; “Add more…” appends to it. A pick is the slot's whole answer rather than an addition to it, which is the right rule for a control that shows one preview — and it is invisible when what it replaced was a set, since the preview is the new head either way. So the button reads Replace… the moment there is something to replace, and a neutral line says what just happened (the 12 inputs on image were replaced by 3). Appending is its own control rather than a second meaning for the same one: Add more… beside the field, and in the batch dialog for when the field is a long way from the bar that reports it.

What another tool was handed is OFFERED, never carried. Attachments are scoped to the operation exactly as typed fields are — a clip reappearing under another graph's video is a picture nobody chose, sent by a form that does not show it. But arriving at the tool next door meaning to hand it the same picture is the ordinary case, and finding the file on disk again is a poor answer to it. So an empty slot offers the last one by name — Use beach.jpg, or Use 12 pictures where a set is what the last tool got — and only where it means the same thing: the same argument name, or the only field of that kind on this form. A second image field under another name is offered nothing, because a mask is not a source photograph.

The separation the page hides is kept at the boundary, in studio_op_target():

operation	backend
generate_image	image-server (:8893) — engines, LoRAs, sampler knobs
the nine edit/face operations	media-tools (:8894) — fixed graphs, hand-written forms
workflow_*	published comfyui.workflows — fields generated from the graph's own schema, minutes-long, possibly video

It is a whitelist per family, not a proxy: transcription and TTS live on :8894 too and are deliberately unreachable from here.

Reloading loses nothing. Renders are jobs on the server, so a reload rejoins one already running; the form's contents are saved as you type and come back with it — including what you attached. The operation is remembered per tab, so passing through People to look at something does not decide what Create was set to: one dropdown, three lists in it, and a tab you press to look is not a tab you press to choose.

Attachments are uploaded when you pick them, not when you press Run, and what the form keeps is the link. That is the whole trick: the box has to receive these bytes anyway (every input is written into the media store so the render can be repeated), so doing it at pick time means the form holds a URL a few dozen bytes long. It goes in localStorage with everything else, the picture is there after a reload — or in another tab, or on your phone — and the request that starts the render carries a link instead of megabytes of base64. Nothing large is kept in the browser at all. A render opened from the gallery skips the upload: its inputs are already in the store, so the form points straight at them.

Video results land in the same place as images: one media store, listed in the gallery.

Below the form, Recent is the gallery itself — the same component the Gallery page mounts, showing the newest dozen with a link to the rest. It is server-backed, so it is what the BOX has made rather than what this browser remembers: a phone that has never opened the studio still sees every render on it. Opening one gives the same viewer as the gallery — the same buttons in the same order, because it is the same component and the list of them is the component's, not the page's. Three of them write into the form: Regenerate (its settings, on the tab and operation that made it, and then Run), Load settings (the same without the Run — what the gallery page calls Open in studio, since there it navigates here first) and Load seed only.

The three are told apart by what they touch, which is why they are worded alike. Load settings replaces the form — operation and tab, prompt, engine, size, steps, LoRAs and their weights, and the inputs an edit started from. Load seed only touches nothing but the seed box, keeping the form you have been typing in; it is offered on renders Load settings is not, since a record can have kept the seed and lost the engine. The include seed tickbox sits after Regenerate and Load settings and qualifies both: it is the difference between "this picture again" and "these settings, rolled again". Unticked, the seed box is left blank — the studio says so on any load it did not start — so Regenerate re-rolls rather than reproduces. It is remembered, because which of those you are doing you are usually doing all afternoon, and it is offered only on a render that recorded a seed. Two more buttons act on the FILE rather than the recipe: Full size opens it in a new tab and Download saves a copy.

A finished run lands in Recent and says so in a line; it does not open itself. A render that went full-screen the moment it was done is right for the one somebody is sitting and watching, and wrong for everything else this page does: renders queue, so several finish while the tab is in the background, and each one opened a modal over the last — a stack to dismiss one at a time before the page could be used again. It also landed on top of whatever was being read, since the viewer stays open through Regenerate and browsing the gallery while a render runs is the ordinary way to use this. So every run — one render, a batch of twelve, an edit that answers with a picture plus an overlay plus a mask — brings Recent level, writes what it made into the notices column above the form (Done · flux · 34s · 1024x1024 · seed 7), and puts Open it (or Review these 3) beside it. Reviewing is one press rather than imposed, and a batch's failures are not hidden behind a full-screen picture of one of the renders that worked.

Settings worth having again get a name — the Templates row on the operation card. Load settings answers half of "that prompt came out right, do it again" and creates the other half: it brings the render's PICTURE back with it, and the picture is often the thing you were about to change. A template is the same object, saved from the form instead of from a render: the prompt, negative, engine, size, steps and LoRAs on Create; every argument of an edit or a swap; every parameter a published workflow generated from its own schema. The seed is left out unless you pin it, because a template carrying one makes every render off it the same picture.

Whether it carries the pictures is its own answer, chosen when you save it, and the two answers are different tools. Left out — the default — the settings land around whatever you have attached, which is "this recipe on that other photo". Kept, it brings its own back and replaces what is there, for a set-up whose reference face, mask or control image is part of the recipe and is tedious to find twice. Each row in the dialog says which it is, because the same button doing two different things to what you have attached is the one difference you need before pressing it.

The stored shape carries that decision rather than a flag beside it: no sources key means "leave what is attached alone", the key present means the template decides — an empty map included, which is a template that deliberately attaches nothing. That is the same question every preset out of the gallery already answers, so one applier serves both and the two cannot disagree. File references never live in request either way, because sources is where an input belongs and one place beats two: they are dropped by name for the hand-written tools and by value for a workflow, whose parameters are its own words. Prose is left alone — a file reference has no spaces in it and a prompt always does — so a prompt that mentions a path survives intact, and the exemplar boxes drawn on a source travel with that source and only with it.

A picture is NAMED, not copied. A store URL is a few dozen bytes where a copy would be a second store nothing prunes — but it means prune, or a delete in the gallery, can take an input out from under a template that still names it. Those are dropped on the way out rather than bound dead, counted, and said out loud both on the row and when you load one, because a slot that is quietly empty gets discovered at Run — or worse, not discovered at all, because whatever was attached before is still sitting in it.

Editing one is saving over it, because the settings are a form: there is nowhere else to change them and no second editor worth writing. Load it, adjust the form, then press Update on its row — which asks first, since nothing else on this box keeps a copy of the old settings, and carries the "keep the pictures" and "pin the seed" switches over set to whatever that template already was. Update appears only on templates for the tool you are on: the form on screen belongs to one operation, so updating another tool's row would store a swap's settings under a workflow's name.

They are yours — kept on the box beside your face names rather than in this browser, so they are on your phone too and survive a cleared profile, and nobody else sees them, administrator included. The dropdown loads one in a press and goes straight back to its placeholder, because a form stops matching the template it came from at the first keystroke. The Templates… button beside it is the rest: save the current form, update, rename, and forget — the last two asking first. Templates saved for other tools are listed there as well, and loading one takes you to its tool.

Every tab carries a Seed box (blank = random). Every render reports the seed it used — in its gallery entry, and as a Load seed only button in the viewer that fills the box for you — so "that one again, but weaker LoRA" is a real request instead of a re-roll and a hope. This is the tool for diagnosing a LoRA: hold the seed still and change one thing. It covers everything that samples; create_mask, list_regions and swap_face_fast have no seed because none of them is stochastic.

The start from an image and People tabs drive the same tools chat calls — edit, reimagine, inpaint, smart-edit, the three face swaps, mask preview — and have its own LoRA picker for the ones that take them. There the model is fixed by the operation rather than chosen to fit the LoRA, so the list shows only what that operation can bind (and follows the inpaint engine you pick); a selection that stops fitting is dropped rather than silently sent.

Naming a region is the part that fails, and there are four ways out. The segmenter behind change part of it answers "where is X" and never "what is here": it has no detect-all head and no label set, so a phrase it does not know and an object that is not in the frame come back the same — empty. That makes guessing synonyms the obvious move and a bad one. So:

• List what I can edit here (list_regions) sweeps a vocabulary and reports which words this picture answers to, with a confidence and how much of the frame each covers. Every word is a button that fills the form in. It is one image encode plus a cheap pass per word, and you can narrow it to just the words you were about to guess.
• A miss says how close it came. "nothing above 0.30, closest 0.24" and "nothing at all, closest 0.01" are different problems: the first is the Threshold box, the second is the wrong word. Clicking a below-cutoff result sets the threshold to suit it.
• Look inside (within) segments a container first and searches that crop. Every picture is seen at a fixed 1008px, so shoes in a full-length photo are a few pixels across — a resolution failure no rephrasing fixes. Finding the person first is what makes them visible.
• Point at it instead drags a box around one example and finds everything like it, with shift-drag marking something to leave alone. No vocabulary at all, which is the way out when no phrase works.

The same knobs are on the chat tools (within, threshold, and list_regions as its own tool), because a chat model hits this wall in exactly the same place and used to answer it by trying another synonym.

Below the form, a live reference explains how checkpoints/checkpoint-merges/fine-tunes (full models = engines) differ from LoRAs (adapters bound to a base architecture), and lists every generation engine, editing pipeline, installed LoRA (with trigger word and chat/API usage), and chat tool — sourced from image-server's /catalog and media-tools' /openapi.json, so it's always current.

3D view (/viz)

The box as a machine: the GPU, RAM, the model store and the CPU, with data moving between them at the rate the counters actually report, and a band of software chips above showing which services are running and what each one is driving.

The rack is a bar chart, so a row is a FILE. Two bays either side of the spine — the language models left, the media store by category right — where a tray's length is what it takes on disk. Several catalog entries routinely share one GGUF, because a second context window on the same weights is a new entry and never a new download; those share the tray, which says how many names are on it (qwen38-27b · 4 names · 36 GB). Drawing them separately counted the same 36 GB four times and put the deck header 150 GB over the truth. The models that take no card — embeddings, rerank, transcription, the CPU chat pair — fold into one always-on row: ~6 GB against a fleet of 826, and five captions saying so was the least informative part of the column. A bay never grows past the frame: rows compress to fit, and past the point where 11px labels stop reading as a column the smallest fold into +N smaller rather than scrolling off the top the way the biggest models used to.

A chip lights up on evidence, in two tiers. A service that publishes what it is doing gets quoted directly — llama-swap names the model it is holding, image-server and media-tools name their running job, and a ComfyUI graph is reported by the media-tools call that queued it (ComfyUI's own /queue is never polled: that would defeat its idle stop). Everything else — LiteLLM, Magentic-UI, Open WebUI, the tool servers — publishes nothing per request and never will, since most of it is somebody else's code. Those fall back to their systemd unit's cgroup CPU (portal.extraServices.*.unit for your own cards), which is a measurement of that service and nobody else, and they draw their control line to the CPU rather than the card.

The comparison is against each service's own idle floor, not a fixed threshold, and that is measured rather than chosen: the shipped services idle anywhere from 0.03% of a core (most of the Python servers) to 13.8% (magentic-ui, which is simply that busy doing nothing). One threshold would either light the busy ones permanently or never notice the quiet ones working.

What no tier will do is guess. A service with no cgroup — anything socket-activated and currently stopped — reports up/down and nothing else, because this view is read to find out who is holding the card, and an invented "busy" is worse than no answer.

The card (/gpu)

One GPU, several people, and every engine here was written as if it were alone with it. Each resolved contention by evicting whatever was there: a render asks llama-swap to unload before it starts, and a chat model starting asked systemd to stop ComfyUI. Both were correct for one person. The second was destructive even for one person — a coding harness left running would stop a video render its own owner had forgotten about, mid-graph, and the render died with its VRAM.

So every path that wants the card asks the portal first, and is told either the id of a lease or what is in the way. A lease carries an expiry its holder renews while it works, which is the part that matters: a crashed render gives the card back by failing to renew rather than by somebody noticing. It is not a scheduler — there is no queue and no fairness. A blocked request waits out gpu.arbitrate.waitSeconds (20 by default, in case the card comes free) and is then refused with a message naming what is in the way.

The one asymmetry is deliberate, in both directions:

wants the card	free	a render is running	chat is running	somebody's hold
a render	yes	yours: yes · theirs: no	yes	yours: yes · theirs: no
chat / an agent	yes	no	yes	yours: yes · theirs: no
training	yes	no	no	yours: yes · theirs: no
takes no card¹	yes	yes	yes	yes

¹ embed, rerank, asr, voice, fast-cpu — everything the catalog marks serve.group = "resident" or serve.device = "cpu", which is the same predicate genai-gpu-claim uses to decide who may skip yielding.

A render takes the card from chat safely — it calls /unload first and waits for the drain — so gating renders on chat would disable the studio for most of the day over a collision that cannot happen. Chat taking it from a render is not safe at all, and that is the direction that was losing work. Set gpu.arbitrate.chatBlocksRender = true to make the card strictly exclusive instead. Anything that does not take the card is exempt entirely — the resident set (~2.6GB of embeddings, rerank and STT), because refusing a memory lookup while somebody renders would break retrieval for no gain, and the CPU models (voice, fast-cpu), which use no VRAM at all and exist precisely so a spoken turn never has to fight for the card. Your own render does not block your next one — the engines have always queued those, and queueing beats refusing.

Refusals look the same everywhere: 503 with a Retry-After, which the OpenAI, Anthropic and Vercel SDKs already back off on, so a coding harness handles a busy card without being taught anything. 429 would be wrong — it means "slow down", not "somebody else has the machine".

Holding the card (gpu.holds.*) is for working in bursts, where the gaps between renders are exactly when somebody else's agent session moves in. It lasts until released, until holds.idleMinutes passes with no work started under it (15 by default — the forgotten-tab guard, and what makes a manual lock safe to offer), or until holds.maxMinutes. An administrator can break one, and the person whose hold it was is told so on their own page rather than left to work out why their next render was refused.

A hold shows in the nav on every page, as a red ◆ on the activity pill beside the health pill — because a hold is the state you forget you are in, which is the whole reason it has an idle timer. It rides on the pill rather than replacing it: an idle-but-held box reads held by you in red, while a hold during a render keeps the render's line and adds the marker, since the render is the more urgent fact and both are true. The marker survives the narrow layout that hides the pill's text, and it disappears when the poll fails — an absent marker must not read as "not held" when nothing was learned.

Stop reaches every surface from one button: studio jobs, image-server renders, chat-tool calls, and in-flight chat or agent streams — closing the upstream socket, which is the only cancellation llama-server offers. Your own work from anywhere; anybody's if you administer the box.

Who you see. An administrator sees who is on the card, because arbitrating a shared machine means knowing who to talk to. Nobody but the owner sees the label or the engine — a prompt is somebody's work, not a fact about the schedule — so everyone else gets "someone else · a video render · 4m". That line is drawn in the API, not the page.

What this does not cover, and it matters. ComfyUI's own interface has no accounts, so a render started there is attributed to nobody and (with gpu.cancel.comfyuiUi) stoppable only by an administrator. The engine ports on the LAN — :8893, :8894, :8188 — take work from anyone who can reach them; genai-gpu-claim refuses to evict a busy ComfyUI and asks the arbiter before a model starts, which covers anything that goes straight to :8080, but the whole arrangement is cooperative. This keeps people from walking into each other. It is not a boundary against somebody determined.

It fails open, everywhere. An unreachable portal means every engine behaves exactly as it did before any of this existed. A box that cannot render because a status page is down would be a worse failure than an unarbitrated one.

Coding harnesses reach the LLM through LiteLLM, which builds a fresh request and drops what the client sent — so general_settings.forward_client_headers_to_llm_api is on, and a harness that sets one of gpu.identityHeaders (X-Genai-User, or Open WebUI's X-OpenWebUI-User-Id) gets named. One that sets none is a single shared unattributed bucket: it may use a free card, it may never hold one, and anybody else's hold refuses it. Never the owner — the failure mode is being turned away, not inheriting somebody's session.

Admin (/admin)

Only present when identity.mode = "trusted-header" — a single-user box has one occupant and nothing to divide, so the page is not routed there rather than rendering a table with one row. It is also left out of the nav for anyone who is not an administrator: the API behind it refuses them whoever reaches it, and an entry that answers "you are not an administrator" is furniture rather than navigation.

One row per person the box has seen, and for each: whether they administer it, whether they are restricted to models that refuse what their vendor made them refuse, whether they may hold the GPU, and a column for each group of settings an optional module declares. Every control is tri-state — on, off, or default, and the third is not decoration. Losing it would make "I decided this" indistinguishable from "I happen to agree with the box", which is the thing the page exists to show; clearing a control deletes the setting rather than storing today's default, so a later change to the host config still moves that person.

Administrator resolves in this order, and the row says which one it landed on:

a setting made here	wins over everything, so a decision is not silently undone by an SSO role changing
the platform's role header	an SSO admin works on arrival, with no rebuild
identity.admins	the operator's build-time list

You cannot remove your own access — the API refuses it and the control is disabled rather than left to fail. This page is the only door, so doing it would leave the box with one fewer administrator and no way for that person to undo it, possibly none at all.

Two limits worth knowing. The role header describes the caller and nobody else, so the box learns what the platform thinks of someone only when that person makes a request; until then their row reads not an administrator, and an explicit setting is the only way to decide about somebody in advance. And there is no user table — people appear here once they have opened the portal or made something, so somebody who has only ever used another service is not listed yet.

Being an administrator does not include reading anybody's renders. The gallery is filtered by ownership for everyone, operator included: private by default has to mean private from the operator too, or it is a setting rather than a property. Prune remains an admin operation and still walks the whole store — deleting by a rule without being shown anything is a different power from browsing, and it is the one an operator running out of disk needs.

An unidentified request is nobody, not the owner. If the proxy does not send the identity header, the caller owns nothing, sees nothing, is not an administrator whatever the role header says, and cannot write — the gallery says so rather than looking empty. The one exception is loopback: the box's own services (the seeder, media-tools, the voice server) reach this port with no proxy in front of them to be labelled by, and act on the box's behalf. This is the failure that has to fail closed. When it did not, an authenticated stranger whose header the proxy happened to omit became the owner — the owner's whole gallery, the owner's admin rights, and no second name anywhere on this page to hint that anything had gone wrong.

Unfiltered models (uncensored.*)

Some models decline whole categories of request — a base model's safety training, or a vendor's political filtering. Others are the same weights with that direction removed, or were never trained to refuse at all. A catalog entry says which with uncensored = true, and this stack reads it.

Both directions of that choice are worth having. It is why the abliterations are on this box at all: a refusal on a legitimate medical, security or historical question is a wrong answer, not a safe one. It is also a reasonable thing to keep an account away from on a shared machine.

uncensored.showDefault (on) is whether a person is offered what is marked — the portal's model and engine lists, the chat tools' schema, the Open WebUI picker. Turned off, those drop everything marked and a request that names one outright is refused. A DEFAULT, overridable per person on /admin, and phrased as show so that every control on that page reads "on means they get it".

On, because what turning it off can do depends on the medium and pretending otherwise would be the mistake. For chat it yields a real subset — official Qwen/gpt-oss against huihui/Heretic — so restricting somebody to it means something. For images essentially every local checkpoint is unfiltered, so an honest marking marks nearly all of them and this alone would leave a restricted person with almost nothing to render on.

It filters the SOURCE lists, never a rendered schema: list_loras() and the engine tables feed the tool spec, the parameter descriptions and the validation, so one filter covers what is offered and what is accepted, and they cannot disagree. It is not a permission system — :8893 and :8894 are on the LAN and take every engine from anyone who can reach them.

Labels this stack does not read (flags)

Every catalog entry carries a free flags attrset, and nothing here reads any of it. A flag is a claim about a model that some other module acts on — one a host imports beside this one — and it is free-form rather than a fixed set of booleans because the module that cares may not be loaded.

That is the whole point. An entry keeps its labels either way, plugins.<name>.* keeps its settings either way, and turning such a module off is a line in a host config rather than an edit to every entry that mentioned it. LoRA sidecars carry them through as well (lora-add --flag, lora-train install --flag, and mediaModels.<name>.flags via genai-fetch-media), so a reader working from the store sees the same labels as one working from the config.

See Optional modules for what such a module can attach to.

Optional modules (plugins, pluginModules, pluginAssets)

Some behaviour does not belong to everybody who runs this box, and the honest place for it is a module a host imports beside this one. A model catalog was the first of those and needed nothing but the module system: catalogs ship at mkDefault, so another module's entries merge with them. Behaviour is harder, because the services are single-file Python and a page is a single file of HTML — there is nowhere for a second module's code to go. Four options are that nowhere:

option	what it supplies
plugins.<name>.*	a settings namespace this stack declares and never reads
pluginModules	Python loaded into the portal and media-tools, asked for the extension points it implements
pluginAssets	browser code concatenated as /assets/plugin.js, which every page loads with the chrome
pluginEnv / pluginPackages	configuration and commands for both halves

Plus lib.flaggedNames, lib.torchEnv and lib.fetchMedia, so a module does not carry its own copy of catalog-name flattening, a second gigabyte of torch, or an instruction to run a fetch by hand.

Absence is not an error. plugins is freeform: a host that sets plugins.foo.something = true and then stops importing foo gets a value nothing reads, not an evaluation failure. That is what makes "try the box without it" a one-line change.

A failed import is fatal. These modules exist to filter, refuse and relabel; one that quietly did not load leaves a box that looks configured and behaves as though it were not. The service fails to start and says which file and why.

This is not a sandbox. A plugin module runs inside the portal's own process with its privileges — it is another way to write part of this stack, kept in another repo, and it is trusted exactly as much as this repo is.

The screen as a status display (kiosk.enable)

Off by default. When it's on, the machine's own monitor becomes the status display: a job starts, and if nobody has touched the keyboard for kiosk.idleSeconds (5 minutes by default) the screen wakes and /viz opens full-screen. The job finishes, and kiosk.offDelaySeconds later the browser closes and the monitor goes back to sleep. Touch the machine at any point and it hands the screen straight back — kiosk closed, no sleep scheduled — and stays out of the way until the session goes idle again.

What counts as a job is kiosk.activities, and the default is the work that takes minutes: a render, a graph on the card, weights loading. LLM inference is deliberately not in it — on a desktop that would wake the monitor for every chat turn — so a box whose LLM work does run for many minutes (an agentic coding harness) wants inference added. Without it the display comes up for loading-model and then leaves as the actual work starts, which is the wrong half of the job to watch.

The off-delay is a minute rather than the few seconds it takes to read the last frame, because "idle" is sampled and real work is intermittent at that resolution: a coding run alternates decode with tool calls the GPU sits out, and a multi-step graph goes quiet between steps. The delay is what bridges those gaps, so it wants to be longer than the longest ordinary pause in the work.

A locked session is what makes this feature work or not. A Wayland session lock is exclusive — nothing draws over it — so a kiosk window opened under one is invisible and all you get is a monitor waking up to show a lock screen. The answer is therefore not to draw over the lock but to keep it from engaging: while a job runs the daemon holds systemd-inhibit --what=idle (kiosk.preventLock, on by default), which puts idle in logind's BlockInhibited — the property an idle daemon reads before firing its lock and screen-off timers. It is released the moment the work stops, and on shutdown, since an inhibitor that outlived its daemon would be a box that quietly stopped locking. It is also the one inhibitor that does not suppress the compositor's own idle notifications, so the daemon can still tell when a human comes back; a raw Wayland idle-inhibitor would blind it.

There is no default for "is the session already locked" (kiosk.lockedCommand) and that is deliberate — see the option.

A locked screen is let into, not drawn over. Nothing can be drawn over a Wayland session lock; that exclusivity is the whole point of the protocol. The way in is therefore not a prettier lock screen but an unlock: with kiosk.unlockCommand set, a session that is locked while nobody is there gets opened for the duration of the job and re-locked (relockCommand) the instant either the work stops or a human touches the machine — the screen is always handed back exactly as it was found, because nobody authenticated to open it. Without an unlock command the daemon leaves a locked screen dark, on the grounds that waking a monitor to show a password prompt is worse than leaving it off.

That requires a locker that can be opened by something other than a typed password. DankMaterialShell's has dms ipc call lock unlock; hyprlock does not (PAM only, no D-Bus unlock), so a host using it gets the dark-screen behaviour above.

services.genai-server.kiosk = {
  lockedCommand = ''test "$(dms ipc call lock isLocked)" = true'';
  unlockCommand = "dms ipc call lock unlock";
  relockCommand = "dms ipc call lock lock";
};

A locked session also short-circuits idleSeconds. The lock means the seat already sat untouched for the session's own idle timeout, and that verdict survives this daemon restarting — an idle notification always counts from zero, so measuring presence only with the daemon's own timer made every rebuild ignore an empty, locked seat for another full period. The shortcut lapses the moment the idle watcher reports input: somebody typing at that lock screen is present, and their session is not this daemon's to open.

(Two other routes were built and measured before this one, and neither shipped: a GTK4 + WebKitGTK layer-shell client dies inside swaylock-plugin's nested compositor for want of xdg_wm_base, and windowtolayer in front of chromium rendered cleanly 1 run in 3 — the rest lost the buffer-size race with the acked configure and left swaylock's default white.)

This is the one piece of the stack that runs as a user service rather than a system one, because waking a monitor and knowing whether somebody is present are questions only the compositor can answer. That is also its one precondition: it needs a graphical session to run in, so a rebooted box that nobody has logged into shows nothing. The console at that point belongs to the display manager's greeter — a separate user, its own compositor, its own idle timer — and it is that timer, not this daemon's offDelaySeconds, blanking the screen. The tell is an empty journal: systemctl --user status genai-kiosk says inactive (dead) because the unit never started. unlockCommand is not the way in either, and deliberately: a greeter is a login prompt rather than a lock, and letting yourself into a lock and putting it back is not the same act as authenticating as somebody. Wanting the view unattended from boot means a console autologin. Presence comes from swayidle against ext-idle-notify-v1 (niri, sway, Hyprland and KDE all implement it); the screen is driven by whichever compositor CLI answers a probe — niri, swaymsg, hyprctl, wlopm, xset. Both are overridable (kiosk.idleCommand, kiosk.wakeCommand, kiosk.sleepCommand) for anything else.

kiosk.activities picks what counts as work. The default is the work that takes minutes — generating-media, gpu-busy, loading-model — and deliberately excludes inference, which is true of every chat turn on the box; a monitor that lights up whenever somebody asks a question is a nuisance rather than a display.

Metrics

Prometheus metrics (metrics.enable, default on) come in two layers:

• :8897/metrics — aggregate gauges from the dashboard's samplers: GPU utilization/VRAM/temperature/power, CPU, RAM, per-service up/down probes, per-model ready/enabled/size.
• :8080/upstream/<model>/metrics — llama-server's own per-model metrics (tokens/s, KV usage, queue depth) for whichever models are loaded.

Host-side scrape config:

services.prometheus.scrapeConfigs = [
  { job_name = "genai";       static_configs = [{ targets = [ "localhost:8897" ]; }]; }
  { job_name = "genai-qwen";  metrics_path = "/upstream/qwen/metrics";
    static_configs = [{ targets = [ "localhost:8080" ]; }]; }
];

Retired 2026-07: genai-gpu-watchdog (and its watchdog.* options) existed solely to restart whisper-server out of a whisper.cpp CUDA-context wedge — GPU utilization pegged at ~100% with the memory controller idle, forever. It gated on a whisper process holding the card, so replacing whisper.cpp with the asr llama-swap model removed both the failure mode and the watchdog's ability to fire.

Web search

Two ways models reach the web, both backed by the local SearXNG (no API keys):

1. Open WebUI search toggle — preconfigured via env; flip the web-search toggle in any chat and results are injected into context. Works with every model.

2. Agentic tool calling — fully declarative: the open-webui-declarative-config service enforces the tool-server registration and, for every chat model, attaches the server:web-search tool and sets native function calling (via the admin API; runs after each activation, preserves other UI customizations). Models decide on their own when to search. Any other agent framework can use the same endpoint:

   curl -X POST http://<host>:8891/search -H 'Content-Type: application/json' \
        -d '{"query": "nixos flakes tutorial", "max_results": 5}'
   

Storage

This is a shared service flake: nothing lives in home folders and no username is hardcoded (see CLAUDE.md for the conventions). All model weights live in one store so no file is ever downloaded twice:

/var/lib/genai-models/
  llm/                LLM GGUFs (llama.cpp HF-cache layout; genai-prefetch)
  diffusion_models/   Z-Image, FLUX.1 GGUFs + fp8, FLUX.2, Wan
  checkpoints/        Pony / epiCRealism / Juggernaut, LTX, ACE-Step
  text_encoders/      clip_l, t5xxl, umt5, Qwen/Mistral encoders
  vae/                Z-Image / FLUX.1 / FLUX.2 / SDXL-fix / Wan VAEs
  loras/              shared LoRA store + {trigger,base} sidecars
  pulid/  insightface/  identity-adapter weights
  hf-cache/           shared HF_HOME (training bases, EVA-CLIP, ...)
/var/lib/genai-comfyui, /var/lib/genai-ai-toolkit, /var/lib/genai-lora-jobs   apps + jobs
/var/lib/genai-media/     everything GENERATED (mediaStore.dir)
  <time>-<hex>.png/.mp4     renders, clips, masks — with a .json record
  sources/                  input images kept so an edit can be repeated

The same rule as the model store, applied to the other direction: one tree, written once. The portal writes studio renders there, media-tools writes every tool artifact there, both serve the same files (:8897/studio/ images/<name> and :8894/files/<name> are two doors onto one file), and the gallery lists it. It is deliberately not a service StateDirectory: those live under /var/lib/private at 0700 root, where the other service cannot follow — which is exactly how the box ended up writing every tool-made picture twice, once in each service's private state, with the gallery able to see only one of them. Same permissions as the model store, and for the same reason: 2775 root:genai, so a member of genai can delete their own media.

stable-diffusion.cpp, ComfyUI (via extra-model-paths), and the trainers all read the same files — one clip_l, one FLUX VAE, one Pony checkpoint for both inference and training, and a LoRA that lands once appears in chat, the API, and ComfyUI's loader nodes simultaneously. The catalog is declarative: services.genai-server.llmModels and .mediaModels (add a model = add an option entry, and the units fetch it — a declared model is a downloaded model. The two bulk sets, comfy and h3, are the exception named in comfyui.optInModelSets: they wait for a host to list them in comfyui.modelSets, which is still a declaration, not a command somebody runs. genai-fetch-media <set> is what the units call and what you reach for when debugging one — never the way a set gets enabled). Dirs are root:genai 2775 — members of the genai group manage models and training jobs without sudo (join it the standard NixOS way: users.users.<name>.extraGroups = [ "genai" ]).

With one limit on "a declared model is a downloaded model": storeReserveGB (default 20). The hardware floors decide whether a machine can run a model — VRAM, host RAM, engine version — and never whether there is room to keep one, so a node with a big card and a small disk passes every check there is and then fills its filesystem. Both fetchers measure free space immediately before each download, which is the only moment the answer is true, and skip with a warning rather than crossing the line — the same call a tier = "cold" entry makes when its archive is unreachable, and for the same reason: a run that aborted the set would turn one tight disk into no models at all. genai-doctor and the portal's health rollup report against the same number, so what they call a problem and what a download refuses to do are one line rather than three opinions.

A second copy elsewhere (remoteStore)

Off by default. Points the store at an archive on another machine — the same relative layout, one tree, reachable two ways:

services.genai-server = {
  remoteStore = {
    enable = true;
    url  = "http://10.0.0.1:8898/genai-models";  # restore: tried before the internet
    path = "/mnt/genai-archive";                 # cold tier: something to symlink into
  };
  # NFS carries numeric ids. Pin these, and match the export's anonuid/anongid.
  serviceUser = { uid = 890; gid = 890; };
};

Restore (url). Every download tries the archive first and falls through to the catalog URL on any miss, so an incomplete or offline archive costs one HEAD request and nothing else. LLM blobs are still verified against Hugging Face's x-linked-etag sha256 whichever source they came from — the mirror can make a fetch faster, never wronger.

Be honest about what this buys. genai-prefetch measures ~104MB/s against Hugging Face with 16 connections — so unless the LAN link comfortably beats that, restoring from the archive is not obviously faster than downloading again, and the win is availability, not speed: repos get re-quantized and deleted, Civitai versions go early-access, and a local copy is the only one that still answers the same bytes next year.

Measure rather than assume, in both directions. A full-duplex wired link is the easy case; a good 5/6GHz link can land in the same range as the CDN, and a mediocre one is nowhere near. genai-store-sync prints the effective rate of every pass that moves anything, so the first backfill answers this for your link without anyone timing it by hand.

Cold tier (path + tier = "cold"). Per catalog entry, weights stay on the archive and a symlink stands in for them at the canonical store path. Every reader opens a path and none of them inspects where it lands, so ComfyUI, image-server and llama.cpp needed no changes at all. Cold LLMs are served with --no-mmap automatically — demand-paging a GGUF across a network filesystem turns one sequential read into tens of thousands of random faults, and llama.cpp touches every weight during a forward pass anyway, so there is no laziness to win.

It trades disk for load time, every load. llama-swap evicts and reloads on every model switch, so a cold chat model pays the network read each time it comes back — this suits an occasionally-used specialist, not anything in a rotation.

On a shared medium — WiFi, or a link with other traffic on it — budget for the variance rather than the average. A cold model's load time is however long its weights take to cross the link right now, so the same model can load in two minutes at 3am and six at 8pm with nothing having changed. That is survivable for something reached a few times a week and quietly infuriating for anything else.

Keeping it current. A genai-store-sync timer backfills declared weights that exist locally and not on the archive, and carries out a tier change in whichever direction the catalog now says (freeze up, thaw down). A freeze copies, verifies the copy, and only then deletes the local file. Run it by hand to watch, or --dry-run to see what would move:

genai-store-sync --dry-run     # what would move, and which way
genai-store-sync               # do it (the timer runs this)
genai-prune --archive          # extend orphan deletion to the archive copies

The archive must be writable, not a read-only export: llama-server writes multimodal projector sidecars next to the weights it loads.

Guards, because the failure modes here are quiet ones. A tier = "cold" entry with no remoteStore.path fails eval rather than becoming a model that is declared and never fetched. The fetchers skip a cold entry whose archive is unreachable instead of downloading it onto the disk it was moved off. genai-doctor reports an unmounted archive, a read-only one, and any dangling cold symlink — the failure that looks like success, since ls shows the model and only open() disagrees — and the portal's /api/health fails on all three. genai-prune counts cold bytes on the archive's ledger rather than claiming to reclaim local disk it never had.

The other end is a HomeFree app (homefree-genai's genai-archive): one directory on the router's pool, exported read-write over NFS and served read-only over HTTP on the LAN address.

Running on different hardware

This covers how much VRAM. Running on a different GPU vendor, architecture, or a multi-GPU box is a larger unsolved problem — see HARDWARE.md for the design and an honest account of what is not built.

The shipped tuning is not advice, it is measurements of one box (32GB card, 128GB RAM). Declare what you have:

services.genai-server.hardware = { vramGB = 24; ramGB = 64; };

Every model declares a serve.minVramGB (and minRamGB) floor measured on that 32GB box. A model over your declared floor is dropped, and the rebuild prints a warning naming which models went and why. Nothing rewrites a model's context or offload to fit a smaller card — it runs at the shipped settings or it does not run. Adjust the catalog yourself if you want the middle ground: llmModels.<name>.serve.context and .kvQuant are ordinary options a host can override.

A hardware.profile option offering named per-card override sets existed from 2026-07-31 to 2026-08-22. It was removed because it was never wired to anything: setting it type-checked, rebuilt cleanly and changed nothing at all. If you are following an older copy of these docs, that is why it appeared to do nothing.

Checking the box against its config (genai-doctor)

genai-doctor        # reports; changes nothing; exit 1 if something failed

hardware.vramGB and hardware.ramGB are declared rather than detected, because Nix evaluation is pure. genai-doctor is what notices when a declaration is wrong — and it treats the two directions differently: claiming more VRAM than the card has is a failure (models are kept that cannot fit), claiming less is only a warning.

It also checks store permissions and free space, portal health, and reports orphans and stale quants via genai-prune.

Measuring what the stack claims (genai-eval)

Runs suites against the live services and writes a report:

genai-eval                 # everything
genai-eval --list          # what suites exist
genai-eval memory rag      # named suites
genai-eval --warm-only     # skip anything that would load a model

Cases are multi-step HTTP calls with deterministic checks, so the same harness grades a chat model and a service. The shipped suites are regression tests for claims this stack makes: that memory supersedes corrected facts, that hybrid retrieval catches both paraphrase and exact identifiers, that the code sandbox cannot see the host filesystem.

Set evals.compareModels = [ "coder-pro" "qwen-dense" ] to run a champion/challenger coding suite: each model writes a program, the sandbox runs it, and the output is checked — so the ranking is what the code did, not what another model thought of it. Pair the winner with a selector to promote it without touching any client. Results render at :8897/evals.

Thinking models and token budgets. A reasoning model spends the budget thinking before it answers, so a fixed max_tokens shared across a comparison measures whether each model finished thinking in time, not what it can do. The shipped suite allows 4000 and checks finish_reason plus non-empty content at the step that produces the code, so a truncated response fails where the message explains itself. Without that check the first three-way run scored a thinking model 0/3 on problems it never attempted — it had burned all 1200 tokens reasoning and returned empty content, which then substituted into the next step as an empty program. If you add your own comparison cases, budget for the reasoning.

A comparison is also treated as failed when no model clears the suite. Numbers alone are not evidence the harness worked; "nothing could pass this" means either every model is bad or the suite is broken, and both deserve a look before anyone quotes a ranking.

The speech suite is a round trip: it synthesizes a sentence with the CPU voice and transcribes it back, covering TTS, STT and the ASR-preamble strip in one case with no audio fixture in the repo. asr is resident, so it evicts nothing and is safe to run on a busy box.

speech-under-load is the same round trip, but only runs when a chat model is already resident. That is not redundancy: asr was once configured with a KV cache too large to fit beside a warm chat model, so transcription failed on a working box and passed on an idle one — and the plain speech suite could not see it. A test that only runs in the easy condition is not covering the hard one.

A suite whose precondition is missing skips rather than fails — memory-reconcile needs a resident chat model, since reconciliation uses one that is already warm and never loads its own. Exit status is non-zero on real failures, so it works from a timer. Reports go to /var/lib/genai-eval; add your own with evals.extraSuites.

Pruning the store (genai-prune)

The store is meant to be exactly what llmModels and mediaModels declare. genai-prune reports the difference; --delete acts on it:

genai-prune                          # report only (default)
genai-prune --stale-quants           # + old quants inside declared repos
genai-prune --delete --stale-quants  # actually remove them

Undeclared is not the same as garbage. Locally trained LoRAs share a directory with catalog ones and can't be re-downloaded, so the tool only deletes what identifies as catalog-managed (source: "manifest" in the sidecar) and is no longer in the catalog. A LoRA with no sidecar, or one from lora-train/lora-add, is always kept and merely listed.

Two classes are opt-in because they're riskier:

Flag	What it removes	Why it's gated
--stale-quants	a .gguf in a declared repo that isn't the declared tag — e.g. a Q4_K_M left behind when the tag moved to UD-Q4_K_XL	latest tags can't be matched locally, so they're never classified
--media-orphans	non-LoRA files absent from mediaModels	this is also where tooling-fetched weights live (PuLID's EVA-CLIP, Florence-2); deleting them costs a re-download and breaks the feature until it happens

Note that removing a model from the portal alone doesn't stick: genai-models-prefetch re-downloads the full catalog at every boot. To reclaim space permanently, drop the entry from llmModels and prune.

Adding a LoRA (or any other model)

mediaModels entries are shaped like the Civitai model card you are copying from — URL, title, type, base model, the specific model instance to run it on, usage tips, trigger words. Add the entry in your host config; host entries are added to the shipped catalog, so nothing else needs restating:

services.genai-server.mediaModels."ghibli-style.safetensors" = {
  type     = "lora";                                    # Civitai's "Type"
  title    = "Studio Ghibli Style";
  page     = "https://civitai.com/models/433138";       # for humans
  url      = "https://civitai.com/api/download/models/482825";
  base     = "pony";        # its "Base Model": Pony → pony, SDXL 1.0 /
                            # Illustrious → sdxl, Flux.1 D → flux,
                            # Z-Image → zimage, Krea 2 → krea
  engine   = "epicrealism"; # the checkpoint the page showcases it on
  triggers = [ "ghibli style" ];
  strength = 0.8;           # usage tips from the page…
  clipSkip = 2;
  notes    = "Keep the trigger early in the prompt.";
};

type picks the store category (lora/lycoris → loras/, checkpoint → checkpoints/, vae → vae/, …), so dir is only needed for oddities. Rebuild, then genai-fetch-media (or just restart image-server, which runs it) downloads the file and writes its {"trigger","base",...} sidecar. From then on the LoRA is listed in chat by name, routed to a compatible engine, and its trigger word, weight, clip skip and step count are applied automatically — lora-list shows all of it. Metadata-only edits apply on the next fetch without re-downloading; enable = false drops a shipped entry.

Not everything has a URL. Locally trained LoRAs are deployed by lora-train install, and a file you already have on disk by lora-add <file.safetensors> "<trigger words>" [base]; both write the same sidecar. Removing a catalog entry never deletes files from the store — those two paths write into the same directory.

A checkpoint works the same way (type = "checkpoint"), but the image server only runs the engines it knows: point one of the services.genai-server.imageServer.models.* options at the new file.

Testing the flake

nix flake check                        # evaluates every check
nix build .#checks.x86_64-linux.vm     # the stack's wiring, in a VM
nix build .#checks.x86_64-linux.wyoming # the same with Wyoming enabled

These boot a real NixOS VM and assert what actually broke during development: units start and answer, every hosted page renders with the shared nav, every enabled service has a portal card and a health path (or an explicit non-HTTP protocol), MCP bridges its tools with forget confirm-gated, the /svc allowlist 404s an unknown name, an uncataloged /api/pull names the option to add it to, genai-prune reports rather than deletes, and disabling a service really removes its unit, card and proxy entry.

They deliberately do not test the models. There is no GPU in a VM, so llama-swap and everything needing weights is switched off. That is a smaller claim than "the stack works" — and it is the claim worth automating, because every bug this repo has shipped to a live machine was in the wiring, not the models. For the models themselves, see genai-eval, which runs against a real box.

One check is worth running before trusting the others: nix eval .#checks.x86_64-linux.vm.drvPath. A check that fails to evaluate is not a failing test, it is an absent one — and it reports as neither.

Usage

# flake.nix inputs
genai-server.url = "git+https://git.homefree.host/homefree/genai-server";

# host modules
inputs.genai-server.nixosModules.default

# host configuration
services.genai-server.enable = true;
# Grant yourself write access to the model store / LoRAs / training jobs
# (standard NixOS group membership; the flake never names users).
users.users.youruser.extraGroups = [ "genai" ];
# Optional: Civitai API token file (civitai.com/user/account → API Keys).
# Needed by any mediaModels entry with `civitaiToken = true` — Civitai gates
# most downloads behind an account. Without it those entries are skipped
# (with a warning) and everything else works.
services.genai-server.civitaiTokenFile = "/run/secrets/civitai-api-token";

Requires: NVIDIA drivers configured on the host, unfree/CUDA nixpkgs allowed. Note the CUDA llama.cpp build compiles from source (no cache). That is the default and the only tuning anybody has measured; hardware.accelerator builds the two C++ engines against Vulkan, ROCm or the CPU instead — see What will not work yet for what that does and does not buy you.

Bringing this up on your hardware

Read this before assuming it will fit. This flake is a very good configuration of one machine — a 32GB RTX 5090 with 128GB of RAM — and being honest about that is more useful than a list of requirements that implies otherwise.

What is actually portable today

The flake evaluates on any hardware, including machines with no NVIDIA GPU. That sounds like a low bar; it was not met until 2026-07-31, and a flake you cannot evaluate is one you cannot try. The VM test runs on a driver-less node specifically so this keeps working.

Declare what you have:

services.genai-server.hardware = {
  vramGB = 24;          # what the card actually has
  ramGB  = 64;
};

genai-hwscan will write that block for you — or rather, print it:

$ genai-hwscan
# GPU: NVIDIA GeForce RTX 5090 (31GB, CC 12.0)
# RAM: 123GB   /dev/kvm: true   free on /var/lib/genai-models: 457GB

services.genai-server = {
  enable = true;
  hardware = { vramGB = 31; ramGB = 123; };
};

It emits config and never writes it — a tool that edits a NixOS configuration is one that eventually loses somebody's edits. It also reports what it could not determine, and leaves those values commented out rather than guessing: a card with no driver loaded looks exactly like no card, and guessing optimistically keeps models enabled that cannot fit, which fails at load after a long download. --json for scripting.

Then run genai-doctor, which compares that declaration against reality and complains. The two share detection and run in opposite directions: hwscan proposes a declaration for a machine that has none, the doctor checks one that exists. Over-declaring VRAM is a hard failure — a model that does not fit fails at load, badly, after a long download — while under-declaring is only a warning.

serve.minVramGB drops any generated model whose floor exceeds vramGB, so a smaller card gets a smaller fleet rather than a fleet that OOMs.

What is measured and what is arithmetic

Every number in the catalog — serve.minVramGB, the contexts, the --n-cpu-moe values, each entry's footprint — was measured on one box: a 32GB RTX 5090 with 128GB of RAM. Nothing here has been run on a smaller card. Treat the floors as the only claim the stack enforces and everything else as tuning you may need to redo. If you measure better numbers for your hardware, they are worth contributing back — genai-eval and the VM test are what make a contributed measurement checkable rather than a claim.

Turning things off

Everything GPU-hungry has an enable:

services.genai-server = {
  comfyui.enable = false;      # torch stack; the big one
  imageServer.enable = false;
  codeSandbox.enable = false;  # needs /dev/kvm
  magenticUi.enable = false;
};

The code sandbox is the model to copy: with no KVM it answers 503 with a reason, never "run it on the host instead". Unavailable-with-an-explanation beats a service that starts and fails at first request.

What will not work yet

• Non-NVIDIA GPUs — selectable, unmeasured. hardware.accelerator (cuda | rocm | vulkan | cpu) is what llama.cpp and stable-diffusion.cpp are built against; nixpkgs takes vulkanSupport and rocmSupport on both, and genai-hwscan proposes vulkan for an AMD or Intel card. So chat, embeddings, vision and transcription are a config change rather than a port. Two things it does not do. It does not move the PyTorch half — ComfyUI, the SAM 3 segment server and the LoRA trainer bring their own wheels and follow their own accelerator, so a non-CUDA host should expect to run the llama.cpp and sd.cpp services and leave the rest off or remote. And it re-measures nothing: serve.minVramGB and the shipped profiles were measured on a 32GB NVIDIA card, and they are floors below which a model answers incorrectly rather than failing, so treat them as unverified until somebody measures them on the machine. CUDA stays the default because changing one silently re-tunes somebody's box.
• Integrated GPUs are bounded by RAM, not by their carve-out. An APU's "VRAM" is a BIOS carve-out that llama.cpp reaches past into system memory through GTT, so hardware.integrated = true makes serve.minVramGB inert and serve.minRamGB govern instead — enforcing a 2GB carve-out as a ceiling would drop nearly the whole catalog on a machine with 92GB of RAM. The corollary bit once: on a carve-out everything "fits", so with a peer declared, anything wanting a card goes to the peer and this host keeps the device = "cpu" models. Left to the RAM floor alone, a router fronting an RTX 5090 pulled 253GB of GGUF onto its own disk and served thirteen chat models on its iGPU while the card sat idle.
• Multiple GPUs. No tensor split, no pinning a model to a card. llama-swap's peers and llama.cpp's RPC workers are the mechanisms and neither is wired up.
• Small VRAM. The shipped model fleet assumes ~32GB. Below roughly 16GB you will be picking models by hand.

The full analysis — five axes of hardware variation, the capability table each component needs, and a proposed genai-hwscan that reads a machine and emits a config — is in HARDWARE.md. It is a design document, not an implementation, and it says so. The honest cost there is in the measuring, not the coding, and this repo has exactly one machine.

Build times, and serving a cache

The CUDA llama.cpp build compiles from source and takes 20–40 minutes on a first build or a nixpkgs bump. Budget for that before your first nixos-rebuild switch.

A machine that has paid that cost can serve it to the others. This box already builds those derivations in order to run them, so serving costs a process and no extra disk:

services.genai-server.binaryCache = {
  enable = true;
  signKeyPath = "/run/secrets/cache-priv-key.pem";
  openFirewall = true;    # LAN or tailnet
};

Generate the key pair first, keeping the private half out of the Nix store:

nix-store --generate-binary-cache-key myhost-1 \
  /var/lib/secrets/cache-priv-key.pem cache-pub-key.pem

Then on each client:

nix.settings = {
  substituters = [ "http://myhost.lan:5000" ];
  trusted-public-keys = [ "myhost-1:<contents of cache-pub-key.pem>" ];
};

Sign it. With no signKeyPath the store is served unsigned, every client refuses those paths unless it turns signature checking off, and the result looks like a working cache until somebody tries to use it — so the flake warns at build time rather than letting you find out later.

It uses harmonia rather than nix-serve: nix-serve is a Perl CGI whose upstream is dormant, and harmonia is the maintained replacement with the same contract.