Local models: llama.cpp, Ollama, LM Studio, vLLM

A local model keeps every word of the campaign on your machine. It needs at least 16 GB of graphics memory (a 16 GB card, or 24 GB of memory on a Mac or another unified-memory machine), and it needs the settings on this page: the difference between a Dungeon Master that rolls dice and one that only talks about them is a handful of flags.

The easy way: the desktop app

The desktop app's Story AI screen has a Local AI on this machine door that measures your hardware, downloads a model sized to it, installs a pinned llama.cpp server with every setting below already applied, and runs it for you. If you host from the desktop app, use that and skip the rest of this page. See Story AI on the desktop. On a self-hosted server, the guided setup gives the llama.cpp commands for your machine, and lists the models of any model server already running there (llama.cpp, Ollama, LM Studio, vLLM, KoboldCpp, TabbyAPI, Jan), a recommendation where one is available.

Which model for your card

Graphics memory is a card's own, or on Apple Silicon and AMD Ryzen AI Max the share of the one memory pool the system leaves the graphics. Below the minimum a smaller model can limp through a scene on short rules the server switches to by itself for any model under 16 billion parameters, but it skips rolls, attacks out of turn and narrates past the dice often enough that the table stops trusting it; problems there are not treated as bugs. Measured turn times are in System requirements.

A 16 GB gaming card: gpt-oss-20b

OpenAI's open model has 21 billion parameters with 3.6 billion active, so it plays the full rules and the full tool set at the speed of a much smaller model, and the whole file fits on a 16 GB card. Its window is cheap: 64K takes about 1 GB beside the file. Take gpt-oss-20b-MXFP4.gguf from ggml-org/gpt-oss-20b-GGUF on Hugging Face:

llama-server -m gpt-oss-20b-MXFP4.gguf \
  -c 65536 --jinja \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --reasoning-budget 1024 \
  --port 8001 --alias gpt-oss-20b

Leave its sampling at llama.cpp's defaults, as it was measured. It reasons in a channel of its own, which the server reads apart from the story, and a tool call it starts in the prose is cut before the table sees it. The 35B also runs on a 16 GB card with 32 GB of system memory, its experts waiting in system memory (llama-server's --n-cpu-moe), but slower on every turn. llama.cpp is the supported way to run it.

The recommended model: Qwen3.6 35B

Qwen3.6 35B-A3B (a mixture-of-experts model, so it runs faster than its size suggests) at Q8 if it fits, Q4 otherwise, served by llama.cpp with a 64K context window. The critical settings, which port to any server:

Context window

The storyteller needs at least 16K tokens to narrate and 24K to run fights; 32K or more is recommended, and 64K is the default. The instructions and the compact tool set take about 15K before any story is told, and every request is fitted to the window the backend really has. Under 16K each turn is refused with a note naming the fix. llama.cpp reports its window; most other servers do not, so the server plans for 16K unless you set OPENAI_COMPAT_CONTEXT. Test backend says which window a campaign gets and whether it is enough.

llama.cpp

llama-server -m Qwen3.6-35B-A3B-Q8_0.gguf \
  -c 65536 --jinja \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
  --reasoning-budget 2048 \
  --port 8001 --alias qwen3.6-35b

Choose the OpenAI-compatible provider and URL http://127.0.0.1:8001/v1, then the model name your server lists (qwen3.6-35b in this example). A blank model follows the endpoint only when it lists exactly one chat model; otherwise choose one in Admin or guided setup. If you start llama-server with --api-key, put the key in OPENAI_COMPAT_API_KEY. The server reads the real context window from llama.cpp, so fixing it there is enough.

Ollama

Ollama is supported, with llama.cpp the server we recommend: Ollama calls the dice and other tools less reliably. The guided setup finds an Ollama running on the server's machine and connects it through Ollama's own API (the Ollama-native provider), which sends the context window with every call: the model's own, up to 64K. Ollama's OpenAI-compatible endpoint ignores a window sent with the request and runs every model at Ollama's own context length, 4K on a card under 24 GB, too small to narrate.

To use that endpoint anyway, the repository ships a Modelfile with the context and sampling settings baked in:

ollama pull qwen3.6:35b-a3b-q8_0
ollama create qwen3.6-dm -f models/qwen3.6-dm.Modelfile
# backend URL http://127.0.0.1:11434/v1, model qwen3.6-dm

That endpoint does not report its window, so set OPENAI_COMPAT_CONTEXT=65536 as well.

LM Studio and vLLM

Both expose an OpenAI-compatible server. Load the model, turn tool calling on (LM Studio: the model must be one with a tool-capable template; vLLM: --enable-auto-tool-choice with the right parser), set the context to 65536, and use the server's URL as the Custom backend. The server treats them as local and sends every sampling parameter; the context is assumed to be 16K unless the endpoint reports it, so set OPENAI_COMPAT_CONTEXT=65536.

From Docker

The container reaches the host as host.docker.internal: http://host.docker.internal:8001/v1 for llama.cpp, :11434 for Ollama. Or use host networking on Linux so the plain 127.0.0.1 defaults work. See Run with Docker.

What a turn costs

A turn is two or three calls: a tool-decision call with thinking (about 50 to 100 seconds on one consumer GPU), the tool results, and a streaming narration call without thinking so the words appear smoothly. Expect 1.5 to 3 minutes for a full turn, combat at the longer end; gpt-oss-20b took 90 to 160 seconds a turn on the 16 GB machine it was measured on (an AMD BC-250). The prompt budget includes the tool descriptions (shown as Tool definitions in Context). Their size depends on the available tools; allow enough window for them, the story and the reply. A request that lands on a connection the model server just closed is asked again once, and so is a reply llama.cpp throws away. Setting DM_THINKING=0 forces thinking off everywhere: fast turns, unreliable tools, a fallback rather than a normal mode.

Checking it

Admin, Server settings, Text model, Test backend, or from a shell:

node scripts/probe-openai-backend.mjs --url http://127.0.0.1:8001/v1 --model qwen3.6-35b

It streams a line, makes a real function call with exact arguments, and continues after the tool's output. A backend that passes rolls dice at the table.

Sharing the GPU

ComfyUI, Kokoro and the text model all want the same memory. On an integrated GPU the server queues media jobs behind each other; a scene picture arrives after its paragraph, and a turn may wait behind a render. Two GPUs, or pictures from an OpenAI key, keep the story moving.