A naive cold start pays the full price every time: allocate GPU, map or read tens of gigabytes into VRAM, compile shaders where applicable, and only then accept user traffic. For this model class that can land in the few-minute range—unacceptable for interactive clients if “idle” is common. Persisting artifacts in a Modal Volume already removes repeated hub downloads; memory snapshots remove the repeated compute initialization once the snapshot exists. See also Modal's very large models example for transfer and mount patterns at this scale.

The trade is intentional: you spend meaningful time once building a checkpoint of a warmed llama-server process. After that, restore should be dominated by orchestration and memory remap—not tensor reload from network or disk in the hot path. If --no-mmap is missing, restore may fail or diverge (Modal snapshot semantics); treat that flag as part of the economic model, not an optimization detail.

Builder image includes compilers, CMake or equivalent, CUDA toolkit headers, and whatever you need to compile llama-server from llama.cpp. It is larger, slower to pull, and may carry a wider security surface. You run it only for compile jobs.

Runtime image holds the produced binary plus minimal shared libraries, systemd-style supervision (often just your Python shim + subprocess), and health/metrics hooks. Smaller images reduce pull time for new workers and shrink the set of packages that must be trusted in production.

The compiled binary then lands on the Volume next to weights so both builder churn and model bytes are versioned independently: bump LLAMA_CPP_TAG without re-downloading GGUF, or refresh weights without rebuilding unless the server ABI expectations changed.

What snapshotting is doing in this stack

Checkpoint/Restore In Userspace serializes a live process tree so a new host can reconstruct it without replaying your full startup script. For inference, that means: threads, register state, virtual memory areas (VMAs), open file descriptors, credentials, and relevant network state. The goal is to hand the next container an image that already has weights resident in the form the runtime expects—not to re-execute disk I/O and GPU initialization from scratch.

Modal's @modal.enter(snap=True) path aligns with memory snapshot semantics: you finish warm work, then seal state. Snapshotting cannot magically fix inconsistent mappings: if restore cannot faithfully recreate how virtual memory points at files or devices, you get subtle corruption, immediate crash, or worse—silent wrong answers.

Why mmap fights checkpoint/restore

Default weight loading often uses memory-mapped files: pages are demand-paged from the GGUF on disk, shared with the page cache, and may be shared read-only across processes. That is efficient for normal servers, but the mapping is a contract between three parties: virtual address, file identity (path, inode, offsets), and kernel bookkeeping. After checkpoint, a new container may mount the same Volume path but still differ in ways that break the mapping contract: inode generation, bind-mount layout, device minor numbers, or timing of lazy faults. Restore metadata must match precisely; when anything drifts, restore either fails or maps the wrong bytes into the address space.

GPU memory is a separate concern: snapshots capture what the Modal stack and platform integration support. Treat GPU state + driver version + binary ABI as part of your release matrix. If you change CUDA driver assumptions or rebuild llama-server without forcing a fresh snapshot cycle, expect restore-time failures that look like “random CUDA errors.”

What --no-mmap buys you

In llama.cpp, --no-mmap forces tensors to be read into ordinary allocated memory rather than file-backed mappings (see the server README). Initial load can be slower and use more RSS during the read, but the steady-state memory image is far closer to “a big blob of anonymous pages + known file descriptors” that checkpoint tooling can round-trip. You are trading optimal Linux page-cache sharing for restore determinism—which is the right trade for snapshot-first scale-to-zero.

Operational guardrails

• Centralize command construction in one function (_build_llama_cmd()) and unit-test that the argv always contains --no-mmap.
• Log the final argv at info level on startup (redact secrets) so production incidents show whether a bad deploy dropped the flag.
• After any change to Volume mount paths or filenames, run one full idle→restore cycle before promoting: snapshot bugs often surface only on second boot.

Related Modal pattern: SGLang with snapshots (different engine, same snapshot idea).

Why a subprocess (not in-process load)

Run llama-server as a child process so the Python Modal class stays a thin supervisor: it can stream logs, enforce timeouts on readiness, capture exit codes, and optionally restart without losing the whole container on a single CUDA fault. If the server segfaults, your wrapper can log, emit metrics, and exit cleanly so the platform replaces the worker rather than wedging a shared library in the same address space as Modal’s runtime.

Logging without deadlocks

If you attach PIPE to stdout/stderr, you must continuously drain those pipes from another thread or async reader. Undrained pipes can block the child once the kernel buffer fills, which looks like a mysterious hang right after startup. Prefer line-buffered reads and forward each line to Modal’s logging with a prefix (e.g. [llama]). Alternatively, let the server log to files on the Volume if you truly need retention—but for most cases, streaming to Modal logs is enough.

Readiness: poll /health, do not sleep

Replace time.sleep(120) with a loop that GETs http://127.0.0.1:<port>/health (or your bound address) with exponential backoff and a hard cap. Log every N failures so stalled boots are visible. The health endpoint should flip to success only when the model can actually accept completions—not merely when the HTTP socket listens.

Warmup: what to send before snap=True seals

Warmup should exercise the code paths you care about in production. At minimum: one short chat completion that touches tokenizer + attention + sampling. If you use multimodal weights, include a tiny image payload so vision towers and mmproj paths are initialized. If you rely on adaptive thinking, send one request with chat_template_kwargs.enable_thinking so the template and response shape you need are hot. Keep warmup token counts modest—enough to prime, not enough to dominate snapshot time.

Where the snapshot commits

@modal.enter(snap=True) means: when this method returns successfully, Modal may treat the resulting process state as the template for future containers. Therefore do not return until health is green and warmup finished. If you return early, you snapshot a half-ready server and pay debugging cost forever. If you need background work after serving starts, schedule it only after you are sure it should be part of the golden image—or accept a second snapshot cycle.

Shutdown and replacement

On container teardown, send SIGTERM to the child, wait with a timeout, then SIGKILL if needed. Unclosed GPU contexts can delay exit; log wall-clock time for shutdown. For long-running deployments, pair this with the operational notes in Operate & compare for recycle and diagnostics.

Phase A — Volume and process contract

• On the build host or a one-off modal run, print the resolved paths passed to --model, --mmproj, and the llama-server binary. Compare against ls -la on the mounted Volume.
• Confirm file sizes are in the expected ballpark (multi-GB GGUF, mmproj on the order of gigabytes). Truncated downloads often pass existence checks but fail at first load.
• Grep runtime logs for the argv or startup banner so --no-mmap appears exactly once and is not overridden by a wrapper script.

Phase B — Authentication and negative tests

• Call /v1/chat/completions without credentials and expect 401/403 (whatever your server is configured to return). Misconfigured auth often ships as “open by accident.”
• Repeat with an intentionally wrong key. Then with the production key. Document the header shape (Authorization: Bearer …) in your runbook so client teams do not guess.

Phase C — Latency: warm path vs idle wake

• Measure time-to-first-token (TTFT) on a warm container (traffic just happened). Then scale to zero, wait until the platform confirms idle, send one request, and measure TTFT again. The second number is what intermittent users feel—optimize that, not only steady-state.
• Log Modal’s transition timestamps if available; correlate with client-side stopwatch to separate queueing from server work.

Phase D — Functional and multimodal correctness

• Text: short deterministic prompt (“Return the word OK only”) to catch garbled output or template failures.
• Multimodal: if --mmproj is enabled, send a minimal valid image payload and assert a structured response—vision failures should not be discovered in production traffic.
• Adaptive thinking: one request with template kwargs enabling thinking; confirm your client can parse both answer and reasoning fields if your product depends on them. Details live under APIs & clients.

Why upgrades are a coordinated release

Bumping llama-server changes HTTP behavior, tokenizer handling, CUDA kernels, and sometimes GGUF expectations. Your Volume still holds the same GGUF bytes, but the binary interpreting them changed. Treat every bump of LLAMA_CPP_TAG as a mini release: compile, run the full validation matrix, then redeploy—never only swap the binary in place on a live snapshot without a new capture cycle.

Recommended sequence

1. Read upstream release notes or compare commits between your old and new tag; search for breaking changes in server, ggml, and CUDA backends.
2. Rebase local patches (for example Gemma chat-template / thinking-related edits in common/chat.cpp). Conflicts here are common—resolve in a branch, not on the production builder.
3. Run build_llama_server (or your equivalent) against the new tag; write the new binary to a staging path or staging Volume first.
4. Boot the server against the same GGUF and mmproj you use in prod; run Phase A–D validation from the checklist above.
5. Deploy to staging Modal app, force at least one idle period, and confirm restore still works. Snapshot incompatibilities often appear only after the second boot.
6. Promote: update production Volume path or swap symlink, redeploy, and let a fresh snap=True cycle capture the new golden image. Keep the previous binary on disk until you are confident—rollback is then a pointer swap.

CUDA / driver coupling

If Modal’s base image or GPU driver revision moves in parallel with your upgrade, test together. A new llama-server built against one CUDA toolkit and run under another is a frequent source of “works on my builder” failures. Pin builder and runtime image digests in version control when possible.

Rollback

Keep the previous LLAMA_CPP_TAG and matching binary artifact addressable (git tag, Volume path, or object name). Rollback is: redeploy known-good binary, run health + one chat completion, trigger a new snapshot. Do not assume an old snapshot image remains compatible after you have upgraded drivers or changed GPU type.