The gguf-runner inference engine

Everlock runs its AI features on its own inference engine, gguf-runner — a self-developed, from-scratch GGUF runtime written for this project. It is embedded in-process through the everlock-ai-runtime crate and runs locally on CPU, so the model lives inside the Everlock binary and answers stay on your machine.

This page explains what the engine is, what it can do, and how it is wired into Everlock. It is the AI-side counterpart of the apimeister-photos engine page.

Where it comes from

Like the apimeister-photos media engine, gguf-runner is maintained in its own repository and vendored into Everlock as a normal Cargo git dependency. The version Everlock ships is whatever commit is pinned in Cargo.lock.

What the engine does

gguf-runner loads quantized models in the GGUF format and runs them directly:

• Text generation with streaming — tokens are produced incrementally and streamed back to the caller as they are generated.
• Vision / image understanding — when a model is paired with its mmproj projector file, the engine accepts an image plus a prompt and returns text. This is what powers image captioning.
• Tool / function calling — the engine can be given a set of callable tools (via a Tool trait) and will invoke them mid-generation, feeding the results back into the conversation, up to a bounded number of calls per turn.
• Conversation history — multi-turn prompts are supported by replaying prior (input, response) pairs.
• Hidden "think" mode — internal reasoning can be kept out of the streamed output so callers see only the final answer.
• Configurable context window — the KV-cache / context length is set per request, so short admin prompts and larger image prompts can use different budgets.
• Quantized weights — models are run from quantized GGUF (e.g. Q8_0, Q3_K_M), keeping the embedded footprint and memory use modest enough to ship inside a single binary.

How Everlock embeds it

The engine itself is transport- and product-agnostic. Everlock wraps it in the everlock-ai-runtime crate, which owns the heavy resources and exposes a small, shared handle.

flowchart TD
GR["gguf-runner (embedded GGUF model + mmproj)"] --> RT["everlock-ai-runtime worker thread"]
RT -->|AiTool — text + tools| SSH["backend-ai-ssh (admin /ai shell)"]
RT -->|ImageInferenceProvider — captions| IMG["backend-image-http (enhance-metadata)"]

Key properties of the embedding:

• Single worker thread. All requests funnel through one dedicated everlock-ai thread that owns the loaded model. The AiRuntime handle is cheap to clone; every clone talks to the same thread, so there is never more than one copy of the model in memory.
• Two consumer traits. Backends depend on small traits rather than the engine directly:
  • AiTool — text prompting plus callable admin tools, used by backend-ai-ssh for the admin SSH /ai shell.
  • ImageInferenceProvider — image captioning, used by backend-image-http.
• One load, shared. backend-ai-ssh registers the runtime in the backend registry; backend-image-http looks it up and reuses it. The runtime starts even when the SSH prompt path is disabled, so captioning still works. There is no second model load.
• Different context budgets. Text turns use a tighter context window; image captioning uses a larger one because image embeddings consume many token slots before the prompt.

For the admin-shell behaviour, tool set, and access model on top of this engine, see AI runtime and access model.

Models are embedded at build time

There is no model file to download at runtime and no model path to configure. everlock-ai-runtime's build.rs fetches the GGUF weights from Hugging Face into target/models/ on first build and bakes them into the binary:

• the default (SmolVLM) build embeds the model with include_bytes!;
• the Qwen3 build embeds it with assembly .incbin, which routes the ~2 GB of data through the system linker and bypasses an LLVM rlib size limit that include_bytes! would hit.

Because the weights are compiled in, an Everlock binary always carries a working model — and which model it carries is fixed at build time.

Models and Docker images

The build embeds one of two vision-capable models, selected by a Cargo feature on everlock-ai-runtime:

Both ship a paired mmproj projector, which is what lets the same runtime serve image captioning as well as text prompts.

This maps directly onto the published container images:

• latest — the default image, built from Dockerfile.smolvlm. It embeds the small SmolVLM-256M model, so the standard Everlock image stays light while still being vision-capable.
• Qwen3 image — a separate image built from Dockerfile.qwen3 (cargo build --release --features qwen3). It embeds the larger Qwen3.5-2B model for stronger answers and captions, at the cost of a much bigger image.

Building the Qwen3 variant locally:

cargo build --release --features qwen3

At a glance

• Everlock's own engine. Inference runs on the in-house gguf-runner project, maintained in the open and vendored into Everlock.
• Local and in-process. The model runs on your CPU, inside the Everlock process; prompts and answers stay on your own hardware.
• Self-contained. The model is embedded in the binary at build time, so each Everlock build ships ready to run with the model it was built for.

Read next

• AI runtime and access model
• AI backend
• Image captioning in the apimeister-photos engine
• Architecture