GPU tier

For production throughput, run the model engines on a GPU host and point Fosnie's providers at them. Fosnie Core still runs as the standard five-container Compose stack; only the inference engines move to the GPU.

Shape

• Chat LLM on vLLM (OpenAI-compatible server).
• Embeddings and reranking on Infinity (or another OpenAI-compatible embed/rerank server).
• Fosnie Core via Docker Compose (default profile), with the provider rows pointed at the engines.

Engines typically run directly on the host (bare-metal or systemd) rather than in the Compose stack, so they get full GPU access.

vLLM (chat)

Serve your chosen model with tensor parallelism across the GPUs and Fosnie-friendly tool/reasoning parsing. A representative launch:

vllm serve <model> \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 65535 \
  --served-model-name fosnie-llm \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

Use a tool-capable model and the matching parsers – Fosnie's agents rely on tool calls. Fosnie learns the context length from the engine's /v1/models, so you don't configure it twice.

Embeddings + rerank (Infinity)

Run an embedding model and a reranker on the GPU. If your rerank server doesn't expose the exact path Fosnie expects (/v1/rerank), put a small reverse-proxy rewrite in front – Fosnie posts rerank requests to {base}/v1/rerank.

Point Fosnie at the engines

In Settings → Providers, set the base URLs. From inside the Compose containers, reach host engines via host.docker.internal (not 127.0.0.1):

• LLM → http://host.docker.internal:8000/v1
• Embeddings → http://host.docker.internal:7997
• Rerank → http://host.docker.internal:7998

Use Test on each role to confirm connectivity, then index a document and ask a question to validate end to end.

Exposing it safely

If you publish the host, terminate TLS in front of Fosnie and set PUBLIC_URL to the https:// URL – the backend refuses to boot with a plaintext public URL. Keep the engine ports off the public interface. See Upgrades & TLS.