Tested configurations
Reference setups Fosnie has been run on, from a laptop to a dual-A100 server to cloud APIs.
Fosnie is model- and hardware-agnostic: it never bundles a model and talks to any OpenAI-compatible engine, local or hosted. These are reference configurations it has actually been run on, so you can size your own deployment against something real. You are not limited to these; they are the span we have exercised.
| Configuration | Compute | LLM | Embeddings + rerank | Notes |
|---|---|---|---|---|
| Laptop, local | CPU only | small local model (~4B) via Ollama | local | The --local profile end to end, fully offline, zero egress. The honest minimum for trying it. |
| Single GPU host | 1 GPU | a mid-size local model via Ollama or vLLM | local | Comfortable for a small team. |
| Dual-A100 server | 2× A100 80GB | Qwen 3.6 27B (dense) on vLLM, tensor-parallel across both GPUs | local | The reference GPU deployment. Code interpreter runs here via the gVisor sandbox (the box has no KVM). |
| Cloud APIs (BYOK) | none of your own | OpenAI, Anthropic, Gemini | Jina | No local GPU: point each capability at a hosted API with your own keys. |
Mixing is expected
Because providers are chosen per capability and per chat, these are not exclusive. A common setup runs a local LLM for private work and keeps a hosted model available alongside it, or uses Jina for embeddings and reranking while the LLM stays local. You switch the LLM per conversation from the composer.
Code interpreter across these
The code interpreter runs on all of the above except a fully locked, unprivileged container: on a KVM host it uses Firecracker, and on a KVM-less host (like the dual-A100 box above) it uses gVisor. Both give the same zero-egress, ephemeral sandbox.
Sizing rule of thumb: retrieval quality tracks the embedding and rerank models more than the LLM, and the LLM size tracks answer quality and latency. Start local to prove the workflow, then move the LLM to a GPU host or a hosted API for production headroom.