Flotilla Corpus Working Paper Apertus 8B July 2026
Is knowledge better baked in, or looked up?
A clean 2×2 on Switzerland's open-weights model, crossing a LoRA fine-tune against the base model, each with and without a cited RAG wiki built from 105 days of a live agent fleet's own logs. The answer was decisive: retrieval is the engine.
$ ollama run hf.co/ursushoribilis/apertus-flotilla:Q4_K_M
The whole paper in one grid
Composite recall score (0–5), blind Opus judge, N = 49 in-corpus questions.
RAG on the base model. Retrieval is the entire engine — a large, consistent lift across every grading dimension.
LoRA alone. With no wiki to consult, the baked disposition adds nothing to clean recall.
RAG on the LoRA model. Retrieval lifts the fine-tuned model by a near-identical margin.
LoRA vs. base, both with RAG. Given identical retrieval, baking the disposition is marginally worse, not better.
Knowledge in weights, or knowledge in retrieval?
The question, framed in Andrej Karpathy's terms: if you bake a model's disposition into its weights via LoRA — specifically the habit of consulting an external wiki before answering and staying grounded in what it retrieves — is that disposition harder to override than simply prompting it every session?
Separately, a cited and auditable RAG wiki holds the facts. The two layers are designed to be complementary and independently updatable: the wiki captures what the organisation knows; the weights capture how it should behave. To test this, we ran a 2×2 factorial across four arms: {base, LoRA-adapted} × {no-RAG, RAG}.
Apertus 8B, held constant
- Base model: Apertus 8B (Swiss open-weights, ETH / EPFL), quantized Q4_K_M at inference.
- LoRA variant: QLoRA fine-tuning (SFT → DPO) on a corpus extracted from the live AI development fleet.
- Both arms run at identical quantization — only the weights differ.
- RAG: BM25 retrieval, k = 3 entries per query from a 66-entry wiki.
- System prompt: a minimal 38-word blurb, identical across all four arms — no rules injection.
The LoRA fuel
Extracted from roughly 105 days of real agent operation logs (March–June 2026), filtered and classified through a five-gate quality pipeline.
| Source | Raw | After gates | Training role |
|---|---|---|---|
| Correction events (mistake + human fix) | 284 | 233 | SFT + DPO |
| Lessons (post-session reflections) | 382 | 107 | SFT · ≥70% confidence |
| Task execution logs (with scratchpad) | 760 | 69 | SFT · quality ≥ 3/5 |
| SFT total | — | 409 | ChatML format |
| DPO pairs | 262 | 51 | Preference alignment |
| Eval holdout | — | 50 | Never in training |
DPO pairs encode explicit before/after signal — prompt is the user turn, rejected the wrong agent answer, chosen the corrected behaviour. This is the “consult-and-comply” disposition in its raw training form.
Facts live here, not in the weights
A frozen, independently citable knowledge base of 66 atomic entries — each one fact, rule, or procedure. Built from the same gold corpus as the LoRA data, but in a different representation.
| Category | Count | Contents |
|---|---|---|
| R01–R30 Rules | 30 | Behavioural protocol: commit hygiene, peer-review, escalation paths |
| F001–F025 Facts | 25 | Operational metrics: agent workload, task counts, circuit-breaker stats |
| I001–I005 Incidents | 5 | Real post-mortems: self-approval violation, Misty spam incident |
| P001–P006 Protocols | 6 | Step-by-step: vault access, PocketBase filter syntax |
EVAL-008 — an in-corpus recall test
50 questions were generated from the 66 wiki entries using Qwen 7B and validated by a 10/10 human acceptance check. Because the questions derive directly from wiki content, this tests whether each arm can retrieve and correctly apply the same facts the wiki holds.
| Arm | System prompt | Retrieval | What it isolates |
|---|---|---|---|
| 0 · base, no-RAG | 38-word blurb | None | Floor: raw Apertus capability |
| 1 · LoRA, no-RAG | 38-word blurb | None | LoRA signal alone |
| 2 · LoRA + RAG | 38-word blurb | BM25 k=3 | Full stack: baked disposition + retrieval |
| 3 · base + RAG | 38-word blurb | BM25 k=3 | RAG without LoRA control |
Graders
Two independent blind judges, arm identities shuffled to X/Y/Z/W per question: Qwen 2.5 7B as a fast preview grader, and Claude Opus 4 as the human-quality blind judge and primary result set. Three dimensions, each 0–5: grounded (references real facts from the gold entry), correct (factual content is right), and faithful_recall (specific numbers, names, procedures recalled).
Retrieval carries it. LoRA does not.
| Arm | Grounded | Correct | Faithful | Composite |
|---|---|---|---|---|
| 0 · base, no-RAG | 1.18 | 1.22 | 0.59 | 1.00 |
| 1 · LoRA, no-RAG | 1.06 | 1.06 | 0.71 | 0.95 |
| 2 · LoRA + RAG | 2.41 | 2.51 | 2.29 | 2.40 |
| 3 · base + RAG | 2.55 | 2.69 | 2.37 | 2.54 |
Qwen 2.5 7B preview scores agree directionally (inter-rater MAE ≈ 1.16/5; Opus is systematically stricter, but both rank the arms identically).
Headline finding
In-corpus recall is driven entirely by retrieval (+1.54 composite). LoRA adaptation shows no recall benefit whether retrieval is absent (−0.05) or present (−0.14). The baked disposition's value, if any, must be demonstrated under adversarial conditions where prompted rules can be overridden but baked ones cannot.
What that looks like, question by question
Example A — Vault access protocol (Q5)
Gold: use vault/agent-fetch.sh (or .ps1 on Windows) to fetch secrets. Never use .env files. Never commit secrets.
Neither the base nor the LoRA-alone model knows this procedural fact. LoRA + RAG retrieves and applies it correctly.
Example B — Self-approval violation incident (Q27)
Gold: Clau approved its own tasks (2 IDs), violating Rule R27 — triggering a no-self-approval lesson and a MANDATORY note in CLAUDE.md.
Example C — PocketBase pagination (Q33) · an edge case
Gold: PocketBase perPage max is 200. Always check totalItems. Use page=2, page=3 for multi-page fetches.
Retrieval quality is the current ceiling: when BM25 mismatches, the RAG arms lose their advantage.
What the 2×2 tells us — and what it doesn't
- Retrieval is the engine. +1.54 composite from adding RAG to the base model, consistent across all three dimensions. The wiki works.
- LoRA alone is neutral-to-negative. −0.05 on clean recall. The baked disposition has nothing to retrieve from — no net benefit without the wiki.
- With the same wiki, the base model is marginally better. −0.14 for LoRA+RAG vs base+RAG. On cooperative queries with clean retrieval, baking the disposition adds a little noise, not signal.
- Retrieval quality is the ceiling. A single BM25 mismatch drops a RAG arm to base level. Dense embeddings + a reranker would lift every RAG arm.
What this does not close
Every arm here got cooperative prompts and clean retrieval — conditions under which the base model already complies, so there is nothing to enforce. The −0.14 says only that baking the disposition doesn't help recall when everything cooperates. It says nothing about what happens when an adversarial instruction tries to override the wiki. That is where the LoRA must earn its keep.
The model is public
The fine-tune is published as a single Q4_K_M GGUF (4.7 GB) with a model card and Modelfile. It runs fully locally, no cloud dependency. One line with Ollama:
ollama run hf.co/ursushoribilis/apertus-flotilla:Q4_K_M
Or build a Modelfile with the fleet-agent system prompt baked in:
# download the GGUF huggingface-cli download ursushoribilis/apertus-flotilla apertus-flotilla-Q4_K_M.gguf # create a Modelfile cat > Modelfile <<'EOF' FROM ./apertus-flotilla-Q4_K_M.gguf SYSTEM """You are a capable engineering agent working within a multi-agent fleet. You follow the heartbeat protocol, respect project scope, respond precisely to corrections, verify work before claiming completion, and never confuse project contexts.""" PARAMETER temperature 0.7 PARAMETER num_ctx 16384 EOF # register and run ollama create apertus-flotilla -f Modelfile ollama run apertus-flotilla
Also available via llama.cpp, LM Studio, vLLM, and Docker Model Runner — see the model card. Corpus, format reference, and training guide are in the companion repository.
Where this goes
Adversarial enforcement test
The experiment the LoRA thesis now depends on. Arm 2 (LoRA+RAG) vs Arm 3 (base+RAG) — identical retrieval, different weight source — under prompt-injection attacks, omitted-context degradation, and role-override instructions. Rubric: resisted / grounded / leaked. If the baked disposition resists where a prompted rule folds, the LoRA is justified as a sovereignty mechanism. If both fold, the product is base + RAG and we've proven it rigorously. Either outcome is publishable.
Role-based pre-prompting
Authenticated role selection generates a tailored system prompt and query macro before the question arrives — a low-cost addition to the Arm 3 stack. Also sidesteps the version-sync problems of maintaining per-vertical LoRA models at scale.
Rolling context compression
Apertus 8B's 64K window fills fast with retrieval + role prompt + multi-turn history. Reset each turn; re-inject the role prompt, fresh retrieval, and a compact summary of the previous exchange. Implementable today, no model change.
Apertus next generation
A new Apertus with multimodal input and a larger window is expected shortly. The wiki and LoRA data are model-agnostic — migration is a re-quantise and re-fine-tune pass. Lock the EVAL-009 baseline on 8B first, then repeat the 2×2 to measure the generation-over-generation lift.
Better retrieval
Replace BM25 with dense vector retrieval + a lightweight reranker. Highest-leverage single improvement to the stack — fewer mismatches, higher faithful recall across every RAG arm, no retraining.
Limitations
- n = 49 questions, one corpus, one 8B model, one grader. Effect sizes are directional; statistical significance is not claimed.
- The −0.14 arm2−arm3 delta is within possible grading noise. The right conclusion is “no recall benefit detected,” not “LoRA is harmful.”
- In-corpus questions only. Out-of-corpus generalisation is a separate run with a different pattern, reported separately.
- Adversarial conditions are untested. The enforceability claim for the LoRA remains an open hypothesis — the subject of EVAL-009.