Where the tokens actually go

Notes on running a two-model coding loop at home — a frontier model to plan, a local model to execute — and designing it so the bill stays small.

There's a good conversation happening right now about whether small, locally-run models are viable for agentic coding. Birgitta Böckeler's recent memo on experiences with local models for coding is the best honest account I've read: a viability funnel, real tasks, and the refreshingly unglamorous conclusion that it's not plug-and-play yet. If you're wondering can this 35B model actually write my code, start there.

This memo is about the layer above that question. I've been running the two-model split she mentions in passing — plan with a big model, delegate execution to a local one — but as a standing piece of infrastructure rather than an ad-hoc handoff. Once you make it a system, a different question becomes the interesting one: not is the small model good enough, but where does the money actually go, and how do you keep it from going there.

Short version: premium tokens should only be spent on the two things a frontier model is uniquely good at — planning and judgment — and every token-heavy, grind-it-out part of coding should happen on local compute you've already paid for. The whole architecture is an argument about where state lives.

The setup

The pieces, genericized:

The loop:

  1. On the planner, I work with the frontier model to produce a task spec — a single markdown file with YAML front-matter (id, type, target_repo, target_files, model) and a structured body: <instructions>, <files>, <acceptance_criteria>, <constraints>. The id is just a UTC timestamp plus a slug.
  2. Authoring and queuing are deliberately separate steps. A half-written spec should never reach the executor. When it's ready, one script commits and pushes it to the bus.
  3. The executor polls the bus. When it sees a new task, it hands the spec to the local model, lets it work, pushes the result to an agent/<id> branch, and opens a PR through the Forgejo API.
  4. Merge is gated on green CI. A script refuses to merge a PR whose combined checks aren't passing, and won't merge at all until I confirm.

That's it. The planner authors, the executor executes, storage replicates, and git is the only thing in the middle.

The one idea that matters: the spec is the message bus, and it's free

Here's the thing I didn't appreciate until I'd built it wrong once.

Naïvely, a two-model system feels like it should be expensive, because you imagine the models talking to each other — the planner explaining the codebase and the task to the executor, the executor reporting back, the planner re-reading the result to plan the next step. Every one of those hops is context you'd have to re-establish, and re-establishing context is exactly what you pay premium tokens for.

None of that happens here, because the models never talk. The spec file is the message, and git is the transport. When the frontier model plans and writes a spec, that spec is the persistent state of the task. The executor doesn't need the planning conversation; it needs the spec, and the spec is sitting in a repo. When the result comes back, it comes back as a diff and a set of CI results — durable artifacts, not a conversation I have to pay to reconstruct.

The general principle, which turns out to govern everything else: persistent state lives in artifacts — specs, repos, CI logs — not in conversational context. Token efficiency in these systems isn't about clever prompt compression. It's about architecture choices that eliminate context-reestablishment. Git-as-protocol isn't a cute homelab flourish: the handoff costs zero model tokens, because the state is already written down and nothing has to be re-read to receive it — and you can't beat zero. Git isn't uniquely cheap there; any artifact on disk hits the same floor. What git buys, for essentially no added cost, is durability, history, and review semantics sitting on top of that zero.

Walking the loop, tagging the cost

If you follow a single task around the loop and mark each hop as premium tokens or free local compute, the shape of the thing becomes obvious.

Planning and spec authoring — premium tokens, but bounded. This is where frontier-model money is spent, and it's the right place to spend it. The codebase context gets loaded once (and cached — more on that below), and the output is a tight, structured spec. You're paying for reasoning and judgment, which is what the expensive model is for.

The handoff — free. A git commit and push. The spec moves from planner to executor for the cost of electrons. No tokens, because no model is re-reading anything.

Execution — free, and this is where the token volume lives. The local model does the grinding: reading files, long reasoning chains, generating and revising diffs, retrying. This is, by volume, the overwhelming majority of tokens consumed in the whole loop — and it costs nothing per token. A local model that's 3x slower and needs two attempts is still free on both the slowness and the retry. That changes what "good enough" means. You're not comparing the local model to the frontier model on quality-per-token; you're comparing free-but-needs-supervision to expensive-but-autonomous.

It also changes what's affordable. The move that made edit tasks work at all was embedding the relevant repo source directly into the task body — 20-odd thousand tokens of context per edit, sent every time. On a metered frontier model you'd never casually dump the repo into the prompt on every small change; here it's the obvious, cheap default, because the prefill is electricity. (It stops being free in a different currency past ~25K tokens, where it turns into a latency problem and you need a retrieval or patch strategy — but that's a scaling wall, not a billing one.)

Local validation — free. Lint, typecheck, and tests run on the executor before a PR is ever opened. This is the cheapest gate in the system and it protects the most expensive resource: my attention.

Review and merge — premium tokens (or human time), but only on green diffs. By the time a change reaches me, it has already passed CI locally. I never spend review tokens — or review attention — on a diff that doesn't compile.

The picture that falls out: the expensive surface is touched exactly twice, for planning and for judgment, and everything in between runs on compute I've already bought. The design goal isn't to minimize total tokens. It's to make sure the premium tokens only get spent on the irreplaceable parts.

The five levers

Everything I do to keep the bill down is a version of "push state into artifacts, keep it out of context." In rough priority order:

  1. The spec is the message bus, and it's free. Covered above. This is the one that matters most, and it's an architecture decision, not an optimization.
  2. Specify the goal and the test — not the recipe. The finding that surprised me most, because it runs against every hosted-model instinct: the less implementation detail I put in the spec, the better the models did. A one-paragraph "build X, here's the acceptance test" beat a fully-pinned spec (schema, API shape, edge cases, the works) — decisively, not marginally. The mechanism, once I saw it, was obvious: every detail you pin is another surface that has to come out right, and more prescribed files means more independent, cross-cutting ways to fail the build. A terse prompt lets the model produce one internally-coherent thing; a detailed one asks it to satisfy a dozen constraints simultaneously, which is exactly what small models are worst at. So the contract is the test, not the prose. Say what "done" looks like, make it verifiable, and say as little as possible about how to get there.
  3. The gate is the asset, not the model. An output gate runs on the executor before any PR opens — it checks that the diff stays in scope, isn't a stub, and (the one that earned its keep) didn't silently shrink a file. Across every round of the bakeoff, that gate caught every bad edit before it became a PR I'd have to review. That's the real finding: the model is interchangeable and free; the guardrail is neither. It's also the cheapest lever by far, because it moves failure detection onto compute I've already paid for. Never let a diff that fails locally consume review.
  4. Cache the stable context. The codebase, the system prompt, the project conventions — anything reused across planning sessions — should be cached rather than re-sent. Prompt caching is roughly 90% off after the first hit. A medium codebase loaded once and reused across many planning sessions is a small fraction of the naïve cost.
  5. Minimum MCP. Don't wire up a tool server until you've actually felt the pain of not having it, because tool definitions are a tax paid on every request. For most local-network orchestration, a coding agent plus SSH beats a bespoke MCP server per node. Build the MCP only when shell composition genuinely fails.

The bakeoff, and the findings I didn't expect

I ran a structured bakeoff across a roster of local models1 to pick an executor. Three results reordered how I think about the whole thing.

One: most "the model is bad" verdicts were the harness, not the model. The single biggest score mover wasn't a model swap — it was discovering that the model's own packaged defaults were sabotaging it. The tag I was running shipped a chat-tuned anti-repetition penalty baked into its config (presence_penalty 1.5 — sensible for stopping a chatbot repeating itself, actively harmful for code), and because I hadn't overridden it per-request, it silently leaked into every code generation, punishing the model for re-emitting tokens it needed to re-emit — like the rest of a file it was editing. Zeroing that one parameter took a model from 1/10 to 5/10. Alongside it: an environment variable that never actually reached the server (so context length and a key attention setting were quietly wrong for months), and thinking models burning their entire output budget on hidden reasoning and returning empty completions — which reads as "broken model" to any client that only sees the final text. None of those are the model. Capture ground truth on the serving engine, the real context length, the exact quant, and what your sampler is actually sending before you rank anything — otherwise you're comparing config accidents, not models. (This rhymes with Böckeler's report of the same model giving different quality on two machines: these models are far more setup-sensitive than hosted APIs have trained us to expect.)

There's a sharp corollary I now use as a triage tool: a settings change that moves one model but not another is a capability probe. The penalty fix jumped one model from unusable to usable and left another completely unmoved — and the one that didn't move had nothing hostile in its config to remove. That asymmetry told me, for free, which model was misconfigured and which was simply at its ceiling. No further tuning will buy the second one anything.

Two: less spec, better output — the lever-2 finding, and it held across the whole matrix. The terse one-paragraph tasks beat the fully-specified ones by a wide margin for the strongest models. More detail pins more files, and more files means more independent ways to fail the build.

Three, and the one I'd most want a reader to take away: none of them actually passed. Plenty of cells compiled and booted a greenfield app from a one-paragraph prompt — a genuinely useful result. But across the entire suite, not one model produced code that passed the full test stage honestly. The single green CI run in the whole bakeoff turned out to be a truncation artifact — the output got cut off before the failing part was written, and less code happened to pass. Green CI on a partial file is survivorship, not skill. So the realistic bar for a free local executor, even the winner, is "a working scaffold you then finish and verify," not "done."

The early rounds also showed every model gutting files on edit tasks — asked to add a method, they'd rewrite the file from scratch and silently drop everything else. I'd have called that an intrinsic limitation. It wasn't: with the penalty leak fixed, an explicit edit preamble, the source embedded in the task, and the shrink-gate watching, the gutting went to zero across fourteen edit cells. The failure mode was configuration plus missing scaffolding, not the model. Worth remembering before you write off a whole capability.

With all of that pinned down, the roster settled on a tuned Qwen3.6-35B-A3B as the executor default — the only model that shipped a booting app and edited it twice without breaking it — with a 27B MLX build emerging as a challenger on a single standout cell (unreplicated, so: a lead, not a conclusion). But honestly, the ranking mattered least of all of this. I can swap the winning model out in a one-line config change. I can't un-learn that the gate did more work than the model, that terse specs win, or that "it compiles and boots" is not "it's done."

When this is worth it, and when it isn't

I don't run this because it's cheaper than just using a frontier model for everything — for a lot of work, it isn't, once you count the time I spend on the orchestration. I run it because it changes what I pay attention to, and because the tasks it's good at are genuinely free at the margin.

It works on a narrower band than the enthusiasm around local models suggests: gated greenfield generation, and bounded edits with the source embedded and the shrink-gate watching. Inside that band it's genuinely useful and genuinely free. Outside it — anything needing discovery or judgment mid-flight, anything correctness-critical, anything where "compiles and boots" is not good enough — it stays on the frontier model, because those are exactly the things I deliberately kept on the expensive side of the fence. And even inside the band, the output is a scaffold: a booting app or a clean additive edit that I still have to finish and verify. None of these models cleared the full test stage on their own.

The discipline is non-negotiable — but note that "discipline" means a sharp acceptance test, not a long spec.2 The dangerous input isn't a terse task; it's a task with weak tests. A confident local model produces plausible, green-looking, wrong code efficiently, and the gate only catches structural failures and broken builds — it does not catch "you asked for the wrong thing." That judgment stays with the frontier model and with me, which is the whole point: it's the one thing worth paying for.

My current setup, genericized:

1. About a dozen models ran through the harness, all served locally through Ollama on a 128GB Mac Studio — some added in later rounds rather than run through every phase. The core suite: Qwen3.6-35B-A3B (the tuned winner), Qwen3-Coder (q4 and q8 — the coder-tuned model I'd expected to win; it hit the same greenfield ceiling once but broke its app on edits), Qwen3-Coder-Next, Devstral-Small-2 24B (cleanest behavior, slowest decode), gpt-oss 120B (a hidden-reasoning leaker until told not to think), and Gemma 4 as a non-coder control (it doubles as the cheap scribe — commit messages, summaries, classification — so the code model never spends cycles writing English). A later extended round added seven more: Qwen3.6-27B in both a GGUF-q8 and an MLX build (the MLX build produced the single best cell of the whole suite, unreplicated), plus EXAONE-Deep 32B, Llama 4 Scout, DeepSeek-Coder-V2 16B, Qwen2.5-Coder 32B, and north-mini-code (another default-on reasoner that burned its whole budget thinking). DeepSeek-R1 70B was benched and excluded — 90GB resident, it coexists with nothing else and has no tool-calling. The genuine frontier-OSS models (GLM-5.2, Kimi K2.7, MiniMax M3) never made the field at all: they don't fit 128GB at any honest quant, which is its own quiet lesson about where the local ceiling actually sits. ↩

2. To make that concrete — the same greenfield task, a link-shortener (POST a URL, get a short code; GET /{code} 302-redirects; persist it), written two ways. Terse: "Build a service that shortens URLs. POST /links returns a short code; GET /{code} redirects to the original. Persist to Postgres. Acceptance: the app boots, /healthz returns 200, and a POST /links then GET /{code} redirects to the URL you posted." Detailed: all of that, plus a full schema (code VARCHAR(8) UNIQUE, target_url TEXT, created_at TIMESTAMPTZ DEFAULT now(), hit_count BIGINT DEFAULT 0), exact request/response JSON, a 400 error envelope, base62-over-row-id code generation, and a prescribed web/ service/ repo/ domain/ package layout — a page of it. Same task; the terse one won, consistently. Not because the model can't code, but because the detailed version hands it a dozen things that must all come out exactly right, each an independent way to red the build: it stores created_at as a local-offset time in a TIMESTAMPTZ column and a round-trip test fails on the offset though the instant matches; it writes shortUrl in one place and short_url in another; it puts the base62 helper where the wrong layer imports it. None of those are the behavior I asked for — they're incidental cross-file constraints I introduced by over-specifying. The terse version asks for one coherent thing plus a way to check it; the model builds that, it boots, the smoke test passes. When I stopped writing the schema for it and started writing the test, the scores roughly tripled. ↩