What $50 of open-weight models gets you in Claude Code

Why I did this

I run Claude Code with a setup I've built up over time:

• A global CLAUDE.md that routes different kinds of work to different agents.
• A handful of skills for recurring rituals like planning and handoffs.
• A Codex plugin on the side for verification.

The part that matters for this post is the agent routing:

• Implementation goes to a coding agent.
• Adversarial review before merge goes to code-review.

At some point I started wondering: what happens if I keep the routing exactly as it is, but swap what's sitting behind each agent slot? Instead of Claude doing the coding and reviewing, what if coding was DeepSeek, or Kimi, and code-review was GLM?

So I picked a real, complete build to run the experiment on: get-good, a spaced-repetition app I'd been building for people who bounce off Anki (more on that soon).

An 8-phase build guide laid out the whole thing in advance. I built it twice from that same guide: once the normal way, on Claude, and once with every agent slot pointed at open-weight models through OpenRouter. This post is about the second build.

Wiring an open-source harness into Claude Code

Mechanically this is simple: Claude Code will point itself at any endpoint that follows the Anthropic API schema, through the ANTHROPIC_BASE_URL variable.

A settings.local.json sets 4 environment variables and you're done:

1
2
3
4
5
6
7
8

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://openrouter.ai/api",
    "ANTHROPIC_AUTH_TOKEN": "sk-or-v1-***redacted***",
    "ANTHROPIC_MODEL": "deepseek/deepseek-v4-pro-0813",
    "ANTHROPIC_SMALL_FAST_MODEL": "deepseek/deepseek-v4-flash"
  }
}

If you're setting this up yourself: rotate the ANTHROPIC_AUTH_TOKEN the moment you're done, avoid pasting it into chat, and never commit it to git.

And in the agent definitions themselves, a model: field pins each one:

• coding -> moonshotai/kimi-k2.7-code
• code-review -> z-ai/glm-5.2

Building get-good, twice

Once the routing settled, the actual build followed the same shape as any Claude Code session: plan mode, clarifying questions, then dispatch to coding, then code-review, then the next phase.

Early on, I queued up an instruction while something else was running: "please switch to a dev branch and work on that." The orchestrator took that literally and created a branch called phase3-journey-note-noteversion. I had to stop it:

No, I mean literally a branch called 'dev'. This project is small enough to not need phasal branches.

It fixed the branch within about 9 seconds of me saying that. Fair enough, that instruction really was ambiguous as written. But it's a good reminder that a less capable model doesn't infer intent the way you've gotten used to; it does something plausible and lets you correct it, which is fine as long as you're actually watching.

DeepSeek, which ran the main thread for 6 of the 8 phases, thinks in public and at length. Its reasoning traces read like genuine first-person deliberation, backtracking included: "Actually, wait.", "Let me just do it now." - at one point it caught itself mid-plan having written a garbled sentence into a file and went back to fix it before I even saw it.

It's also why some single responses took 4 minutes: on this model, 4 out of every 5 output tokens were reasoning.

The single clearest pattern across all 8 phases was about instructions, not model choice. In Phases 1 through 3, the coding worker was told to run its own tests before handing off, and all 3 passed review on the first attempt. In Phases 5 and 6, it was told not to, and both failed review on real bugs.

There was also a quieter money guzzler I only found afterward in the billing export: short, 9-to-12-output-token calls to Claude Sonnet kept firing in the background, even though nothing in the harness was configured to call it. That points at Claude Code's own auto-mode safety classifier, running on a real Claude model no matter what the rest of the harness was pointed at.

On top of that, money nearly ran out once already before any of this. Earlier that same night, the implementation agent for one of the last build steps was killed mid-task by the same wall:

1
2
3

API Error: 402 This request requires more credits, or fewer max_tokens.
You requested up to 32000 tokens, but can only afford 4902.
To increase, visit https://openrouter.ai/settings/credits and add more credits

What it actually cost

Total spend across the whole experiment, per my own OpenRouter activity export, was $50.80 over 1,725 individual API calls. It breaks down like this:

Two things in that table surprised me enough to dig into them:

• The Sonnet tax added up to more than it looked like in the moment. That auto-mode classifier from earlier: 304 calls, 9 to 12 output tokens each, added up to $5.00, 14.2% of total spend, more than every open-weight worker dispatch combined. Point being: "fully open-source" wasn't quite true.

• One provider tripled a portion of the bill. deepseek-v4-pro was served by 3 different providers over the course of the build, and one of them, GMICloud, is responsible for a huge anomaly. For every other provider, prompt caching hit 93 to 98% of the time. For the 2 hours and 42 minutes GMICloud was in the rotation, cache hits collapsed to 13%. The effective cost per prompt token went from about $0.14 per million to $1.02 per million.

On the reliability side, DeepSeek, which held the main thread the longest, made 666 tool calls, and the error rate was about 1.2%. GLM 5.3 had a short stint at the end, and that too was similarly clean.

How it stacks up against the Claude build

I had Fable 5.1 do a blind comparison of the two finished apps. Unfortunately, the Claude build came out ahead on 5 of 6 axes.

Some of it isn't a model difference at all: the open-source build's project folder was missing a CLAUDE.md, which meant nobody ever told it to run ruff check, ruff format, djlint, or pyright before calling a phase done. The Claude build's project came with that file, ran clean, and the open build shipped with 18 lint findings, 12 unformatted files, and 17 type errors, none of which were about whether the code worked, only about whether it was tidy. That's a harness gap I introduced, not something DeepSeek or Kimi got wrong.

Worth noting in the other direction: on the actual test suite, the open build came out with 74 passing tests against a real Postgres database and 0 tracebacks across 88 browser-flow requests, against 83 tests on SQLite with 1 traceback on the Claude build. I don't have timing or cost data for the Claude build at all, since a Claude Max subscription doesn't produce a billing export the way OpenRouter does, so none of this is a clean apples-to-apples comparison.

Would I do this again?

Yes, for the specific thing it was actually good at: cheap, well-specified worker dispatches. All 19 coding and code-review calls combined, the actual implementation and review work, cost $4.47.

No, not as a full replacement for the main thread:

• The reasoning-heavy models are slow in a way that's hard to work around, since the thinking itself is most of what you're paying for and waiting on.
• Provider routing is a lottery; one bad provider assignment cost me as much as the rest of the build combined.
• The credit model itself is a hard stop with no warning: a review that's mid-flight simply dies when the balance runs out (unless you have auto-recurring billing, and I don't have that much money!)

If I ran this again, I'd cap max_tokens more conservatively and keep Claude on the main thread for anything I actually needed to finish on schedule.