GPT-5.6 Sol scores 80 on the Artificial Analysis Coding Agent Index against 77 for Claude Fable 5, at a lower cost per task (about $1.04). That post went up on July 9.

I called model parity back in April, in a piece with a section headed “GPT-5.4 Caught Up.” What I missed was everything around the model: that OpenAI would close the product gap, and buy a decade of compute while doing it.

Over the last few months I’ve been using ChatGPT more and more for work-like tasks. Not Duetto work, because we don’t have an enterprise plan and Duetto material stays out of it. Work-like things. Both the capabilities and the UX have improved to the point where I’d put the product at least on par with claude.ai, to my surprise. There are still gaps. It’s still good.

TL;DR

• GPT-5.6 Sol leads the Artificial Analysis Coding Agent Index 80 to 77 over Claude Fable 5 (July 9) and Terminal-Bench 2.1 85.77% to 84.64% over Claude Opus 5 (July 22), the second one with a methodology caveat that widens the gap.

• Claude Opus 5 still leads the broader Artificial Analysis Intelligence Index at 60.7 against GPT-5.6 Sol’s 58.9 (July 29). Anthropic’s remaining model lead is general capability, not agentic coding.

• Anthropic leads on revenue ($47B run-rate, May 28, 2026) and valuation ($965B), and leads enterprise API share 40% to 27% (Menlo, December 2025). OpenAI closed the capability gap and most of the product gap while trailing commercially.

• OpenAI is far ahead on planned compute (Stargate at nearly 7GW toward a stated 10GW commitment, the Broadcom “Jalapeño” inference ASIC). In July it moved its usage limits three times in seventeen days, in both directions, around a near-global outage on the 25th. Anthropic has been raising its limits since May.

• No open-weight provider I found fields an app ecosystem within reach of ChatGPT or Claude Code, which keeps this a two-horse race even as the weights converge. Chinese domestic app numbers could complicate that.

Where the coding lead sits

Two current measures put GPT ahead on agentic coding. The Coding Agent Index lists GPT-5.6’s three effort tiers at Sol 80, Terra 77.4 and Luna 74.6, with Claude Fable 5 at 77.2 (Terra and Fable 5 land 0.2 apart). Sol’s per-task cost runs below both Fable 5 and Opus 4.8. On Terminal-Bench 2.1 (89 tasks, Terminus 2 harness, snapshot dated July 22), GPT-5.6 Sol leads Claude Opus 5 by about a point, 85.77% to 84.64%.

The Terminal-Bench result comes with a caveat from the publisher. Opus 5’s run used Claude Opus 4.8 as a refusal fallback. Count the nine affected passing results as failures instead and Opus 5 drops to 81.27%, which widens the GPT lead to roughly four and a half points. The lead is real. Its size depends on how you count.

Now the other direction. On the broader Artificial Analysis Intelligence Index, as of July 29, Claude Opus 5 sits first at 60.7 and Claude Fable 5 second at 59.9, with GPT-5.6 Sol third at 58.9. Anthropic holds the top two spots on general capability while losing the coding-agent tables. My “diminishing edge” framing was directionally right if imprecise. The edge moved. It didn’t disappear.

SWE-bench Pro, the number everybody reached for six months ago, has no usable current standing. Scale’s public-set leaderboard doesn’t list GPT-5.6, Claude Opus 5 or Claude Fable 5 at all, and the top of it reshuffled again this month. The aggregator sites quoting newer figures disagree with each other and cite at least one model name I can’t confirm exists. Skip it.

The product story is a two-way race now

This is the first half of what I missed. Codex spans CLI, IDE, web, and a hosted cloud mode for long-running tasks. The Codex CLI changelog for the last two weeks of July includes resumable sessions with paginated thread history, session naming and branching, configurable sub-agents, audio input, and an /import command that migrates project-scoped memories out of Cursor and Claude Code. That last one reads like a decision made by somebody thinking hard about switching costs. On July 9 OpenAI also folded the standalone Codex desktop app into a single ChatGPT desktop app with Chat, Work, and Codex modes. Existing Codex users were moved across automatically, and Codex remains a full mode inside the merged app.

What I notice in use isn’t a benchmark, it’s flow. Codex and GPT live in the same product, and when I go looking for something I worked on weeks ago, ChatGPT finds it wherever I left it. Claude’s Projects still behave like sealed folders for me. That isolation was an advantage once, back when context was scarce and a hard wall around what the model could see was the safer design. With current context budgets it mostly means I go hunting. GPT is also the far better image renderer, which makes for a short comparison, since Claude doesn’t ship a first-party image model at all.

Anthropic didn’t stand still while any of this happened. Claude Design shipped out of Anthropic Labs on April 17, and it’s excellent: you talk to it and get slides, wireframes, one-pagers and pitch decks, with export to PPTX or PDF, inline comments, adjustment sliders, and handoff into Claude Code. Codex has no equivalent that I’m aware of (and I’d bet real money they’re building one). Claude Cowork expanded to web and mobile on July 7. Both labs shipped serious B2B product work inside the same four months.

Capacity: a much bigger future, a strained present

The second half is compute. OpenAI is buying a much bigger future than Anthropic is, and is visibly straining in the present.

On committed buildout it isn’t close. OpenAI, Oracle and SoftBank put combined planned Stargate capacity at nearly 7 gigawatts across the Abilene flagship plus five more sites, with over $400B invested and the full commitment still stated as $500B and 10GW. Tomasz Tunguz’s aggregation of the announced vendor deals puts OpenAI’s 2025–2035 infrastructure commitment at roughly $1.15 trillion across seven suppliers, which is derived rather than OpenAI-confirmed. On June 24, OpenAI and Broadcom unveiled Jalapeño, OpenAI’s first custom inference ASIC, designed to tape-out in about nine months, with engineering samples already running Codex workloads in the lab and first production deployment targeted at gigawatt scale in late 2026 (CNBC’s coverage has the same details).

Anthropic’s disclosed commitments are significant but much smaller: up to 5GW with AWS announced April 20 (with nearly 1GW of Trainium online by year end and $100B+ committed over ten years), well over 1GW of Google TPU capacity in 2026 with the Broadcom-built expansion arriving from 2027, and more than 300MW from SpaceX’s Colossus 1. Its custom silicon is at the conversation stage, reportedly early talks with Samsung on a 2nm accelerator with nothing shipping before late 2027.

The strain shows up in the usage limits, which OpenAI moved three times in seventeen days, in both directions. Sol’s launch week doubled traffic inside 48 hours, so on July 12 and 13 it lifted the five-hour cap on Plus, Pro and Business tiers to absorb the load. On July 25 came a near-global outage, 9:17 to 11:08 AM ET, elevated errors across ChatGPT, the API and Codex, resolved the same morning. On July 29 it reset banked limits for ChatGPT Work and Codex users, whose quota Sol was burning faster than the math anticipated, then put the five-hour cap back the next day. Anthropic has been moving the other way since spring. On May 6 it permanently doubled Claude Code’s five-hour limits for Pro, Max, Team and seat-Enterprise users and dropped peak-hour throttling for Pro and Max, on the back of that SpaceX capacity.

Where Anthropic still leads

Anthropic’s remaining, potentially durable, advantages are enterprise adoption and commercial scale. Anthropic is ahead on total revenue and on valuation, and well ahead on enterprise API share. Its Series H, a $65B round closed May 28, 2026, disclosed a $47B run-rate and a $965B post-money valuation. OpenAI’s most recent confirmed figures are roughly $25B annualized (about $2B a month) at the March 31, 2026 close of its $122B round, at an $852B post-money valuation. Menlo Ventures’ December 2025 enterprise report put Anthropic at 40% of enterprise LLM API usage against OpenAI’s 27%, and at 54% in AI coding specifically. No 2026 Menlo report exists yet, so December is the current number.

So OpenAI has closed the capability gap and most of the product gap while trailing on money and on enterprise accounts. That’s the actual shape of the comeback. The lead Anthropic still holds on the Intelligence Index is general capability, and general capability is the harder thing to put in front of a CTO who’s buying a coding agent this quarter.

Apple versus Microsoft, again?

I reached for that analogy myself in May 2025, when OpenAI spent close to $10 billion in three weeks on Jony Ive’s io Products and on Windsurf, and I called it OpenAI’s Apple moment: control the stack from silicon to screen, consumer-first, design-led. Anthropic in that mapping is Microsoft, enterprise and developer-led, winning the accounts while nobody writes headlines about it.

I don’t fully trust it. Apple versus Microsoft was never settled by who had the better quarter. It ran on which company owned the layer everybody else had to build on top of, and on switching costs measured in years. Nobody owns that layer here. Both companies rent it, from Amazon, Google, Microsoft, Nvidia, Broadcom and now SpaceX. For a developer, changing frontier models is a config change and an eval run. What costs real time is changing the tool you work in, which is a much smaller lock than owning the platform. The Apple/Microsoft dynamic assumed you couldn’t leave.

The mapping also keeps slipping. Anthropic is the enterprise player in it, which is right, but it’s also the one with the bigger revenue base and the higher valuation, which is not the role the analogy assigns it.

Can either of them make money

Neither is profitable and both are burning billions, so anything past that is projection.

OpenAI’s advertising business is on pace to miss its own five-year forecast by about 90%, OpenAI projected $2.5B in ad revenue this year, scaling to $100B by 2030. eMarketer puts the entire US chatbot-ad market, not just OpenAI, under $1B this year and around $5.4B by 2030, per 24/7 Wall St. on July 21. The widely circulated 2026 cash-burn estimates in the $25B range are analyst syntheses rather than disclosures, because OpenAI is private and doesn’t publish the figures that would settle it. Same caution applies to the gross-margin numbers people quote for both companies.

Anthropic’s topline looks better but is contested. Ed Zitron argues in ”Anthropic’s ‘Profitability’ Swindle” that the near-term EBITDA profitability claim is a one-quarter accounting artifact of temporarily discounted SpaceX compute during exactly the months profitability was claimed, and that costs still rise linearly with revenue. He also flags CFO Krishna Rao’s March 9 sworn filing citing cumulative revenue “exceeding $5 billion to date” against the $19B run-rate announced six days earlier. That second point is weaker than it reads, since cumulative revenue and annualized run-rate aren’t the same measure. The first point is the one I’d want answered, and it hasn’t been.

Hardware doesn’t resolve it. Chris Lehane said at Davos in January that OpenAI’s first consumer device would debut in the ”latter part” of 2026, which is an unveiling and explicitly not an on-sale date. MacRumors reported in February that the thing is a smart speaker with a camera launching in 2027. Everything else circulating about it (screenless, voice-first, 360-degree camera, Foxconn in Vietnam) is unconfirmed. As of today nothing has shipped and no date is confirmed, so I’d file the device as a 2027 question and stop considering it this year.

Open weights are a side argument

The open-weight families are close to the frontier on raw capability. Kimi K3, the highest-scoring open model on the Artificial Analysis Intelligence Index, sits at 57, about four points behind Claude Opus 5’s 60.7 and under two behind GPT-5.6 Sol’s 58.9, the same index cited above. On LMArena the best open-weight entry, Kimi K3-max at 1491, trails Claude Fable 5 at the top by 17 points.

What none of them have is the app ecosystem. No open-weight provider fields anything within reach of ChatGPT or Claude Code on integration depth or developer reach, at least none I found, though I didn’t dig into DeepSeek’s or Qwen’s domestic app numbers in China, which could complicate that. Weights converging matters much less than it sounds like it should when the surface people work in belongs to two companies.

Which is where I end up. The model tables are the smallest part of it. OpenAI put Codex and ChatGPT in one app with an import command pointed straight at Claude Code’s users, and it has nearly 7 gigawatts of planned capacity standing behind that. Claude Opus 5 still holds the top of the Intelligence Index at 60.7, and Anthropic still held 40% of enterprise API spend as of last December. OpenAI is knocking on the door. A year ago I’d have told you that door was closed.

A coda. Could we imagine a world in which Anthropic and OpenAI run products on other models? Seems very unlikely now. But the cost of developing new models is astronomical, the cost of derived models much less (even though both companies are fighting distillation tooth and nail). So there is a world in which they use the model building work they’ve done as an audience and tool building exercise, and focus on expanding that share by expanding options, let the models fight for their own justification. Farfetched, as I mentioned, but if they start losing to application clones that use open weight models, they’re essentially inviting someone to compete. The existing assumption that the app ecosystem was just to drive captured inference falls apart if that inference is cost prohibitive. We shall see.

Bob Matsuoka is CTO of Duetto and also writes about AI business at AI Power Ranking.

Related reading:

• OpenAI’s ‘Apple Moment’: Building a Walled-Garden AI Stack — Where I first made the Apple analogy, three weeks and $10 billion into OpenAI’s acquisition run.

• It’s the Harness, Stupid — The April piece where I already conceded model parity, under a section header that says so. Everything I got wrong after that is in this article.

• I Switched to Claude.AI from ChatGPT As My Main AI Assistant — The May 2025 call I’m revisiting here, with the context-management and stability reasons that drove it.

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners.

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders.

I made the first commit to my project, called claude-mpm, on July 24, 2025. It was a multi-agent project manager built on top of Claude Code: a Python package that gave you specialist agents, a ticket workflow, a thin memory layer, and a way to route work between them. Building the agents was most of the work, because at that time Claude Code had no notion of a subagent. You wrote your agents as Markdown, loaded them yourself, and orchestrated the handoffs by hand.

Custom subagents shipped in Claude Code on July 25, 2025, the day after that first commit.

I’m not telling this story because of that coincidence. I’m telling it because it marks a moment. In July 2025, multi-agent orchestration was something novel. A year later it’s commonplace. And that single move (capability migrating out of the code you write and into the tool you run) is a line connecting the past twelve months in agentic coding, measurable in two of my own repositories.

TL;DR

• Same author, same subscription, two eras. claude-mpm (Python) took roughly twelve months and 4,552 commits, with memory and search either hand-rolled or bolted on from outside. trusty-tools, a 25-crate Rust monorepo with first-class memory, semantic search, worktree orchestration, and PR review, came together in about two months.

• The budget held. The developer improved, but the harness and the models leapt. I spent the year getting better at driving a team of agents. Even so, most of the difference sits with the tooling, not with me. Both projects ran on the same Claude Max subscription. What changed underneath was the frontier model and everything Claude Code learned to do on its own.

• A year ago you hand-built the PM layer. Agents-as-Markdown, an external vector-search dependency, a memory subsystem you maintained yourself. Working in tickets felt like an edge.

• Now the harness carries it. Subagents nest and run in the background, worktrees are a native flag, skills and memory and search are first-class. Ticket-driven development is table stakes. Running many worktrees against one repo is my working model now.

• The safe extrapolation: The observed path is tickets → worktrees → many concurrent sessions per repo. If the next year rhymes with the last, the harness manages more parallelism than a person can hold in their head.

A year ago: you built the orchestration yourself

claude-mpm was shaped by what the harness couldn’t do at that time.

The frontier models in late July 2025 were Claude Opus 4 and Sonnet 4, which had reached general availability on May 22, 2025. Claude Code itself was about two months into GA, capable but young. Subagents didn’t exist until the day after I started. Agent Skills wouldn’t launch until October 2025. There was no native worktree support. If you wanted isolated parallel work, you ran git worktree add yourself and wired it up by hand. MCP was maybe eight months old. Memory and semantic search were not primitives the harness handed you.

So we (claude-mpm and me) built all of it. The agents were Markdown templates the package loaded and orchestrated. The ticket workflow lived in the CLI. Memory was thin by necessity, a couple of files under src/claude_mpm/memory/, and even that was on its way out. The changelog shows the memory hooks being removed and handed off to an external successor rather than maintained in-tree. Semantic search wasn’t native either. It came in as an outside dependency, mcp-vector-search, referenced across dozens of files. Worktree awareness existed at the level of a consumer that knew the concept, not a subsystem that owned it.

That was the state of the art then, and it worked. Over about twelve months claude-mpm accumulated 4,552 commits (roughly 380 a month) and grew into a real system: multiple MCP channel servers built into the Python package, a plugin path exposing 56-odd skills, agent templates for a spread of specialist roles. Ticket-driven development, routing discrete units of work through a queue instead of narrating one long conversation, felt like an advance at the time. You had to construct the scaffolding to get there, and the scaffolding was the hard part of the project.

Now: the harness carries it

trusty-tools took its first commit on May 19, 2026. It was still under active development the day I pulled these numbers. In roughly two months it reached 1,726 commits on main, about 860 a month. Same author, same Claude Max subscription, a different era.

That first commit wasn’t a blank slate. It already carried claude-mpm’s PM scaffolding, and one absorbed component holds claude-mpm session logs dated May 11, a week before the new repo existed. The Python predecessor was building its Rust successor. Then the successor began building itself: on July 6 the commit attribution switched to “generated with trusty-mpm,” in a commit that was fixing trusty-mpm’s own guard for its own subagents. Three days later the instructions moved out of CLAUDE.md into trusty-mpm’s own convention. About seven weeks were built by its predecessor before it took over its own development.

A lot shipped in that window. The frontier moved to Claude Opus 4.8 (late May 2026) and Claude Sonnet 5 (end of June 2026), with a Mythos-class model, Fable 5, arriving in June at a million tokens of context. The million-token context window had gone GA at standard pricing in early 2026. Inside Claude Code, subagents now nest several deep and run in the background by default. /fork and /subtask landed in mid-2026, and a native --worktree flag had arrived in 2026. The harness I was building on top of in mid-2026 was a different animal from the one I started claude-mpm against.

And trusty-tools reflects that, because it didn’t have to build the parts the harness now provides. It could spend that effort building capabilities further out. The result is a 25-crate workspace consisting of about 600K lines of Rust. The pieces claude-mpm hand-rolled or imported are now first-class crates of their own:

• Memory is trusty-memory, a dedicated crate with knowledge-graph operations, “dream” consolidation that compacts and reorganizes stored facts, multiple namespaced memory “palaces,” and chat-session persistence. claude-mpm had no equivalent. Its two-file memory layer was being handed off precisely because maintaining that by hand no longer made sense.

• Search is a first-class crate (semantic, lexical, and knowledge-graph search, call-chain lookup, typeahead, indexing) rather than an external MCP dependency referenced across the codebase.

• Worktree orchestration is heavy and native to the design: EnterWorktree/ExitWorktree as real operations, and fifteen-plus simultaneous live worktrees as the ordinary way of working, not a party trick.

• Ticketing is a dedicated subsystem spanning GitHub issues and JIRA, not an example command.

• And then the crates with no claude-mpm analogue at all: PR and diff review, git analytics, a code intelligence layer, a TUI, an embedding daemon.

• It also has an original (not meta) harness called trusty-code (still a work in progress but early indications are that it will perform similarly to open-code), and a personal agents harness called trusty-agents. Both leverage many of the same orchestration tools that trusty-mpm uses, which is why I’m including them in the package.

On top of that sits a catalog of 37 specialist agents over five foundation layers, and a live skill catalog surfacing more than 190 skill names, triple claude-mpm’s 56. The prompting style changed too. A year ago you spent your prompt budget teaching the model how to be an agent. Now you spend it telling a competent agent what you want, because the harness supplies the how (in the form of memory, search, specs, and tickets).

Speaking of, ticket-driven development, the edge a year ago, is now the assumed baseline. The frontier moved up a level: many worktrees against a single repo or monorepo, several sessions of work in flight at once.

The measured contrast

Two projects, one author, one subscription. Here’s the comparison. Effort and wall-clock framing are estimates. Commit counts and dates are exact. The productivity story they imply is inference.

Time to build a comparable system: claude-mpm ~12 months versus trusty-tools ~2 months, roughly a sixth of the wall-clock time.

A more capable system (25 crates with first-class memory, search, worktree orchestration, PR review, and git analytics) came together in about a sixth of the calendar time.

Commit velocity: claude-mpm ~380/month versus trusty-tools ~860/month, roughly 2.3x, though squash-merges across many worktrees likely undercount trusty-tools’ true activity.

Velocity roughly 2.3x per month. A year of tech-lead work sharpened the exact skills this way of working rewards, driving a team, now a team of agents, and holding SDLC discipline as the branches multiply. And it shows in the git record, not just in my own say-so. Three signals I can actually measure. Commit messages that name a driving issue or decision rose from roughly one commit in ten to about nine in ten. The code now logs why, not only what. The eight near-identical agent files I copy-pasted early in claude-mpm collapsed into one composed base definition, a fix I started mid-project and carried further into trusty-tools. And architecture decision records went from essentially none to eighteen numbered ADRs plus a per-crate decisions taxonomy, a habit that matured across both projects rather than one trusty-tools invented. Developer growth and harness capability compound. They push the same direction. But even with that, a person going from good to better doesn’t buy you a sixth of the calendar time. The impressive part is still the tooling.

Capability checklist across three columns, a year ago, now, and a speculative year from now, for orchestration, memory, search, worktrees, skills, ticketing, and PR review.

Every row that read “you build this” a year ago reads “the harness provides this” now. Multi-agent orchestration: hand-built then, native flag now. Memory: thin and hand-maintained then, a knowledge-graph crate now. Search: external dependency then, first-class now. Worktrees: DIY shell commands then, a --worktree flag and fifteen live trees now.

The harness now enforces the workflow, not just supplies the parts. Spec-linked documentation was a prompt hint in claude-mpm, off by default. In trusty-tools it’s a build-blocking lint gate (trusty-sld-lint) wired into CI and pre-commit. Documentation that fails the build, not documentation you’re reminded to write. Ticket discipline hardened the same way: that rise in issue-referenced commits now sits inside a codified chain (spec → issue → PR-linked-to-issue → review gate → squash-merge), process-enforced, not yet a hard check that rejects an unlinked PR. And claude-mpm’s main branch required zero approvals and no passing checks, protected in name only. trusty-tools’ main requires a review approval and six passing CI checks, with an LLM review pass (trusty-review) as a named gate. A year ago a disciplined developer chose these. Now the harness refuses to merge without a review and passing checks.

A word on the subscription plan

Claude Max was announced in April 2025 at $100/month (Max 5x) and $200/month (Max 20x), with usage shared across chat and Code. Those price points appear to have held from then through mid-2026. The most visible change over that whole span was the five-hour rate limits for Claude Code, which Max subscribers draw on, being doubled in mid-2026. More headroom, not a different product tier.

So the input that stayed roughly constant was the money and the person. The input that changed was the model quality and the harness capability. When you hold the developer and the budget still and the output grows by this much, the variable that moved most would be the model and the harness (mostly the model, though in my experience good memory and search make a huge difference), even granting the developer improved and the language changed.

A year from now

The trajectory has a clear shape: tickets → worktrees → multi-session parallelism. Working in tickets was novel in mid-2025 and is common sense now. Native worktrees arrived in early 2026 and, within months, running many of them at once became my working model rather than a demo. Each step took a workflow that used to live in the developer’s head (tracking the units of work, isolating the parallel branches) and moved it into the tool. There are no hard adoption statistics for this progression. I’m describing a tooling timeline and a direction, not a measured majority practice. The tooling timeline itself is real, though: worktree support arriving across editors and agents through late 2025 and into 2026 follows the same curve.

The next frontier isn’t hard to name. If tickets became common sense and worktrees became my working model, the thing after worktrees is many concurrent sessions against one repo or monorepo. Enough parallel work in flight so that no human is tracking all of it, and the harness keeps the branches, the memory, and the review gates coherent. The /fork and /subtask primitives that landed in mid-2026, and subagents running in the background by default, are early moves in exactly that direction. The developer’s job shifts further from writing the steps toward specifying the outcome and reviewing the merge.

That’s not a prediction of artificial general anything. It’s the same migration, run one more turn: complexity leaving the code you write by hand and entering the tool you run. A year ago the hard part of the project was the scaffolding. This year it’s the harness. Next year it’s the coordination of more parallel work than one person can follow. So that one person gets pushed further up into the spec and the design.

What a difference a year makes

I built claude-mpm, and I’m very proud of it. For its moment it was a good answer to a real constraint. That’s the ordinary fate of scaffolding once the platform grows the feature underneath it, the design working as intended.

The measure of the year isn’t that I got better, though I hope I did. It’s that the same person, on the same max plan, with the same working habits, could build a materially more capable system in a fraction of the time, because the models got sharper and the harness absorbed the work that used to be mine to do. Twelve months of hand-built PM layer on one side, two months of composing first-class parts on the other. Same author. Same subscription.

What a difference a year makes.

Bob Matsuoka is CTO of Duetto and also writes about AI business at AI Power Ranking.

Related reading:

• It’s the Harness, Stupid — Why orchestration, not raw model quality, drives the spread in outcomes — the mechanism underneath this whole comparison.

• What Is Harness Engineering? (And Do You Need to Learn It?) — The durable skill under the tooling: designing the loop, not just the scaffolding.

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners.

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders.

Every engineering org I’ve worked in has a folder like this somewhere. A docs/ directory, a Confluence space, a Google Doc linked from a wiki page nobody visits. Inside it: the spec. Written carefully before a feature shipped. Reviewed by three people. Approved. And then, the moment the code merged, functionally dead — a fossil of what the team intended, drifting from what the system does.

I don’t say that to complain about lazy documentation habits. Nothing in the workflow ties the spec to what ships — nothing breaks when the code changes and the doc doesn’t, and nothing in CI cares. The traceability matrix that maps requirements to implementation gets updated once, at launch, because keeping it current isn’t anyone’s job. Six months later an engineer greps the spec for the retry policy, finds three paragraphs of prose, and can’t tell whether they match the code or a decision quietly overridden in a hotfix a year back.

Spec-driven development, in its traditional form, fixes this with process: write the spec first, get sign-off, then build to it. That helps at the start of a project and does nothing for the middle, where the spec is still a separate artifact checked by separate tooling — if it’s checked at all. The argument here is narrower than “write better specs.” The fix isn’t a better document; it’s making the document part of the same system that builds and tests the code, so it can’t drift without something noticing.

TL;DR

• Nothing enforces the spec-to-code link, so it rots. A standalone spec has nothing tying it to what ships; nothing breaks when it drifts, so it does.

• The reframe: the spec becomes part of the workflow — referenced inline from code comments, parsed as data by the build, versioned in the same commits as the implementation.

• trusty-mpm is a working instance, not a proposal: a spec requirement runs verbatim inside the prompt the model reads each session, code comments cite spec sections by ID, and a Rust module parses spec markdown at build time to check code against it — flagging code that cites a spec revision that’s since changed.

• This isn’t new — literate programming, Gherkin’s living documentation, rustdoc doctests, and design-by-contract all tried versions of it — but AI-generated code gives it new urgency: the volume of change makes a hand-maintained spec even less credible than before.

• The industry is converging on the same shape: GitHub Spec Kit, AWS Kiro, OpenSpec, and Tessl all bet that spec and code must co-evolve as tracked artifacts, not a document and its distant cousin.

• A minimal frontmatter schema — stable ID, version, status, applies_to globs — is the concrete thing to adopt this quarter, whether or not you touch any of the tools above.

The reframe: the spec as workflow, not deliverable

Here’s the alternative. Instead of describing the system from outside, the spec becomes a component the system consumes. Code references it by a stable ID; the build (or a linter, or a review bot) parses the spec and checks the referenced section still says what the code assumes. Editing the spec means touching something with known consumers, not filing a document into a drawer.

None of this is new. Knuth’s literate programming interleaved explanation and code decades ago; Gherkin made acceptance criteria executable; Rust’s doctests and mdbook test run documentation’s code examples in the test suite, so a doc that lies about the API fails CI; design-by-contract (Eiffel, Ada/SPARK) puts pre- and postconditions in the interface; RFC 2119 gave the field MUST/SHOULD/MAY, precise enough for a machine to check.

What’s changed is the pace, and who’s writing the code. When a team ships every two weeks, a stale spec is an annoyance you catch next planning cycle. When an agent generates hundreds of lines and opens a PR before you’ve finished coffee, an unchecked spec is stale before the reviewer finishes the diff. The old techniques weren’t wrong; they just didn’t have to survive this throughput.

Why code alone isn’t enough

Open a function you didn’t write and you can read exactly what it does. What you can’t read is whether that’s what it was supposed to do. Code is a precise record of behavior and a silent one about intent — in the source tree a bug and a feature look identical, and the only thing that tells them apart is what someone wanted, which lives outside the code.

Tests don’t fill that gap, even though we reach for them as if they do. A test locks in behavior — it keeps the code doing what it does now — but it can’t tell you that behavior was the one you wanted. A test that a webhook retries three times passes just as happily whether “three” was a deliberate choice or a number someone typed on a Friday. The spec carries the intent, separate from the code meant to deliver it.

The deadline hits, so you take the shortcut. The hotfix goes in at 2 a.m. and nobody circles back. Every codebase is a running negotiation with expediency, and each shortcut gets absorbed into the code, where it stops looking like a shortcut and starts looking like a decision. A year later nobody can tell which lines were chosen and which were just the fastest way through. The spec is the one term of that negotiation you agree not to move — a survey marker: the code shifts around it, and because it stays put, you can measure how far. Take it away and a shortcut is just the code, indistinguishable from what’s near it.

That’s what trusty-mpm’s drift check does, later in this piece: when a spec section moves to a new revision and the code still points at the old one, the check flags it. The marker moved; the tool says so.

That agreement is the line between formal engineering and vibe coding. Vibe coding keeps the output and throws away the intent behind it. It works until the output has to change and neither the humans nor the model can reconstruct what it was for. Sean Grove, whose argument I reach below, calls this version-controlling the binary and shredding the source. AI sharpens the line rather than softening it: ask an agent to fix the case in front of it and it will often comply by quietly narrowing a behavior — solving the immediate problem, dropping an edge case nobody mentioned — and without a spec, that narrowing is just the new code, shipped with passing tests. The difference between formal engineering and vibe coding is the spec: the idea that doesn’t drift for expediency.

An exemplar: trusty-mpm

trusty-mpm — the tm binary — is a Rust-based multi-agent orchestration harness I’ve been building: the layer that composes the prompt assets, agent definitions, and skills a Claude Code session consumes. (Not the unrelated Python project of a similar name.) The spec-as-workflow pattern here wasn’t designed up front; it emerged from a failure. What follows is what it does and why it matters — the file paths and function names are collected in the appendix.

A spec the running system reads

A live trusty-mpm session once misidentified its own harness — told the user it was running under something it wasn’t. The fix wasn’t a prompt patch and a changelog note. It was a spec: a short document stating what the system must know about itself, and when.

What makes it more than a postmortem is where that spec ended up. Its central requirement doesn’t just sit in a docs folder — it runs, close to verbatim, inside the prompt the model reads at the start of every session. And the live instruction cites its own source, naming the exact spec section that specified it. Change that section without updating the prompt and the citation is still pointing at it, flagging the mismatch. The requirement can’t quietly fall out of the running system, because the running system reads it every time.

Links that run both ways

For that to hold up across a whole codebase, specs need addresses. Each spec section has a stable ID, and the code that implements it carries a comment naming the section it satisfies. The spec points back the other way: each requirement lists the modules that implement it. Either side can find the other — which means either side drifting from the other gets caught, rather than surfacing months later as a mystery.

A check that catches an outdated pointer

Those section IDs carry a revision number. When a spec section changes in a way that matters, its revision moves forward. Code that still points at the old revision was written against a spec that has since moved — so a check flags it instead of trusting the stale link (the specifics are in the appendix). It’s the traceability-matrix problem from the opening, turned into something a machine checks on every change instead of a spreadsheet cell a human forgets to update.

A test that fails when the shipped artifact drifts

That self-awareness spec has acceptance criteria: things the shipped prompt must contain. A CI test asserts exactly those. If a later edit strips a required section out of the prompt, the test fails on the pull request that caused it, not in a bug report three sprints later. The spec’s requirement and the test’s pass condition are the same statement, written once.

The smallest version of the same idea

The discipline scales all the way down. Inside individual functions, a short comment states why a behavior exists and names the tests that enforce it, right beside the code. And the house rule is explicit: a spec is a behavior contract — it says what, not how — and code must link to its spec section from the first pull request that implements it, even while the spec is still a draft. Load-bearing before it’s finished.

The industry is formalizing the same idea

trusty-mpm isn’t the only place this is showing up; the convergence from different directions is itself evidence the constraint is real.

The clearest version of the thesis comes from Sean Grove at OpenAI, in a talk transcribed as “The New Code”. Code, he argues, captures maybe 10-20% of a piece of software’s value; the rest is the structured communication — intent, constraints, tradeoffs — that produced it. Most teams version-control the binary and shred the source: they keep the generated code and throw away the spec. He cites OpenAI’s own Model Spec as one meant to stay living and cross-functional rather than one team’s document the rest of the org ignores.

Several tools are building toward the same shape. GitHub Spec Kit runs spec → plan → tasks → implement with artifacts colocated in the repo (specs/[branch]/spec.md), FR-001-style IDs, and a project constitution.md of non-negotiables — though it doesn’t use YAML frontmatter, a gap I address below. AWS Kiro structures each feature as requirements.md, design.md, and tasks.md, plus persistent “steering files” for conventions. OpenSpec is the closest to spec and code co-evolving as tracked diffs: a propose → apply → archive cycle built on delta specs (ADDED/MODIFIED/REMOVED/RENAMED, in GIVEN/WHEN/THEN form). And Tessl raised $125M on the strongest version of the bet — the spec is the source, the code a regenerable output marked // GENERATED FROM SPEC - DO NOT EDIT — though as of mid-2026 still a thesis, not a practice with a track record; I cite it as direction, not proof.

None of these do exactly what trusty-mpm’s drift check does — parse the spec at build time, check it against the linked code, and flag an outdated pointer automatically, in CI rather than argued for as a future state.

The spec stops being about the system

The through-line is narrower than “documentation matters” — most teams agree it matters, and stale docs keep happening anyway. The claim is mechanical: a spec with a stable ID, cited from code, parsed by a build step, checked for drift, behaves differently than one in a wiki — not because anyone tries harder, but because something other than good intentions is now checking.

That’s the demarcation from the opening, made concrete: the difference between formal engineering and vibe coding is the spec — the idea a team agrees not to drift for expediency. The mechanisms in this piece exist to keep that agreement from decaying back into a good intention.

I should hold trusty-mpm to that standard, and it half meets it. The conventions are there: document numbers, stable section IDs, links from code to the spec sections it implements, the drift check, a status / owner / last-updated header on each spec. What’s missing is the last mile I’m prescribing to other teams: that header is semi-prose, not a machine-readable schema — no formal frontmatter, no published schema, no CI gate validating it. trusty-mpm parses spec prose today, not spec metadata, because the metadata isn’t structured enough to parse.

So the next step is to close that gap in trusty-tools: a formal frontmatter block on each spec, a schema behind it, and a CI check that fails when a spec’s metadata drifts from the code it governs. That’s what adopting the standard means here — making the build enforce conventions the project only half-follows, instead of prescribing to others a discipline I’ve left to good intentions in my own repo. The spec becomes a component of the system at the point where the build won’t let it drift. That’s the part still to build.

Bob Matsuoka is CTO of Duetto and also writes about AI business at AI Power Ranking.

Related reading:

• It’s the Harness, Stupid — What a harness is and why orchestration, not model quality, drives the spread in outcomes — the same infrastructure that makes spec-as-workflow possible.

• What Is Harness Engineering? (And Do You Need to Learn It?) — The durable skill underneath the harness: designing the loop, not just the scaffolding — spec resolution is one piece of that loop.

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners.

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders.

Appendix: how it actually works

The practical takeaway: a frontmatter schema that plugs into the build

If you want a piece of this without a whole tool, start with a frontmatter schema for your specs — the structured metadata block at the top of a markdown file — that your build can validate. A precedent to copy: Structured MADR ships a CI-validated YAML schema (JSON Schema plus a GitHub Action) with fields like title, status, created/updated, author, and related. Kubernetes’ KEP (kep.yaml) and Python’s PEP headers are older versions of the same idea: metadata structured enough for tooling to check, not just a reviewer.

Here’s a schema to start from, adapted for code-adjacent specs rather than pure design docs:

Field Example Purpose id SPEC-042 Stable anchor code comments point at (e.g. // impl of SPEC-042#retry-policy) title "Retry policy for outbound webhook delivery" Human-readable name status accepted One of draft / review / accepted / deprecated / superseded version 1.2.0 Bump on any semantically meaningful change → enables revision-drift detection owners ["@handle"] Who owns the spec applies_to ["src/webhooks/**"] Globs this spec governs; CI flags code changed under a glob with no spec touch (and vice versa) supersedes [] IDs of specs this one replaces requirement_ids ["FR-014", "FR-015"] Requirement IDs this spec governs last_verified 2026-06-01 Last time someone confirmed spec still matches code

Each field does something a build can act on, not just something a human can read. Two do the heavy lifting: version makes drift checkable — without it, “the code cites an old section” is a judgment call, not a fact a machine can determine — and applies_to lets CI flag a PR that touches a governed glob with no spec commit, and the reverse. Together they turn the staleness signal a traceability matrix only promised into something a linter checks, not a column a human forgets.

trusty-mpm grew its own version of this organically, out of that self-identification failure. Before any schema existed, its section IDs were already doing the job id does above, and its revision-tagged sections plus the drift check were doing the job version does. Adopting this from scratch, you don’t need to rediscover that path — the frontmatter above is the same mechanism, explicit from day one.

How it works

The mechanics behind each mechanism in the exemplar section — file paths, identifiers, and the code that backs them.

• The self-awareness spec — docs/specs/trusty-mpm-self-awareness.md, tracked as DOC-28. Its requirement R2 runs close to verbatim in the session prompt, BASE_SM.md, and in the output-style file. The live instruction cites it inline: “…the active palace carries an is_fact triple identifying this framework (see docs/specs/trusty-mpm-self-awareness.md §5).”

• The ID grammar — each spec carries a DOC-N document number; each governed section a stable ID in the SPEC-{SUBSYSTEM}-{NN}~{rev} grammar (e.g. SPEC-CONFORMANCE-02~draft), where ~{rev} is the revision the drift check watches. A {#SPEC-…} heading marker anchors each section — like an HTML id — so code links to the section, not the whole file.

• Code-to-spec links — a # Spec References block in a module’s doc comment (the rustdoc Rust renders into API docs). From front_gate.rs:

  //! # Spec References
  //! - [`SPEC-CONFORMANCE-02~draft`](docs/specs/intent-conformance.md#SPEC-CONFORMANCE-02~draft) (§5.1 FRONT gate)
  //! - [`SPEC-CONFORMANCE-01~draft`](docs/specs/intent-conformance.md#SPEC-CONFORMANCE-01~draft) (§4 decision matrix)

The reverse direction: each spec requirement ends with an “Implementing Modules” table plus inline file:line citations.

• The parser and drift check — spec_resolve.rs. parse_spec_refs() scans Rust source for # Spec References blocks; resolve_spec_section() reads the spec markdown, finds the anchored section, and extracts its “Behavior Contract” and “Rationale.” Two gates run it through the same path — front_gate.rs before work starts, conformance.rs (in trusty-review) after — so they can’t diverge on what a section requires. If code links ~v1 of a section that’s since become ~v2, it flags revision_drift = true rather than trusting the citation.

• The CI test — bundle_tests.rs, test output_styles_carry_identity_protocol_and_load_marker, asserts every bundled output style contains the non-overridable Identity protocol section and the marker <!-- trusty-mpm-instructions-loaded: v1 -->. Drift from DOC-28’s acceptance criteria fails the PR that introduced it.

• The doc-comment convention — harness_doc.rs pairs a Why / What / Test triad inside /// comments: the rationale plus the exact test names that enforce it. The house rule is stated in docs/specs/README.md — “a behavior contract… without prescribing the implementation” — and requires code to link to spec sections from the first implementation PR, even while the spec is ~draft.

An engineer at work asked me a good question last week. “What’s harness engineering, and do I actually need to learn it — or is it going to be obsolete by the time I do?” Fair question. The term is about five months old, half the people using it mean different things by it, and the people who build the most capable coding agents around keep going on record to say the thing you’d build will get absorbed into the next model.

So here is my answer, stated plainly, because I think most engineers are getting it wrong: harness engineering is the most important skill you can build right now beyond coding and architecture themselves. And the strongest evidence that it matters is that most engineers don’t yet believe they need it.

I’ve been writing harnesses for over a year, so treat that as a disclosed bias rather than a neutral survey. What follows argues against the smartest version of the other side — the case that harness work is disposable scaffolding the models will eat for breakfast — because that case is largely correct, and it still doesn’t touch the skill I’m talking about.

TL;DR

• Harness engineering is real but young. The phrase traces to Mitchell Hashimoto in February 2026 and an OpenAI Codex case study days later; Fowler and Böckeler formalized it as “Agent = Model + Harness.” Report it as an emerging frame, not a settled discipline — and no “harness engineer” job title exists yet.

• The labs are right that the crutch layer shrinks. Anthropic’s Cat Wu says “the models will eat your harness for breakfast”; Boris Cherny says scaffolding gets “pushed into the model itself.” No one at Anthropic said “don’t learn it.”

• The benchmark fight lives at the crutch layer. Same model, different scaffold moves scores by up to 28 points on GAIA and 18.6 points on Terminal-Bench 2.0 — but Agents’ Last Exam shows model choice drives roughly 3x the spread of harness choice. Scores were beside the point.

• The skill lives at a layer the benchmarks don’t measure. Not a better crutch — a different unit of work: research → spec → ticket → build → PR → review → merge → deploy, driven by the harness. OpenAI ran that loop with 3 engineers to ~1M lines and 1,500 PRs with zero human-written code.

• Harness engineering is not IDE engineering. If you treat the harness as an extension of your old editor workflow, you cap your ceiling. The durable move is letting it drive and jumping in when needed.

• Yes, you need to learn it. The specific syntax is throwaway. That’s exactly why the skill matters — you’re learning to operate a new unit of work, not one vendor’s config file.

The question, stated bluntly

Start with what “harness engineering” is even asking you to do, because the word carries two arguments at once and people talk past each other constantly.

I already made the case that orchestration beats model quality in It’s the Harness, Stupid back in April — same model, large spread in outcomes, the competitive edge moving from model superiority to ecosystem superiority. That piece defined what a harness is and showed that it dominates results. I’m not going to re-argue it. This is the follow-up question that piece left open: if the harness matters that much, is building one a skill worth learning — or a treadmill that resets every model release?

That distinction is the whole article. Because the answer the evidence points to is: parts of it reset every release, and the part that doesn’t is the part almost nobody is naming. Most of the public argument is being had about the parts that reset.

What a harness actually is

The cleanest definition comes from Martin Fowler and Birgitta Böckeler: “the harness” is everything in an AI agent except the model itself. Agent = Model + Harness. They split it into guides that push instructions forward and sensors that feed results back, and they frame the whole practice as a specific form of context engineering. Simon Willison puts it the same way from the other direction: a coding agent is software that acts as a harness for an LLM.

When I say “using a harness,” here’s the concrete inventory I mean:

• Agents — the loop that calls the model and routes its tool calls, plus any sub-agents you dispatch work to.

• Skills — reusable capabilities you can invoke by name instead of re-explaining every time.

• Hooks — deterministic gates that fire on events: run the tests, block a commit, reformat on save.

• Workflow — the orchestrated path from research to deploy, and who (or what) drives each step.

• Harness-specific instructions — how this harness should behave, kept distinct from project-specific instructions about this codebase. Conflating those two is one of the most common configuration mistakes I see.

• Memory and search — what persists across sessions, and how the agent retrieves it.

Internalize that inventory before you touch any tool, because every product arranges these pieces differently and calls them different things. Hugging Face is the one source I’ve found that formally separates the scaffolding (the behavior layer — prompts, tool descriptions, memory) from the harness proper (the execution layer that calls the model and decides when to stop), then notes that most products just call the whole bundle a harness. That ambiguity is real, not something to paper over: the term is contested, some practitioners call it an old idea in new packaging, and Latent Space literally ran a piece titled “Is Harness Engineering Real?”. I don’t want to oversell a five-month-old buzzword. I want to separate the durable part from the disposable part, and to do that I have to give the skeptics their strongest swing first.

“The models will eat your harness for breakfast”

Here’s the skeptical case in its own words, and it’s a good case.

Cat Wu, who heads product for Claude Code, has a line for it: the models will eat your harness for breakfast. Her team does a system-prompt audit on every new model and deletes the reminders the model no longer needs. Her example: a to-do enforcement tool built to stop Claude Code from overclaiming that a refactor was finished became dead weight once newer models completed multi-step refactors on their own.

Boris Cherny, who created Claude Code, says the same thing from the architecture side. In an Every.to interview, he described the tool as “the thinnest possible wrapper over the model” and said that as models advance, “stuff that used to be scaffolding... gets pushed into the model itself.” His team builds harness features they expect to delete: “we build most things... even if that means we’ll have to get rid of it in three months. If anything, we hope that we will get rid of it in three months.” Disposable on purpose. And at Sequoia’s AI Ascent this spring he extended it forward — prompt-injection defenses, static command verification, permission modes, human-in-the-loop gates would all become less critical, he argued, “because models will do the right thing themselves.”

This isn’t only an Anthropic view. It’s the bitter lesson applied to agents: building in how we think the work should be structured tends to lose, over time, to raw capability and scale. Han Lee makes the practitioner version bluntly: “Almost all of it is going to dissolve into the next generation of models... build each production harness like you mean to replace it.” Tool wrappers dissolve because models read OpenAPI specs directly now. Elaborate memory layers collapse into “plain text in progress.md plus git log.”

Two concessions I’ll make up front, because the fair version of this argument requires them. First: nobody at Anthropic said “don’t learn it.” The claim that they did is a paraphrase — a real cluster of “the harness shrinks” statements from Wu and Cherny, compressed by repetition into something stronger than anyone actually said. Second, and this is the one that stings: even the benchmark evidence for harnesses says model choice usually wins. On Agents’ Last Exam, swapping models with the harness fixed produced an 18-point pass-rate spread; swapping harnesses with the model fixed produced about 6. “The model accounts for about 3x the pass-rate spread of the harness.” If your goal is a higher number on the leaderboard, buy the better model before you tune the scaffold.

So the skeptics have a benchmark, a bitter lesson, and the people who build the reference implementation all pointing the same way. If I stopped here, the answer to “do you need to learn it” would be “not really — wait for the next model.” I don’t stop here, because all of that is arguing about a layer I don’t mean.

Why both are right — and why it doesn’t touch the real skill

The reconciliation is that “harness” names two different things, and the argument above is entirely about the first one.

The first layer is scaffolding-as-crutch. A hook that reminds the model to actually run the tests. A tool wrapper that translates an API the model can’t yet read. A permission gate that catches a mistake the model still makes. Anthropic’s own framing nails why this dissolves: “every component in a harness encodes an assumption about what the model can’t do on its own.” When the model can suddenly do that thing, the component becomes dead weight — exactly Wu’s deleted to-do tool. This layer is supposed to shrink. Anthropic builds it disposable on purpose. The benchmark gaps live here too: the 28-point GAIA swing and the 18.6-point Terminal-Bench spread measure how much a scaffold props up a fixed model’s score. Prop-up value falls as the model climbs. That’s the whole skeptical case, and it’s correct.

The second layer is workflow orchestration. Letting the harness drive the entire loop — research → spec → ticket → build → iterate → PR → review → merge → deploy — as one continuous operation instead of a sequence of prompts you babysit. This layer does not dissolve into a better model, because a better model doesn’t decide what’s safe to run unattended, what the blast radius of an autonomous change is, or where a human judgment call has to sit. Better models make the loop run better. They don’t make the loop design itself.

Blake Crosley draws the same line and I think it’s the sharpest version: harness syntax is ephemeral and gets absorbed, but verification judgment is durable — knowing what’s safe to run without watching, what the acceptable failure modes are, where the loop needs a gate. The config file you write today is throwaway. The judgment about how to structure autonomous work is not.

The killer piece of evidence sits in the phrase’s own origin story. When OpenAI published its harness-engineering case study, the headline number was three engineers producing roughly a million lines of code across 1,500 pull requests, with zero human-written code, by engineering the harness around Codex. Read that carefully. That is not a better crutch bolted onto a fixed workflow. It’s a different unit of work — the engineers stopped writing lines and started operating a loop. No model upgrade alone produces that shape of output, because the shape is a workflow-design decision, not a capability. Ryan Lopopolo’s summary of what changed is the tell: the only scarce resource left was synchronous human attention. That’s an orchestration problem, and no amount of model progress makes it go away.

This is why I can concede the entire benchmark argument without losing anything. Model choice beats harness choice on scores — sure, roughly 3x on Agents’ Last Exam. But the fight over scores is being had at the crutch layer, and the skill I mean lives at the orchestration layer, which those benchmarks don’t even measure. Even the strongest model still needs someone who knows how to hand it a whole workflow instead of a single task.

Harness engineering vs. IDE engineering

Here’s the crux, and it’s where I think most engineers cap their own ceiling without noticing.

The biggest difference between harness engineering and IDE engineering is what the unit of work is. In the editor era — including the AI-autocomplete-in-your-editor era — the unit is a change you make, assisted. You’re still driving. The tool suggests, you accept, you commit. A good harness inverts that. The unit becomes an outcome you delegate: research through deploy, handled by the harness, with you supervising the loop rather than typing inside it.

If you approach a harness as an extension of your existing editor workflow — a faster autocomplete, a smarter pair — you’ll get some lift and you’ll hit a ceiling fast, because you’re still the bottleneck on every step. The results I’ve gotten that actually surprised me came from letting the harness drive the whole loop and jumping in only where my judgment was needed: at the spec, at the review, at the “is this safe to merge” gate. That maps exactly onto Crosley’s durable skill. You’re not writing less carefully. You’re spending your attention on the decisions that don’t delegate, and letting the loop own the ones that do.

This is a materially different workflow from anything I did before agents, and I say that as someone who’s lived through several supposed paradigm shifts that turned out to be the same job with new keybindings. This one isn’t. The muscle you build isn’t “prompt the model well.” It’s “decompose an outcome into a loop a machine can run mostly unattended, and know precisely where to stand in it.” Harrison Chase, who runs LangChain, frames harness engineering as an extension of context engineering, and that lineage is right — context engineering is about a year old and settled, harness engineering is the newer, contested layer on top. But the operative verb changed. You’re not composing a context window. You’re operating a workflow.

Do you need to learn it? Yes — here’s how

Yes. The fact that it still feels optional is the problem, not a reason to wait.

Here’s the path I’d suggest, and it’s roughly the one I took.

Start by understanding what using a harness means — the inventory from earlier: agents, skills, hooks, workflow, harness-specific versus project-specific instructions, memory, search. Not as vocabulary. As the actual pieces you’ll arrange. If those seven words don’t map to concrete settings you can change, start there before you touch a workflow.

Try several, and configure them well. Don’t judge the category from one tool on defaults. Learn to tune Claude Code yourself until it performs, rather than running it out of the box and concluding the harness “doesn’t matter.” Try Codex, Gemini, Auggie, OpenCode. I built claude-mpm, so weight my enthusiasm for the multi-agent approach accordingly — the point isn’t which one wins, it’s that you can’t feel the shape of the skill from a single vendor’s config.

Lean into the differences instead of smoothing them over. The instinct is to find the tool that feels most like your old editor and stop. The results live in the opposite direction — in the workflows that feel least familiar, where the harness drives and you supervise. Best results come from leaning into what’s different, not translating it back into what you already knew.

Let it drive, and know where to stand. This is the whole skill in one sentence. Hand the loop the outcome, supervise at the gates your judgment actually owns, jump in when the blast radius or the ambiguity demands it. That standing-in-the-right-place instinct is what transfers across every tool and survives every model release.

And that’s the reframe I’ll close on. Everything about the specific harness you learn this quarter is throwaway. The config syntax, the exact hooks, the tool wrappers — Han Lee is right, the next model eats away at it. But that’s precisely why the skill is worth building, not a reason to skip it. You’re not learning one vendor’s settings file. You’re learning to operate a new unit of work — an autonomous loop from research to deploy — and that competence is the thing the model upgrades keep raising the value of, not erasing. The syntax is disposable. The judgment about how to run the loop is what compounds.

Most engineers will figure this out eventually, when the workflow shift is obvious in hindsight. The ones who figure it out now get a head start measured in the gap between “my editor got smarter” and “my unit of work changed.” I’d rather be early on that one.

One layer up: loop engineering

There’s a move past the harness, and it picked up a name while I was writing this.

The stack is starting to read like a ladder: prompt engineering, then context engineering, then harness engineering — and now loop engineering on top. Each rung stops being where you spend your attention once the rung below it gets good enough to trust. You quit hand-tuning prompts when context engineering settled into a roughly year-old, mostly-solved practice. The bet underneath loop engineering is the same shape one level up: once your harness is solid, you stop prompting the agent and start designing the loops that prompt it for you.

The term isn’t mine. Addy Osmani formalized it in June, and his definition is the one I’d hand someone first: “Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.” He also collected two lines that land the shift faster than I can. Peter Steinberger’s version: you should be designing the loops that prompt your agents. And Boris Cherny — the same Cherny from the “eat your harness for breakfast” section — put it flatly: “I don’t prompt Claude anymore… my job is to write loops.” LangChain picked it up a week later, and their framing is the practical one: you don’t build a loop, you stack them — an agent loop inside a verification loop inside an event-driven loop inside a hill-climbing loop, each one checking the one below it.

Here’s the seam between the two layers, drawn plainly. Harness engineering is the discipline of building the scaffolding — the agents, hooks, skills, gates, and workflow from the inventory up top. Loop engineering is what you do once that scaffolding is good enough that you stop typing prompts and start directing repeatable cycles. The harness is what you build. The loop is what you run on it, again and again, with your attention moved up to which loops to run and when to trust them unattended. That’s the same “let it drive, know where to stand” instinct from a few paragraphs back, pushed one rung higher: now you’re not standing inside the loop at all — you’re choosing which loops get to run.

I’m not planting a flag here. The people already naming it are out ahead of me, and that’s the reason to point at it — this is where the harness work goes next, not a term I’m coining. If harness engineering closes the gap between “my editor got smarter” and “my unit of work changed,” loop engineering is what shows up on the far side of that gap, once the unit of work is a cycle you supervise instead of a task you run. I’m watching that one closely. And I’d start learning it before it feels obvious.

Bob Matsuoka is CTO of Duetto and also writes about AI business at AI Power Ranking.

Related reading:

• It’s the Harness, Stupid — The predecessor to this piece: same model, wide spread in outcomes, and why the competitive edge moved from model quality to orchestration. It defined what a harness is; this piece argues that building one is a skill worth learning.

• HyperDev’s Three Golden Rules — The working rules I keep coming back to for professional AI work, and the discipline that keeps a driven-by-the-harness loop from running off the rails.

• Claude Sonnet 5 Takes the Default Driver Slot — A concrete example of the crutch layer shrinking: adaptive thinking folds interleaved reasoning into the model, removing work harness authors used to do by hand.

• Loop Engineering — Addy Osmani’s June 2026 piece that named the layer above the harness: once the scaffolding holds, you stop prompting the agent and design the loops that prompt it. The forward edge of the arc this article traces.

• The Art of Loop Engineering — LangChain’s treatment of loop engineering as stacked loops — agent, verification, event-driven, hill-climbing — each one checking the one beneath. The practitioner’s map of where harness work heads next.

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners, including the coding-agent leaderboards this piece leans on.

Claude Sonnet 5 landed this week, June 30, 2026, as the new default on Claude Free and Pro and the new occupant of the middle tier: Haiku below it, Opus 4.8 above, Fable 5 and Mythos 5 at the top. The pitch from Anthropic is the one practitioners care about — Opus-adjacent capability at Sonnet economics, aimed squarely at high-volume agent loops where Opus pricing compounds fast. One wrinkle sharpens the switch decision. Fable 5 is back in Claude Code, redeployed the morning after Sonnet shipped, which puts the top of the lineup back in reach and moves the ceiling any switch has to weigh against.

Two things matter more than the headline. First, the architecture shift: extended thinking is gone, replaced by adaptive thinking that runs on by default and interleaves reasoning between tool calls. The effort parameter now defaults to high in both the Claude API and Claude Code. Second, the pricing has a catch the announcement does not foreground. The list price looks flat against Sonnet 4.6. The new tokenizer means your invoice may not be.

If you drive coding agents for a living, this is the model you will be pointing your harness at by default. The question is whether to switch today[1], reach past it, or wait two weeks and measure.

TL;DR

• New default everywhere. Sonnet 5 is the default model on Free and Pro at launch. API ID claude-sonnet-5, Bedrock anthropic.claude-sonnet-5, Vertex coming soon.

• Extended thinking removed; adaptive thinking is always on. Manual thinking: {type: "enabled", budget_tokens: N} now returns a 400 error. Depth is set by effort (low, medium, high, xhigh, max; high by default on the API and in Claude Code), and the model reasons between tool calls without prompt engineering — the architecturally significant change for agentic work.

• 1M context, 128k output, January 2026 cutoff. Repo-level reasoning at Sonnet pricing.

• Agentic coding 63.2%, per Anthropic’s comparison chart, versus 58.1% for Sonnet 4.6 and 69.2% for Opus 4.8. Likely SWE-bench Pro, not Verified — and no Verified score is published yet.

• Pricing looks flat, isn’t quite. $2/$10 per MTok intro through August 31, then $3/$15 — same list price as Sonnet 4.6. But the new tokenizer generates 1.0–1.35x more tokens for the same text, so “same price” is misleading at the invoice level.

• Fable 5 is back in Claude Code (July 1) as the practical ceiling above Opus — priced higher, with a retrained safety classifier that reroutes offensive-cyber work down to Opus 4.8. Details below.

What Anthropic shipped

Sonnet 5 sits in the workhorse slot of the lineup. Haiku 4.5 at $1/$5 handles cheap high-volume work. Opus 4.8 at $5/$25 is the heavy lifter. Fable 5 and Mythos 5 sit above Opus, both dark at Sonnet 5’s launch, though Fable returned to Claude Code on July 1 (more below). Sonnet is the tier you point at the bulk of your agent traffic, and the one whose economics decide whether a coding-agent product is viable.

The specs are current-generation and unsurprising on paper: 1,000,000-token context window, 128,000 max output tokens (300,000 on the Batch API with the beta header), text and image input, January 2026 knowledge cutoff. Anthropic calls it “the most agentic Sonnet model yet,” which is marketing, but the architecture underneath the phrase is the actual story.

Extended thinking is gone. If your code sends thinking: {type: "enabled", budget_tokens: N}, Sonnet 5 returns a 400 error. That is a breaking change for any harness that sets thinking budgets explicitly. Grep your codebase for it before you flip the model ID. In its place is adaptive thinking: always on, with depth allocated dynamically by the model and steered through the effort parameter — low, medium, high, xhigh, or max. The default is high on both the API and Claude Code, with xhigh available above it for the hardest coding and agentic tasks.

The piece that matters for agents is that adaptive thinking automatically enables interleaved thinking. The model reasons between tool calls. It reflects on what a tool returned before deciding the next action, and it does this without you wiring up a scratchpad or a reflection prompt. For anyone who has built an agent loop by hand, that is the part you used to engineer yourself, now folded into the default behavior of the model.

About that benchmark number

Anthropic’s launch chart gives an agentic coding comparison:

Model Agentic coding score Claude Sonnet 4.6 58.1% Claude Sonnet 5 63.2% Claude Opus 4.8 69.2%

Source: Anthropic’s comparison chart, via TechCrunch. Now the caveats, because this is where launch-day coverage tends to get sloppy.

That 63.2% is almost certainly SWE-bench Pro — the harder multi-file agentic eval — not SWE-bench Verified. The two are not interchangeable, and the numbers live on different scales. The Opus 4.8 figure in the same chart (69.2%) matches its published SWE-bench Pro score, which supports the Pro read. Anthropic has not published a standalone SWE-bench Verified number for Sonnet 5. The comparison chart is an image, with no underlying table released at launch.

So here is the trap. If you go looking, you will find a 82.1% SWE-bench figure attached to “Claude Sonnet 5.” Do not use it. That number comes from February 2026 pre-launch speculation about a different model iteration — a phantom Sonnet 5 with a “Dev Team Mode” that never shipped in the form described. The model that launched today has different characteristics. Any spec sheet dated before June 30 is describing something else.

What can you say with confidence? Sonnet 5 sits meaningfully above Sonnet 4.6 on agentic coding and a few points below Opus 4.8. For reference, the most recent third-party leaderboard before launch had Sonnet 4.6 at 79.6% SWE-bench Verified and Opus 4.8 at 88.6%. A reasonable expectation puts Sonnet 5’s Verified score somewhere in the low-to-mid 80s — but that is my read of where the gap lands, not a number Anthropic has confirmed. Treat it as judgment, not data.

On the safety side, Anthropic reports lower hallucination and sycophancy rates than Sonnet 4.6, better refusal of malicious requests, and stronger resistance to prompt injection. By design, it also carries intentionally weaker cybersecurity exploit capability than Opus 4.8.

Pricing and the tokenizer tax

The list price is the easy part:

Model Input $/MTok Output $/MTok Claude Haiku 4.5 $1.00 $5.00 Sonnet 5 (intro, through Aug 31) $2.00 $10.00 Claude Sonnet 4.6 $3.00 $15.00 Sonnet 5 (standard, from Sep 1) $3.00 $15.00 Claude Opus 4.8 $5.00 $25.00

At standard rates, Sonnet 5 carries the identical list price to Sonnet 4.6: $3/$15. Through August 31 you get an introductory $2/$10. Read quickly, that says “same price, free discount for two months.” Read the footnote and it says something else.

Sonnet 5 uses the same tokenizer as Opus 4.7, which encodes the same text into roughly 1.0–1.35x more tokens than pre-Opus-4.7 models, depending on content type. List price is per token. So at the standard September rate, identical workloads can cost up to 35% more than they did on Sonnet 4.6 — same sticker, more tokens on the meter. The introductory pricing is doing real work here: the 33% input discount roughly offsets the tokenizer inflation through the end of August, which is almost certainly the point. Come September 1, the offset disappears and the effective increase shows up on the invoice.

The practical move is dull and necessary. Run a representative slice of your actual workload through Sonnet 5 and measure token consumption directly. Do not assume cost neutrality from the matching list price. The teams that got burned on the 4.6-to-4.7 transition were the ones who read the sticker and skipped the meter.

Where it sits in the market

The framing that fits the launch is that the workhorse-tier fight has moved off “which model is smartest” and onto “how cheaply and reliably does it run without a human watching.” That is the right frame for anyone shipping coding agents, where margin lives or dies on the per-task cost of the driver model.

Anthropic positions Sonnet 5 as cheaper than OpenAI’s GPT-5.5 and Google’s Gemini 3.1 Pro at introductory rates, and more expensive than Gemini 3.5 Flash, which holds the budget slot. I’d treat the competitor pricing as directional rather than precise — those comparisons come from launch coverage, not from cross-checked primary pricing pages, and competitor list prices move. The shape is more reliable than the digits: Sonnet 5 is priced to undercut the premium workhorses and to sit above the bargain tier, which is exactly where Anthropic wants the default coding driver to land.

The use case Anthropic names is the one this audience will actually hit: high-volume agent loops where Opus pricing compounds. CI review bots. Test generation. Batch transforms. Multi-step autonomous workflows that run for a while without supervision. Daniel Shepard at Zapier told TechCrunch that a two-part automation which “used to stall halfway” now finishes end to end. A single data point, and a vendor-friendly one, but it points at the right capability: completion of long autonomous chains, not raw single-shot smarts.

What this means for agentic coding

The interesting architecture is the harness story. Default-high effort plus interleaved reasoning means the model arrives already configured for the agent pattern most teams hand-build: think, act, observe the result, reflect, act again. You do not prompt-engineer your way to a reflective agent anymore. It is the out-of-box behavior. For harness authors, that shifts the work — less coaxing the model into reasoning between steps, more managing the effort dial so a 12-file rename does not quietly run at high and burn tokens it never needed.

Watch that dial. effort defaults to high, and high effort spends more tokens than the job often warrants. The same lesson from the Opus 4.8 cycle applies: the cost surprises come from leaving the reasoning budget maxed on work that did not need it. Drop to medium or low for mechanical tasks. Reserve high for the hard ones.

And the 1M context window at Sonnet pricing is the underrated piece. Repo-level reasoning — cross-file refactors, full-codebase audits — without chunking, on the model you were already going to run for volume.

The ceiling moved: Fable 5 is back

Then there is the ceiling, and in the days around this launch it moved. The most capable public Claude model most teams could actually run was Opus 4.8, at roughly 88.6% SWE-bench Verified on the Morph leaderboard. Fable 5 sits higher on that same board, in the mid-90s, though that figure is a third-party read and Anthropic has published no Verified score of its own. What moved here was less capability than access: at Sonnet 5’s June 30 launch, Fable was dark. Anthropic had suspended Fable 5 and Mythos 5 on June 12 to comply with a U.S. export-control order, and with no way to verify user nationality in real time, it pulled both models for all users.

That reversed a day later. On June 30 the Commerce Department lifted the order, and on July 1 Anthropic redeployed Fable 5 globally across the Claude Platform, Claude.ai, Claude Code, and Cowork. As I write this, my Claude Code /model picker offers Fable 5, and I’ve set it as my default for new sessions. So the practical ceiling is no longer Opus. It is Fable — with two asterisks.

The first is access and price. Through July 7, Fable 5 counts against up to 50% of weekly usage limits on Pro, Max, Team, and select Enterprise plans; after that it shifts to usage credits, and cloud-provider access on AWS, Google Cloud, and Microsoft Foundry is still being re-enabled in phases. Per token it prices well above the Opus tier: Anthropic’s model docs list it at $10/$50 per MTok, double Opus 4.8’s $5/$25, and the new tokenizer encodes the same text into more tokens again, so the effective gap is wider than the sticker. This is not the model you point at bulk agent traffic. It is the one you reach for on the problems that stall everything below it.

The second asterisk is the reason it came back at all, and it lands squarely in coding work. The suspension traced to a report from Amazon researchers who found a prompt that got Fable 5 to identify software vulnerabilities and start describing how one could be exploited, before its guardrails blocked the attempt from reaching a working exploit. Anthropic’s fix was not to weaken the model but to retrain the safety classifier sitting in front of it, and the new one blocks that specific technique in more than 99% of cases. When the classifier fires, the request doesn’t error. It reroutes to Opus 4.8 in the same session, re-run and labeled with the model that actually answered. The documented trigger categories are offensive cybersecurity work (building exploits, malware, or attack tooling), plus most biology and chemistry, distillation attacks on Fable itself, and frontier-model development.

Here is where it gets concrete. Ask Fable 5 to write exploit code against a security vulnerability and you can watch it hand the task down to Opus 4.8 mid-session, a quieter, older model finishing what the newer one declined. For defensive security work that reversion is a real cost, not a hypothetical. Anthropic is candid that the tighter cyber filter routes more benign coding and debugging requests to Opus than teams would like, and developers have already reported the classifier over-firing on routine defensive-security work, auto-switching to Opus on tasks like CVE triage. The billing follows the block: a request stopped on input is charged at Opus rates; one stopped midstream bills Fable rates for the tokens already produced, then Opus for the rest.

Should you switch your default driver?

For most coding-agent work, yes — but measure first, and mind the calendar.

Switch now if you are running Opus 4.8 on tasks that do not strictly need it. The capability floor rose; a real share of Opus traffic will run acceptably on Sonnet 5 at a meaningful discount, and the intro pricing through August 31 makes the test cheap. This is the clearest win in the release.

Switch now, with care if you are upgrading from Sonnet 4.6. You get better agentic coding, interleaved reasoning by default, lower hallucination and sycophancy. But grep for explicit thinking budgets first — they will 400 — and run your token measurement before September 1, when the tokenizer tax stops being masked by the intro discount.

Wait, or stay put if you are on Priority Tier with Sonnet 4.6 — it is not available on Sonnet 5 at launch, so the choice is staying on 4.6 or jumping to Opus 4.8. And if your cost model is tight and unmeasured, the matching list price is a trap; spend the two weeks measuring before you commit budget to it.

There is now a fourth move the launch-day framing didn’t include: reach past Sonnet entirely. With Fable 5 back in Claude Code as of July 1, the hardest problems that Sonnet 5 and even Opus 4.8 stall on have a home again, at a price that keeps it off your bulk traffic and with a security governor that bounces exploit work down to Opus. For volume, Sonnet 5 is the driver. For the few tasks that justify the top of the lineup, the top of the lineup is reachable again.

Anthropic built Sonnet 5 to be the default driver model for the agent era, and on the architecture, it earns the slot. Adaptive thinking with interleaved reasoning is the correct shape for a coding agent, and shipping it as the default behavior removes work that harness authors used to do by hand. The asterisk is the invoice. The list price tells you one thing; the tokenizer tells you another. For the next two months, the introductory rate hides the difference. After that, the teams that measured will know what they are paying for, and the teams that read the sticker will get a surprise in their September bill.

Bob Matsuoka is CTO of Duetto and also writes about AI business at AI Power Ranking.

Related reading:

• I Tracked Every Token — What a $1.07 bug fix reveals about AI coding economics, and why per-token pricing hides the real bill.

• The Agent Unlock: Why Opus 4.5 Changed How I Work — When a top-tier model crossed the line into autonomous coding that holds up under real work.

• Breaking: Opus 4.6 and Agent Teams — A model release read for what it changes in day-to-day agent workflows, not just the benchmark chart.

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

[1]: Complicating “today”: with Fable 5 back as a Claude Code option (and, for some of us, the new default for hard sessions), the switch question isn’t only Sonnet-5-or-not. It’s which tier each task belongs to, Fable included. See “The ceiling moved: Fable 5 is back” below.

In 2019, comedian Gary Gulman released an HBO special called The Great Depresh. The title puns on the 1930s collapse, but the special is about Gulman’s own clinical depression — the hospitalization, the electroconvulsive therapy, the long climb back. He calls it “depresh” throughout. The diminutive is the point: “depression” was too heavy to say out loud for years, so he found a smaller word that let him talk about it at all. The special ends on a line he delivers like a weather report: “My depresh is in remish.”

Something adjacent is moving through the developer community, and most people don’t have a name for it yet. It isn’t burnout, and it isn’t fear of replacement, though that gets blamed for it. It’s a specific malaise in senior engineers who, by outward measures, are doing fine — shipping more, faster, with tools that work. They feel worse anyway. Call it the Coding Depresh.

The Depresh is real, it’s documented, and there’s a path out that doesn’t require pretending the loss isn’t a loss. I write from an unusual position. I spent eight years as CTO of TripAdvisor managing more than 560 engineers, mostly not writing code, and I missed it. I came back to hands-on coding in March 2025, entirely through AI tools — my re-entry and AI-assisted development are inseparable. I never had to grieve a pre-AI coding identity, because I didn’t have one to defend. That gives me a strange vantage point on the engineers who did.

TL;DR

• The Coding Depresh is identity disconfirmation, not job-loss fear: when the thing you built your professional self around gets automated, “who am I as a developer?” stops having an easy answer.

• It’s documented. A 15-year veteran describes shipping an AI-built pipeline and feeling grief “for an identity.” Stack Overflow’s 2025 survey shows distrust in AI accuracy (46%) outrunning trust (33%) for the first time, even as usage climbed to 84%.

• The most-cited evidence that AI slows experienced developers — METR’s 2025 trial showing them ~19% slower — has been retired by the same researchers, whose redesigned data now points the other way. Even the skeptics’ own number moved.

• A different camp — the Energesh — reports feeling more capable, not less. The fault line isn’t seniority or skill. It’s your answer to “who are you as a developer.”

• The most useful question I’ve seen comes from an Anthropic engineer: “I thought that I really enjoyed writing code, and I think instead I actually just enjoy what I get out of writing code.” Where you land tells you where you live.

• For engineers in the Depresh, and the CTOs managing them: the craft instinct that makes the tools feel wrong is the most valuable thing you own. It doesn’t have to die for you to adopt the tools.

The Depresh is real, and it’s documented

Start with the most precise account I’ve found. George Violaris, a developer with fifteen years’ experience, wrote in March 2026 about shipping a data pipeline with AI assistance in an afternoon — work that would have taken him three days by hand. He expected pride. He got this instead:

“That evening, I felt something I can only describe as grief. Not for a person. For an identity.”

He goes on: “Three years ago, writing code wasn’t just what I did. It was who I was.” Then: “The ego death was real. ‘I am a person who writes excellent code’ had to die.”

This is not a man worried about his next paycheck. He shipped the thing. The pipeline is in production. What broke wasn’t his employment; it was the relationship between his sense of self and the work. Most of the public conversation treats developer anxiety as a labor-market story — real, severe for junior developers, but separate, and not the one I’m writing about.

The Depresh I’m describing hits a different person: the experienced developer whose answer to “who are you?” was “I am the person who writes excellent code.” For that person, the tools don’t threaten the job first. They threaten the identity first.

The numbers underneath the mood are strange. Stack Overflow’s 2025 Developer Survey, the largest census of working developers we have, shows two lines crossing. Favorable sentiment toward AI tools fell from 77% in 2023 to 60% in 2025, while usage or intent to use rose from 70% to 84%. People are adopting tools they feel worse about. Trust in AI accuracy dropped to 33% against 46% who distrust it — the first year distrust outran trust — and the top frustration, cited by 66%, was “AI solutions that are almost right, but not quite.” The most experienced developers are the most skeptical: only 2.5% “highly trust” the output, and use it all day anyway. Working with something you don’t trust, that demands verification you can’t delegate — that’s the exhaustion of vigilance without resolution.

Then there’s the METR study — less for the number most people quoted than for what happened to it. In 2025 the nonprofit METR ran a controlled trial with experienced open-source developers and found them 19% slower with AI assistants while they believed they’d been roughly 20% faster. That became the most-cited evidence that the tools don’t pay off. Then in February 2026 the same team retired it, writing that the original no longer reflects “the current impact of AI models on open-source developer productivity”; their redesigned measurement runs the other way, an estimated 18% speedup for the same returning developers. The intervals are wide, so the magnitude is soft — but the direction reversed, and why it had to be rebuilt is the tell: 30 to 50% of developers refused to submit tasks they expected AI to speed up, and many refused to work without AI at all. The measurement broke down because people won’t give up the tools. I read the original not as “the tools are bad” but as disorientation: when your instincts and the stopwatch disagree, then a year later the stopwatch reverses, your competence comes unmoored.

There is another camp, and it’s also documented

Here is what keeps the Depresh from being the whole story. A different group of engineers is having close to the opposite experience, and they’re not naive optimists or vendors.

Andrej Karpathy named the moment in February 2025: “a new kind of coding I call vibe coding, where you fully give in to the vibes, embrace exponentials, and forget that the code even exists.” He was describing a feeling, not a methodology. Simon Willison later coined “vibe engineering” for the responsible professional version — engineers using these tools deliberately to accelerate real work.

The output side is loud. Pieter Levels built a browser flight simulator with no game-development background and reached around $1M in annualized revenue in 17 days. Boris Cherny, who leads Claude Code at Anthropic, runs five parallel instances and ships 20 to 30 pull requests a day: “once there is a good plan, it will one-shot the implementation almost every time.”

This is where my own story sits. When I came back, the mechanics of typing code line by line weren’t part of my working identity — I’d been away from the keyboard for years. What I found waiting was the part I’d missed: building things, deciding what should exist, watching it take shape, fixing what’s wrong with it. The joy of building and the mechanics of writing code were never the same thing.

I can be concrete, because I’ve done it in the open. Since March 2025 I’ve shipped real, full-cycle software through these tools — not snippets, complete projects. The flagship is claude-mpm, a multi-agent orchestration platform for Claude with a real user base. Most of 2025 was Python; then in 2026 I taught myself Rust to build the trusty-* ecosystem — code search, memory, PR review, orchestration — something I wouldn’t have attempted by hand while running an org. Architecture, review, releases, the full loop, at a volume I couldn’t reach typing every line. Most of it is public on GitHub under bobmatnyc, so the claim is checkable. I’m not theorizing about the Energesh. I’m living in it.

What actually separates the two camps

Not seniority. Not raw skill. Not the domain you work in, though each shades the experience. The fault line is the answer you’d have given, before any of this started, to a single question: who are you as a developer? There’s an older version of the same split.

In the early nineteenth century the canuts were the silk weavers of Lyon, clustered in the Croix-Rousse district, working intricate patterns by hand. It was master-craft work, identity-defining — the skill lived in the fingers. Then Joseph Marie Jacquard demonstrated a loom that wove those same complex patterns automatically, driven by punched cards, doing in a single pass what had taken the weaver and an assistant by hand. The canut whose sense of self lived in the act of weaving, in the tight and skilled handwork itself, was displaced at the identity layer. The canut whose relationship was with the silk itself, the finished cloth, still had somewhere to stand. The fabric still got made. More of it, in fact. If your identity comes from the tight weaving rather than from seeing the cloth made, you are a Canut.

Coding has the same shape. If your answer to “who are you” was inseparable from “I am the person who writes the code” — the line-by-line authorship, the mechanical understanding of every decision, the craft of it — then AI arrives at the identity layer, not the job layer. Violaris’s grief is the correct, proportional response; it scales with how much of himself he invested. If your answer was closer to “I am the person who builds the thing that exists at the end,” the same tools feel like a gain. You were always pointed at the cloth, not the weave. The result just got cheaper to make.

There’s a quiet irony in the comparison. Jacquard’s punched cards are a direct ancestor of the computer — they ran on into Babbage’s Analytical Engine and later Hollerith’s tabulators and the IBM punch card. The mechanism that displaced the weaver became the machine the coder works on, now automating a layer of the coder’s craft too. Same lineage, one more turn.

The cleanest articulation comes from an Anthropic engineer, quoted in Anthropic’s own internal research on AI-assisted work:

“I thought that I really enjoyed writing code, and I think instead I actually just enjoy what I get out of writing code.”

That is the skeleton key. Most developers in the Depresh believe they love writing code, and they’re not wrong, exactly — but the two halves of that sentence always came bundled. You couldn’t get the output without the authorship. AI unbundles them. Do you love the writing, or what the writing gets you? Where you land tells you which camp you’re in, and it’s not always where you assumed.

Kent Beck’s vocabulary dissolves a false worry. He distinguishes augmented coding from vibe coding. In vibe coding you don’t care about the code, only the behavior. In augmented coding you care deeply about the code, its complexity, its tests — “it’s just that I don’t type much of that code.” The Energesh isn’t about lowering your standards: the taste and architectural judgment that took decades to build are still doing the real work.

David Heinemeier Hansson gives the sharpest version of the craft objection. DHH spent most of 2025 resisting agent-first coding, and his reason was a craft reason. On Lex Fridman’s podcast he said the joy is to type the code himself; he keeps AI in a separate window because letting it drive made him “feel competence draining out of [his] fingers.” He has since switched to an agent-first workflow, on the logic that the tools finally met his standard, not that his standard moved. His values stayed put; what changed was his assessment of whether the tools honored them. That’s the template: the resister and the convert are the same man with the same principles.

Which is why the craft instinct deserves defending, not demolishing. The tools feel wrong partly because you’re judging them against that instinct, not just against output. Its firing isn’t a malfunction — it’s the sharpest instrument you own, calibrated over years, telling you when something is almost right but not quite, the same complaint 66% named in the Stack Overflow data. The mistake is concluding it has to be put down for the tools to be picked up. It doesn’t.

The path from Depresh to Energesh

This part is for the reader sitting in the Depresh right now. It isn’t a pep talk, and I’m not going to tell you the feeling is irrational, because it isn’t.

Grieve first. The ego death Violaris describes is real, and you’re allowed to mourn it. The craft identity you spent years building had value — it shipped real systems and earned you a career. Skipping the grief doesn’t work. The developers I’ve watched leap straight to enthusiasm tend to adopt the tools resentfully and use them badly, half-hoping they’ll fail. Let the loss be a loss before you look for what’s on the other side.

Then ask the real question. The Anthropic engineer’s version: do I love writing code, or what I get from writing code? You’ve probably never had to answer it, because the two were never separable before. They are now. The answer isn’t a verdict on your worth as an engineer. It’s a map of where you actually live, and either answer is fine — but you can’t find the path until you know which is true for you.

Learn harness engineering. The on-ramp for a senior engineer goes up a level, not down. Vibe coding pulls you toward the model’s altitude; this pulls you above it. The harness is the tooling layer around the model — orchestration, context management, verification scaffolding, the agent configuration that decides what the model sees and whether you can trust what comes back. Engineering that layer rewards the judgment you spent years building: systems thinking, architecture, the discipline of making an unreliable component dependable. My own claude-mpm and trusty-* tools are harness engineering and nothing else. So skip the vibe-coded toy app. Pick something real, build the harness that drives it, and watch how it feels — whether you feel more like yourself, or less.

Reframe from author of code to author of outcomes. Your craft instinct — what good architecture looks like, what breaks at scale, when a design is quietly wrong — isn’t going away, and the model doesn’t have it. The model is fast, capable, judgment-free. You are slow by comparison and you have taste. The job becomes directing the thing and knowing whether what came back is right. Not a demotion from engineer to button-pusher. It’s the part of the work that was always hardest to teach, now occupying most of your day.

A word for the CTOs reading this: you have a version of this problem you may not have named. The Depresh shows up in your metrics before your one-on-ones: review cycles stretching out, code-quality variance widening, your most experienced engineers going quiet in design reviews. The path for them is the path for an individual — don’t push adoption before you’ve made room for the grief. And understand who you’re dealing with: the engineers who feel the Depresh most sharply are frequently your best ones, who invested most in the craft. Their standards are the feature, not the bug. Burn that instinct down to force faster adoption and you’ll get the adoption and lose the standards.

Where this leaves you

Gulman didn’t end his special by announcing he was cured. He said his depresh was in remish — smaller word, smaller claim, a thing managed rather than defeated. That’s about the right register for where the industry is.

The craft you built was real, and so is the grief if you’re feeling it. But the thing you actually loved — if it turns out to be the building, the deciding, the watching something work that didn’t exist this morning — that part never depended on you typing every character yourself. The canut whose love was the cloth still had cloth to make. You can find out which one you are. Most developers go a whole career without having to ask. You get to.

Your depresh can be in remish. The energesh is on the other side of one unsparing question, and you already know how to ask those. You’ve been debugging your own assumptions for years.

Bob Matsuoka is CTO of Duetto and writes about AI business trends at AI Power Ranking.

Related reading:

• The Tide Has Turned: Senior Developers Are Finally Adopting AI Tools — Why the holdouts changed their minds, and what shifted to make it happen

• The Era of the Leader/Practitioner — How AI tools made the hybrid leader-who-builds role viable again

• We’ve Turned a Corner — On the shift from skepticism to working practice

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

This is Part 3 of a three-part read on the Fire Horse year in AI coding, the payoff the whole series was building toward. Parts 1 and 2 walked twelve coding models across the twelve animals of the zodiac and two hemispheres — the Chinese front-runners and the $60 billion SpaceX-buys-Cursor deal in Part 1, the Western incumbents and the collapse of the Western open flank in Part 2. Now the question all of it was built around: who controls the inference, what they do with your prompts, and whether the answer can change without your consent.

TL;DR

• The community read is convergence with a ceiling — and the real endorsement is behavioral: Chinese models reached ~61% of top-10 token consumption on OpenRouter in one February week, with DeepSeek alone near 17.6%.

• For enterprise, the practical data-safety ranking is: self-host > AWS Bedrock / Azure AI Foundry > US inference hosts > OpenRouter ZDR > direct Chinese API.

• China’s National Intelligence Law means a written no-train promise from a China-domiciled vendor is not a clean answer for regulated work, whatever the privacy policy says.

• CrowdStrike measured DeepSeek-R1 producing insecure code at a ~50% higher rate on politically sensitive prompts — and the bias persisted when running the open weights locally. Self-hosting is not the escape hatch you assume.

• Provenance was a proxy. The DeepSeek risk and the SpaceX-Cursor risk are the same underlying exposure, and the Cursor deal makes the flag-over-the-lab heuristic useless.

What developers are actually saying

Cut through the benchmarks and the community lands on one sentence: convergence with a ceiling.

The gap has collapsed for the bulk of coding work. For autocomplete, summarization, refactoring known code, and the high-volume long tail, Chinese open-weight models are competitive and dramatically cheaper. Where they still lose is the hardest agentic, multi-file, novel-algorithm work — the jobs where finishing is the whole point. Every hands-on test in my research told the same story from a different angle, and the animal sections in Parts 1 and 2 carry the specifics: Kimi’s “could not put it all together,” Flash’s confusion off-script, the Dog that finished when the Monkey could not.

Two things stand out about the endorsements. No prominent named engineer — no Karpathy, no Willison — has publicly endorsed a Chinese model for production coding; the praise is essentially all pseudonymous Hacker News handles. But the real endorsement is behavioral. On OpenRouter, Chinese models reached roughly 61% of token consumption among the top-10 models in one week this past February (around 45–51% of platform-wide tokens by April), with DeepSeek alone at about 17.6% top-10 share — reportedly exceeding Google and OpenAI combined in one snapshot. One caveat on the volume story: the Chinese models dominate tokens, not revenue. Anthropic accounts for roughly 12% of OpenRouter tokens but about 46% of its revenue, which is the cheap-volume-versus-paid-difficulty split showing up in the billing. Developers are voting with their tokens even when they will not put their name on a blog post.

The behavior that dominates is routing. Use the cheap or local model for the long tail; escalate to Claude or GPT for high-stakes correctness and novel work. That is not a compromise people settled for. It is the rational architecture, and it is what most serious teams already run.

The enterprise question

Here the conversation stops being about capability and starts being about who can read your prompts. The three Chinese vendors are not equivalent, and the differences are material.

DeepSeek is the hard case. Its terms of service permit training on user-submitted data by default. There is no zero-data-retention option. Data is processed in China. No SOC 2, no HIPAA BAA. Its own privacy policy describes collecting prompts, chat history, uploaded files, voice, device and OS details, IP, device identifiers, crash logs, and — the line that stops procurement officers cold — “keystroke patterns or rhythms,” stored on “secure servers located in the People’s Republic of China,” retained “as long as needed,” which analysts read as indefinite.

Moonshot (Kimi) is better but not clean. Its policy permits using API content to “develop and improve the services” — no ZDR — though it hosts in Singapore. Singapore residency is a real improvement over data-in-China, but Moonshot’s core operations are Beijing-based, which matters for the legal point below.

Alibaba (Qwen) has the strongest commitment of the three. Alibaba Cloud Model Studio states explicitly that it “will never use your data for model training,” offers multiple regions including US and EU, and Alibaba carries institutional accountability the others lack — Hong Kong-listed, with an international regulatory track record.

The caveat that overrides all three privacy policies is China’s National Intelligence Law, which requires Chinese companies to “support, assist and cooperate” with state intelligence work regardless of what their terms say. A written no-train commitment from a China-domiciled vendor is subject to that law. For regulated industries, the only clean answers are self-hosting open weights or routing through Western-managed infrastructure where the inference never touches a Chinese-operated service.

Which gives a practical ranking. From safest to least:

1. Self-host the open weights. Bedrock, vLLM, Ollama, your own GPUs. The weights are open; the inference is yours. (One asterisk, in the security section below.)

2. AWS Bedrock or Azure AI Foundry. DeepSeek and Qwen run in the provider’s environment, data stays in your selected region, no interaction with Chinese-operated services. AWS contractually guarantees no training on your data; Azure runs Qwen in your tenancy.

3. US inference hosts. Together AI, Fireworks AI, DeepInfra run the open weights on US infrastructure.

4. OpenRouter with ZDR. OpenRouter supports zero-data-retention per-account, per-key, or per-request ("zdr": true), but does not enumerate which specific Chinese-model endpoints honor it, and the National Intelligence Law caveat still applies upstream.

5. Direct Chinese API. Convenient, cheapest to wire up, and the option you cannot defend in a regulated procurement review.

“It’s complicated” is a non-answer. The ranking is the answer.

A cautionary security finding

One result deserves its own section because it survives self-hosting, which most people assume is the escape hatch.

CrowdStrike tested DeepSeek-R1 and found it produced insecure code at a higher rate on politically sensitive prompts. The baseline vulnerable-code rate was about 19%; it rose to roughly 27% when prompts contained CCP-sensitive terms — a relative increase of about 50% off that baseline, not a 50-point jump. CrowdStrike frames this as emergent misalignment, not a deliberate backdoor, so read it as a behavioral artifact rather than intent. The operational detail: the bias persisted when running the open weights locally. Self-hosting does not fix alignment baked into the weights. You can air-gap the model and the bias rides along.

Booz Allen found a related pattern: three of four Chinese models produced more security flaws when the user was described as US government, with Qwen3-Coder adding roughly 130% more vulnerabilities under a government persona. The authors stop short of alleging deliberate backdoors, and you should too — the mechanism is unproven. But the measured effect is real.

The counterpoint keeps this from being a blanket indictment. In one test, Kimi K2.5 posted the lowest aggregate vulnerability score of the group, below the US comparison model. The Monkey’s brilliance shows here too. The concern is not uniform. It is model-specific, prompt-specific, and measurable — which means it is something you test for, not something you assume.

For completeness on the public record: South Korea’s PIPC found in April 2025 that DeepSeek transferred user prompts and device data to a ByteDance-affiliated cloud without consent. Wiz found a publicly exposed, unauthenticated DeepSeek database in January 2025 leaking over a million log lines including chat history and API keys (secured quickly after disclosure). Government-device bans exist in Italy, Australia, Taiwan, South Korea, the Netherlands, the Czech Republic, Germany, India, and roughly 17 US states. But note what does not exist: a nationwide US API ban. “DeepSeek is banned in the US” overstates the situation. The Congressional pressure is real — the House Select Committee sent formal letters to Airbnb and Cursor in April 2025 over Chinese-model data concerns — but it is pressure, not prohibition.

The flag was never the variable

Now back to the governance question, with the whole zodiac in hand.

Every concern in the enterprise section comes down to control of the model layer. Who controls inference. What they do with your prompts. Whether a no-train promise is durable. Whether the governance can change without your consent. None of it is intrinsically about China — China is the jurisdiction where the control question has the sharpest legal teeth, because of the National Intelligence Law.

The Cursor acquisition raises the same question from the other direction. Read the enterprise objection to DeepSeek and the enterprise objection to a SpaceX-owned Cursor side by side and they rhyme. Both are: a single owner now controls the layer that sees all your code, the no-train and retention guarantees can be revised by that owner, and you have limited visibility into what happens upstream. The DeepSeek version has a foreign-intelligence-law wrapper. The Cursor version has a change-of-control wrapper. The underlying exposure — your IP flowing through infrastructure you do not control, governed by terms the owner can change — is the same exposure. Beijing on one side, Boca Chica on the other. That is where Sanchit Vir Gogia’s line from Part 1, Cursor sitting “close to the intellectual-property bloodstream,” stops being an analyst soundbite and becomes the whole argument.

That symmetry is what the zodiac quietly demonstrated. A Chinese frame walked through twelve animals and landed on Anthropic, OpenAI, Google, Meta, and Mistral as readily as on Moonshot, DeepSeek, Alibaba, and Zhipu. The frame fit the whole field because the flag over the lab was never the real variable. Provenance was a proxy for the question that matters — who controls the inference and what they are permitted to do with it — and the Cursor deal makes that proxy useless. You can no longer reason about model risk by asking which flag flies over the lab.

Practical routing guide

Strip out the geopolitics and a workable default emerges. This is roughly how I would set up a team today, animals included where they help. (The capability and access details behind each label live in Parts 1 and 2.)

The long tail — autocomplete, summarization, classification, tagging, structured extraction, refactoring code you already understand. This is 80–90% of volume, and it belongs to the Rabbit and the Rat. Use the cheapest capable model. Gemini 3 Flash if you are already in Google’s ecosystem. A self-hosted Qwen or Kimi if cost and data residency both matter. Gemma locally for pure structured-text utility, Gemma 3n on the edge, Codestral for fill-in-the-middle tab completion. The quality difference against a frontier model on this work is hard to detect, and the cost difference is 5–20x.

Routine multi-file work in a known codebase. Composer 2.5 as Cursor’s default — the Snake — or DeepSeek V4 / Kimi K2.6 through a vetted inference path. Cheap, capable, fine when the task stays on the rails.

High-stakes correctness, novel algorithms, off-script agentic work, deep architecture. This is the Dog and the Tiger. Claude Opus or GPT-5.5. This is where “Kimi just could not put it all together” and “Flash gets confused when a task goes off-script” stop being quotes and start being your Tuesday. Pay for the model that finishes.

Regulated or IP-sensitive work. Ignore the leaderboard and start from the data-safety ranking above. Self-hosted weights or Bedrock/Azure first; provenance second. If EU residency is the binding constraint, the Rat’s sovereignty pitch (Mistral on-prem, EU-only) is the one place provenance actually buys you something — though weigh it against a free self-hosted Qwen. And run the CrowdStrike test yourself — generate security-relevant code with and without sensitive context and diff the output before you trust any model in this tier, Chinese or otherwise.

The meta-point: the right answer is almost never one model. It is a routing policy. The teams getting real value here are not picking a winner. They are matching each class of task to the cheapest model that clears the bar for that task, and escalating only when the work demands it.

Where this leaves us

A Fire Horse year is supposed to be bold and unruly, and the field obliged. The Chinese models have largely arrived in the high-volume, low-stakes parts of your workflow — cheaper, open, and good enough that you would struggle to tell the difference. On the hardest work, the Western frontier still holds a real lead, and the cost of getting that work wrong is exactly where the price premium earns out. The clearest casualty of the year is the Western open-weight story: Meta and Mistral built the movement and got passed on code, and the open coding lead now sits with the Chinese labs.

But the durable lesson of this year is not about China. It is that the model layer inside your IDE is a governance surface, and ownership of that surface is now in motion — a $60 billion acquisition on one side, a foreign-intelligence statute on the other, and the same question underneath both. Where does my code go, and who can read it, and can the answer change without my consent? The zodiac was a device, but it made the point cleanly enough: twelve animals, two hemispheres, one set of questions. Provenance was the comfortable proxy. After this year, the proxy is gone. You have to ask the real question now, and you have to ask it of everyone — including the editor you have trusted by default.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

This is Part 2 of a three-part read on the Fire Horse year in AI coding. Part 1 set up the conceit — twelve coding models mapped to the twelve animals of the Chinese zodiac, in a year (丙午, the once-in-sixty Fire Horse) that earned its reputation for upheaval — and walked through the Chinese front-runners and the $60 billion SpaceX-buys-Cursor deal that reframed the whole field. (See Part 1 for the benchmark caveats; every version number and vendor-versus-independent distinction below assumes that three-question rule.)

This part is the Western field, and it carries its own argument. The Western frontier still leads the hardest work, the off-script agentic jobs where finishing is the whole point. But the Western open-weight story collapsed. Meta and Mistral built the open movement, and both got outrun on code by the Chinese open models. The open coding lead crossed an ocean, and that is the thread this part follows to its end.

TL;DR

• The Western frontier still leads the hardest agentic work, and on that tier the price premium earns out — the Sharda “$200 beat the $30” verdict lands here.

• Gemini 3 Flash dethroned its own bigger sibling on coding (78% vs 76.2% SWE-bench Verified). The “cheap weak Flash” framing is obsolete; watch the citation error that attributes the 78% to the old 2.5 Flash.

• Gemma 4 jumped roughly 3x on coding in one generation (LiveCodeBench v6 80.0% vs Gemma 3 27B’s 29.1%), and Gemma 3n runs multimodal in 2–3GB on the edge.

• Llama stalled — stale lineup, a closed pivot (Muse Spark), 15.6% Aider against ~5x-higher Chinese open models, and Yann LeCun saying the Llama 4 results “were fudged a little bit.”

• Mistral sells sovereignty more than raw capability now, and the EU-data edge is narrowing. Set Llama and Mistral side by side and the open coding lead has crossed an ocean.

The animals that hold the line

The order here runs roughly down a visibility-and-strength gradient: the frontier reasoners first, then the local family, then the two foundational Western open-weight players that built the movement and got passed.

Dog — Claude (Anthropic)

The dog is loyal, faithful, and protective, the one that finishes the job and guards the gate. Claude Opus is the model developers reach for when correctness matters and the one that holds the line on guardrails. Repeatedly in the research, it is also the one that finished.

The capability story is about consistency under pressure. DataScienceDojo ran Kimi K2.6 against Claude Sonnet 4.6 and found Kimi capable but Claude more consistent — Claude added a DELETE endpoint nobody asked for, flagged a Redis warning, and applied type-level validation unprompted. Composio’s harder test landed on the line Part 1 first quoted from the Monkey’s side: “Opus was expensive, but it finished. Kimi just could not put it all together once the task got real.”

And the GLM thread from Part 1 closes here, on the economics. Ashish Sharda’s much-quoted “I Tested GLM-4.6 for 2 Weeks and Went Back to Claude” landed on the argument against pure cost optimization: “The $200/month AI model beat the $30/month alternative. Sometimes expensive is worth it.” That was an older GLM, and the framing has aged — GLM-5.2 now posts the top open-weight index score in the field and sits #2 on Code Arena, so the cheaper model is no longer a lightweight you outgrow. The point that survives is narrower and still holds: on the hardest agentic work, Opus is the one that finishes, and the developers pairing GLM for volume with Claude for the hard problems are routing to exactly that tier. The dog is expensive to keep. It also guards the thing you cannot afford to lose.

Tiger — GPT (OpenAI)

The tiger is bold, fierce, and competitive, a predator that leads from the front. GPT-5.5 leads the field on terminal work, topping Terminal-Bench 2.0 by 13 points.

The tiger does not appear much in the open-weight cost debate because it is not playing that game. Its claim is one hard surface: the command line, where an agent has to chain real operations against a real environment and not lose the thread. Cursor’s routing guidance sends shell-heavy terminal work to GPT-5.5 for exactly this reason. On its territory, it leads.

Rooster — Gemini (Google)

The rooster is showy, observant, punctual, and loud at dawn. The whole Gemini family struts in fast and crowing, and the loudest crow is internal: the lean Gemini 3 Flash dethroned its own bigger sibling, Gemini 3 Pro, on coding. The framing most people carry for Flash is obsolete. “The cheap, weak sibling” stopped being true at the end of last year.

Gemini 3 Flash shipped December 17, 2025, and beat Gemini 3 Pro on coding: 78% SWE-bench Verified for Flash against 76.2% for Pro. The smaller, cheaper model won. That 78% figure is the source of a common citation error — people attribute it to Gemini 2.5 Flash, which is the previous generation. The number belongs to the 3 Flash family. If you see “Flash beats Pro at 78%,” confirm which Flash before you repeat it.

The current Flash earns the upset. Gemini 3 Flash runs roughly 4x cheaper than 3 Pro (around $0.50/M input), about 3x faster (218 tokens/sec), scores 90.4% on GPQA Diamond, and Cursor, Cline, JetBrains AI, and Gemini CLI adopted it immediately. The latest, Gemini 3.5 Flash, beats Gemini 3.1 Pro on Terminal-Bench 2.1 at 76.2% and is described as Google’s strongest agentic model.

None of which retires Pro. The developer consensus is hybrid routing, not replacement. Flash handles 80–90% of the work — summarization, classification, tagging, structured pipelines, autocomplete — indistinguishably from Pro. Pro still earns its keep on architectural reasoning, complex multi-file refactors, novel algorithms, and the off-script agentic tasks where, as one developer put it, “Flash gets confused when a task goes off-script.” Cursor’s routing still sends deep-architecture and long-context work to the heavyweight reasoning tier, and Pro is the Gemini family’s answer there. The rooster reasons deep in one body and runs fast in the other, and that “off-script” line could be the epigraph for this entire field.

Rabbit — Gemma (Google’s open-weight family)

The rabbit is gentle, quiet, and home-bound, and Gemma is Google’s open-weight family — the calm local helpers that never pretended to be frontier coders, then quietly got 3x better at code in one generation. The Gemma arc runs across three things: the workhorse, the leap, and the tiny one that runs where nothing else will.

Start with the leap, because it dates everyone’s article. Gemma 3 is previous-generation. Gemma 4 landed April 2, 2026 (up to 31B dense plus a 26B MoE) under a full Apache 2.0 license, and the jump is large. On LiveCodeBench v6, Gemma 4 31B reportedly scores 80.0% against Gemma 3 27B’s 29.1% — roughly 3x better at coding in a single generation. That number is vendor and secondary-source, not yet on an independent leaderboard, so hold it loosely; the generational jump is the durable part.

Gemma 3 was the quiet local utility the family was known for, not a coder: agentic tool use on τ²-bench Retail at 6.6%, and Fixstars’ hands-on found hallucinations on technical detail and no capacity for complex agentic VSCode work. What Google marketed was the LMArena 1338 score, which beats GPT-4o and Claude 3.7 Sonnet — but that measures chat preference, not coding ability, and the two should not get conflated. None of which made Gemma 3 useless: it earned real adoption as a free local utility for JSON extraction, log parsing, and code explanation.

Then there is Gemma 3n, the tiny edge variant that fits where nothing else does. Clever architecture lets the full 5B and 8B models run in 2GB and 3GB of memory, multimodal across image, audio, video, and text. A privacy and offline play, not a coding rival. The rabbit wins by being where the others cannot go: edge devices, air-gapped machines, the laptop with no connection.

Access runs through Ollama (override the default 2048 num_ctx or context keeps falling out), llama.cpp, LM Studio, and Vertex AI. No native Cursor or Windsurf integration.

Pig — Llama (Meta)

The pig is the sign of abundance and generosity. That was Llama: Llama 2 and Llama 3 became the base layer for the open-weight movement, the foundation under thousands of derivatives, fine-tunes, and quantizations. Meta’s generosity built the ecosystem everyone else now competes in. The pig is also the sign of complacency, and the 2026 Llama story is a provider that got too well-fed to move while leaner animals ran past it on code.

The lineup went stale, then went closed. The Llama 4 models shipped in April 2025 and were never refreshed; the ~2T Behemoth was shelved as of May 2026; Muse Spark, released that same month, is Meta’s first closed-weight, API-only model, reportedly lagging on coding; and Llama 5 slipped to roughly 2027. Andrew Ng called the retreat from open weights “a significant loss for the developer community.” The biggest model in the family is stuck in the pen, and the family stopped being fully open.

The coding numbers are why none of that got forgiven. On Aider Polyglot, Llama 4 Maverick scores 15.6% — against Kimi K2 at 59.1%, Qwen3-235B at 59.6%, and DeepSeek-V3.2 Reasoner at 74.2%, roughly a 5x gap to the Chinese open models. And it doubles as the cleanest benchmark-trap example in the series, the callback to Part 1’s rule. For LMArena, Meta submitted a conversationality-tuned variant that ranked around #2; the actual public release ranked #32, and LMArena rebuked Meta for the swap. Yann LeCun, who left Meta in November, told the Financial Times in January 2026 that the Llama 4 results “were fudged a little bit.” One case study in why a vendor number means nothing until an independent leaderboard confirms it.

Llama still ships under the Llama Community License, not an OSI-approved one, and has been passed in both openness and downloads — DeepSeek (MIT) and Qwen (Apache 2.0) are more open, and Qwen overtook Llama as the most-downloaded open family on Hugging Face, with Chinese models holding four of the top five open-weight slots — GLM-5.2 now the #1 open-weight model on the Artificial Analysis index, ahead of Qwen, Kimi, and DeepSeek. The money did not buy the code scores: Meta runs the best-funded open lab there has ever been, and r/LocalLLaMA’s reaction to Llama 4’s coding was negative anyway. Access is everywhere, and no first-party coding default anywhere, because the scores do not earn one.

Rat — Mistral (France)

The rat is first in the zodiac, clever and nimble, the outsider that thrives in the cracks and punches above its weight. Mistral fits cleanly: the European challenger named for the cold, fast wind off the Alps, a fraction of OpenAI’s war chest, surviving against far larger labs by being efficient and by owning a niche nobody else can — sovereignty. It made its name with Mistral 7B in September 2023, which beat Llama 2 13B at half the parameter count. The rat against the pig, literally, on the first release.

The current flagship open model is Mistral Large 3 (December 2025), a 256K-context MoE under Apache 2.0 that Microsoft markets as “the strongest fully open model developed outside of China” — praise and an admission in one sentence.

On coding, keep two models straight, because the names invite a mistake. Codestral (codestral-2508) is a fill-in-the-middle specialist: its FIM pass@1 is around 95.3% (vendor-reported, class-leading), and it quietly wins the tab-completion slot in a lot of editors. Its Aider Polyglot score is only 11.1% (independent) — but that is a FIM model measured on an agentic task, the same “which benchmark, which task” trap the Llama section just walked through. The agentic model is Devstral, which scores 53.6–61.6% on SWE-bench Verified. Use Devstral, not Codestral, for SWE-bench comparisons — and ignore any source citing “Codestral 2 with Apache 2.0,” which does not exist.

The rat’s real moat is jurisdiction, not a leaderboard score. The pitch is sovereign AI: data never leaves the EU, GDPR and EU AI Act native. The French Ministry of Armed Forces signed a 2026–2030 framework; Macron told citizens to “download Le Chat rather than ChatGPT.”

Be fair about the limits. The EU AI Act edge is narrowing — Mistral, OpenAI, Anthropic, Google, and Microsoft all signed the EU AI Act Code of Practice in July 2025, while Meta declined and the Chinese firms did not, so the real divide is US/EU signatories versus China and Meta, not Mistral alone. And the skeptic’s line is hard to answer on capability: why pay Mistral on-prem when you could run Qwen for free?

The coding lead crossed an ocean

Set the Pig and the Rat side by side and the thesis arrives from a fresh direction. Llama and Mistral are the two foundational non-Chinese open-weight players, the labs that built the Western open movement, and both have been outrun on coding by the Chinese open models — the Dragon, Ox, Monkey, and Goat from Part 1. The Western open-weight story in 2026 is Meta retreating to a closed model and Mistral surviving on a sovereignty niche rather than on raw capability. The coding lead did not just shift between companies. It crossed an ocean.

That leaves the frontier still in Western hands for the hardest work, and the open flank fallen. Which sets up the question the whole series was built around. Twelve animals across two hemispheres, and underneath all of them, one question: who controls the inference, and can you trust it with your code? Part 3 is the governance reckoning.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

This is Part 1 of a three-part read on the Fire Horse year in AI coding, the one that sets up the frame and walks through the disruptors.

2026 is the Year of the Horse in the Chinese zodiac, and not the ordinary kind. It is a Fire Horse year — 丙午, Bing Wu — the pairing that comes around once every sixty years. The last one was 1966. Tradition gives the Fire Horse a reputation for boldness, intensity, and upheaval, the kind of year people are warned about rather than wished. As a frame for the state of AI coding tools, it is almost too neat. This is the most chaotic, multi-polar stretch the field has seen, and a large share of the disruption comes from Chinese labs. The animal fits twice.

I had a different article planned. A straight survey of Chinese coding models, Kimi and DeepSeek and Qwen, and whether you should trust them with your code. Then SpaceX agreed to acquire Cursor for $60 billion in stock. Anysphere, Cursor’s parent, would become a wholly owned SpaceX subsidiary; the deal was announced June 16 and is expected to close in Q3. SpaceX absorbed xAI back in February, which means the editor sitting inside 64% of Fortune 500 development workflows now answers, eventually, to the same org chart as Grok. The acquisition reframed the whole piece.

Because the question developers keep asking about DeepSeek and Kimi is “where does my code go, and who can read it?” That now applies to the most popular AI editor in the enterprise. The model layer inside your IDE has become a governance problem, and it does not much matter whether the party on the other side is in Beijing or in Boca Chica. The shape of the concern is the same. Only the jurisdiction’s law changes.

So I kept the Chinese-model framing and pushed it further. To organize the field, I mapped twelve models to the twelve animals of the zodiac, one animal per model family, which forced the labs that ship a half-dozen variants into a single sign with their internal rivalries inside the section. The mapping is most certainly a device (but a fun one -- don’t hate!), and it makes a quiet point on its way: the animals land on Chinese and Western labs alike, and a Chinese frame ends up describing the whole field. That is the through-line this series pays off in Part 3 — provenance was always a weak proxy for the question that matters, who controls the inference and what they are allowed to do with what passes through it.

This part covers the frame, the methodology behind every number below, and the front-runners pushing the field. Part 2 turns to the Western incumbents and the collapse of the open-weight flank. Part 3 is the governance reckoning the series builds toward.

TL;DR

• It is a Fire Horse year, the once-in-sixty-years sign of upheaval, and AI coding earned the label: a $60B acquisition, a collapsing capability gap, and a dozen credible model families from both hemispheres.

• Almost every benchmark number you have seen for a Chinese coding model is either self-reported or conflates versions. Kimi K2 alone has shipped six named releases in eleven months. Treat headline scores as marketing until an independent leaderboard confirms them.

• DeepSeek competes on price, not the hardest work: V4-Pro runs 34–86x cheaper than Opus 4.8, scores 80.6% on SWE-bench Verified, and falls behind where the work gets difficult.

• Qwen has the cleanest enterprise story of the Chinese front-runners — Bedrock, Azure, Apache 2.0 licensing, and it just passed Llama as the most-downloaded open family on Hugging Face.

• Cursor’s own default model, Composer 2.5, is built on Moonshot’s Kimi K2.5. The strongest model in the most-deployed enterprise editor is already a Chinese-origin model fine-tuned by an American company.

• The quiet spoiler is GLM-5.2 (Zhipu): the #1 open-weight model on the independent Artificial Analysis index (#4 across all models) and #2 on LMArena’s Code Arena, from a lab most Western developers were not tracking.

The benchmark trap

Start with some discomfort. Most of the numbers in this debate are unreliable, and not because anyone is lying outright. They are unreliable because of two structural problems: version conflation and contamination. This section is the methodological spine for the whole series, so it sits up front.

Take Kimi K2. Here is the release history: K2 in July 2025, K2-Instruct-0905 in September, K2 Thinking in November, K2.5 in January 2026, K2.6 in April, and K2.7 Code this month. Six checkpoints. When a blog post says “Kimi scores 65.8% on SWE-bench,” that was the original K2. K2.6 scores 80.2%. Those are different models with the same nickname, and people cite them interchangeably. The result is a comparison that means nothing.

Contamination is the second problem, and it is worse because it is invisible. SWE-bench Verified is a static dataset. Models trained after its release have, to varying and unmeasurable degrees, seen the answers. SWE-rebench, presented at NeurIPS 2025, rebuilt the evaluation on decontaminated tasks and watched scores fall across the board: GPT-4.1 dropped from 31.1% to 26.7%, LLaMA-3.3-70B from 18.1% to 11.2%. The Llama drop returns in Part 2, where Meta’s benchmark numbers get a chapter of their own. The headline numbers are inflated, and the inflation is not uniform.

So here is the rule I use, and the one I would suggest you adopt. When you see a coding score, ask three questions. Which exact version? Self-reported or independent? On a contaminated benchmark or a fresh one? The most trustworthy independent sources right now are the Aider Polyglot leaderboard, the Artificial Analysis Intelligence Index, and Scale AI’s SWE-bench Pro. Everything else is a starting point, not a conclusion. A zodiac of twelve animals is also twelve moving targets. Every name below has a version number, and sometimes a whole product line, hiding behind it.

The animals that lead

With that caveat doing its work, here is the field as of mid-June, one animal per family. The Horse and Dragon open the series — the acquisition that reframed it and the lab that moved the market — then the rest of the Chinese front-runners, before a closing reveal that ties the East to the West.

(Fire) Horse — Grok (via SpaceX)

The year’s own animal, and the only one that arrives by acquisition. The Fire Horse is fast, wild, and short on guardrails, which is a fair description of how Grok is charging into developer tooling — not by building a foothold but by buying one. SpaceX merged with xAI in February 2026. Grok itself has about 6% enterprise adoption and no developer-tools presence to speak of. Then, on June 16, SpaceX agreed to pay $60 billion in stock for Cursor — announced, not yet closed, with the deal expected to complete in Q3.

The strategic logic is plain: buy the distribution Grok could not earn. Cursor reportedly sits in 64% of the Fortune 500 and runs roughly $4 billion in annualized revenue, up from $2 billion in February. Grok had the model and none of the reach. The horse does not wait for permission.

No changes to model access have been announced, and no commitment to keep Cursor model-agnostic post-close has been made either. The analyst concerns are specific. Jason Andersen of Moor Insights: “xAI’s models and treatment of guardrails are very different than what Cursor has stood for,” and “Will Cursor be able to point at models other than Grok?” Justin Greis of Acceligence on the data posture: “For many enterprise customers, Cursor’s zero-data-retention policy was not simply a security feature” — it was foundational to procurement approval. Sanchit Vir Gogia of Greyhound put the structural point plainly: Cursor “sits inside the act of software creation, close to the intellectual-property bloodstream.”

Hold that last quote. It is the hinge of the governance argument, and Part 3 picks it up where this part leaves it.

Dragon — DeepSeek

The dragon is the only mythical creature in the zodiac, the sign of power and good fortune, and it goes to the only model that actually moved the market. DeepSeek is the lab that produced “the DeepSeek moment,” the release that made Western labs recalculate. It earns the dragon.

First, kill a rumor. There is no DeepSeek R2. The CEO was reportedly dissatisfied with it, Reuters noted no timeline, and every “R2 spec sheet” circulating online is speculation that contradicts the next one. If a comparison cites R2 numbers, the comparison is fiction.

The real timeline runs through the V-series, ending at V4, released April 24, 2026, as the current flagship — a trillion-plus-parameter MoE under an MIT license.

The much-cited V4 figure, SWE-bench Verified at 80.6%, is vendor-reported and does not reproduce independently — treat it as a DeepSeek claim, not a confirmed result. On the harder SWE-bench Pro, V4-Pro scores 55.4 against Claude Opus 4.7’s 64.3 — meaningfully behind where the work gets difficult. Same pattern runs through the whole series.

Price is where DeepSeek actually competes. One dollar buys about 1.15 million output tokens from V4-Pro versus roughly 40,000 from Opus 4.8 — call it 34x cheaper on input and 86x on output. For high-volume, low-stakes generation, that math is hard to ignore.

The hands-on reports split the way you would expect: capable, cheap, uneven. One developer found V4-Pro generated “subpar slightly buggy code” on sequential tasks; another said V4 Flash “often outdoes Kimi 2.6 on problems involving complex spatial reasoning.” One operational note that will bite people: the deepseek-chat and deepseek-reasoner endpoints retire July 24, 2026. If you have them wired into anything, migrate now.

Ox — Qwen (Alibaba)

The ox is the methodical workhorse, dependable and unglamorous, and Qwen is the Chinese player most developers have not actually evaluated. That is a mistake, because it has the cleanest enterprise story of the three: the broadest legitimate access, and the compliance paperwork to back it.

The current flagship is Qwen3-Coder-Next, released February 2026. Vendor-reported SWE-bench Verified lands at 70.6–71.3%, but the number to trust is independent: Scale AI scored last July’s Coder-480B at 38.70 (±3.55) on SWE-bench Pro, sobering against the vendor figures. The original Qwen3-235B marketing was inflated too — Artificial Analysis put it at 17, below average for its class — while the 2507 refresh is legitimately competitive at 25.

The hands-on reads are mixed. A Better Stack test pitted a smaller Qwen 3.5 variant against Claude Sonnet 4.5 and Claude won decisively; Qwen produced a blank project and could not self-diagnose. Artificial Analysis also flags Qwen3-235B-2507 as “very verbose,” burning 15M tokens to run their index versus an 8.1M average, which quietly inflates real-world cost even when the per-token price looks low.

Now the part that decides things for anyone in a regulated shop. Qwen has the broadest legitimate access of any Chinese model, and it just passed Llama to become the most-downloaded open family on Hugging Face. Apache 2.0 licensing on most releases (verify per version). It runs in your own environment on AWS Bedrock — where AWS guarantees customer data never trains Qwen — and on Azure AI Foundry inside your own tenancy, with local options through Ollama and vLLM. In Cursor it is not native; you need Cursor Pro+ and a custom-model config. That compliance posture is the Qwen story, and Part 3 ranks exactly where it lands on data safety.

Monkey — Kimi (Moonshot AI)

The monkey is clever, versatile, mischievous, and the sharpest tool-user in the zodiac — exactly the K2 family’s profile. It is the agentic one, brilliant and inconsistent in the same breath. “High ceiling, low floor” is a monkey’s whole personality.

The K2 family is the open-weight model that made Western labs pay attention. It is a trillion-parameter sparse mixture-of-experts design built for repo-scale work, and that is where developers praise it most — large context, real understanding of how files relate to each other. The current flagship is Kimi K2.7-Code (June 12, 2026): 1T total parameters, 32B active, 256K context, thinking mode forced on, and reportedly about 30% fewer reasoning tokens per task than K2.6 (independent). Pricing runs $0.95 per million input, $4.00 output.

The independent numbers are respectable. On Aider Polyglot, the original K2 scored 60.0%, ahead of Claude Sonnet 4 at 56.4% and behind Opus 4 at 70.7%. On the Artificial Analysis Intelligence Index v4.1, K2.7-Code sits at 42 (K2.6 was 43), strong among open models but now behind GLM-5.2’s 51 for the open-weight lead. Where Kimi keeps its edge is agentic stability, not the raw index — the troop holds together under load better than the score alone suggests.

The community read is more textured. K2.6’s launch pulled 592 points and 303 comments on Hacker News, with several people calling it “another DeepSeek moment.” The praise centers on context and cost — one developer reported dropping Claude Pro at $20/month for Kimi via Ollama and “haven’t hit usage limits a single time.” The skepticism centers on reliability: “high ceiling but low floor” came up more than once, meaning capable but inconsistent run to run.

That inconsistency shows under load. In Composio’s hard agentic test, the verdict was blunt: “Opus was expensive, but it finished. Kimi just could not put it all together once the task got real.” Kimi is excellent until the task stops being routine. Part 2 returns to that line from the Dog’s side.

Access is easy. OpenRouter lists it at roughly $0.60–0.74 per million input tokens and $2.50–3.50 output; the direct API runs through platform.moonshot.cn; it works in Cursor as a custom model via OpenRouter. And the detail to hold onto: Cursor’s own Composer is built on Kimi K2.5. The Snake section, below, is where that lands.

Two vendor claims to flag. Kimi’s Agent Swarm reportedly spawns up to 100 sub-agents across roughly 1,500 tool calls — the monkey troop in literal form, interesting and unverified. And treat the HumanEval 99.0 figure for K2.5 with suspicion: HumanEval is near saturation, so a score that high tells you the benchmark is exhausted, not that the model is.

Goat — GLM (Zhipu)

The goat is gentle, creative, and persistently underrated — mild on the surface, a pleasant surprise once you give it a chance. GLM is the quietly capable one. In one widely shared account it was “the first open model that was actually usable” for code, with a Hacker News developer (handle MintsJohn) rating it around ChatGPT-4.0 level, more than most expected from an open model at the time.

GLM also anchors the most-quoted cautionary tale of the season. Ashish Sharda’s “I Tested GLM-4.6 for 2 Weeks and Went Back to Claude” reads as a knock, but it is two weeks of a developer living inside GLM before deciding the premium model was worth the money. The goat held its own long enough to make the comparison a real one — a better daily driver than the headline suggests. Part 2 carries the verdict that comparison reached, on the Dog’s home turf.

Snake — Composer (Cursor’s own model)

The snake keeps hidden wisdom and waits in the grass until you step on it. Composer fits because it conceals what it is: Cursor’s in-house model wears an American badge over a Chinese-origin base — Moonshot’s Kimi K2.5 — and most people who run it daily have no idea.

The naming confuses people, so be precise. There is no third-party product called Composer. It is Cursor’s in-house coding model, a mixture-of-experts design RL-trained for software engineering, built on Moonshot’s open Kimi K2.5.

The version history runs Composer 1 (October 2025, shipped with Cursor 2.0), Composer 1.5 (February 2026), Composer 2 (March 2026, Kimi K2.5 base confirmed by Cursor), and Composer 2.5 (May 18, 2026, current). Composer 2.5 scores 79.8% on SWE-bench Multilingual and 63.2% on CursorBench v3.1, trained on roughly 25x more synthetic tasks than Composer 2. Cursor claims it “matches Opus 4.7 at about 1/10th the cost.”

Keep the layers straight: Composer is the model, Cursor Agent is the harness that runs it, and Composer 2.5 is the default model in that harness. Cursor’s own May routing guidance: deep architecture and long context to Claude Opus 4.7, shell-heavy terminal work to GPT-5.5, general default to Composer 2.5.

Sit with the implication. The default model in the most-deployed enterprise AI editor — the one SpaceX just paid $60 billion for — is a Chinese-origin open-weight model fine-tuned by an American company. The boundary a lot of enterprises thought they were enforcing (”no Chinese models in our stack”) was already crossed before the acquisition was announced. Provenance was the easy part to police. It turns out it was also the part nobody was actually policing.

That is where the East-rises story stops being about which labs ship from Beijing. The boundary enterprises thought they enforced was already inside their default editor. Part 2 turns to the Western field that is supposed to be the safe choice — the incumbents that still lead the hardest work, and the open-weight movement outrun on code.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

I watched a Claude Code session run a multi-workstream engineering job last week, and I caught myself doing something I do not usually do with these tools: I stopped intervening. The orchestrator — the PM layer, running claude-opus-4-8 — handled the kind of coordination work I would expect from a team lead who has been on the project a year, decomposing it, predicting conflicts, routing to the right specialist. Not a faster coder. A technical lead.

This time the session was recorded turn by turn, with per-token cost telemetry on every move. So these are not impressions. Below are the specific behaviors I saw, described as precisely as the instrumentation allows, each one carrying its timestamp, the orchestrator’s own words, and what it cost. The conclusion I’ll leave to you.

The job ran on trusty-tools, a Rust workspace, on 2026-06-10. One human (me), one PM coordinator, a fleet of subagents, and a memory layer plus code search and an adversarial PR reviewer wired in over MCP. It ran for 14 hours and 37 minutes. I authored nine content prompts in that window. The longest was 84 characters.

TL;DR

• A single Claude Code + claude-mpm session ran 14h 37m on a real Rust codebase and cost $485.96 at rack rate — $264.65 for the PM (claude-opus-4-8, 265 turns) and $221.30 for the subagents (sonnet at 4,626 turns, haiku at 605). It made 63 delegations. On a turn basis, 93% of the work was autonomous; my share was 7%.

• I’m on Anthropic’s Max plan — $200/month flat. This session ran inside that subscription. The rack-rate figure is the right way to price the economic value of the work; the actual cost to me was zero marginal dollars.

• That 7% was nine human-authored prompts in 14.5 hours. Several were one word — proceed, let's do the top candidates. The longest ran 84 characters. The orchestrator supplied the structure; I supplied the direction changes.

• The economics work because of cache. The sonnet agents read 3,531 cached tokens for every fresh input token (482M cache-read against 137K fresh). The PM ran at 587×, haiku at 34,868×. Most of the context window each turn is priced as cache, not fresh input.

• My idle time shows up on the bill. The most expensive PM turns are cache-cold context rebuilds after I left a long gap: $7.26 to resume after proceed (1.5 hours of silence), $5.82 to pick back up after a 4.5-hour wait.

• The orchestrator ran adversarial reviews on its own engineers’ work 13 times, unprompted — “Per my verification ownership I won’t take that at face value” — and bounced a starred approval back for more work. It absorbed two infrastructure failures without escalating them to me. None of these behaviors is impressive alone. A competent IC does them all before lunch. They appeared together, in one session, with a cost attached to each.

You can explore the full annotated session timeline — a turn-by-turn companion infographic showing the user-visible output alongside the underlying mechanism for each move.

A note on what this is and isn’t

I am not claiming the model understands anything, and I am not grading it on a benchmark. I am describing observed behavior from one session, the way you would describe an animal in the field: here is what it did, here is the order it did it in, here is what it cost, draw your own conclusions. Twelve behaviors stood out. They are below, roughly in the order they appeared.

The Stack in November

This session would not have run the same way seven months ago, not even close. Three layers changed in that window, and they changed together.

Start with the model. The session above ran on claude-opus-4-8, released May 28, 2026. Seven months earlier the comparable model was Opus 4.5, which shipped November 24, 2025. Better code is the obvious place to look for the shift, but the one that mattered here sits elsewhere: 4.8 is substantially less likely to let a flaw in its own work pass unremarked — it surfaces its own errors instead of quietly shipping them. That is the difference between a contractor who does their best and one who tells you when he thinks something is wrong. The 1M context window also moved from beta on 4.5 to generally available on 4.8, which matters across a 14-hour run, and in Claude Code 4.8 defaults to xhigh effort — the cost figures above reflect that setting.

Then the Claude Code harness around it. In November 2025, Claude Code was effectively a single-session tool; background agents existed, but worktree isolation did not. WorkTree support is the critical unlock. Each subagent in this session ran in its own isolated git worktree — its own branch, its own working tree, no chance of stepping on a parallel agent’s files mid-run. The merge-conflict prediction in behavior #6 is only tractable because the agents writing code cannot collide on disk while they work. Parallel execution without merge confusion needed a harness feature that wasn’t there last November. Added in the same window: 5-level nested subagent hierarchies, claude agents for session-wide visibility, and Dynamic Workflows for the PM coordination layer.

The third layer is the orchestration framework. claude-mpm went from v4.26 to v6.5.44 over those seven months — two major versions, roughly 450 releases. Two changes carry most of the behavioral weight. trusty-memory became mandatory session context, so the PM reads project history before it does anything else, which is behavior #1 above. And worktree-first became the framework default, which is what enables behaviors #3, #6, and the parallel fan-outs in #9. The bench is deeper too: 57 specialized agents now versus about 30 in November, so the routing in behavior #9 has more specialists to route to.

None of these three improved in isolation. The model’s self-auditing is only useful if the harness can surface those audits inside a multi-agent pipeline. The worktree isolation is only useful if the orchestration framework knows to reach for it by default. The memory priming is only useful if the model is good enough to act on what it finds. When all three shift in the same six-month window, the effect is not additive.

Strip away the version numbers and the feature lists, and the functional picture is narrower than it looks. The evolution since November clusters around three areas. Context: the 1M window is generally available now, and trusty-memory loads project history before the PM does anything. Recall: the PM enters each session knowing what happened before, with a deeper bench of specialists to route to. Error checking: the model flags its own flaws, and the orchestration layer runs adversarial review unprompted. None of the twelve behaviors below trace to the model writing better code. They trace to the system getting better at knowing what it knows, remembering what it has done, and catching what it gets wrong.

Session Vitals

Before the behaviors, the stat block. This is the anatomy laid flat.

Metric Value Total cost (rack rate) $485.96 PM cost (claude-opus-4-8, 265 turns) $264.65 Subagent cost (sonnet 4,626 turns + haiku 605 turns) $221.30 Delegations (agent calls) 63 Skill + MCP tool calls 18 Wall-clock duration 14h 37m Human share (turn basis) 7% Autonomous share (turn basis) 93% Human-authored content prompts 9 Longest human prompt 84 characters

The cache numbers sit underneath all of it. The sonnet agents pulled 482,137,941 cache-read tokens against 136,569 fresh input tokens. The PM pulled 70,287,449 cache-read against 119,632 fresh. Those two figures are the reason a 14-hour run costs what a junior contractor costs for an afternoon. I come back to them below.

1. Context First

The first action after my prompt was not a plan and not code. It was three memory and context calls in a row. The first one failed: memory_recall: missing 'palace' (no --palace default configured). The PM did not abort or report the error to me. It adjusted the call — added the palace parameter — retried, and got a stored finding from earlier work.

Then it read Cargo.toml and README.md before planning anything. In its own words at 21:48: “I’ll start by orienting myself. Let me query project memory and check what ‘core services’ and ‘console’ refer to, since neither maps obviously to a crate in this workspace.” One minute later it had resolved both terms and declared the scope — the equivalent of reading the ticket history before touching the keyboard.

2. Plan Before Acting

Rather than starting, it named the parallel streams and stated why they could run independently before dispatching anything: “These are independent, so I’ll dispatch them in parallel” (21:49). Structure stated before action taken.

The pattern held under pressure. When I said let's do the top candidates at 22:03, it refused to treat that as a green light: “’The top candidates’ spans everything from a critical bug cluster to a 5,400-line refactor to design/ADR work — very different sizes and risk. Before I fan out agents (and tokens), let me confirm how aggressively to go.”

3. Agents, Narrated

Two agents went out in a single PM turn that cost $0.34 — one to ticketing to pull the open GH issues for the core services, one to local-ops to investigate and deploy the console. Both came back inside two minutes. Each got a defined, bounded job, and the PM said what each was for before sending it. Later waves followed the same shape: at 23:27 three agents launched in parallel for Wave 1, redeploying trusty-search, splitting a 5,421-line file, and drafting an architecture decision record.

4. Verify Before Committing

When I said let's do the top candidates, the PM stopped and asked rather than fanning out. It called AskUserQuestion and waited 19 minutes for my answer before dispatching a single engineer.

The same instinct showed up against its own team’s reports. After the rust-engineer reported PR #1097 done with tests, clippy, and fmt all green, the PM declined to take it: “Per my verification ownership I won’t take that at face value — especially since #1088 and #1089 were collapsed into one commit, and #1089’s core complaint ... may be only partially addressed.” It ran an adversarial PR review and a CI check in parallel before accepting the work. The verification was budgeted into the run.

5. Name What You Don’t Know

On the architecture decisions baked into ADR-0010, the PM drew the line cleanly: “it’s an architecture decision so I’ll draft it for your sign-off rather than implement blind.” It surfaced four open questions, each labeled blocking or non-blocking, and answered none of them unilaterally.

It applied the same judgment to a failure. When a memory write was blocked because the trusty-memory daemon held the palace write lock, the PM diagnosed the cause and made a call: “Memory write is blocked ... not worth a detour; the work is durably captured in the merged squash commit, closed issues, and commit messages.” It marked the gap, decided closing it wasn’t worth the interruption, and said so.

6. Conflict Prediction

Before any Wave 1 agent was dispatched, the PM named the collision: “#1096 and #607 both edit .line-cap-allowlist.tsv (splitting an allowlisted file requires removing/lowering its entry), so running them in parallel guarantees a merge conflict on that file.” It named the file, the two items that would collide, the mechanism, and the mitigation — sequential waves — in the same breath. The same pattern recurred in Wave 2, where it held #607 until #993 landed for the identical reason.

7. Unprompted Checkpoint

Unprompted, at 02:02, the PM produced a formatted two-section status checkpoint: a “Shipped to main” table listing each merged PR with its commit hash and a verification note, and an “In flight / staged” list with the state of each item still moving. Self-organized visibility, not a dashboard anyone designed for it.

8. Risk-Stratified Options

For the ADR-0010 decisions, the PM laid out four questions, each with named options and the cost of each. On the unknown-tag handling: Option P (permissive — risk: typo’d tags survive silently), Option L (allowlist — more control, requires per-index config), Option H (hybrid, the one it proposed). It recommended where it had a view and left the choice with me. At 03:14 it paused and asked what to work on next rather than self-selecting — which is where the 4.5-hour gap in the log comes from. It was waiting on me.

9. Specialist Routing

Routing stayed consistent by agent type. Issue reads and epic filing went to ticketing, daemon deploys to local-ops, implementation to rust-engineer. CI polling and merges were version-control’s; runtime QA went to api-qa, architecture feasibility to research. Across all 63 delegations, the PM did not hand an implementation task to ticketing or a CI task to the engineer.

10. Plans Sharpen with Information

When recon came back, the plan went from vague to specific. After the first two agents returned, the summary carried exact PR numbers, exact crate versions (trusty-search 0.24.4, trusty-memory 0.15.2, trusty-analyze 0.7.0), exact ports, and one specific stray process to clean up — none of which existed in my prompt.

After the issue specs arrived, “fix the top candidates” became a coordinated analysis: “these five cluster around one shared concern — the indexes.toml persistence / colocated / warm-boot scan paths — and #1088/#1089/#1090 in particular interlock through the config-write path, so a coordinated fix is safer than five isolated ones.” The prompt set a direction; the detail came from what the agents found.

11. Self-Maintained Cross-References

Throughout, the PM tracked issue numbers, file paths, commit hashes, and the relationships between them. It noticed that #1088 and #1089 had been collapsed into one commit and independently checked whether that collapse was safe — a cross-reference that was in no agent’s report, only in the PM’s own check against the original spec.

It also caught something I’d have missed: an external contributor, maui314159, had a live PR (#1082) covering a dependency that the #819 work needed. The PM surfaced it as a coordination point rather than duplicating or ignoring it. Later it drew the boundary precisely: “#819 isn’t blocked by conflict — it’s gated on accepting a contributor’s architecture decision, which is your call, not something I’ll auto-merge.” Then it reviewed the external PR in full before going further.

12. Clean Handoff

The session ended on a handoff, not on more output. After filing the final epic (#1119), the PM declared the scope complete, named the one item still in flight, and gave the resume mechanism: “Session stays paused (session-20260611-161949). That clears everything you asked for in this window. The only thing still in flight is #819 (KG ingest endpoint, building in kg-ingest) — I’ll relay its PR when it lands but won’t start anything new. Resume anytime with /mpm-session-resume.” I issued /exit 18 minutes later.

This is the strong version of stopping. The PM did not keep going to look busy, and it did not stop arbitrarily. It identified the boundary — what was done, what was still moving, where the human picks back up — and stopped there.

What the Instrumentation Adds

The twelve behaviors are what you’d see watching over the shoulder. The telemetry adds three things you can’t see that way.

The 7% is even smaller than it sounds. Nine human-authored prompts in 14.5 hours. Three were a word or a phrase — proceed, let's do the top candidates, in console running?. The longest content prompt I wrote all session was 84 characters: it doessnt show the individual consoles, theses should be tabs - and it should be am spa — typos and all. The orchestrator did not need a spec. It needed a direction and the occasional course correction.

The economics are a cache story. The sonnet agents read 3,531 cached tokens for every fresh input token. The PM ran at 587×; haiku, doing high-volume narrow tasks, hit 34,868×. Most of each turn’s context is pulled from cache at cache prices, not re-sent as fresh input. A 14-hour run that re-paid full freight for context on every turn would cost a multiple of $486. The caching is why an orchestration this long is economically viable at all. A note on what I actually paid: I’m on Anthropic’s Max plan at $200/month. The $485.96 above is rack rate — the right figure for understanding the economic value of the output. My marginal cost for this session was zero.

Waiting costs money, and you can see exactly where. The most expensive PM turns share one trait: cache_read=0 — the turns where context had to be rebuilt cold after a cache miss, each one corresponding to a long gap I left. The $7.26 turn, the single priciest of the session, came right after I typed proceed, following 1.5 hours of silence; it wrote 364,618 tokens of cold context to restart the pipeline. The $5.82 turn came after my 4.5-hour wait. The orchestrator picked up where it left off both times, with full context. It just had to pay to reconstruct it. In a system like this, my latency has a line item.

And one finding the behaviors undersell: the failures. The first memory call errored and the PM fixed and retried it. The memory write later failed against a held lock, and the PM diagnosed the daemon contention and moved on. A subagent dropped its connection mid-run on an infra hiccup, and the PM recorded the socket error and later resumed the job to finish it. Three failures, three absorptions, zero escalations to me. That is the part that reads most like a year-one IC: not that nothing broke, but that what broke got handled below my line of sight.

So What?

Four threads run under all of this.

The first is where the line now sits between supervision and delegation. For most of these tools, the answer has been “supervise closely” — read every diff, catch every drift. This session moved the line. I supervised at the level of direction and architectural calls; I delegated everything from PR review to conflict avoidance to failure recovery. The PM did its own adversarial review 13 times and bounced a starred approval. Verification did not disappear; it moved inside the loop, and it left a receipt.

The second is what changes when the coordination work — not the typing — is the part the tool does well. The code these systems write stopped being the interesting question a while ago. The interesting development is that the orchestration layer now decomposes work, predicts conflicts, routes to specialists, tracks provenance, and knows when to hand back. That is project management. When the typing is cheap and the coordination is the hard part, a tool that coordinates well is worth more than a tool that types fast.

The third is the one underneath everything else. No single behavior above is impressive on its own. A competent IC reads the ticket history, names the parallel streams, predicts a merge conflict, routes to the right person, and stops at the decision boundary — without being told. What changed is that they now appear together, unprompted, in one session — and this time there’s a receipt for every one of them. The dollar figure is not the headline. The instrumentation is. We can finally watch the whole thing work and count what it cost, behavior by behavior.

The fourth is about where to point the question. For a while the useful question was “what can the model do.” After this session I think the better question is “what can the trifecta do,” because none of the behaviors above trace cleanly to a single component. They come out of the interaction: a model that audits its own work, a harness that isolates parallel work in separate worktrees, and an orchestration layer that routes to the right specialist with project memory already loaded. Pull any one of the three and the session degrades — the self-audit goes nowhere without a pipeline to surface it, the parallel waves collide without isolation, the routing misfires without memory. The capability lives in the seams between the parts, not in any one of them.

One last note on the economics. The Max plan is $200 a month. For that, I ran a session that bills at $486 rack rate, and the output is good enough that going back to a metered model is hard to imagine. Dealers give the first one away for a reason. The Max plan is that first bag: the work is compelling enough to be structurally addictive, and the flat rate strips out the per-token friction that might otherwise make you stop and think. I don’t mean that as a complaint. It’s a description of how the pricing works on the user.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Explore the annotated session timeline — the turn-by-turn companion infographic for this session

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

The Math Doesn’t Work (Yet): Inside the AI Profitability Problem

OpenAI’s own projections show losses getting bigger as revenue gets bigger. Leaked investor documents reported across WSJ, Fortune, and The Information put the company at roughly $74 billion in operating losses in 2028 — on roughly $100 billion in projected revenue. That pairing is the headline. Not a smaller loss as scale arrives. A loss that grows faster than the top line.

That single relationship is the whole story. We tend to model AI companies as software businesses that will eventually grow into their cost structure the way SaaS companies did before them. The numbers say otherwise. These are capital-intensive infrastructure plays wearing software-company clothing, and the unit economics underneath them run in the opposite direction from the SaaS playbook most of us internalized over the last fifteen years.

A caveat before the numbers, because it matters for what’s below: neither OpenAI nor Anthropic publishes audited financials. Most figures here come from leaked investor decks, run-rate annualizations the companies announce in funding rounds, or SEC filings made by their cloud partners. The uncertainty is part of the analysis, not a footnote to it. When a specific timeline or quarterly figure couldn’t survive cross-checking against primary sources, I left it out.

TL;DR

• OpenAI’s leaked projections show operating losses widening as revenue grows — roughly $74B in losses on ~$100B revenue projected for 2028, with cumulative cash burn near $115B through 2029.

• In 2025 OpenAI spent about $1.69 for every dollar of revenue (~$9B net loss on ~$13B revenue), per leaked documents confirmed by multiple outlets.

• Anthropic’s revenue trajectory is steep — roughly $1B annualized at the end of 2024 to a figure announced in the tens of billions by mid-2026 — with Claude Code alone reported at $2.5B annualized by February 2026.

• Inference token prices fell about 75% in a year. Selling more AI makes the per-unit economics cheaper, which makes revenue growth harder, not easier.

• AI-native gross margins sit near 45% versus 75–85% for mature SaaS — a structural gap of 23–33 points no company has yet closed.

• No company has a verified path to profitability. Every specific breakeven-by-year claim I tried to confirm fell apart under scrutiny.

Two Companies, Two Shapes

OpenAI ended 2025 at roughly $20 billion in annualized revenue, a figure CFO Sarah Friar has stated directly. That is a large business by any normal measure. It is also a business that, in the same year, spent about $1.69 for every dollar it took in — somewhere around a $9 billion net loss on roughly $13 billion in recognized revenue, according to leaked documents that WSJ, Fortune, and The Information each reported. The company is majority funded by Microsoft, runs its compute primarily on Azure, and its strategy is scale-first: build the largest models, capture the most usage, and trust that revenue follows the curve.

Anthropic’s shape is different. Its revenue trajectory is steeper and more concentrated. The company grew from roughly $1 billion annualized at the end of 2024 to a figure it announced in the tens of billions by mid-2026 — the number it cited in its Series H materials. I’m deliberately not pinning an exact figure to a month here, because the company has grown several-fold inside a single five-month window and any precise number is stale by the time you read it. What’s verifiable is the slope, and the slope is steep.

The more interesting detail is the concentration of value. Claude Code, one product, was reported at roughly $2.5 billion annualized by February 2026. A single coding tool driving that much of a company’s run rate tells you something about where the margin-bearing demand actually lives. Anthropic is backed by Amazon (over $8 billion invested) and Google (over $2 billion), and it has committed to spend more than $100 billion on AWS over ten years, with roughly 1 GW of Trainium capacity targeted by the end of 2026. Two large companies funding it; one of them also selling it the silicon it runs on.

Why Compute Is the Problem

What separates these companies from every SaaS business you’ve evaluated: they don’t own their infrastructure. They rent it, at hyperscaler rates, from the same companies that fund them.

OpenAI’s Azure spend reportedly ran around $3.7 billion in 2024 and roughly $8.7 billion across the first three quarters of 2025. Treat those numbers as medium-confidence — they come from leaked documents, and Microsoft pushed back that the figures “aren’t quite right.” But the direction is consistent with everything else: compute cost is the dominant line item, it’s largely fixed, and it grows with usage.

Anthropic’s arrangement produced one of the stranger details in modern enterprise finance I’ve read in some time. The company signed a deal for compute from Colossus 1 — Elon Musk’s Memphis data center, operated by xAI — at roughly $1.25 billion per month for 300 MW of capacity, running through May 2029. That’s not from a leak. It surfaced in SpaceX’s S-1 SEC filing and was confirmed by CNBC, Axios, and Data Center Dynamics, with a potential total value above $40 billion. There’s a 90-day mutual cancellation clause, so the headline total overstates the firm commitment. Still: Anthropic — funded by Google and Amazon — is paying Elon Musk’s company more than a billion dollars a month for compute. The AI capital world is stranger from the inside than the press releases suggest.

Zoom out and the renter problem gets sharper. Hyperscaler capex for 2026 is projected at $660–690 billion. Against that, OpenAI’s $20 billion ARR is roughly 3% of a single year’s data-center buildout by its suppliers. The companies selling AI applications are small tenants in an infrastructure market they don’t control and can’t currently price against.

The Unit Economics Trap

This story inverts an instinct most of us trust.

In normal software, even in the Cloud, scale is your friend. Marginal cost trends toward zero, gross margin climbs as you grow, and a mature SaaS business lands at 75–85% gross margin because serving the millionth customer costs almost nothing. Volume is the cure.

Inference doesn’t behave that way. Every token generated costs compute — real, metered, non-zero compute — so the marginal cost of serving usage stays stubbornly positive. And the price you can charge for that token is collapsing. Enterprise transaction data from Ramp shows inference prices falling roughly 75% in a single year, from around $10 per million tokens to around $2.50. Capability per dollar is improving fast, which is good for buyers and brutal for sellers, because it means the revenue you booked at last year’s prices reprices downward while your compute bill does not.

Put the two forces together and you get a squeeze that worsens with success. The better you are at selling inference, the more usage you drive; the more usage you drive, the more the per-unit price falls; the more it falls, the harder it is to grow revenue against a compute bill that scales with that same usage. Volume isn’t the cure here. Under these dynamics it’s part of the disease.

The survey data puts a number on how far this world sits from SaaS. ICONIQ Capital polled about 300 software executives and pegged AI-native gross margins at 41% in 2024, 45% in 2025, and a projected 52% in 2026. Improving — but starting from a base 30-plus points below mature SaaS, and closing the distance slowly. A 52% gross margin is a respectable hardware business. It is a structurally difficult software business, especially one still spending heavily to grow.

Two Different Bets

OpenAI and Anthropic are running different experiments on how you eventually close that gap. Neither has been validated.

OpenAI’s bet is scale and breadth. Build the broadest platform, capture consumer and enterprise and API demand simultaneously, and assume that at sufficient scale you gain pricing power over compute, model-efficiency gains compound, and the revenue base grows fast enough to absorb the fixed cost. The leaked projections embody the risk in this bet: they show losses widening through 2028 even as revenue approaches $100 billion, with cumulative cash burn near $115 billion through 2029. The theory requires the curve to bend after the window we can currently see.

Anthropic’s bet is narrower and more product-led. Find a wedge where the work is valuable enough that buyers tolerate real prices, prove the margin there, and expand outward. Claude Code is that wedge made concrete — $2.5 billion annualized from developers who pay because the output is worth more than the inference under it. Coding, agents, and enterprise automation are higher-value work than chat, and higher-value work supports prices that don’t immediately erode under token deflation. The risk: revenue concentration in a single product line, and an infrastructure bill — AWS commitments plus the xAI deal — that’s enormous relative to a company still proving the model.

Two theories of the same problem. Scale your way past the margin gap, or find work valuable enough that the gap doesn’t bind. We don’t yet have the data to say either works.

What Would It Actually Take

I’ll skip the timeline speculation — every specific breakeven-by-year claim I tried to verify died on contact with the sources. The structural requirements are clearer than the dates.

Three things have to move. First, gross margins have to climb from the mid-40s toward something defensible — call it 60-plus — and stay there while volume grows. That means model-efficiency gains (cheaper inference per unit of capability) have to outrun price deflation, rather than getting passed straight through to buyers as lower prices.

Second, these companies need pricing power over compute, which today they don’t have. At current scale they’re tenants. The open question is at what ARR a vendor becomes large enough to negotiate compute like a partner instead of a customer — or to build its own. Anthropic’s Trainium commitment and OpenAI’s various infrastructure moves are bets that vertical integration eventually changes the cost equation. That’s unproven, and it’s expensive in the interim.

Third, the product mix has to keep shifting toward work that resists deflation — enterprise agents, coding tools, automation that’s measured against labor cost rather than against the falling price of a token. Claude Code is the cleanest evidence that this category exists and that buyers will pay. Whether it’s a large enough share of total volume to lift blended margins across a company is the question that decides the whole thing.

The Open Questions

What we don’t know outweighs what we do.

We don’t know the actual, current gross margins at either company — only the AI-native sector estimate of roughly 45%. Neither company publishes the number that would settle the argument. We don’t know whether Anthropic’s stack of compute commitments, the decade-long AWS deal alongside the month-by-month xAI arrangement, creates structural tension or healthy redundancy. A company hedging across three infrastructure providers is either diversifying supply or revealing that no single supplier can meet its demand. We don’t know the ARR threshold at which compute pricing becomes negotiable, which is the hinge the entire margin story turns on.

And there’s the strategic risk that has no clean precedent: your infrastructure supplier is also your competitor. Microsoft ships Copilot. Amazon and Google both build models that compete with Anthropic’s. xAI builds Grok. Every dollar these companies pay for compute partly funds a rival’s model program. In normal software you don’t hand your gross margin to the company trying to beat you. Here it’s the default arrangement.

So the real question isn’t whether the AI labs are growing. They obviously are, faster than almost any companies in history. The question is whether revenue growth and margin improvement are the same trend or opposing ones. The SaaS era trained a generation of operators to believe that scale fixes economics. The leaked numbers describe a business where scale, so far, makes the loss bigger. Until one of these companies publishes a gross margin that shows the curve bending, that’s the math we have. And the math doesn’t work yet.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• The First 70% Era — Where agentic AI delivers value and where it stops, and why the higher-value work resists token deflation

• AI and the Rise of the Hyperdev — Why developers pay real money for AI tooling, the demand side of the margin story

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Most of the best ideas in software engineering aren’t new. They’ve been written up in books, argued over at conferences, taught in every “best practices” deck since the late 1990s. And most teams quietly don’t do them.

Not because anyone thinks they’re wrong. Test-driven development, design by contract, architecture decision records, mutation testing — ask a room of senior engineers whether these are good ideas and you’ll get nods. Ask the same room who practices them consistently under deadline pressure and the hands stay down. I’ve been in that room for twenty-five years, on both sides of the question. I’ve also been the engineering leader who let those practices slip because shipping the feature mattered more this quarter.

There’s a single economic reason these practices lose. The upfront cost is high, the payoff is real but distant, and human attention is the binding constraint. Write the test before the code, document the decision, specify the invariant — every one of those is a tax you pay now against a benefit you collect later, maybe, if the project lives long enough. Under deadline pressure, that’s a losing trade for a human. So we skip it, ship, and pay the interest later in bugs and confusion. Call it the impatience tax.

Agents don’t pay that tax. They have infinite patience for upfront rigor and roughly zero marginal cost for the tedious work that rigor demands. Writing a thorough test suite for code that doesn’t exist yet is psychologically brutal for a person and completely fine for a model. That single shift — the cost of patience going to zero — quietly inverts the economics of a whole list of practices we knew were right and gave up on anyway.

This isn’t a piece about what AI makes possible. Lots of things are possible. It’s about a narrower, more useful question: which disciplines did we already agree were correct, fight about for decades, and abandon for reasons that no longer hold?

Here are nine.

TL;DR

• These nine practices share one structure: high upfront cost, distant payoff. Human attention is the constraint that kills them under deadline pressure.

• AI removes the constraint. A failing test is the clearest prompt you can hand an agent; a spec is its input; an ADR is its context. The discipline becomes the interface.

• TDD, design by contract, and property-based testing turn from “things we should do” into the most effective way to constrain agent behavior and prevent hallucinated correctness.

• Documentation, ADRs, and living docs get a bilateral ROI: agents generate them from code, and they make agents far more effective in your codebase.

• The catch is real. A 2025 METR randomized trial found experienced developers were about 19% slower with AI assistance. These practices pay off only when AI is used with discipline, not as autocomplete.

1. Test-Driven Development

Start with the practice most teams abandoned first.

Writing tests before code was always the theoretically superior move. It forces you to define the interface before you build behind it, catches bugs at the moment of definition instead of during integration, and leaves behind a living specification of what the code is supposed to do. Kent Beck made the case decades ago and the case held up.

Almost nobody did it consistently. The reason is psychological, not technical. Writing detailed tests for code that doesn’t exist yet, while a deadline breathes on your neck, feels like building scaffolding for a house you haven’t designed. Your brain screams at you to just write the function. So you write the function, promise yourself you’ll add tests after, and — well. You know how that goes.

Now flip the perspective. To an agent, a failing test isn’t scaffolding. It’s the clearest possible specification of intent you can provide. “Make this pass, don’t break anything else” is an unambiguous, machine-checkable instruction, which is exactly what a probabilistic system needs to stay honest. The test suite becomes a guardrail that prevents the most dangerous failure mode in AI-assisted coding: confident, plausible, wrong. Hallucinated correctness dies against a red bar.

TDD went from the discipline most teams couldn’t sustain to one of the best tools we have for bounding what an agent is allowed to claim it did. Same practice. Opposite economics.

2. Spec-Driven Development and Design by Contract

Bertrand Meyer formalized design by contract in the 1980s and built it into the Eiffel language: specify preconditions, postconditions, and invariants, then let the implementation follow from the contract.

The idea was sound and the adoption was thin, for one stubborn economic reason: the contract only pays off if someone else writes the implementation from it. If you’re writing both the spec and the code, the spec is overhead — you already know what you meant. The contract’s value lives in the handoff, and for most of software history there was no cheap handoff to hand it to.

Now there is. You write the contract; the agent writes the implementation from it. Spec-driven development stops being a documentation chore and becomes the actual control surface for delegation. The spec is the part requiring human judgment about what the system should do. The implementation — the part that used to eat the hours — is the part you delegate. Meyer’s economics finally close, forty years late, because the missing party in the transaction showed up.

3. Architecture Decision Records

Why does the codebase look like this? Why Postgres and not Dynamo, why this queue, why the weird module boundary that everyone trips over?

ADRs were the right answer to that question — a short dated record of each significant decision, the context, and the alternatives rejected. The discipline almost never held. Same shape as everything else here: the cost is immediate (stop, write the thing) and the value accrues slowly, mostly to some future engineer who isn’t in the room yet.

Two things flipped at once, which makes this one more interesting than the rest. First, agents can generate ADRs from an existing codebase — read the git history, the dependency choices, the structure, and reconstruct the decisions that produced them. The retroactive cost of documentation drops toward zero. Second, and this is the part people miss: existing ADRs dramatically improve what an agent can do in your codebase. An agent that can read why you chose eventual consistency won’t keep proposing changes that assume strong consistency.

So the ROI went bilateral. Agents help you write ADRs, and ADRs help agents help you. The practice that used to only cost now pays on both ends.

4. Continuous Code Review

“Catch issues early” has sat on every best-practices list since Extreme Programming put continuous review on the map. The advice was never controversial. The bottleneck was always the same: human reviewer attention is finite, expensive, and easily exhausted. So review collapsed into batch PR review — a tired engineer reading a 600-line diff on a Friday afternoon, approving most of it on faith.

AI review on every commit — not batched at the PR boundary, but running as code lands — is moving from aspirational toward baseline: increasingly the default on teams that have wired it in, though not yet universal. The marginal cost of a careful read went to nearly nothing, and the read happens while the context is still warm.

But this one comes with an emergent problem worth naming, because it bites teams that adopt the tooling without rethinking the model. PRs are getting larger and arriving faster under AI-assisted development. An agent can produce a 2,000-line change in an afternoon. If your review model still routes everything through a human approver at the end, that human is now the rate limiter, drowning in volume they didn’t generate and can’t realistically read. AI review on every commit is part of the answer. The harder part is restructuring what the human reviews — architecture, intent, the decisions a model shouldn’t make alone — and letting the machine handle line-level correctness continuously. Adopt the tool without rethinking the workflow and you’ve just built a faster way to overwhelm your best reviewer.

5. Pair Programming

Pairing always looked expensive in the most obvious way: two engineers, one task, double the salary against a single unit of output. That intuition was wrong on the numbers — the measured overhead from the pair-programming studies was closer to 15%, often recovered through fewer defects — but the 2× gut feeling is what drove the decisions. The benefits were real — knowledge transfer, real-time review, fewer dumb mistakes — but the perceived math meant most teams reserved it for critical paths, gnarly bugs, or onboarding a new hire. A luxury, rationed.

The pair is now a human and an agent, and it’s available to every engineer continuously, not rationed to the critical path. The knowledge-transfer benefit generalizes — the agent can explain unfamiliar parts of the codebase on demand. The real-time-review benefit generalizes — a second set of eyes on every line, every time, without scheduling two calendars. The economics that made pairing a rationed luxury simply don’t apply when one half of the pair has near-zero marginal cost.

Worth a caveat: agent-as-pair is genuinely good at the review and explanation half of pairing, and weaker at the part where a human partner pushes back on a bad design before you’ve written a line. You still need humans pairing with humans for that. But the day-to-day, line-by-line version of pairing just became free, and that’s most of what pairing was for.

6. Mutation Testing

Coverage numbers lie, and most engineers know it. Eighty percent line coverage tells you eighty percent of your lines got executed by a test — not that any of those tests would notice if the behavior broke. Mutation testing is the honest measure: it deliberately introduces bugs (flip a comparison, drop a line, change a constant) and checks whether your test suite catches them. If a mutant survives, you have a test that runs code without actually validating it.

Mutation testing was always the gold standard and almost never run continuously, for one reason: it’s computationally expensive. You’re effectively running your whole suite many times over, once per mutation. On a real codebase that’s brutal. So it lived in research papers and the occasional heroic CI job that someone eventually disabled for being too slow.

That constraint is mostly gone — compute is cheap and parallel, and we got more comfortable spending it. And the practice arrived right when we suddenly need it most. AI-generated tests have a characteristic failure mode: they drift toward coverage metrics without meaningful assertions. The model writes a test that calls the function, exercises the path, and asserts almost nothing of substance — green checkmark, zero protection. Coverage looks great. Mutation testing is the thing that catches exactly that. It’s the verification layer for a verification layer, and it matters more now than when it was invented, because now a machine is writing the tests.

7. Living Documentation

Documentation was always supposed to be a first-class artifact. It almost never was, and the reason is by now familiar: writing docs is tedious, and the penalty for stale docs accrues slowly and lands on someone else. So docs rotted. Every team has a wiki that’s a graveyard of half-true pages from two reorgs ago.

AI changes both halves of the equation at once. It generates docs from code, so the writing cost drops. And it consumes docs as context, so the docs earn their keep immediately — a well-documented codebase is a measurably more useful codebase for an agent working in it. The ROI is immediate and bilateral, same structure as ADRs.

There’s a quiet shift hiding in there. Documentation used to be written for humans who’d mostly never read it. Now it’s also written for the agent that will read it on every task, which means stale docs don’t just confuse a future engineer — they actively degrade your tooling today. The feedback loop tightened from months to minutes. That’s the kind of change that actually moves behavior, because the cost of skipping it shows up now instead of later.

8. Runbook Generation from Incidents

On-call always leaned too hard on tribal knowledge. The person who knows why the payment service wedges at 3 a.m. is asleep, on vacation, or left the company last spring. Writing a runbook after each incident was obviously the right move and reliably the thing nobody did, because the incident was over and everyone wanted to go back to bed.

Incidents become runbooks automatically now. The agent has the incident timeline, the chat transcript, the commands that resolved it, the postmortem — and it can turn that into a structured runbook while the details are fresh, without asking an exhausted engineer to relive the night. The cost that used to fall right when motivation was lowest now falls on a system that doesn’t get tired or resentful.

I’d treat the generated runbook as a draft a human still signs off on, not gospel. But “imperfect draft, reviewed in five minutes” beats “blank page nobody ever fills in,” and that was always the real competition.

9. Property-Based Testing

Example-based tests check the cases you thought of. Property-based testing is stronger: you specify the invariants a system must always satisfy — reversing a list twice returns the original, a serialized-then-deserialized object equals the original, the account balance never goes negative — and the framework generates hundreds of adversarial inputs trying to break them. QuickCheck pioneered the approach; it finds the edge cases you’d never have written by hand.

It never went mainstream outside a few communities, and the bottleneck wasn’t tooling — good property-based libraries exist for most languages. The bottleneck was writing good property specifications. Identifying the right invariants requires deep domain reasoning: you have to understand the system well enough to state what must always be true, which is harder than writing a few example cases. Most engineers, under pressure, defaulted to the easier thing.

This is where AI helps in a way that’s less obvious than “it writes the code.” A model can generate property suites from a spec, and — more usefully — it can reason about what invariants a system should satisfy in the first place, surfacing properties you hadn’t articulated. That’s the expensive, judgment-heavy part it actually offloads. Combined with mutation testing to keep the generated properties honest, you get a testing approach that was always more powerful than example-based testing and was always too expensive in human reasoning to adopt widely.

The Catch

I’d be selling you something if I stopped there, and the data won’t let me.

In July 2025, METR ran a randomized controlled trial with experienced open-source developers working on real tasks in repositories they knew well. The developers expected AI assistance to speed them up. It slowed them down — by roughly 19%. METR’s February 2026 follow-up found that gap narrowing, and reversing on some measures, as the same kind of developers gained real experience with the tools — which is to say the 19% was a snapshot of the unfamiliar, undisciplined path, and it closes precisely as people pick up the habits this piece is about.

That finding is real and it isn’t a contradiction of everything above. It’s the missing condition. Every practice in this piece works because it imposes structure on the agent — TDD as a guardrail, the spec as input, the ADR as context, mutation testing as the check on the check. Used that way, with discipline, AI is constrained toward correctness. Used the other way — as autocomplete, as a vibe-coding partner you don’t supervise — you get more code, faster, with less correctness and a slower path to done once you account for the cleanup. The METR developers, working in code they already understood deeply, may well have been paying exactly that tax: accepting plausible suggestions that took longer to vet and fix than writing it themselves would have.

So the inversion isn’t automatic. The cost of patience dropped to zero, which makes the rigorous path finally affordable. It does not make the undisciplined path good. If anything it makes discipline more important, because a tool that produces plausible output at high volume is precisely the tool that most needs a guardrail you can’t talk your way past. A red test bar doesn’t care how confident the model sounds.

What To Do With This

The interesting question was never “what does AI make possible.” That list is enormous, mostly speculative, and not very actionable. The better question is the one this whole piece is built on: which practices did we already know were right, argue about for decades, and quietly give up on?

That list is short, specific, and yours to write. Go pull your own team’s “we should really do this but we don’t” backlog — the standing items in retros that everyone agrees with and nobody owns. I’d bet most of them have the same economic shape: high upfront cost, distant payoff, killed by human impatience under deadline. Test coverage on the legacy module. The runbooks. The ADRs for the three decisions everyone keeps re-litigating. The integration tests that would’ve caught last quarter’s outage.

Run each one through a single question: was this abandoned because it was wrong, or because it was expensive in human patience? The wrong ones, leave abandoned. The expensive-in-patience ones just got cheap. Those are the ones to pick back up first.

The impatience tax got repealed. The disciplines it used to make unaffordable are sitting right there, mostly unchanged, waiting for someone to notice the price changed.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• The Other Shoe Has Dropped — Why enterprise AI bills don’t match the per-token price collapse

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Something shifted in the last year, and it took me a while to name it.

A growing number of people are running organizations while still doing real hands-on technical work. Not as a hobby, not on weekends, not as a vanity exercise to keep their commit graph green. They are building things their teams depend on — and they are doing it as a deliberate part of the job, not in the cracks between meetings. The work has a specific shape. It is rarely a production feature. It is the layer underneath: developer productivity tooling, internal services, agentic harnesses, MCP connectors, the infrastructure that unblocks everyone else.

For most of my career this combination didn’t really hold together. You could code or you could lead, and the moment you tried to do both seriously, one of them rotted. I’ve watched plenty of technical executives keep a foot in the codebase and slowly become the bottleneck everyone routed around politely. The pattern was familiar enough to be a warning.

What changed is not that leaders suddenly got more disciplined. It’s that the time cost of meaningful technical contribution collapsed. Agentic coding made a hybrid role viable that wasn’t viable before — and the more I look at it, the more I think this isn’t a new invention at all. It’s the recovery of a very old idea that modern specialization interrupted.

I’m writing this as someone living in the middle of it. I spend roughly 30% of my time coding. And when I say coding, I mean directing a team of agents — much closer to that than hands-on work, which is probably the whole point. That’s not a full-time IC’s week, and it isn’t meant to be. It’s enough to stay close to the work that matters and to build the enabling infrastructure I think is worth my own hands on the keyboard.

TL;DR

• A distinct role is emerging: leaders who run organizations and still do deep technical work — specifically enabling/institutional work (tooling, harnesses, internal services), not critical-path product features.

• This satisfies Charity Majors’ actual advice. Her line was never “stop coding.” It was “stop writing code in the critical path.” Enabling work fits that exactly.

• The integration of strategist and practitioner has deep cross-cultural precedent — Japan’s bunbu-ryōdō, Rome’s Marcus Aurelius, China’s wen-wu, the Renaissance polymath, Mattis’s “warrior monk.” The modern role is a recovery, not a novelty.

• Agentic coding is what makes it newly viable: focused sessions now deliver output that once required sustained, uninterrupted immersion. The Anthropic 2026 data shows ~27% of AI-assisted work is work that “wouldn’t have been done otherwise.”

• Directing agents feels like delegation — the same skill leaders already use with human reports. Which is why experienced leaders adapt to it more naturally than juniors do.

The pattern, named

The difficulty with the coding executive was never philosophical. It was attentional.

Charity Majors mapped this years ago in The Engineer/Manager Pendulum, and the piece holds up because she was precise about the mechanism. Management is interruptive by design — your job is to be available, to unblock, to absorb the chaos so your team doesn’t have to. Serious engineering is the opposite. It requires blocking interruptions for long enough to hold a complex system in your head. Two incompatible attention modes. Try to run both at once and you do neither well.

But here is the part people skip when they quote her. Majors never said managers should stop coding. Her actual advice was sharper: don’t write code in the critical path. Don’t be the person others are waiting on. Stay technical, stay sharp, just don’t make yourself a dependency that blocks shipping. “The best frontline eng managers in the world,” she wrote, “are the ones that are never more than 2-3 years removed from hands-on work.”

That distinction carries the whole argument. Because there is a category of technical work that is consequential without being critical-path, and it turns out to be exactly the work senior people are best positioned to do.

Call it enabling work. Internal tooling. Developer productivity infrastructure. Agentic harnesses. MCP services that other teams plug into. Architectural prototypes that prove a direction before anyone commits to it. None of this is what blocks a release on Thursday. All of it multiplies whoever comes after. It tolerates interruption — you can pick it up Tuesday afternoon and put it down when a real fire starts — precisely because nobody is standing at your desk waiting for it.

This is also where the industry is putting its money. Gartner named platform engineering a top strategic trend for two years running and projects that 80% of large engineering organizations will run dedicated platform teams by 2026. The structural reason a leader can work here without becoming the bottleneck is built into the definition of the domain: its output unblocks others rather than blocking them.

Will Larson’s Staff Engineer archetypes circle the same territory without quite landing on it. His “Architect” sits in a permanent argument — some organizations demand the Architect stay deep in the code, others forbid it. The leader/practitioner resolves that argument by relocating it: deep in the enabling and infrastructure work, absent from production product code. Not the pendulum, not the staff IC. A real hybrid, and a newly coherent one.

This is not new

Here is where I want to slow down, because the most interesting thing about this role is how old it is.

The idea that a leader should be both a strategist and a practitioner — not separate modes to alternate between but a single integrated way of operating — shows up independently across at least five civilizations. That kind of convergence usually means a culture has found a durable answer to a real problem.

Japan gave it a name: bunbu-ryōdō (文武両道), the way of both the literary and the martial. Bun is letters, cultivation, strategy. Bu is the martial, the active, the practitioner’s hand. Ryōdō means both ways, together — not balanced, not traded off, but held at once. By the mid-fourteenth century the dual-talented warrior was already established as the model leader, and during the Edo period the Tokugawa shogunate made it official policy for the samurai class. The phrase that survives captures the stakes: culture without power is ineffective, and power without culture is barbarous.

The archetype is Miyamoto Musashi. Undefeated in more than sixty duels, often fighting with a wooden sword against live steel. He founded a two-sword school, and in the last months of his life he wrote The Book of Five Rings in a cave. He was also a recognized master of ink painting and calligraphy — his Shrike on a Withered Branch survives as a designated Important Cultural Property of Japan. The same hands that won sixty duels produced fine art a nation still protects. He didn’t oscillate between the sword and the brush. He held both, and each sharpened the other. “When I apply the principle of strategy to the ways of different arts and crafts,” he wrote, “I no longer have need for a teacher in any domain.” Mastery in one discipline illuminating all the others.

Rome had Marcus Aurelius, the philosopher-king made historical rather than theoretical. He ran the empire and commanded its armies on the Danube, and he wrote the Meditations in the war camps — fragmentary notes to himself, composed in the middle of campaigning and administration. That book was never published philosophy. It was a working journal, the most powerful man in the ancient world writing to stay grounded while doing the job.

China institutionalized the same ideal as wen and wu — civil cultivation and martial capability — and ran it for roughly thirteen centuries through the scholar-official class. Zeng Guofan is the canonical case: he rose through the imperial examinations to high Confucian office, then built and commanded an army of more than a hundred thousand, reportedly keeping a diary on Neo-Confucian ethics even as he directed the campaigns. The integration wasn’t left to personal taste. It was built into the examination and career structure.

And the archetype still lands today. James Mattis earned the nickname “Warrior Monk” — battlefield commander and devoted reader, a 7,000-book library, the Meditations carried into combat. The chain from Aurelius to Mattis is literal: the same book, eighteen centuries apart. That we still reach for “warrior monk” as a compliment for a leader tells you the integration never stopped resonating.

Across all of it, the answer is the same. The contemplative and the active were not specializations to assign to different people. They were a single discipline, each half informing the other. Modernity — with its org charts, its clean role boundaries, its professional specialization — interrupted that. The leader/practitioner is not a tech-industry novelty. It’s an old integration becoming feasible again.

Why now

So what actually changed? Not the wisdom. The economics.

The thing that made the coding executive a bad idea was the attention math. Serious technical work demanded long, unbroken stretches of focus — the exact resource a leadership schedule cannot reliably provide. You cannot design a system in the fifteen minutes between a board prep and a one-on-one. The pendulum was a real constraint, not a failure of will.

Agentic coding changes that math directly. The unit of work moved up a level. Instead of holding every implementation detail in working memory across a four-hour session, you specify intent, direct an agent, review what comes back, correct course, and direct again. A focused 30-minute session now produces what used to require an afternoon of immersion — not because the thinking got easier, but because the implementation cost collapsed.

The Anthropic 2026 Agentic Coding Trends Report puts numbers on the shift. Average session length has climbed to 23 minutes in the agentic era, up from about 4 in the autocomplete era — the work got denser, not just faster. 78% of Claude Code sessions now involve multi-file edits, up from 34% a year earlier. Teams running multi-agent workflows report 2–4x faster delivery from task creation to deployment. And the figure that matters most for this argument: roughly 27% of AI-assisted work consists of tasks that “wouldn’t have been done otherwise” — the scaling projects, the nice-to-have tools, the exploratory infrastructure that was never quite worth the manual hours.

That 27% is the enabling work. It is the category that lives or dies on time cost, and it’s the category a leader/practitioner is best placed to take on.

The arithmetic is what makes 30% credible. I documented a 6–10x multiplier on focused technical sessions in The Irreducibles earlier this year — a project I estimated at 150–200 billable hours compressed into roughly 50–70 hours of wall-clock time, most of which wasn’t coding at all. If directed work runs several times faster than hand-coding, then a day and a half a week can produce what once consumed a full-time engineer’s week. That’s not a marginal gain. It’s a change in what’s structurally possible.

There’s another dimension that doesn’t show up in the productivity numbers: the work itself is unstable. Agentic coding patterns are still shaking out. There aren’t many experienced practitioners, the field is moving fast, and we don’t yet have good consensus on which patterns are load-bearing and which are fashion. A manager who’s only reading about it can’t make that distinction on behalf of a team. You have to be in it to know.

There’s a counterintuitive wrinkle worth naming: the people best positioned to exploit this are the senior ones. A University of Chicago working paper from late 2025 found experienced developers were 5–6% more likely to succeed with AI agents for every standard deviation of work experience, largely because they worked plan-first — laying out objectives, alternatives, and steps before invoking the tool. That’s the opposite of the assumption that AI flattens the seniority curve. Expertise improves your ability to delegate to a model for the same reason it improves your ability to delegate to a person. AI doesn’t change what senior engineering is. It reveals what it always was.

Directing agents is delegation

This is the part that interested me the most, and it’s the bridge between the leadership job and the technical one.

A year or so ago, working with Claude Code felt like coding. Now it feels like delegating. I wrote about that shift when it first became undeniable — the move from being a programmer who uses AI to being something closer to a technical project manager who directs it. Anthropic’s report uses the same vocabulary, describing engineers moving “from writing code to orchestrating the systems that write it.”

What I didn’t fully appreciate at the time is how directly that maps onto the muscle leaders already have. Delegating to an agent feels, in practice, just like delegating to a human engineer. You frame the problem, set the constraints, hand it off, and come back to assess the result. Give a directive, walk away, return to completed work. That loop is the daily reality of management. Leaders developed it because they had to, and it transfers to agents almost without friction.

So the leader/practitioner doesn’t have to become a coder again in the old sense. The skill in demand is judgment plus delegation, and that’s the skill leadership has been building all along. The hands-on knowledge tells you what to ask for and whether the answer is any good. The delegation instinct does the rest.

This is why the enabling work and the agentic tools fit together so cleanly. Enabling work tends to be well-defined, non-user-facing, and long-horizon — exactly the profile agents handle well and exactly the profile that tolerates a leader’s interrupted schedule. The hands-on contribution mostly takes the form of specifying constraints and patterns, which is what I’ve called pattern mastery: when you write the pattern down, you’ve written the spec, and the spec multiplies everyone else’s output.

What it actually looks like

Let me ground this without turning it into a war story.

The concrete examples from my own work are the kind of thing I mean. I built an agentic harness — the orchestration layer I argued is the real determinant of AI coding outcomes, where the same model can swing more than a quality point depending on the scaffolding around it. I built MCP services, the org-specific connectors that replaced a handful of standalone tools in a few hours of directed work each. None of that was a production feature. All of it was infrastructure other people now depend on.

Here’s a detail that may make the point. I now have an “AI architect” on my leadership team helping maintain the very infrastructure I originally built — not just the harness, but our inference relationships, our training program, office hours, the real human work I no longer have the time, or the right, to be doing myself. And I expect to hand off more over time. The enabling work I do today partly becomes the system that does tomorrow’s enabling work. That handoff is the role in miniature: you build the thing that multiplies the team, then you put someone in place to build the next version.

The proportion matters. Around 30% hands-on keeps judgment fresh without putting me in the critical path. Even full-time senior ICs aren’t full-time coders — Bain’s Jue Wang, quoted in MIT Technology Review last December, put developer coding time at 20–40%, with the rest going to analysis, strategy, and the surrounding work. A leader at 30% isn’t doing something exotic. They’re spending their technical budget on the layer where it compounds.

The decision is not “how do I find time to code.” It’s “what enabling work is worth my own hands?” Those are different questions. The first leads to the bottleneck I watched so many executives become. The second leads somewhere useful.

The choice

I’ll resist overselling this, because it isn’t for everyone and it isn’t automatic.

This is a deliberate role, not a default. Staying technically current costs ongoing investment, and the work is often invisible — enabling infrastructure rarely shows up in a quarterly review the way a shipped feature does. The role is easy to misread, too. From the outside, a CTO who codes can look like a CTO who hasn’t let go. The defense against that reading is the discipline Majors named: stay out of the critical path. Build the multipliers, not the blockers.

The returns are real, though. Fresh judgment — the kind that lets you evaluate not just what and why but how. Trust from engineers who see you in the work rather than above it. And institutional infrastructure that makes the whole team faster, built by the person with both the technical depth and the positional authority to prioritize it.

There’s a closing note in the history worth keeping. Bunbu-ryōdō wasn’t only a personal aspiration. The Tokugawa shogunate institutionalized it — built career and class structures around the assumption that a leader should be both. China did the same with its examination system. We’re not there yet. For now, the leader/practitioner is an individual choice, made one person at a time, made viable by tools that finally collapsed the cost of staying hands-on.

But the precedent suggests where this could go. When a way of working proves durable, organizations eventually build structures around it. The era of the leader/practitioner is early. It is also, I’d argue, a return — to an integration we knew was valuable long before we had the means to make it practical again.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• From Coding with AI to Managing AI — When agentic coding starts to feel like delegation

• It’s The Harness, Stupid! — Why orchestration quality dominates AI coding outcomes

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Two stories from the last two weeks. Uber burned through its entire 2026 AI budget in four months on Claude Code, with COO Andrew Macdonald telling the Rapid Response podcast that the link between that spend and shipped consumer features “is not there yet.” The Information had the underlying numbers a few weeks earlier: engineer adoption from 32% to 84% between December and March, heavy users running $500–$2,000/month in tokens, and CTO Praveen Neppalli Naga torching $1,200 in a two-hour demo. Same week, Microsoft told thousands of engineers in its Experiences + Devices division that their Claude Code access is going away. Windows Central, summarizing The Verge’s Notepad scoop, has the cutoff at June 30 — end of fiscal year — with cost as the actual driver even though EVP Rajesh Jha framed it publicly as convergence on Copilot CLI.

Two of the most AI-forward enterprises on the planet, same tool, same week. The “AI is failing” takes were live within hours.

I don’t buy that framing.

The headlines are getting it wrong. Uber didn’t cancel anything — adoption ran ahead of the budget and the company blew its annual spend keeping up. That’s a planning failure, not a verdict on the tool. Microsoft didn’t divorce Anthropic either; they’re still consuming Claude through Azure Foundry and M365 Copilot. What they cancelled is a specific license — Claude Code at the engineer-seat level — because engineers preferred it over GitHub Copilot CLI and the division was paying for that preference.

What both stories show: AI is a new tool and we haven’t learned to use it well yet. The teams over budget pointed it at problems it wasn’t the cheapest way to solve, then let it decide for itself how much work to do per task.

I’ve made the cloud parallel here before. Early cloud was expensive and misused. Lift-and-shift workloads routinely ran two or three times their on-prem cost — I watched that play out across teams I ran, and it took years to correct through architecture. Then the industry learned: right-sizing, reserved instances, autoscaling, serverless where it fit, on-prem where it didn’t. The bills came down. Not because compute got dramatically cheaper, but because we got more careful about what we asked the cloud to do. AI is in the same phase. Cheap per-token, expensive per-task, and the gap is architectural.

A few weeks ago I ran controlled head-to-head tests on Opus 4.6 and Opus 4.7 against identical coding tasks. Both models passed every test. Opus 4.7 cost 3.6× more to do it. Same outcomes, same rate card, dramatically more tokens.

Finout’s analysis of production deployments tells the same story at scale: up to a 35% cost increase overnight, driven by tokenizer changes that don’t show up on the per-token rate card. Not one team’s bad luck — the shape of the bill across the enterprise AI buyer base right now. The second of two shoes on AI economics.

I wrote about that version-to-version cost drift in detail. Providers can collapse per-token prices in public while the per-task bill drifts upward in private. The first shoe was the per-token price collapse that made everyone optimistic. The second is the behavioral and architectural cost overhang now landing on quarterly P&Ls.

TL;DR

• Per-token costs at GPT-3.5-equivalent performance are down roughly 280× since late 2022, per Stanford’s AI Index 2025. Vendor revenue tells the opposite story: Anthropic’s annualized revenue went from $1B in January 2025 to $30B by April 2026 — a 30× move in 15 months, coming from enterprise inference, not consumer subscriptions.

• Gartner’s April 2026 survey: just 28% of AI use cases fully meet ROI expectations, 78% of IT leaders report material AI charges that didn’t show up in any procurement model.

• Gartner estimates agentic workflows consume 5–30× more tokens than equivalent chatbot interactions; Stanford’s Digital Economy Lab puts the upper bound for coding agents at 1,000×. The cost driver isn’t the model — it’s the workflow architecture wrapped around it.

• Two patterns hold the line in production. Search-first architectures put inference at the end of a deterministic pipeline. Consolidated single-shot designs replace multi-call chains.

• Inference is a power tool, not a default. Use it with specific ROI goals per call, apply it to code solutions rather than to directly solve problems, and bound it with deterministic structure on both ends.

Why the Per-Token Savings Didn’t Reach the Invoice

Per-token economics of frontier models have been collapsing for two years. Stanford’s AI Index 2025 puts the decline at roughly 280× from a late-2022 baseline at GPT-3.5-equivalent performance. Most enterprise budget conversations in 2024 started from that headline. The implicit assumption: bills should be going down.

They’re not. The clearest read comes from the vendor side. Anthropic’s annualized revenue went from $1B in January 2025 to $30B by April 2026, with roughly 80% from enterprise and API usage rather than consumer subscriptions. Anthropic now discloses 1,000+ customers spending more than $1M per year — a cohort that doubled in under two months — alongside roughly 300,000 business customers. The mid-tier ($100K–$1M/year) grew 7× year over year.

Menlo Ventures’ 2025 State of Generative AI report cross-checks at the market level: enterprise GenAI spend tripled to $37B in 2025, with LLM API consumption alone at $8.4B by mid-year. The high tier shows up in named deals: Snowflake’s $200M multi-year partnership implies a $50–70M annual run rate from one customer, and Deloitte is deploying Claude across 470,000 employees.

For a typical enterprise running multiple production AI workloads, $500K–$2M per year is now the realistic floor. Fortune 100 is running $10M–$50M+, the most AI-intensive past $100M. The Gartner numbers point the same direction: just 28% of AI use cases fully meet ROI expectations and 20% fail outright, and 78% of IT leaders report material AI charges that didn’t show up in any procurement model.

The per-token chart is real. The invoice is also real. What closes the gap is behavior. Three behaviors specifically.

Models do more work per task. Reasoning models reason. Agentic loops loop. The prompt that used to consume 4K tokens now consumes 40K because the assistant explores, plans, second-guesses, and verifies. Some of that is valuable. Much of it is the model performing thoroughness in a way that costs you money. The Opus 4.6-to-4.7 jump I documented earlier: same task, same outcome, 2.9× more output tokens and 4.8× more cache reads.

Workflows fan out. A “single” task in a modern agentic system might trigger a planner, researcher, coder, reviewer, and summarizer. Each makes its own LLM calls over overlapping context. Gartner’s March 2026 analysis puts agentic workflows at 5–30× the token consumption of an equivalent chatbot ask. Stanford Digital Economy Lab’s April 2026 arXiv paper goes further: coding agents can consume 1,000× more tokens than equivalent chat completions. The agent isn’t more expensive because it’s smarter. It’s more expensive because it’s louder.

Context windows fill themselves. Long context is a feature in marketing and a bill in practice. In our own enterprise Claude.AI usage — 82,852 messages from 329 employees over 3.5 months, audited via the Anthropic Compliance API — the average request carried 366,000 input tokens, mostly from 10-turn conversations dragging accumulated history forward into every new turn. Most production systems I’ve audited show the same fingerprint: pipelines paying for context they aren’t actually using.

None of this is fraud and none of it is mysterious. It’s the natural consequence of letting probabilistic systems decide how much work to do on every call. The savings from cheaper tokens were real. They just got consumed by an order of magnitude more tokens per task.

What I’ve Found Shipping These Systems

The teams handling this well aren’t the ones cutting AI usage. They’re changing the shape of how they use it.

The pattern I keep coming back to: treat inference as the expensive step at the end of a mostly deterministic pipeline. Do the cheap, structured work in code. Reserve the model call for the part that actually requires judgment. Then bound the call hard — context budget, output budget, quality gate on whether the call even runs.

Two examples from systems I’ve been building illustrate this from different angles.

Example 1: Code-Intelligence — Search-First Architecture

The naive version of a code review tool is obvious: dump the changed files into Claude, ask for a review. That works. It also costs roughly $0.03 per operation, scales linearly with repo size, and produces a lot of review output you didn’t need. Claude.AI offers a code review service — it ended up costing us thousands a month for just a few repos. Augment Code offers a well-regarded one as a GitHub app, but charges a platform fee (a meaningful fraction of our Anthropic spend) just to connect.

So we built our own. It leverages a multimodal search/RAG/KG engine I’d already built, so this wasn’t from scratch.

The version I actually ship uses a multi-tier search pipeline with the LLM call at the very end:

Stage 1: Vector Search    (~$0.0002 per query, semantic similarity)
Stage 2: BM25 Reranking   (~$0.0001 per query, lexical relevance)
Stage 3: Static Analysis  (~$0.0001 per query, AST + symbol resolution)
Stage 4: Quality Gate     (free, deterministic threshold check)
Stage 5: Single LLM Call  (~$0.03 per call, only if Stages 1-4 passed)

The first four stages cost about $0.0004 combined. They do the bulk of the work: deciding what code is actually relevant, ranking it, pulling structural relationships, and deciding whether the result is even worth asking an LLM about.

Hard budget controls run through the whole pipeline:

# Budget enforcement, not aspiration
MAX_CONTEXT_FILES = 6          # cap on what we send to the model
MAX_REVIEW_WORDS = 500         # cap on what the model returns
RELEVANCE_FLOOR = 0.005        # quality gate before calling the model

if combined_relevance_score < RELEVANCE_FLOOR:
    # no point spending $0.03 to get a review of weakly-related code
    return SkipReason("below relevance floor")

context = select_top_n(ranked_results, MAX_CONTEXT_FILES)
review = llm.review(context, max_output_tokens=MAX_REVIEW_WORDS * 1.4)

The RELEVANCE_FLOOR check is the part to underline. A meaningful percentage of review requests in real codebases don’t justify an LLM call at all — the changes are mechanical, the related code trivial, or the search signal weak enough that whatever the model says will be hallucinated context. Refusing to spend $0.03 on those cases is where most of the savings come from.

Rough economics across a quarter of usage:

Approach Cost per operation LLM calls per 1K operations Direct “review the diff” ~$0.030 1,000 Search-first with gates ~$0.0034 average ~430

About 80% of the workflow logic ends up deterministic: search, ranking, static analysis, gating. The model handles the last 20% — judgment on curated context. Same outcome from the user’s perspective, roughly an order of magnitude cheaper, with more predictable failure modes because most of the pipeline is debuggable code rather than prompt behavior.

The limitation: this is more work than wiring up a single LLM call, and the gates are only as good as your search infrastructure. The payoff is on the cost and determinism side, not on speed of initial implementation.

Example 2: duetto-intelligence — Context Injection Instead of Replacement

The second pattern comes from duetto-intelligence, internal tooling I’ve been building against that same enterprise Claude.AI usage — 82,852 real employee messages over 3.5 months, not a thought experiment. The problem here isn’t “should we call the LLM at all.” It’s: given that our people are already routing structured-data questions through a $0.274-per-request multi-turn Sonnet conversation, what’s a cheaper path that doesn’t degrade the answer?

The audit data made the gap concrete. Average request: 366K input tokens, ten-turn conversation, $0.274 to Anthropic. The same query answered through a Haiku single-pass against deterministically-retrieved internal data: $0.0009. A 300:1 cost ratio on the slice of traffic about structured product knowledge, CRM/account prep, JIRA, people and org lookups.

Not all traffic. Somewhere in the 35–40% range based on classified samples. About half of remaining queries genuinely need full Sonnet or Opus reasoning — writing, debugging, free-form analysis — and shouldn’t be intercepted at all.

The framing matters, because it’s easy to mis-read this as “replace Claude with a smaller model.” It isn’t. duetto-intelligence acts as a context injection layer in front of the user-facing model. When a query has structured-data intent, we route a sub-query to DI, get back a bounded structured result, and inject that into the prompt the larger model sees. The expensive model still does the reasoning — it just stops being responsible for the deterministic data retrieval it’s bad at and expensive for.

The naive design for the routing layer looks like this:

1. Classify the user's intent             → LLM call (~150 tokens)
2. Plan which subsystems to query         → LLM call (~250 tokens)
3. Call subsystem A, summarize response   → LLM call (~300 tokens)
4. Call subsystem B, summarize response   → LLM call (~300 tokens)
5. Synthesize a final answer              → LLM call (~250 tokens)
                                          Total: ~1,250 tokens, 5 calls

Each call is plausible on its own. Together they’re a tax on every user interaction. Latency stacks linearly with calls, and any one of the five can hallucinate in a way that corrupts the rest of the chain.

The consolidated design replaces three steps with deterministic code:

1. Classify intent                  → LLM call  (~80 tokens, tight tagger prompt)
2. Fan-out to subsystems            → code      (0 tokens, intent → call map)
3. Consolidated synthesis           → LLM call  (~200 tokens, structured input)
                                       Total: ~280 tokens, 2 calls

The trick is the intent classifier. It produces a tag from a fixed vocabulary of 87 tags — revenue.query.ytd, forecast.compare.year_over_year, account.lookup.contact, and so on. Each tag maps deterministically to a set of downstream calls in plain Python. No LLM in the routing step. The model isn’t asked “what should we do?” It’s asked “what is the user asking about?” — a much smaller, bounded question.

We validated against a 50-query test corpus drawn directly from the compliance data — real questions people had asked the model in production. After tuning, 100% of those queries land on the fast path with no LLM call required for routing. That’s the proof the deterministic-discipline part holds at the boundary; the routing isn’t quietly falling back to a second model call to bail itself out.

Budget enforcement is explicit in every prompt template:

SOURCE_CHAR_BUDGET = 600    # per data source pulled into context
OUTPUT_TOKEN_BUDGET = 200   # cap on synthesis response
INTENT_TAG_VOCAB = load_intent_taxonomy()  # 87 tags, versioned

def synthesize(intent_tag: str, sources: list[Source]) -> str:
    trimmed = [s.truncate(SOURCE_CHAR_BUDGET) for s in sources]
    return llm.complete(
        prompt=template(intent_tag, trimmed),
        max_tokens=OUTPUT_TOKEN_BUDGET,
    )

The economics, against measured baselines rather than estimates: current Claude.AI Chat spend across the 329-user population runs $15,264 over 3.5 months. Roughly $4,360/month, driven by that $0.274-per-request multi-turn average. If DI intercepts the 35–40% of traffic that’s structured-retrieval underneath, projected savings come in around $1,500–1,700/month, or $18–20K/year on this single user population. The leverage isn’t from picking a cheaper model. It’s from refusing to pay Sonnet rates to answer questions a deterministic system already has the data for.

The intent vocabulary is the contract. New capability means a new tag, a new downstream mapping, a new prompt template. The model never has to invent structure on the fly. This is what people mean by “use the LLM to code solutions, not to solve problems directly” — the routing logic lives in code, the tagger is a thin call, the synthesis is bounded.

The source-character budget matters more than the output budget. The compliance audit confirmed it: production overspend is on the input side. 366K input tokens against a few hundred output. Models will happily consume whatever context you hand them. Trimming at the source — 600 characters per source, no exceptions — is how you keep per-call cost from drifting upward as the system gets more capable.

The limitation: this only works on the routable slice. The pattern isn’t “eliminate inference.” It’s “stop spending $0.274 to answer questions that have a structured answer at $0.0009.”

Why “Just Use a Cheaper Model” Doesn’t Save You

A reasonable objection: aren’t the cheaper, smaller models supposed to handle this? Why not route everything to Haiku or an open model and call it solved?

The pricing math seems to support it. The behavioral math doesn’t.

Two things go wrong when you swap a cheaper model into an unstructured workflow. Cheaper models are usually less efficient per task — more turns to converge, more exploration, more hallucination, which means more retries and verification calls. A model 5× cheaper per token can run 1.5–2× more expensive per completed task if your workflow lets it spin.

And the workflow itself is where most of the cost lives. The 5–30× multiplier is structural, not modal — it exists regardless of which model you point at it. Switching from Sonnet to Haiku inside an unbounded agent loop changes the per-token cost. It doesn’t change the loop.

Model choice is a 2–5× lever. Architecture choice is closer to an order-of-magnitude lever in the systems I’ve shipped — consistently larger than what model swaps deliver. Most teams are over-tuning the model selection and under-tuning the structure around it.

The default assumption — including from vendors with strong incentives to sell you more tokens — is that the answer to AI cost is buying inference more cleverly. The actual answer is using inference less, more deliberately, with hard bounds on what each call is allowed to do.

How I Think About Inference Now

Inference is a power tool. Not a default. You don’t reach for it when a search query, a regex, or a switch statement would do. You reach for it when you need probabilistic judgment over unstructured input. Every call you don’t make is the cheapest call.

Use it to code solutions, not to solve problems. The highest-leverage use of LLMs in my workflow is generating the deterministic code that then handles the workflow without further LLM calls. A model that writes you a 50-line classifier is more valuable than a model that acts as the classifier on every request forever. The first costs tokens once. The second costs tokens every transaction for the life of the system.

Wrap every call in a budget. Context budget on the input, token budget on the output, quality gate on whether the call runs at all. Treat the LLM call as you’d treat a paid API with rate limits and SLA penalties. Because it is.

Set specific ROI targets per call. “AI-assisted code review” is too coarse to optimize. “Reviewing files with relevance score > 0.005, capped at 6 files, returning 500 words” is something you can measure cost-per-outcome on. Even loose ROI math at the call level surfaces where you’re paying for theater.

Treat behavioral cost as the primary risk. Model rate cards will keep coming down. They are not your problem. Your problem is what your pipeline asks of the model and what the model decides to do once asked. That’s the line item that grew while the unit cost dropped 280×. That’s the shoe that just dropped.

What This Means If You’re Running an AI Budget

Three things to look at, in order of how much they’ll move the line item.

Audit the call graph, not the rate card. Pull a representative day of production traffic and trace the actual LLM calls per user task. Count them. Most teams find a handful of workflows producing the majority of cost, and most of those have 2–4 LLM calls that could be replaced by deterministic code. That’s the consolidated-design pattern from the duetto-intelligence example. 50–80% reductions are common when you actually look.

Put quality gates in front of inference. For any workflow where the LLM call is expensive and the input quality is variable, add a deterministic check that decides whether the call is worth making. That’s the search-first pattern from the code-intelligence example. The savings come from the calls you don’t make, which never show up on the invoice.

Set hard context budgets and enforce them in code. Per-source character limits, per-call token caps, no “just in case” context stuffing. The output budget gets attention because it’s visible. The input budget is usually where the actual money goes.

None of this requires changing models, switching providers, or making bets on the next frontier release. It’s architectural work inside the pipeline you already have — work the per-token price chart has been letting people defer.

The teams that do it over the next two quarters will look like they got a 5–10× cost improvement from “AI getting cheaper.” The teams that don’t will look like AI got 3–4× more expensive while everyone else’s costs fell. Same providers, same models, same rate cards. Different shoe.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Opus 4.6 vs 4.7: The Real Cost of Incremental AI Improvements — The first shoe, on per-task cost drift between model versions

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

My Duetto colleague Jake Becker is sharp. He’s been ahead of AI adoption on our team — experimenting early, pushing for new tools, staying current. Last week he messaged me on Slack: “I wish I had my own CTO Assistant. Like what you have.”

I paused.

I have one. I’ve had several, in different forms, going back months. But I hadn’t said anything about it publicly.

That gap — between an AI-forward person who knows the tools and a practitioner who actually has the thing — is what this piece is about.

TL;DR

• Off-the-shelf AI tools are capable but contextually blind. You still have to leave your domain to get help.

• Serious knowledge workers — writers, researchers, historians — have always built their own knowledge systems. Never trusted a vendor to hold their material.

• Coding is shifting toward what writing has always been: directing, curating, maintaining a living body of knowledge.

• Building your own AI memory and search layer is the new rite of passage. The tool you build IS the skill being developed.

• The latest versions of my own stack: trusty-memory and trusty-search — both open source, both installable today.

Everyone Wants One. Few Have One.

Jake’s comment revealed something I’d been taking for granted.

The major vendors are actively working on this — memory features, RAG pipelines, personalization layers. But those solutions are optimized for breadth, not for one person’s actual work across functions.

ChatGPT, Claude, Copilot — these are all capable. They’re also still contextually blind to you. You have to leave your environment, paste in context, explain your situation from scratch, then interpret the output back into your actual work. Vendors are working on it. But the solutions remain generic, and every session still starts from zero.

I wrote about this in March — Everyone Blamed Clawd Bot’s Execution. The Concept Was the Problem. The structural flaw of universal assistants isn’t fixable. They require you to leave your context to get help. What actually works is the opposite: your tools get assistant capabilities, and assistance comes to where your context lives.

Off-the-shelf tools haven’t solved this. They’ve gotten more powerful — better reasoning, longer context, faster inference — but they still don’t know your codebase, your decisions, your institutional history, your current sprint. They know a lot about the world, and very little about you.

Jake wanted my assistant. But what he actually wants is his assistant. The one that knows what he knows.

That’s a different problem entirely.

The job changed first.

Coding Is Becoming Writing

Practitioners feel it before analysts name it.

A few years ago, being a strong engineer meant writing a lot of code quickly and correctly. Today, with agentic AI coders at their disposal, the best engineers I watch spend their time directing, reviewing, specifying, and curating. The unit of work has moved up a level. Implementation is increasingly delegated. Judgment — about architecture, trade-offs, what to build and why — is the differentiator.

This is not what happens when automation replaces a skill. It’s what happens when a new discipline appears.

Writers have always worked this way. A novelist doesn’t produce words per minute as a primary metric. They produce decisions — what to say, in what order, with what emphasis. The words are the output of the decisions, not the work itself. What makes a writer productive over a career isn’t typing speed. It’s having a system: notes, research, accumulated material, patterns of thought that compound over years.

Some (typically senior) engineers are arriving at the same realization. Your value isn’t the code. It’s the judgment, the accumulated context, the knowledge of what was tried and why it failed. The question is whether that accumulates in your head alone — which doesn’t scale, and doesn’t survive a context switch — or whether it lives in a system.

Good engineers who learn to use their AI tools effectively generate better code — the stack amplifies judgment and accumulated knowledge. The gap isn’t between skilled and unskilled engineers in isolation; it’s between engineers who’ve wired their knowledge into their tools and those who haven’t. The knowledge is real in both cases. Only in one case does it compound.

The writers I’ve observed who sustain serious output over decades all have the same property: they know where things are. Their research is retrievable. Their earlier thinking is available to their current thinking. The system makes the person bigger than their working memory.

That’s the gap.

Writers Don’t Trust Vendors With Their Material

Niklas Luhmann, a German sociologist working in the 1950s, produced 70 books and nearly 400 articles over his career. I haven’t written a book yet — but I have published over 200 articles at HyperDev. Worth naming the parallel. He credits his output to his Zettelkasten — a slip-box system of 90,000 interconnected index cards, each with a unique identifier linking it to related thoughts. Not a filing cabinet. A network of ideas that got richer with every addition.

And it’s worth asking: is that so different from what an AI knowledge system does today? The Zettelkasten was an analog precursor to what trusty-memory and trusty-search do programmatically — indexing ideas, linking related thoughts, surfacing connections that wouldn’t otherwise be visible. Luhmann was doing manually what these tools do automatically. Same architecture. New substrate.

He didn’t use a vendor product. He built a system that reflected how he thought. The architecture of his Zettelkasten was itself an expression of his intellectual method.

This isn’t a historical quirk. It’s a pattern. Serious knowledge workers have always built their own systems — commonplace books, research archives, private wikis. The reason is structural: the schema you design reflects how you think. That’s not something a product gives you. A product gives everyone the same schema.

Andrej Karpathy pointed at something similar last month with his LLM Wiki gist. His framing: use LLMs not just to write code, but to build and maintain a personal knowledge base. “Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase.” Three folders, structured Markdown, a large context window. He concluded: “I think there is room here for an incredible new product.”

He’s right there’s room. I wrote about his framing in What’s In Your Second Brain? The product comment is where I’d push back. You can build tooling around the pattern. You can’t productize the schema. The schema is the moat — because it reflects how you think, not how a product manager thinks you think. The ones who get it aren’t waiting for a product.

The Lightsaber Rite of Passage

In Star Wars canon, a Padawan doesn’t receive a lightsaber. They build one.

The ritual is called the Gathering. Initiates travel alone to the Crystal Caves of Ilum. They have to find their kyber crystal — the crystal that’s attuned to them through the Force. The caves are shaped by the initiate’s own fears and insecurities. The crystal doesn’t go to the strongest or the fastest. It bonds with the person who confronts what’s in the way.

Then they build it themselves, guided by Professor Huyang.

You can’t buy this. You can’t inherit it. The construction is the training. The tool reflects the builder.

I’m not the first to reach for this metaphor in tech. But I think it lands differently now. Building your own AI memory and search layer isn’t just useful. It’s diagnostic. You can’t do it without confronting what you actually know, how you actually think, what deserves to persist and what doesn’t. The schema you design for your knowledge base is a statement about your mind.

The engineers I know who are operating at the highest level right now — CTOs, senior architects, tech leads at places moving fast — they all quietly roll their own. They don’t announce it. They just have it.

My Own Lineage

I’ve been building versions of this for months.

trusty-izzie was the first — a simple wrapper. Then ai-commander, a more structured approach to context management. Then open-mpm and claude-mpm, which was where I started thinking seriously about multi-agent orchestration. Then kuzu-memory, a graph-backed memory layer. Then mcp-vector-search, semantic search over my entire codebase.

Each iteration taught me something about what I actually needed. Not what I thought I needed. What the practice revealed.

This piece was drafted with a configured writing assistant — claude-mpm loaded with my publication style guide, my voice patterns, my article archive. That’s a saber too. Not a generic chat interface. A tool shaped around how I think and write, producing work I can actually publish rather than work I have to fix. The saber list keeps growing.

The latest two are the most capable tools I’ve built.

The Latest Sabers

trusty-memory is a machine-wide AI memory daemon written in Rust. It uses what I call the Memory Palace architecture — multiple named palaces, each for a different domain. Sub-5ms baseline retrieval on Apple Silicon. It runs as an MCP server for Claude Code, which means my assistant stores and retrieves memories automatically, across sessions, without me managing any of it explicitly.

cargo install trusty-memory

Available at crates.io/crates/trusty-memory.

trusty-search is a machine-wide hybrid code search daemon, also in Rust. Always-on, one install per machine. It combines BM25 lexical search with HNSW vector search (all-MiniLM-L6-v2 INT8) and a Knowledge Graph with 1-2 hop expansion, fused via Reciprocal Rank Fusion. It exposes an MCP server with 11 tools. Stdio and HTTP/SSE transports drop straight into Claude Code.

cargo install trusty-search

Available at crates.io/crates/trusty-search.

These tools, along a set of custom reporting pythons apps along with a custom Slack Bot I use to access the data remotely, comprise my digital brain.

These aren’t products I bought. These are tools I built, iterated, and use daily. They know my codebase the way a Zettelkasten knows a scholar’s intellectual territory — not because a vendor configured them, but because I did.

To be precise: trusty-memory and trusty-search are infrastructure utilities — the memory layer and the search layer. Building the actual assistant that uses them is a separate act of customization. That’s where the lightsaber metaphor completes: the kyber crystal is only part of it. The construction — what you build with the crystal — is the saber.

When Jake said he wished he had a CTO Assistant, this is what he was gesturing at. Not a prompt template. Not a workflow. A living knowledge layer that compounds.

The Right Question

Jake asked: “Can I get a CTO Assistant?”

That’s the wrong question. It assumes the thing is available off the shelf, and the task is finding and configuring it.

The right question is: “What would it take to build one that knows what I know?”

That question is harder. It requires confronting the shape of your knowledge, what’s worth persisting, how to structure retrieval. It’s uncomfortable in the same way the Crystal Caves are uncomfortable — not because the work is technically difficult, but because you have to be honest about what you actually have.

Not everyone needs to write Rust. The specific technology isn’t the point. The point is that the engineers asking the right question are already operating differently. They’re working like writers — maintaining a living body of knowledge, building systems that compound, treating their accumulated context as an asset rather than a liability.

Writers’ discipline has been creeping into engineering for a while. AI made it urgent.

If you’re waiting for a product to hand you the thing, you’re waiting for someone to build your lightsaber. It won’t be yours.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• What’s In Your Second Brain? — Karpathy’s LLM Wiki and the case for a compounding knowledge layer

• Everyone Blamed Clawd Bot’s Execution. The Concept Was the Problem. — Why universal assistants are architecturally broken

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Six months ago, I abandoned my Rust/Tauri-based writing app. Not because Tauri was bad—it wasn’t. Obsidian simply worked better for my needs. But that experience with Rust left an impression: the compiler was unlike anything I’d encountered. It didn’t just catch errors; it made entire classes of bugs impossible.

Then I built AI Commander for managing tmux sessions. Python would have been the obvious choice—async programming, process management, plenty of libraries. But I chose Rust, partly out of curiosity, partly because I needed rock-solid thread control. The result was revelation: code that compiled cleanly worked correctly, and performance was extraordinary.

Now I’m deep into replacing both mcp-vector-search and kuzu-memory with pure Rust implementations. My new trusty-search outperforms the Python predecessor by an order of magnitude, even though that version used compiled Python extensions. This isn’t an isolated case—it’s part of a fundamental shift happening in AI development.

Rust isn’t just another systems language. It’s becoming the infrastructure language of AI, and the reason has everything to do with letting the compiler handle the heavy lifting—especially as AI systems increasingly write their own code.

The Performance Reality Check

The performance gains are measurable and significant. Hugging Face’s tokenizers library, rewritten in Rust, delivers 10-100x speedups over pure Python implementations. Polars, the Rust-based DataFrame library, consistently outperforms pandas in data processing benchmarks. In embedded systems, Rust achieves 98% of C performance while eliminating memory safety issues entirely.

This isn’t just raw speed. Memory efficiency matters significantly in production AI systems. Where Python applications might consume gigabytes for large-scale data processing, equivalent Rust implementations often require 3-5x less memory. The difference compounds when deploying AI systems at scale.

This performance advantage stems from a fundamental philosophical difference in how Rust approaches the safety-speed trade-off.

When your AI system processes millions of sensor events per second in an autonomous vehicle, or makes split-second trading decisions with millions at stake, runtime failures aren’t just inconvenient—they’re catastrophic. Python’s flexibility, its greatest strength for research and experimentation, becomes a liability in these production environments.

The Compiler Does the Heavy Lifting

Here’s where Rust’s philosophy is revolutionary. Most languages force you to choose between safety and performance, between readable code and efficient code. Rust rejects this trade-off entirely through what it calls “zero-cost abstractions.”

The principle, borrowed from C++ but perfected in Rust, is elegantly simple: “What you don’t use, you don’t pay for. What you do use, you couldn’t hand code any better.”

Consider async programming, critical for AI systems handling multiple data streams. In Python, async operations carry runtime overhead—coroutine scheduling, context switching, memory allocation for tasks. In Rust, the compiler transforms your high-level async/await code into efficient state machines. No runtime scheduler, no hidden allocations, just bare-metal performance wrapped in readable syntax.

The genius is that the compiler absorbs the complexity. You write code that looks high-level and readable:

async fn process_ai_requests(stream: &mut DataStream) -> Result<Vec<Response>, Error> {
    let mut responses = Vec::new();

    while let Some(request) = stream.next().await {
        let response = ai_model.infer(request).await?;
        responses.push(response);
    }

    Ok(responses)
}

But the compiler generates assembly that rivals hand-optimized C. The ownership system prevents data races without locks. The borrow checker eliminates memory leaks without garbage collection. The type system catches logic errors that would be runtime failures in dynamic languages.

This is the heavy lifting: converting human-friendly abstractions into machine-efficient reality, all at compile time, with zero runtime cost.

This compiler-centric philosophy becomes even more critical as we enter an era where AI systems increasingly write their own code.

The AI Code Generation Protection Paradox

As AI-powered coding tools like Copilot, Claude Code, and Cursor become mainstream, a new challenge emerges: how do you ensure correctness when the code author might not fully understand what they’ve written?

When Claude or Copilot generates Python, Java, TypeScript, or C code, human oversight becomes the primary error-checking mechanism. Code reviewers need to catch type mismatches, memory leaks, race conditions, and logic errors that might not surface until production. As AI-generated code becomes more prevalent and complex, human review processes struggle to keep pace with the volume and complexity of generated code.

Think about it: an AI might generate a perfectly logical-looking concurrent data structure in Python that works flawlessly in single-threaded tests but creates subtle race conditions under load. A human reviewer might miss the threading implications entirely. The same AI generating Rust code? The compiler would reject it outright if the data sharing wasn’t provably safe.

Rust’s compiler fundamentally changes this dynamic. It doesn’t matter if the code was written by a human, an AI, or a collaboration between both—if it compiles, entire categories of dangerous bugs simply cannot exist. Memory safety violations? Caught at compile time. Data races in concurrent code? Impossible. Use-after-free errors? Eliminated by the ownership system.

This creates a fascinating inversion: Rust may be the best language for AI-generated code precisely because it doesn’t trust the programmer. While other languages rely on developer discipline and code review to catch errors, Rust enforces correctness through the type system and ownership model. The compiler becomes an automated, exhaustive code reviewer that never gets tired, never misses subtle bugs, and never waves through “probably fine” code.

As Microsoft research demonstrates, 70% of security vulnerabilities stem from memory safety issues that could be eliminated at compile time. When AI systems generate infrastructure code, this shifts from a productivity concern to an existential safety issue.

The compiler isn’t just doing heavy lifting for performance; it’s providing safety guarantees that scale beyond human oversight.

Memory Safety in the Age of AI

Rust’s creator, Graydon Hoare, designed the language after getting stuck in a broken elevator caused by software crashes. His goal was simple: write fast, small code without memory bugs. That mission has become critical as AI systems move from labs to production infrastructure.

In traditional software, memory safety issues mean patches and updates. In AI systems controlling physical infrastructure—autonomous vehicles, medical devices, financial trading systems—they can mean catastrophic failure.

Rust’s ownership model doesn’t just prevent crashes; it eliminates entire categories of bugs that plague concurrent AI workloads. While C++ applications in multi-threaded environments regularly exhibit race conditions during testing, Rust’s compile-time guarantees make such issues structurally impossible.

But here’s the crucial part: this safety comes at zero runtime cost. Unlike garbage-collected languages that trade performance for memory safety, Rust enforces safety through compile-time checks. Your production AI system runs with the performance of C but the safety guarantees that traditional systems languages can’t provide.

The Python-Rust Symbiosis

Here’s where the narrative gets interesting. The emerging trend isn’t Rust replacing Python—it’s Rust and Python working together. The PyO3 bridge, Rust-powered Python tools like Ruff and uv, and the growing number of “Python API, Rust engine” libraries show that the two languages are becoming symbiotic.

Python remains dominant for AI research and experimentation. The ecosystem is unmatched: PyTorch, TensorFlow, scikit-learn, Hugging Face Transformers. PyTorch’s 100,000+ GitHub stars reflect an ecosystem that won’t disappear overnight. When you’re prototyping models, analyzing data, or building internal tools, Python’s flexibility and ecosystem make it irreplaceable.

But production tells a different story. When performance, memory usage, and reliability matter—when you’re building the infrastructure that powers AI systems rather than the models themselves—Rust increasingly dominates.

The pattern emerging across tech giants is clear: Python for the interface layer, Rust for the infrastructure layer. Experimentation happens in Jupyter notebooks; production happens in compiled Rust binaries.

Looking Forward: Infrastructure vs. Experimentation

This division isn’t arbitrary—it reflects the maturation of AI from research curiosity to critical infrastructure. Research requires flexibility, rapid iteration, and access to cutting-edge libraries. Production requires reliability, performance, and safety guarantees.

Rust excels at infrastructure because the compiler handles the complexity of building robust systems. Memory safety, concurrency, error handling—all the concerns that make production systems hard to build correctly—are enforced by the type system rather than relying on developer discipline.

Consider the trajectory: Microsoft aims to eliminate C and C++ from their codebase by 2030, replacing it with Rust. The Rust Foundation officially positions the language as ideal for “ultra-reliable AI systems.” Major tech companies are increasingly choosing Rust for performance-critical AI infrastructure. These aren’t experimental projects—they’re billion-dollar bets on where AI infrastructure is heading.

The Compiler Advantage

What makes Rust uniquely suited for AI isn’t just performance or safety—it’s the fundamental approach of front-loading complexity into the compiler rather than dealing with it at runtime.

In AI systems, runtime failures are exponentially more expensive than compile-time failures. A training job that crashes after days of computation. An inference system that memory-leaks during peak traffic. A edge AI device that locks up in a safety-critical situation. These failures cascade through systems in ways that research environments rarely encounter.

Rust’s compiler is exhaustive in a way that benefits AI development specifically. It catches data races in concurrent model training. It prevents buffer overflows in tensor operations. It enforces proper error handling in distributed inference systems. It does this without performance overhead, without runtime monitoring, and without hoping that comprehensive testing caught every edge case.

The compiler becomes your co-pilot in building reliable AI infrastructure.

The Personal Proof

My own experience validates this thesis. AI Commander needed complex thread management for tmux session control—exactly the kind of concurrency that’s error-prone in traditional languages. Rust’s ownership system made data sharing across threads not just safe, but intuitive. The code that compiled was correct.

With trusty-search, the performance gains over mcp-vector-search weren’t just about Rust being “faster than Python.” They were about Rust enabling architectural choices—zero-copy string processing, efficient memory layouts, fine-grained concurrency control—that would be risky or impossible in garbage-collected languages.

These aren’t micro-optimizations. They’re fundamental design differences that become critical at scale.

The Real Limitations of Rust for AI

Before declaring Rust the future, it’s essential to acknowledge where it falls short compared to Python in AI contexts.

The learning curve is steep. Rust’s ownership model requires a fundamental mental shift that can slow initial development. Developers comfortable with garbage-collected languages often struggle with borrowing rules, especially when building complex data structures. The compiler’s strict requirements, while beneficial long-term, can feel obstructionist when prototyping.

The ecosystem gap remains significant. While Rust has impressive performance libraries, Python’s AI ecosystem is vast and mature. TensorFlow, PyTorch, scikit-learn, and thousands of specialized ML libraries have no Rust equivalents. Building production ML pipelines often requires library combinations that simply don’t exist in Rust.

Compilation time can kill iteration speed. Where Python allows instant feedback during development, Rust’s compilation process—especially for complex projects—can introduce friction that slows experimentation. This matters significantly in AI research where rapid iteration drives discovery.

Talent pool challenges are real. Finding experienced Rust developers is substantially harder than finding Python developers. The language’s complexity means onboarding takes longer, potentially impacting team velocity and project timelines.

Not every AI workload benefits from Rust’s strengths. Data analysis, model training with existing frameworks, and one-off scripts often don’t require the safety and performance guarantees Rust provides. Using Rust for these tasks can be overkill that reduces productivity without meaningful benefits.

These limitations explain why the Python-Rust symbiosis model makes sense: leverage each language where it excels rather than forcing one-size-fits-all solutions.

The Future is Compiled

As AI systems become infrastructure—as they move from experimental notebooks to production systems handling millions of users—the languages that power them will need to provide the guarantees that infrastructure requires.

Python will continue to dominate the experimental and interface layers. But the backbone, the high-performance inference systems, the real-time edge deployments, the safety-critical applications—these will increasingly be built in languages where the compiler, not the runtime, ensures correctness.

Rust represents a fundamental shift in how we approach building reliable systems: moving complexity from runtime to compile time, from testing to proving, from hoping to knowing.

This shift becomes critical as we enter the era of AI-generated code. When AI systems are writing the infrastructure that runs other AI systems, traditional human oversight breaks down. We need tools that can verify correctness automatically, at compile time, without relying on human reviewers to catch subtle but catastrophic errors.

The compiler does the heavy lifting so production systems can focus on their actual job: running AI workloads reliably, safely, and at scale. In a world where AI infrastructure is becoming as critical as power grids and financial networks—and where that infrastructure is increasingly written by AI itself—that guarantee isn’t just nice to have, it’s existential.

The age of agentic AI isn’t just about better models. It’s about building the reliable, high-performance infrastructure those models need to operate in the real world, written by AI systems that can’t be trusted to write safe code in traditional languages. And increasingly, that infrastructure is being built in Rust, one compile-time guarantee at a time.

Bob Matsuoka writes about the intersection of software engineering and AI at HyperDev. His latest Rust projects replace Python infrastructure with performance gains that would make even a compiler blush.

Leonard Shelby can’t form new memories. In Memento, after the attack, every fifteen minutes or so his short-term recall resets. He has a problem most engineers would recognize: a fast working scratch space and no durable storage. So he builds a system. Polaroids with handwritten captions on the back. A wall of pinned notes. Tattoos for the load-bearing facts — the ones he can’t afford to lose, can’t afford to misfile, can’t afford to look up wrong.

What he doesn’t do is try to remember everything. He doesn’t carry a thicker notebook. He builds infrastructure to find what matters, fast, when he needs it. The pocket holds what’s relevant right now. The system decides what makes it into the pocket.

I’ve been thinking about Leonard a lot this year, because the prevailing argument in my corner of the internet is that LLMs no longer need him. The pocket got bigger. Frontier models can hold a million tokens of context. Token prices fell. So just stuff the pocket. Skip the indexing. Skip the tattoos. Read everything.

I think that’s the wrong read of where we’re going.

The real argument, fairly stated

The “RAG is dead” position isn’t crazy. It goes roughly like this: a handful of frontier models — Gemini 1.5, Claude’s extended context tiers — can now handle contexts approaching 1M tokens without choking. Token prices have fallen to around $0.60 per million for several frontier models. Standing up a real retrieval pipeline — chunking strategy, embedding model, vector store, hybrid scoring, reranker, eval harness — is 40 to 80 engineering hours, easily. For a small, static corpus you query a few thousand times a month, just dumping the whole thing into context with prompt caching is simpler and arguably cheaper.

Claude Code is the existence proof people point at. It doesn’t ship a vector database. It uses ripgrep, file globs, and reads files into context. And it works pretty well. So why bother with the rest?

I’ll concede the envelope. For a 300K-token corpus, single tenant, low query volume, mostly static — yeah, stuff it. Cache it. Read it. Don’t build a search stack to feel sophisticated. That’s a real position and I don’t want to strawman it.

But that envelope isn’t where most of the work lives. And the argument the loud version makes — that grep replaces search, that long context replaces retrieval — confuses two different things; there are cracks in that argument.

The pocket problem

Here’s the first crack. “Lost in the middle.” Liu et al., 2023, TACL: when you stuff long contexts, model performance forms a U-shape. Strong recall at the beginning, strong recall at the end, degraded recall in the middle. The 2026-gen frontier models have improved on this — early evals suggest the curve is flatter — but it’s not gone.

So the first thing the bigger pocket buys you is the ability to put a lot of important stuff exactly where the model is most likely to underweight it. You can’t reorder a 700K-token dump by relevance unless you’ve already done retrieval. Which means the question doesn’t go away — it just hides.

Second crack: cost. RAG-style queries — retrieve relevant chunks, feed 8–16K of context — run around $0.005–$0.008 per request at $0.60/million input tokens, when you count the full prompt. A naive full-context pass against a 500K-token corpus runs $0.30 in input tokens alone, before output. That’s a 40–60x ratio at list price, and it widens if you have high cache-miss rates or large output windows. At 50K daily queries — not enormous, this is a mid-sized internal tool — you’re choosing between a few hundred dollars a day and tens of thousands. That’s not a rounding error you absorb. That’s a re-platforming decision that wakes someone up.

Third crack, and this is the one that actually makes me grumpy. Grep is not search. Grep is lexical pattern matching. Searching for auth_token finds the string auth_token. It finds it in commented-out dead code from 2023. It finds it in test fixtures with hardcoded sample values. It finds it in a vendor library you don’t even own. It misses the function called refresh_session_credential that does what you’re actually looking for, because the words are different. Vector search finds that function. Grep doesn’t. They’re not the same tool. Pretending they are is how you ship the wrong fix at 2 AM.

When people say “Claude Code uses grep, so search is over,” they’re describing a tool optimized for a bounded, file-system-structured corpus where the author controls the index shape. That’s a legitimate architecture for that problem. It does not generalize to “ten million product images” or “a 30-year codebase across 800 repos” or “tenant-scoped documents under per-row access control.” Long context can’t replace a knowledge graph, either. You can serialize a graph into text — flatten the edges, encode the relationships — but then you’ve lost typed edges and efficient query-time traversal. You can’t join across it. You can’t reason about edge types. You’d need to flatten the graph to feed it in, at which point you’ve discarded the structure that made it useful.

With me so far? The argument isn’t “long context is bad.” Long context is great. The argument is that long context is a consumer of retrieval, not a replacement for it.

What good retrieval actually looks like

A modern retrieval stack, the kind that holds up under real query volume, isn’t one thing. It’s a layered pipeline.

Lexical first — BM25, TF-IDF, the unfashionable stuff. It’s still excellent at exact tokens, identifiers, error messages, version numbers. The things vector embeddings smudge.

Dense second — vector search over embeddings. all-MiniLM-L6-v2 is the workhorse at 384 dimensions. Catches semantic neighbors, paraphrases, the function-named-different-but-doing-the-same-thing case.

Fuse them — Reciprocal Rank Fusion at k=60 is the standard, and it’s standard because it works. You don’t pick BM25 or vector. You merge their rank lists.

Rerank — a cross-encoder over the top-k from RRF. Slower per-pair, but you only run it on the candidates that made it through the cheap layers.

The numbers from a 2026 benchmark pass I was reading — “From BM25 to Corrective RAG,” April 2026 — make the layered case cleanly:

• Dense vectors alone: Recall@5 of 0.587

• BM25 alone: 0.644

• Hybrid via RRF: 0.695

• Hybrid + neural reranker: 0.816

That last number is 39% better than dense-only. It’s also 17% better than RRF without the reranker. The layers compound. In systems that need both recall and precision at scale, you usually want all four layers. And — this part matters — none of them are replaced by a bigger context window. You still have to pick what goes into the pocket.

There’s a piece on top of this, too: intent. A query like “where is process_payment defined” is not the same shape as “how does the refund flow handle partial captures.” The first is a definition lookup; BM25 should dominate, you want exact identifier matching, you don’t want semantic neighbors fuzzing the result. The second is conceptual; dense embeddings should dominate, you want the chunk that explains the flow even if the words don’t match. Treating those queries identically is how you get plausible-but-wrong answers.

The EMNLP 2024 “Self-Route” paper made a related point I keep coming back to: let the model itself decide whether a query needs retrieval or full-context, based on complexity. They got better accuracy AND lower compute cost. The two approaches aren’t enemies. They’re complements with different cost/quality envelopes, and a working system uses both.

(A note on scope: I’ve been using “search” and “memory retrieval” somewhat interchangeably here, which isn’t quite right. They’re the same pipeline mechanics — retrieve, rank, inject — but different problems. Memory retrieval has a temporal model search doesn’t: recency matters, decay matters, and the source is agent history rather than a document corpus. Memory is also primarily graph-based, because what you’re trying to recover is relationships and context across time, not just chunks of text. For the purposes of this argument — context window vs. retrieval discipline — the distinction doesn’t change anything. But it’s worth naming.)

What I’m building

I have two projects in this space. I’ll be specific about what each one is and isn’t, because the field is full of demos that don’t survive contact with real corpora.

mcp-vector-search is the older one. Python, per-project, designed to live next to a codebase as an MCP server. The retrieval core is hybrid: BM25 plus HNSW over MiniLM embeddings, with knowledge-graph expansion built from tree-sitter AST parsing — function definitions, call edges, import edges, type relationships. Fifteen-plus edge types in the graph. Cross-encoder reranking on top of RRF. MMR for diversity so you don’t get five near-duplicates in the top results. Temporal decay weighted by git blame age. Cyclomatic complexity factored into ranking, because dense, gnarly code is more often what you’re looking for than the trivial wrappers around it.

trusty-search is what I’m building toward. Rust daemon, machine-wide rather than per-project, multi-tenant across all my work. Same retrieval core — RRF at k=60, MiniLM embeddings, BM25 plus HNSW — but with intent classification on the front. Queries get tagged Definition, Usage, Conceptual, or BugDebt, and each intent has pre-tuned alpha/beta weights between lexical and dense. Sub-10ms p50 on warm queries.

I’ll be honest: trusty-search is early-stage. Some of the features I just listed are scaffolding with stubs underneath. The intent classifier is rule-based at the moment and needs to become a small model. The two tools are complementary rather than competing for now — mcp-vector-search is the deeper analysis layer, trusty-search is the fast facts store you hit constantly during a session — though the long-term plan is for trusty-search to replace mcp-vector-search.

The reason I’m building both is that I keep running into the same wall: I can give an agent a million-token window and it will still ask the wrong question of the wrong file. The bottleneck is not how much I can stuff in. The bottleneck is which 8K of the 800K matters for this query. That’s a search problem. It’s been a search problem for sixty years. It’s still a search problem.

Back to Leonard

The reason Leonard works as an analogy isn’t the amnesia. It’s the discipline. He decides what makes it onto a polaroid. He decides what gets tattooed and what stays loose. He writes “DON’T BELIEVE HIS LIES” on the back of a photo because he won’t have the context to evaluate trustworthiness later — so he encodes the conclusion now, into a system he’ll find when he needs it.

The context window is the pocket. Whatever Leonard pulls out and looks at right now. It can be enormous. It can be cached. It can be cheap. None of that decides what goes in.

Retrieval is the tattoo on his wrist. The polaroid pinned to the wall. The system that decides which fact survives, which one is one query away, which one is buried. The job of that system is exactly the job that doesn’t go away when the pocket gets bigger.

The question was never “do I need RAG?” That framing turned a design discipline into a vendor category and made it easy to dismiss. The question is the one Leonard asks every morning: what do I actually need to find, and have I built the thing that finds it? At a million tokens, you’re not freed from that question. You’ve just made it more expensive to answer incorrectly.

It feels crazy that I wrote this only a year ago…

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Context Memory and Search: The Secrets to Effective Agentic Work — Why context management, memory, and retrieval are the three pillars of effective agentic systems

• What’s In Your Second Brain? — Tooling for the modern CTO: how to build external memory that actually works

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

TL;DR: I spent years avoiding AWS due to its overwhelming complexity, preferring the simplicity of Vercel and composable stacks. Then Claude MPM changed how I work with AWS. Now I’m running sophisticated multi-service AWS deployments with GPU instances, comprehensive monitoring, and infrastructure-as-code—all managed through AI-assisted tooling. The lesson: agentic approaches transform ops workflows just as profoundly as they do coding.

Remember when AWS felt like digital quicksand? Every innocent “let me just deploy this simple app” spiraled into an afternoon lost in IAM policies, VPC configurations, and security group rules that made no sense. I’d start with what should be a five-minute deployment and emerge three hours later, bleary-eyed, with a working application and absolutely no confidence I could recreate the process.

That’s why I became a Vercel evangelist. One git push, automatic deployments, zero configuration. The developer experience was everything AWS wasn’t: predictable, fast, and actually enjoyable. For most projects, this composable stack approach—Vercel for frontend, managed databases, serverless functions where needed—delivered exactly the right balance of power and simplicity.

But somewhere along the way, that changed.

The Claude Code/MPM Turning Point

Today I’m running the kind of infrastructure I would have delegated to an ops team a year ago. GPU instances for ML workloads. Multi-AZ deployments across six subnets. Sophisticated monitoring pipelines that integrate CloudWatch metrics, Cost Explorer analysis, and automated GitHub issue creation. Terraform managing multi-account infrastructure with cross-service dependencies.

The difference? I don’t even need AWS’s Q assistant. I have something better: purpose-built AWS skills in Claude MPM that handle service deployment, infrastructure analysis, and operational workflows.

This transformation illustrates something crucial about the agentic revolution: it’s not just changing how we write code. It’s fundamentally altering how we approach operational complexity.

The Infrastructure Reality Check

Let me show you what I mean with real numbers. Here’s what I’m actually running across three active projects to support internal tools:

CloudWatch Reporting Service (Serverless):

• 12 Lambda functions handling health checks, metrics aggregation, and MCP server functionality

• API Gateway HTTP API with sophisticated CORS and authentication

• DynamoDB tables for state management and external directory lookups

• SNS/SQS for alerting and dead letter queue handling

• Direct Bedrock integration with Claude 4.5 Haiku for automated analysis

• CloudWatch Events scheduling 5-minute monitoring cycles

• Secrets Manager for GitHub app credentials and API keys

Code Intelligence Platform (Compute):

• Two EC2 instances: t3.xlarge for web serving, g4dn.xlarge for GPU-accelerated indexing

• EBS volumes with gp3 storage and custom IOPS configuration

• VPC with six subnets across availability zones

• Application Load Balancer with Route53 DNS and ACM certificates

• EFS for shared file storage across instances

• CloudWatch Synthetics for endpoint monitoring

• Lambda-based Slack notifications triggered by SNS topics

Enterprise Infrastructure (Multi-Account):

• Terragrunt-managed infrastructure across production and staging accounts

• S3 backend for Terraform state with DynamoDB locking

• Cross-account IAM policies and service integration

• Integration with external providers (Sentry, GitHub, Kubernetes clusters)

A year ago, this list would have been my personal infrastructure horror story. Today, it’s Tuesday.

What Changed

The transformation wasn’t gradual. It was a step function that happened when I realized AI assistants could handle the cognitive overhead that makes AWS painful.

Before: AWS documentation as archaeological expedition. Digging through service guides, trying to understand the relationship between VPC route tables and security groups, wondering if I need an Internet Gateway or a NAT Gateway or both. Every deployment felt like solving a puzzle where half the pieces were hidden.

After: Natural language infrastructure requests. “Set up monitoring for this Lambda function with alerting to Slack” becomes a series of guided steps where the AI handles the AWS-specific implementation details while I focus on the business requirements.

The key insight is that AWS’s complexity isn’t inherently bad—it’s just cognitively expensive. When you remove that cognitive load through AI assistance, you can appreciate what all those services actually enable.

Take my monitoring setup. Previously, I would have settled for basic uptime checks because configuring comprehensive CloudWatch metrics, Cost Explorer integration, and automated issue creation felt like a weekend project. With claude-mpm AWS skills, it became an hour of guided configuration that resulted in production-grade observability.

Or consider the GPU instance management. The g4dn.xlarge for ML indexing runs sophisticated start/stop automation, monitors for runaway processes, and automatically scales EBS volumes based on data requirements. Setting this up manually would have required deep expertise in EC2 lifecycle management, CloudWatch alarms, and Lambda automation. With AI assistance, I focused on defining the business logic while the tooling handled the AWS implementation.

The DX Philosophy Still Matters

None of this means AWS wins every comparison. Vercel’s developer experience remains superior for the use cases it targets. When I need to ship a marketing site or a straightforward web application, git push deployment still beats any infrastructure-as-code workflow.

The difference is recognizing when complexity serves a purpose versus when it’s just complexity. Vercel abstracts away infrastructure concerns because most web applications don’t need granular control over compute, storage, and networking. But when you’re building systems that do need that control—ML pipelines, high-throughput data processing, complex service topologies—AWS’s granularity becomes valuable rather than burdensome.

AI assistance changes the cost-benefit calculation. When configuring VPC networking takes 20 minutes of guided conversation instead of three hours of documentation archaeology, you can choose AWS for projects where you previously would have compromised on requirements to avoid operational overhead.

But there’s an honest accounting problem buried in that logic. Claude Code isn’t free. API costs, subscription fees—if you’re running significant conversation volume to figure out your infrastructure, you’re spending real money. At some point, you’re spending more on AI assistance than a Vercel seat would cost. The “AWS saves money at scale” argument gets complicated fast when you factor in the cognitive tooling required to get there.

So let me be direct about where each wins. Pure self-service developer experience—one engineer, a web app, ship it fast? Vercel, and it’s not particularly close. The moment you need an AI co-pilot to configure your infrastructure, you’ve added a cost layer that Vercel eliminates by design. But complex multi-service deployments—ML pipelines, GPU compute alongside serverless, multi-account Terraform, monitoring infrastructure that spans six services—those don’t live in Vercel’s world. That’s where the math inverts and AWS earns its complexity premium.

The Broader Implications

This transformation reveals something important about how agentic approaches will reshape technology adoption. We’re not just making individual tasks more efficient—we’re changing which categories of tools become accessible to developers.

I see this pattern across the infrastructure stack. Database migrations and performance tuning become approachable when AI translates business requirements into specific configuration changes. Kubernetes stops being “too complex for small teams” when you can describe desired behavior in natural language and get helm charts and operators generated automatically. IAM policies, security groups, and compliance frameworks become manageable when AI can analyze your application requirements and generate least-privilege configurations.

These tools were always powerful. They were just too expensive to learn and maintain for many use cases. AI assistance changes that economics.

Where This Goes Next

We’re still early in AI-assisted infrastructure management. Today’s tooling handles deployment and configuration. Cost optimization, security posture, performance tuning—those are coming. Full system architecture from high-level requirements is probably further out than the hype suggests, but it’s not science fiction.

But the fundamental lesson remains: complexity isn’t always the enemy. Sometimes it’s just temporarily inaccessible. When AI removes the accessibility barriers, you can choose tools based on their actual capabilities rather than their learning curves.

For now, I’m running infrastructure that would have seemed impossible to manage solo a year ago. And it’s kind of fun.

AWS might still feel like quicksand sometimes. But now I have a helicopter.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

The modern CTO toolkit isn’t just apps and coding tools. The real differentiator is a custom knowledge layer — databases, search indices, memory graphs, behavioral instructions that compound over time. No product gives you this. You build it.

Andrej Karpathy gestured at something similar last month when he posted a GitHub Gist he called “LLM Wiki.” His framing: stop using LLMs just to write code, use them to build and maintain a personal knowledge base instead. “Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase.” Three folders, structured Markdown, a large context window, a few Python scripts. No RAG, no vector database. He concluded with: “I think there is room here for an incredible new product.”

He’s right that there’s room. But the product comment is where I’d push back, and I’ll get to that. What Karpathy is describing isn’t a note-taking system. It’s a personal operational knowledge layer. For CTOs specifically, that layer needs to be broader than a personal wiki — it needs live organizational data, agent-connected search, and context that persists across months of decisions. No app hands you that.

TL;DR

• Karpathy’s LLM Wiki shows the direction: LLMs as knowledge compilers, not just code generators

• A modern CTO’s “second brain” is more than PKM — it’s live databases, custom agents, and contextual search across organizational data

• When I joined Duetto as CTO, my custom toolkit let me synthesize a 150-person R&D org in weeks instead of months

• The power isn’t Obsidian. It’s what you connect to it — MCP servers, search indices, knowledge graphs

• Productizing this is theoretically possible and practically very hard, because the schema is the moat

The toolkit article got it half right

In What’s In My Toolkit: Claude Code and Family, I wrote about vanilla Claude Code’s core limitations: context evaporates, code search is keyword-based, memory doesn’t persist, execution is single-threaded. The tools I built — Claude MPM, mcp-vector-search, kuzu-memory — address each of those gaps.

But that article was about coding workflows. The real story is broader.

The same architecture that makes a coding session more effective — persistent memory, semantic search, specialized agents pulling from structured data — turns out to be extraordinarily useful for executive work. Understanding an organization, tracking decisions over time, querying data across systems, maintaining context across months of meetings and analysis. The toolkit I built for software development became the toolkit I used to onboard as a CTO.

That onboarding story is documented in detail elsewhere. Short version: I pointed a multi-agent framework at GitHub, JIRA, Slack, Confluence, and budget spreadsheets, and synthesized a 150-person R&D organization in the weeks before my start date. The difference between doing that with a chat interface versus a CLI-based orchestration layer with parallel agents and persistent memory wasn’t 2x or 5x. It was closer to 10x.

But the onboarding was just the starting gun. The second brain I assembled keeps compounding.

What’s actually in my second brain

Let me be specific. Because when people (now) hear “second brain” they usually think Obsidian vaults with color-coded tags and pretty Markdown files. That’s part of it. It’s the surface layer.

The actual power comes from what’s underneath.

The memory layer

kuzu-memory is a KuzuDB-backed knowledge graph that persists across every AI session. It stores learnings from conversations, code commits, decisions, patterns. When I start a new Claude Code session on a problem I’ve touched before, the context isn’t blank — it’s enriched with what was learned the last time.

This is the thing people underestimate. A project-specific memory that accumulates over months of work develops a kind of organizational intelligence you can’t replicate in a single conversation. It knows why a particular architectural decision was made. It knows that a vendor was evaluated and found lacking. It knows the terminology your team uses internally that differs from industry standard.

KuzuDB isn’t a product choice for its own sake — it’s graph-native, which means it handles relationships well. The connections between people, systems, decisions, and code are as important as the facts themselves.

The search layer

mcp-vector-search provides semantic search across all project files. Not keyword search — semantic search with AST parsing. When I ask “where is the analysis I did on contractor productivity last quarter,” it finds it even if the document never uses those exact words.

At Duetto, this covers everything in my CTO project: architecture records, meeting notes pulled from Granola, emails I’ve synthesized, analysis documents, planning artifacts. Months of accumulated context, all searchable in seconds. The underlying code intelligence for the engineering organization runs as a separate service — mcp-vector-search is for my working knowledge, not the codebase itself.

The databases

My CTO project has three:

• cto.db — SQLite. Work classification, people analysis, contributor data, commit history. The operational database for running analyses and reports.

• analytics.duckdb — DuckDB. OLAP queries and analytics. When I need to slice engineering output data in different ways or run something that would be painful in SQLite, it goes here.

• duetto_knowledge.db — The RAG-queryable knowledge base backing a Flask web app for interactive exploration.

These aren’t a product I bought. They’re a schema I designed, built incrementally, and own completely. The schema reflects how I think about the organization, which is precisely why it’s useful.

The connectors

gworkspace-mcp handles Drive, Docs, Sheets, Gmail, Calendar, and more. I wrote my own rather than using the off-the-shelf options — Google’s first-party integration and Anthropic’s default both have significant tool coverage gaps. Mine exposes substantially more of the Workspace API surface and integrates transparently with Claude MPM, so agents can use Google Workspace tools without any special configuration at the call site.

Beyond Workspace: Notion API for product specs and planning documents. Extraction scripts for JIRA, Confluence, Slack, Datadog, and AWS. Each system outputs to as raw data, which feeds analysis pipelines that generate reports stored in a project directory.

For company-wide memory, two more tools: duetto-memory and duetto-directory. These handle shared organizational context — information that needs to flow between tools and across team members rather than staying in a single session. Memory persists within our VPC, encrypted to individual users’ OAuth keys. Not even our own IT has access to it. Context shared from Claude Code shows up in Claude.ai, and vice versa, without any manual sync.

The entire flow is queryable. From a single Claude session, I can ask about budget trends, team velocity, specific architectural decisions, or what a particular engineer has been working on for the last three months. Because it’s all in the same context-addressable system.

Obsidian as the front door

Yes, I use Obsidian. But it’s a front door, not the building. The vault holds my personal notes, research captures, and synthesized analysis. The Obsidian Web Clipper feeds raw material into the knowledge pipeline. Templates enforce consistent structure.

Karpathy’s insight about Obsidian as IDE is right in the narrow sense: it’s the interface you use to read and organize. But the interesting work happens outside it — in the databases, the agents, the search indices, the custom scripts.

CLAUDE.md files everywhere

The context layer isn’t just data. It’s also behavioral instructions.

Every major directory in my project has a CLAUDE.md. The root CTO project one is 400 lines of conventions, routing logic, document lifecycle rules, and architectural decisions. Every subdirectory has a more focused version. Every specialized agent has its own constraints.

These files are my second brain’s schema, expressed as instructions rather than data. A single routing rule — “if the prompt mentions meeting notes, save to projects/meetings/2026-W##/“ — sounds trivial. But it means twelve months of meeting notes accumulate in consistent, queryable locations rather than wherever an agent happened to save them. Multiply that by forty routing rules across fifteen subdirectories, and the entire corpus becomes navigable. The CLAUDE.md files are what make the databases useful. Without them, the data is just data.

Karpathy put it well: “You share the schema, not the code.” The schema is the valuable part. The schema is what compounds.

My schema took months to build. It will keep getting better. No product ships with the right schema for my organization, because no product knows what I know about how Duetto’s R&D works.

The productization question

Karpathy said there’s room for an incredible product. He’s not wrong about the gap. He might be wrong about the solution.

The structural problems with productizing a second brain:

Context compounds, products don’t. My system gets smarter with every commit, meeting, and conversation. A SaaS product serves thousands of customers and maintains no one’s specific context. The more I use my system, the wider the gap between it and any off-the-shelf alternative.

The schema is the moat. My knowledge architecture reflects how I think about engineering organizations. Someone else’s knowledge architecture would be different. Products that force their schema on you — and every product does — are imposing someone else’s way of thinking on your problem. That friction is small at first and grows over time.

Privacy is structural, not incidental. My databases contain org structures, salary data, performance patterns, vendor negotiations. Routing that through third-party infrastructure creates risk that’s practically impossible to contain. When I built duetto-memory for enterprise use, the entire stack stays within our VPC, with memories encrypted to individual users’ OAuth keys. Not even IT can read them. That level of isolation is nearly impossible to provide as a multi-tenant SaaS.

Some layers could be productized — the infrastructure, not the intelligence. A well-designed memory MCP with sensible defaults. Semantic search that works without configuration. Privacy-preserving graph storage you don’t have to host yourself. The plumbing.

The schema, the decisions, and the accumulated context can’t be productized. Those are yours. That’s the point — and it’s also why the product gap Karpathy sees will remain open even after someone tries to fill it.

Can anyone do this?

There’s an access problem here, and I’d be dishonest not to acknowledge it.

Building what I’ve described requires knowing Python well enough to write extraction scripts, understanding enough about graph databases to design a schema, and being comfortable with CLI-based tooling and MCP server configuration. Not every CTO has that background. Not every technical leader wants to spend weekends building personal infrastructure.

The irony is that the people who most need better organizational intelligence — executives without deep engineering backgrounds — are least equipped to build these systems. And the people who are most capable of building them are often less interested in the executive problems the systems could solve.

Tiago Forte, who wrote Building a Second Brain, has been making this point for years. His PARA method and CODE framework are accessibility layers — ways to make the underlying ideas approachable without requiring you to build a graph database. The methodology is sound. But it was designed for knowledge workers, not for CTOs running engineering organizations who need live data pipelines, not filing systems. Well-designed for whom?

Karpathy’s LLM Wiki is explicitly a system for someone comfortable writing Python and working with file systems. His Gist has code in it. That’s a feature for his audience and a barrier for everyone else.

What I’d watch for

A few trends that will determine whether this remains a DIY space or gets productized:

MCP as infrastructure. The Model Context Protocol creates a standard interface for exactly this kind of knowledge infrastructure. Memory servers, search servers, database connectors — they all expose the same interface to any compatible AI client. The ecosystem is growing fast. As more MCP servers mature, the configuration burden drops.

Searchable context beats raw window size. Karpathy argues for plain Markdown because ~400K words fit in a modern context window. That’s true, and the window is getting larger. But the more important shift is that structured, searchable context doesn’t have a ceiling. A well-organized knowledge base that spans years of meetings, decisions, and analysis delivers more than any single context window can hold — and the value scales with the quality of the organization, not the size of the model.

Local model quality. Karpathy runs Anthropic agents via Claude Code. But local model quality is improving fast. A system that uses the cloud API for synthesis and queries but runs a local model for routine indexing tasks would be significantly cheaper and more private. Not ready yet. Getting closer.

The product Karpathy thinks exists — if it gets built — probably looks like a well-designed local MCP server with clean configuration, sensible defaults, and a plugin ecosystem for connectors. Not a SaaS. Not a cloud database. Something you install and own.

The people who need it most will have already built their own before any product ships. And in the process of building it, they’ll have accumulated the one thing no product can give them: months of their own operational context, organized the way their own mind works.

That’s not a consolation prize. That’s the whole point.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• What’s In My Toolkit: Claude Code and Family — The coding layer of the stack

• I Built a Coding Tool. Then I Used It to Onboard as CTO — Applying agent orchestration to organizational analysis

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

So if you’ve been using Claude Code and noticed it felt... off... you weren’t imagining it.

Anthropic published a full breakdown yesterday and it’s actually three separate bugs that compounded into what looked like one big degradation. The developer community was right to be concerned, and the evidence they collected was instrumental in getting this fixed.

Here’s what actually happened:

1. They silently downgraded reasoning effort (March 4)

They switched Claude Code’s default from high to medium reasoning to reduce latency. Users noticed immediately. They reverted it on April 7.

Classic “we know better than users” move that backfired. From their postmortem:

“This was the wrong tradeoff. We reverted this change on April 7 after users told us they’d prefer to default to higher intelligence and opt into lower effort for simple tasks.”

The UI was appearing frozen in high reasoning mode, so they made an executive decision to sacrifice quality for speed. Developers immediately felt the difference and pushed back hard.

2. A caching bug made Claude forget its own reasoning (March 26)

This one was particularly insidious. They tried to optimize memory for idle sessions—clear old thinking after an hour of inactivity to speed up resumption. Sounds reasonable, right?

A bug caused it to wipe Claude’s reasoning history on EVERY turn for the rest of a session, not just once. So Claude kept executing tasks while literally forgetting why it made the decisions it did.

The cascading effects were brutal:

• Every request became a cache miss

• Usage limits drained faster than expected

• Claude appeared “forgetful and repetitive”

• Sessions felt like they were constantly resetting

3. A system prompt change capped responses at 25 words between tool calls (April 16)

They added this seemingly innocent instruction: “keep text between tool calls to 25 words. Keep final responses to 100 words.”

It caused a measurable 3% drop in coding quality across both Opus 4.6 and 4.7. They caught this through ablation testing—removing the instruction and measuring the performance difference.

Reverted April 20.

The community evidence was damning

While Anthropic was investigating internally, the developer community was building their own case. Stella Laurenzo from AMD’s AI group published the most comprehensive analysis—6,852 Claude Code sessions and over 234,000 tool calls.

Her findings:

• Median visible thinking length collapsed 73% (2,200 → 600 characters)

• API calls per task spiked up to 80x from February to March

• Claude was choosing “simplest fix” over correct solutions

BridgeMind’s testing showed Opus 4.6 accuracy dropping from 83.3% to 68.3%.

The data was undeniable.

The perfect storm effect

Here’s what made this particularly hard to pin down: all three bugs affected different traffic slices on different schedules. The combined effect looked like random, inconsistent degradation.

Hard to reproduce internally. Hard for users to isolate the exact cause. It just felt... wrong.

Some sessions hit the reasoning downgrade. Others hit the caching bug. The unlucky ones hit multiple issues simultaneously. No wonder it seemed like Claude was having random bad days.

What this reveals about AI product development

This postmortem is actually refreshing in its transparency. Most AI companies would have quietly fixed the issues and moved on. Anthropic owned the mistakes publicly.

But it also highlights a fundamental tension in AI product development: users often prefer maximum capability over convenience optimizations. The reasoning effort downgrade was done for user experience (reduce perceived latency), but developers would rather wait for better output.

The lesson: don’t optimize away what users value most without asking them first.

All fixed now (v2.1.116)

As of April 20, all three issues are resolved:

• Default reasoning is now “xhigh” for Opus 4.7, “high” for others

• Caching bug squashed

• Verbosity limits removed

• Usage limits reset for all subscribers

Anthropic is also committing to more transparency going forward with a dedicated @ClaudeDevs account for deeper technical communication with developers.

The community was right to raise hell about this. And Anthropic’s response—full transparency with concrete fixes—sets a good precedent for how AI companies should handle quality regressions.

Your coding assistant is back to full strength.

Independent Validation

The technical analysis backing this story comes from multiple independent sources. Stella Laurenzo’s comprehensive audit of 6,852 sessions provided the quantitative foundation. BridgeMind’s testing offered controlled benchmark data. These weren’t isolated complaints—they were systematic investigations with reproducible findings.

When a company publishes a detailed postmortem acknowledging specific engineering decisions that degraded their product, and that postmortem aligns with community-gathered evidence, we’re seeing transparency in action. The developer community did the work to document the problems. Anthropic owned the solutions.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders