I watched a Claude Code session run a multi-workstream engineering job last week, and I caught myself doing something I do not usually do with these tools: I stopped intervening. The orchestrator — the PM layer, running claude-opus-4-8 — handled the kind of coordination work I would expect from a team lead who has been on the project a year, decomposing it, predicting conflicts, routing to the right specialist. Not a faster coder. A technical lead.

This time the session was recorded turn by turn, with per-token cost telemetry on every move. So these are not impressions. Below are the specific behaviors I saw, described as precisely as the instrumentation allows, each one carrying its timestamp, the orchestrator’s own words, and what it cost. The conclusion I’ll leave to you.

The job ran on trusty-tools, a Rust workspace, on 2026-06-10. One human (me), one PM coordinator, a fleet of subagents, and a memory layer plus code search and an adversarial PR reviewer wired in over MCP. It ran for 14 hours and 37 minutes. I authored nine content prompts in that window. The longest was 84 characters.

TL;DR

• A single Claude Code + claude-mpm session ran 14h 37m on a real Rust codebase and cost $485.96 at rack rate — $264.65 for the PM (claude-opus-4-8, 265 turns) and $221.30 for the subagents (sonnet at 4,626 turns, haiku at 605). It made 63 delegations. On a turn basis, 93% of the work was autonomous; my share was 7%.

• I’m on Anthropic’s Max plan — $200/month flat. This session ran inside that subscription. The rack-rate figure is the right way to price the economic value of the work; the actual cost to me was zero marginal dollars.

• That 7% was nine human-authored prompts in 14.5 hours. Several were one word — proceed, let's do the top candidates. The longest ran 84 characters. The orchestrator supplied the structure; I supplied the direction changes.

• The economics work because of cache. The sonnet agents read 3,531 cached tokens for every fresh input token (482M cache-read against 137K fresh). The PM ran at 587×, haiku at 34,868×. Most of the context window each turn is priced as cache, not fresh input.

• My idle time shows up on the bill. The most expensive PM turns are cache-cold context rebuilds after I left a long gap: $7.26 to resume after proceed (1.5 hours of silence), $5.82 to pick back up after a 4.5-hour wait.

• The orchestrator ran adversarial reviews on its own engineers’ work 13 times, unprompted — “Per my verification ownership I won’t take that at face value” — and bounced a starred approval back for more work. It absorbed two infrastructure failures without escalating them to me. None of these behaviors is impressive alone. A competent IC does them all before lunch. They appeared together, in one session, with a cost attached to each.

You can explore the full annotated session timeline — a turn-by-turn companion infographic showing the user-visible output alongside the underlying mechanism for each move.

A note on what this is and isn’t

I am not claiming the model understands anything, and I am not grading it on a benchmark. I am describing observed behavior from one session, the way you would describe an animal in the field: here is what it did, here is the order it did it in, here is what it cost, draw your own conclusions. Twelve behaviors stood out. They are below, roughly in the order they appeared.

The Stack in November

This session would not have run the same way seven months ago, not even close. Three layers changed in that window, and they changed together.

Start with the model. The session above ran on claude-opus-4-8, released May 28, 2026. Seven months earlier the comparable model was Opus 4.5, which shipped November 24, 2025. Better code is the obvious place to look for the shift, but the one that mattered here sits elsewhere: 4.8 is substantially less likely to let a flaw in its own work pass unremarked — it surfaces its own errors instead of quietly shipping them. That is the difference between a contractor who does their best and one who tells you when he thinks something is wrong. The 1M context window also moved from beta on 4.5 to generally available on 4.8, which matters across a 14-hour run, and in Claude Code 4.8 defaults to xhigh effort — the cost figures above reflect that setting.

Then the Claude Code harness around it. In November 2025, Claude Code was effectively a single-session tool; background agents existed, but worktree isolation did not. WorkTree support is the critical unlock. Each subagent in this session ran in its own isolated git worktree — its own branch, its own working tree, no chance of stepping on a parallel agent’s files mid-run. The merge-conflict prediction in behavior #6 is only tractable because the agents writing code cannot collide on disk while they work. Parallel execution without merge confusion needed a harness feature that wasn’t there last November. Added in the same window: 5-level nested subagent hierarchies, claude agents for session-wide visibility, and Dynamic Workflows for the PM coordination layer.

The third layer is the orchestration framework. claude-mpm went from v4.26 to v6.5.44 over those seven months — two major versions, roughly 450 releases. Two changes carry most of the behavioral weight. trusty-memory became mandatory session context, so the PM reads project history before it does anything else, which is behavior #1 above. And worktree-first became the framework default, which is what enables behaviors #3, #6, and the parallel fan-outs in #9. The bench is deeper too: 57 specialized agents now versus about 30 in November, so the routing in behavior #9 has more specialists to route to.

None of these three improved in isolation. The model’s self-auditing is only useful if the harness can surface those audits inside a multi-agent pipeline. The worktree isolation is only useful if the orchestration framework knows to reach for it by default. The memory priming is only useful if the model is good enough to act on what it finds. When all three shift in the same six-month window, the effect is not additive.

Strip away the version numbers and the feature lists, and the functional picture is narrower than it looks. The evolution since November clusters around three areas. Context: the 1M window is generally available now, and trusty-memory loads project history before the PM does anything. Recall: the PM enters each session knowing what happened before, with a deeper bench of specialists to route to. Error checking: the model flags its own flaws, and the orchestration layer runs adversarial review unprompted. None of the twelve behaviors below trace to the model writing better code. They trace to the system getting better at knowing what it knows, remembering what it has done, and catching what it gets wrong.

Session Vitals

Before the behaviors, the stat block. This is the anatomy laid flat.

Metric Value Total cost (rack rate) $485.96 PM cost (claude-opus-4-8, 265 turns) $264.65 Subagent cost (sonnet 4,626 turns + haiku 605 turns) $221.30 Delegations (agent calls) 63 Skill + MCP tool calls 18 Wall-clock duration 14h 37m Human share (turn basis) 7% Autonomous share (turn basis) 93% Human-authored content prompts 9 Longest human prompt 84 characters

The cache numbers sit underneath all of it. The sonnet agents pulled 482,137,941 cache-read tokens against 136,569 fresh input tokens. The PM pulled 70,287,449 cache-read against 119,632 fresh. Those two figures are the reason a 14-hour run costs what a junior contractor costs for an afternoon. I come back to them below.

1. Context First

The first action after my prompt was not a plan and not code. It was three memory and context calls in a row. The first one failed: memory_recall: missing 'palace' (no --palace default configured). The PM did not abort or report the error to me. It adjusted the call — added the palace parameter — retried, and got a stored finding from earlier work.

Then it read Cargo.toml and README.md before planning anything. In its own words at 21:48: “I’ll start by orienting myself. Let me query project memory and check what ‘core services’ and ‘console’ refer to, since neither maps obviously to a crate in this workspace.” One minute later it had resolved both terms and declared the scope — the equivalent of reading the ticket history before touching the keyboard.

2. Plan Before Acting

Rather than starting, it named the parallel streams and stated why they could run independently before dispatching anything: “These are independent, so I’ll dispatch them in parallel” (21:49). Structure stated before action taken.

The pattern held under pressure. When I said let's do the top candidates at 22:03, it refused to treat that as a green light: “’The top candidates’ spans everything from a critical bug cluster to a 5,400-line refactor to design/ADR work — very different sizes and risk. Before I fan out agents (and tokens), let me confirm how aggressively to go.”

3. Agents, Narrated

Two agents went out in a single PM turn that cost $0.34 — one to ticketing to pull the open GH issues for the core services, one to local-ops to investigate and deploy the console. Both came back inside two minutes. Each got a defined, bounded job, and the PM said what each was for before sending it. Later waves followed the same shape: at 23:27 three agents launched in parallel for Wave 1, redeploying trusty-search, splitting a 5,421-line file, and drafting an architecture decision record.

4. Verify Before Committing

When I said let's do the top candidates, the PM stopped and asked rather than fanning out. It called AskUserQuestion and waited 19 minutes for my answer before dispatching a single engineer.

The same instinct showed up against its own team’s reports. After the rust-engineer reported PR #1097 done with tests, clippy, and fmt all green, the PM declined to take it: “Per my verification ownership I won’t take that at face value — especially since #1088 and #1089 were collapsed into one commit, and #1089’s core complaint ... may be only partially addressed.” It ran an adversarial PR review and a CI check in parallel before accepting the work. The verification was budgeted into the run.

5. Name What You Don’t Know

On the architecture decisions baked into ADR-0010, the PM drew the line cleanly: “it’s an architecture decision so I’ll draft it for your sign-off rather than implement blind.” It surfaced four open questions, each labeled blocking or non-blocking, and answered none of them unilaterally.

It applied the same judgment to a failure. When a memory write was blocked because the trusty-memory daemon held the palace write lock, the PM diagnosed the cause and made a call: “Memory write is blocked ... not worth a detour; the work is durably captured in the merged squash commit, closed issues, and commit messages.” It marked the gap, decided closing it wasn’t worth the interruption, and said so.

6. Conflict Prediction

Before any Wave 1 agent was dispatched, the PM named the collision: “#1096 and #607 both edit .line-cap-allowlist.tsv (splitting an allowlisted file requires removing/lowering its entry), so running them in parallel guarantees a merge conflict on that file.” It named the file, the two items that would collide, the mechanism, and the mitigation — sequential waves — in the same breath. The same pattern recurred in Wave 2, where it held #607 until #993 landed for the identical reason.

7. Unprompted Checkpoint

Unprompted, at 02:02, the PM produced a formatted two-section status checkpoint: a “Shipped to main” table listing each merged PR with its commit hash and a verification note, and an “In flight / staged” list with the state of each item still moving. Self-organized visibility, not a dashboard anyone designed for it.

8. Risk-Stratified Options

For the ADR-0010 decisions, the PM laid out four questions, each with named options and the cost of each. On the unknown-tag handling: Option P (permissive — risk: typo’d tags survive silently), Option L (allowlist — more control, requires per-index config), Option H (hybrid, the one it proposed). It recommended where it had a view and left the choice with me. At 03:14 it paused and asked what to work on next rather than self-selecting — which is where the 4.5-hour gap in the log comes from. It was waiting on me.

9. Specialist Routing

Routing stayed consistent by agent type. Issue reads and epic filing went to ticketing, daemon deploys to local-ops, implementation to rust-engineer. CI polling and merges were version-control’s; runtime QA went to api-qa, architecture feasibility to research. Across all 63 delegations, the PM did not hand an implementation task to ticketing or a CI task to the engineer.

10. Plans Sharpen with Information

When recon came back, the plan went from vague to specific. After the first two agents returned, the summary carried exact PR numbers, exact crate versions (trusty-search 0.24.4, trusty-memory 0.15.2, trusty-analyze 0.7.0), exact ports, and one specific stray process to clean up — none of which existed in my prompt.

After the issue specs arrived, “fix the top candidates” became a coordinated analysis: “these five cluster around one shared concern — the indexes.toml persistence / colocated / warm-boot scan paths — and #1088/#1089/#1090 in particular interlock through the config-write path, so a coordinated fix is safer than five isolated ones.” The prompt set a direction; the detail came from what the agents found.

11. Self-Maintained Cross-References

Throughout, the PM tracked issue numbers, file paths, commit hashes, and the relationships between them. It noticed that #1088 and #1089 had been collapsed into one commit and independently checked whether that collapse was safe — a cross-reference that was in no agent’s report, only in the PM’s own check against the original spec.

It also caught something I’d have missed: an external contributor, maui314159, had a live PR (#1082) covering a dependency that the #819 work needed. The PM surfaced it as a coordination point rather than duplicating or ignoring it. Later it drew the boundary precisely: “#819 isn’t blocked by conflict — it’s gated on accepting a contributor’s architecture decision, which is your call, not something I’ll auto-merge.” Then it reviewed the external PR in full before going further.

12. Clean Handoff

The session ended on a handoff, not on more output. After filing the final epic (#1119), the PM declared the scope complete, named the one item still in flight, and gave the resume mechanism: “Session stays paused (session-20260611-161949). That clears everything you asked for in this window. The only thing still in flight is #819 (KG ingest endpoint, building in kg-ingest) — I’ll relay its PR when it lands but won’t start anything new. Resume anytime with /mpm-session-resume.” I issued /exit 18 minutes later.

This is the strong version of stopping. The PM did not keep going to look busy, and it did not stop arbitrarily. It identified the boundary — what was done, what was still moving, where the human picks back up — and stopped there.

What the Instrumentation Adds

The twelve behaviors are what you’d see watching over the shoulder. The telemetry adds three things you can’t see that way.

The 7% is even smaller than it sounds. Nine human-authored prompts in 14.5 hours. Three were a word or a phrase — proceed, let's do the top candidates, in console running?. The longest content prompt I wrote all session was 84 characters: it doessnt show the individual consoles, theses should be tabs - and it should be am spa — typos and all. The orchestrator did not need a spec. It needed a direction and the occasional course correction.

The economics are a cache story. The sonnet agents read 3,531 cached tokens for every fresh input token. The PM ran at 587×; haiku, doing high-volume narrow tasks, hit 34,868×. Most of each turn’s context is pulled from cache at cache prices, not re-sent as fresh input. A 14-hour run that re-paid full freight for context on every turn would cost a multiple of $486. The caching is why an orchestration this long is economically viable at all. A note on what I actually paid: I’m on Anthropic’s Max plan at $200/month. The $485.96 above is rack rate — the right figure for understanding the economic value of the output. My marginal cost for this session was zero.

Waiting costs money, and you can see exactly where. The most expensive PM turns share one trait: cache_read=0 — the turns where context had to be rebuilt cold after a cache miss, each one corresponding to a long gap I left. The $7.26 turn, the single priciest of the session, came right after I typed proceed, following 1.5 hours of silence; it wrote 364,618 tokens of cold context to restart the pipeline. The $5.82 turn came after my 4.5-hour wait. The orchestrator picked up where it left off both times, with full context. It just had to pay to reconstruct it. In a system like this, my latency has a line item.

And one finding the behaviors undersell: the failures. The first memory call errored and the PM fixed and retried it. The memory write later failed against a held lock, and the PM diagnosed the daemon contention and moved on. A subagent dropped its connection mid-run on an infra hiccup, and the PM recorded the socket error and later resumed the job to finish it. Three failures, three absorptions, zero escalations to me. That is the part that reads most like a year-one IC: not that nothing broke, but that what broke got handled below my line of sight.

So What?

Four threads run under all of this.

The first is where the line now sits between supervision and delegation. For most of these tools, the answer has been “supervise closely” — read every diff, catch every drift. This session moved the line. I supervised at the level of direction and architectural calls; I delegated everything from PR review to conflict avoidance to failure recovery. The PM did its own adversarial review 13 times and bounced a starred approval. Verification did not disappear; it moved inside the loop, and it left a receipt.

The second is what changes when the coordination work — not the typing — is the part the tool does well. The code these systems write stopped being the interesting question a while ago. The interesting development is that the orchestration layer now decomposes work, predicts conflicts, routes to specialists, tracks provenance, and knows when to hand back. That is project management. When the typing is cheap and the coordination is the hard part, a tool that coordinates well is worth more than a tool that types fast.

The third is the one underneath everything else. No single behavior above is impressive on its own. A competent IC reads the ticket history, names the parallel streams, predicts a merge conflict, routes to the right person, and stops at the decision boundary — without being told. What changed is that they now appear together, unprompted, in one session — and this time there’s a receipt for every one of them. The dollar figure is not the headline. The instrumentation is. We can finally watch the whole thing work and count what it cost, behavior by behavior.

The fourth is about where to point the question. For a while the useful question was “what can the model do.” After this session I think the better question is “what can the trifecta do,” because none of the behaviors above trace cleanly to a single component. They come out of the interaction: a model that audits its own work, a harness that isolates parallel work in separate worktrees, and an orchestration layer that routes to the right specialist with project memory already loaded. Pull any one of the three and the session degrades — the self-audit goes nowhere without a pipeline to surface it, the parallel waves collide without isolation, the routing misfires without memory. The capability lives in the seams between the parts, not in any one of them.

One last note on the economics. The Max plan is $200 a month. For that, I ran a session that bills at $486 rack rate, and the output is good enough that going back to a metered model is hard to imagine. Dealers give the first one away for a reason. The Max plan is that first bag: the work is compelling enough to be structurally addictive, and the flat rate strips out the per-token friction that might otherwise make you stop and think. I don’t mean that as a complaint. It’s a description of how the pricing works on the user.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Explore the annotated session timeline — the turn-by-turn companion infographic for this session

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

The Math Doesn’t Work (Yet): Inside the AI Profitability Problem

OpenAI’s own projections show losses getting bigger as revenue gets bigger. Leaked investor documents reported across WSJ, Fortune, and The Information put the company at roughly $74 billion in operating losses in 2028 — on roughly $100 billion in projected revenue. That pairing is the headline. Not a smaller loss as scale arrives. A loss that grows faster than the top line.

That single relationship is the whole story. We tend to model AI companies as software businesses that will eventually grow into their cost structure the way SaaS companies did before them. The numbers say otherwise. These are capital-intensive infrastructure plays wearing software-company clothing, and the unit economics underneath them run in the opposite direction from the SaaS playbook most of us internalized over the last fifteen years.

A caveat before the numbers, because it matters for what’s below: neither OpenAI nor Anthropic publishes audited financials. Most figures here come from leaked investor decks, run-rate annualizations the companies announce in funding rounds, or SEC filings made by their cloud partners. The uncertainty is part of the analysis, not a footnote to it. When a specific timeline or quarterly figure couldn’t survive cross-checking against primary sources, I left it out.

TL;DR

• OpenAI’s leaked projections show operating losses widening as revenue grows — roughly $74B in losses on ~$100B revenue projected for 2028, with cumulative cash burn near $115B through 2029.

• In 2025 OpenAI spent about $1.69 for every dollar of revenue (~$9B net loss on ~$13B revenue), per leaked documents confirmed by multiple outlets.

• Anthropic’s revenue trajectory is steep — roughly $1B annualized at the end of 2024 to a figure announced in the tens of billions by mid-2026 — with Claude Code alone reported at $2.5B annualized by February 2026.

• Inference token prices fell about 75% in a year. Selling more AI makes the per-unit economics cheaper, which makes revenue growth harder, not easier.

• AI-native gross margins sit near 45% versus 75–85% for mature SaaS — a structural gap of 23–33 points no company has yet closed.

• No company has a verified path to profitability. Every specific breakeven-by-year claim I tried to confirm fell apart under scrutiny.

Two Companies, Two Shapes

OpenAI ended 2025 at roughly $20 billion in annualized revenue, a figure CFO Sarah Friar has stated directly. That is a large business by any normal measure. It is also a business that, in the same year, spent about $1.69 for every dollar it took in — somewhere around a $9 billion net loss on roughly $13 billion in recognized revenue, according to leaked documents that WSJ, Fortune, and The Information each reported. The company is majority funded by Microsoft, runs its compute primarily on Azure, and its strategy is scale-first: build the largest models, capture the most usage, and trust that revenue follows the curve.

Anthropic’s shape is different. Its revenue trajectory is steeper and more concentrated. The company grew from roughly $1 billion annualized at the end of 2024 to a figure it announced in the tens of billions by mid-2026 — the number it cited in its Series H materials. I’m deliberately not pinning an exact figure to a month here, because the company has grown several-fold inside a single five-month window and any precise number is stale by the time you read it. What’s verifiable is the slope, and the slope is steep.

The more interesting detail is the concentration of value. Claude Code, one product, was reported at roughly $2.5 billion annualized by February 2026. A single coding tool driving that much of a company’s run rate tells you something about where the margin-bearing demand actually lives. Anthropic is backed by Amazon (over $8 billion invested) and Google (over $2 billion), and it has committed to spend more than $100 billion on AWS over ten years, with roughly 1 GW of Trainium capacity targeted by the end of 2026. Two large companies funding it; one of them also selling it the silicon it runs on.

Why Compute Is the Problem

What separates these companies from every SaaS business you’ve evaluated: they don’t own their infrastructure. They rent it, at hyperscaler rates, from the same companies that fund them.

OpenAI’s Azure spend reportedly ran around $3.7 billion in 2024 and roughly $8.7 billion across the first three quarters of 2025. Treat those numbers as medium-confidence — they come from leaked documents, and Microsoft pushed back that the figures “aren’t quite right.” But the direction is consistent with everything else: compute cost is the dominant line item, it’s largely fixed, and it grows with usage.

Anthropic’s arrangement produced one of the stranger details in modern enterprise finance I’ve read in some time. The company signed a deal for compute from Colossus 1 — Elon Musk’s Memphis data center, operated by xAI — at roughly $1.25 billion per month for 300 MW of capacity, running through May 2029. That’s not from a leak. It surfaced in SpaceX’s S-1 SEC filing and was confirmed by CNBC, Axios, and Data Center Dynamics, with a potential total value above $40 billion. There’s a 90-day mutual cancellation clause, so the headline total overstates the firm commitment. Still: Anthropic — funded by Google and Amazon — is paying Elon Musk’s company more than a billion dollars a month for compute. The AI capital world is stranger from the inside than the press releases suggest.

Zoom out and the renter problem gets sharper. Hyperscaler capex for 2026 is projected at $660–690 billion. Against that, OpenAI’s $20 billion ARR is roughly 3% of a single year’s data-center buildout by its suppliers. The companies selling AI applications are small tenants in an infrastructure market they don’t control and can’t currently price against.

The Unit Economics Trap

This story inverts an instinct most of us trust.

In normal software, even in the Cloud, scale is your friend. Marginal cost trends toward zero, gross margin climbs as you grow, and a mature SaaS business lands at 75–85% gross margin because serving the millionth customer costs almost nothing. Volume is the cure.

Inference doesn’t behave that way. Every token generated costs compute — real, metered, non-zero compute — so the marginal cost of serving usage stays stubbornly positive. And the price you can charge for that token is collapsing. Enterprise transaction data from Ramp shows inference prices falling roughly 75% in a single year, from around $10 per million tokens to around $2.50. Capability per dollar is improving fast, which is good for buyers and brutal for sellers, because it means the revenue you booked at last year’s prices reprices downward while your compute bill does not.

Put the two forces together and you get a squeeze that worsens with success. The better you are at selling inference, the more usage you drive; the more usage you drive, the more the per-unit price falls; the more it falls, the harder it is to grow revenue against a compute bill that scales with that same usage. Volume isn’t the cure here. Under these dynamics it’s part of the disease.

The survey data puts a number on how far this world sits from SaaS. ICONIQ Capital polled about 300 software executives and pegged AI-native gross margins at 41% in 2024, 45% in 2025, and a projected 52% in 2026. Improving — but starting from a base 30-plus points below mature SaaS, and closing the distance slowly. A 52% gross margin is a respectable hardware business. It is a structurally difficult software business, especially one still spending heavily to grow.

Two Different Bets

OpenAI and Anthropic are running different experiments on how you eventually close that gap. Neither has been validated.

OpenAI’s bet is scale and breadth. Build the broadest platform, capture consumer and enterprise and API demand simultaneously, and assume that at sufficient scale you gain pricing power over compute, model-efficiency gains compound, and the revenue base grows fast enough to absorb the fixed cost. The leaked projections embody the risk in this bet: they show losses widening through 2028 even as revenue approaches $100 billion, with cumulative cash burn near $115 billion through 2029. The theory requires the curve to bend after the window we can currently see.

Anthropic’s bet is narrower and more product-led. Find a wedge where the work is valuable enough that buyers tolerate real prices, prove the margin there, and expand outward. Claude Code is that wedge made concrete — $2.5 billion annualized from developers who pay because the output is worth more than the inference under it. Coding, agents, and enterprise automation are higher-value work than chat, and higher-value work supports prices that don’t immediately erode under token deflation. The risk: revenue concentration in a single product line, and an infrastructure bill — AWS commitments plus the xAI deal — that’s enormous relative to a company still proving the model.

Two theories of the same problem. Scale your way past the margin gap, or find work valuable enough that the gap doesn’t bind. We don’t yet have the data to say either works.

What Would It Actually Take

I’ll skip the timeline speculation — every specific breakeven-by-year claim I tried to verify died on contact with the sources. The structural requirements are clearer than the dates.

Three things have to move. First, gross margins have to climb from the mid-40s toward something defensible — call it 60-plus — and stay there while volume grows. That means model-efficiency gains (cheaper inference per unit of capability) have to outrun price deflation, rather than getting passed straight through to buyers as lower prices.

Second, these companies need pricing power over compute, which today they don’t have. At current scale they’re tenants. The open question is at what ARR a vendor becomes large enough to negotiate compute like a partner instead of a customer — or to build its own. Anthropic’s Trainium commitment and OpenAI’s various infrastructure moves are bets that vertical integration eventually changes the cost equation. That’s unproven, and it’s expensive in the interim.

Third, the product mix has to keep shifting toward work that resists deflation — enterprise agents, coding tools, automation that’s measured against labor cost rather than against the falling price of a token. Claude Code is the cleanest evidence that this category exists and that buyers will pay. Whether it’s a large enough share of total volume to lift blended margins across a company is the question that decides the whole thing.

The Open Questions

What we don’t know outweighs what we do.

We don’t know the actual, current gross margins at either company — only the AI-native sector estimate of roughly 45%. Neither company publishes the number that would settle the argument. We don’t know whether Anthropic’s stack of compute commitments, the decade-long AWS deal alongside the month-by-month xAI arrangement, creates structural tension or healthy redundancy. A company hedging across three infrastructure providers is either diversifying supply or revealing that no single supplier can meet its demand. We don’t know the ARR threshold at which compute pricing becomes negotiable, which is the hinge the entire margin story turns on.

And there’s the strategic risk that has no clean precedent: your infrastructure supplier is also your competitor. Microsoft ships Copilot. Amazon and Google both build models that compete with Anthropic’s. xAI builds Grok. Every dollar these companies pay for compute partly funds a rival’s model program. In normal software you don’t hand your gross margin to the company trying to beat you. Here it’s the default arrangement.

So the real question isn’t whether the AI labs are growing. They obviously are, faster than almost any companies in history. The question is whether revenue growth and margin improvement are the same trend or opposing ones. The SaaS era trained a generation of operators to believe that scale fixes economics. The leaked numbers describe a business where scale, so far, makes the loss bigger. Until one of these companies publishes a gross margin that shows the curve bending, that’s the math we have. And the math doesn’t work yet.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• The First 70% Era — Where agentic AI delivers value and where it stops, and why the higher-value work resists token deflation

• AI and the Rise of the Hyperdev — Why developers pay real money for AI tooling, the demand side of the margin story

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Most of the best ideas in software engineering aren’t new. They’ve been written up in books, argued over at conferences, taught in every “best practices” deck since the late 1990s. And most teams quietly don’t do them.

Not because anyone thinks they’re wrong. Test-driven development, design by contract, architecture decision records, mutation testing — ask a room of senior engineers whether these are good ideas and you’ll get nods. Ask the same room who practices them consistently under deadline pressure and the hands stay down. I’ve been in that room for twenty-five years, on both sides of the question. I’ve also been the engineering leader who let those practices slip because shipping the feature mattered more this quarter.

There’s a single economic reason these practices lose. The upfront cost is high, the payoff is real but distant, and human attention is the binding constraint. Write the test before the code, document the decision, specify the invariant — every one of those is a tax you pay now against a benefit you collect later, maybe, if the project lives long enough. Under deadline pressure, that’s a losing trade for a human. So we skip it, ship, and pay the interest later in bugs and confusion. Call it the impatience tax.

Agents don’t pay that tax. They have infinite patience for upfront rigor and roughly zero marginal cost for the tedious work that rigor demands. Writing a thorough test suite for code that doesn’t exist yet is psychologically brutal for a person and completely fine for a model. That single shift — the cost of patience going to zero — quietly inverts the economics of a whole list of practices we knew were right and gave up on anyway.

This isn’t a piece about what AI makes possible. Lots of things are possible. It’s about a narrower, more useful question: which disciplines did we already agree were correct, fight about for decades, and abandon for reasons that no longer hold?

Here are nine.

TL;DR

• These nine practices share one structure: high upfront cost, distant payoff. Human attention is the constraint that kills them under deadline pressure.

• AI removes the constraint. A failing test is the clearest prompt you can hand an agent; a spec is its input; an ADR is its context. The discipline becomes the interface.

• TDD, design by contract, and property-based testing turn from “things we should do” into the most effective way to constrain agent behavior and prevent hallucinated correctness.

• Documentation, ADRs, and living docs get a bilateral ROI: agents generate them from code, and they make agents far more effective in your codebase.

• The catch is real. A 2025 METR randomized trial found experienced developers were about 19% slower with AI assistance. These practices pay off only when AI is used with discipline, not as autocomplete.

1. Test-Driven Development

Start with the practice most teams abandoned first.

Writing tests before code was always the theoretically superior move. It forces you to define the interface before you build behind it, catches bugs at the moment of definition instead of during integration, and leaves behind a living specification of what the code is supposed to do. Kent Beck made the case decades ago and the case held up.

Almost nobody did it consistently. The reason is psychological, not technical. Writing detailed tests for code that doesn’t exist yet, while a deadline breathes on your neck, feels like building scaffolding for a house you haven’t designed. Your brain screams at you to just write the function. So you write the function, promise yourself you’ll add tests after, and — well. You know how that goes.

Now flip the perspective. To an agent, a failing test isn’t scaffolding. It’s the clearest possible specification of intent you can provide. “Make this pass, don’t break anything else” is an unambiguous, machine-checkable instruction, which is exactly what a probabilistic system needs to stay honest. The test suite becomes a guardrail that prevents the most dangerous failure mode in AI-assisted coding: confident, plausible, wrong. Hallucinated correctness dies against a red bar.

TDD went from the discipline most teams couldn’t sustain to one of the best tools we have for bounding what an agent is allowed to claim it did. Same practice. Opposite economics.

2. Spec-Driven Development and Design by Contract

Bertrand Meyer formalized design by contract in the 1980s and built it into the Eiffel language: specify preconditions, postconditions, and invariants, then let the implementation follow from the contract.

The idea was sound and the adoption was thin, for one stubborn economic reason: the contract only pays off if someone else writes the implementation from it. If you’re writing both the spec and the code, the spec is overhead — you already know what you meant. The contract’s value lives in the handoff, and for most of software history there was no cheap handoff to hand it to.

Now there is. You write the contract; the agent writes the implementation from it. Spec-driven development stops being a documentation chore and becomes the actual control surface for delegation. The spec is the part requiring human judgment about what the system should do. The implementation — the part that used to eat the hours — is the part you delegate. Meyer’s economics finally close, forty years late, because the missing party in the transaction showed up.

3. Architecture Decision Records

Why does the codebase look like this? Why Postgres and not Dynamo, why this queue, why the weird module boundary that everyone trips over?

ADRs were the right answer to that question — a short dated record of each significant decision, the context, and the alternatives rejected. The discipline almost never held. Same shape as everything else here: the cost is immediate (stop, write the thing) and the value accrues slowly, mostly to some future engineer who isn’t in the room yet.

Two things flipped at once, which makes this one more interesting than the rest. First, agents can generate ADRs from an existing codebase — read the git history, the dependency choices, the structure, and reconstruct the decisions that produced them. The retroactive cost of documentation drops toward zero. Second, and this is the part people miss: existing ADRs dramatically improve what an agent can do in your codebase. An agent that can read why you chose eventual consistency won’t keep proposing changes that assume strong consistency.

So the ROI went bilateral. Agents help you write ADRs, and ADRs help agents help you. The practice that used to only cost now pays on both ends.

4. Continuous Code Review

“Catch issues early” has sat on every best-practices list since Extreme Programming put continuous review on the map. The advice was never controversial. The bottleneck was always the same: human reviewer attention is finite, expensive, and easily exhausted. So review collapsed into batch PR review — a tired engineer reading a 600-line diff on a Friday afternoon, approving most of it on faith.

AI review on every commit — not batched at the PR boundary, but running as code lands — is moving from aspirational toward baseline: increasingly the default on teams that have wired it in, though not yet universal. The marginal cost of a careful read went to nearly nothing, and the read happens while the context is still warm.

But this one comes with an emergent problem worth naming, because it bites teams that adopt the tooling without rethinking the model. PRs are getting larger and arriving faster under AI-assisted development. An agent can produce a 2,000-line change in an afternoon. If your review model still routes everything through a human approver at the end, that human is now the rate limiter, drowning in volume they didn’t generate and can’t realistically read. AI review on every commit is part of the answer. The harder part is restructuring what the human reviews — architecture, intent, the decisions a model shouldn’t make alone — and letting the machine handle line-level correctness continuously. Adopt the tool without rethinking the workflow and you’ve just built a faster way to overwhelm your best reviewer.

5. Pair Programming

Pairing always looked expensive in the most obvious way: two engineers, one task, double the salary against a single unit of output. That intuition was wrong on the numbers — the measured overhead from the pair-programming studies was closer to 15%, often recovered through fewer defects — but the 2× gut feeling is what drove the decisions. The benefits were real — knowledge transfer, real-time review, fewer dumb mistakes — but the perceived math meant most teams reserved it for critical paths, gnarly bugs, or onboarding a new hire. A luxury, rationed.

The pair is now a human and an agent, and it’s available to every engineer continuously, not rationed to the critical path. The knowledge-transfer benefit generalizes — the agent can explain unfamiliar parts of the codebase on demand. The real-time-review benefit generalizes — a second set of eyes on every line, every time, without scheduling two calendars. The economics that made pairing a rationed luxury simply don’t apply when one half of the pair has near-zero marginal cost.

Worth a caveat: agent-as-pair is genuinely good at the review and explanation half of pairing, and weaker at the part where a human partner pushes back on a bad design before you’ve written a line. You still need humans pairing with humans for that. But the day-to-day, line-by-line version of pairing just became free, and that’s most of what pairing was for.

6. Mutation Testing

Coverage numbers lie, and most engineers know it. Eighty percent line coverage tells you eighty percent of your lines got executed by a test — not that any of those tests would notice if the behavior broke. Mutation testing is the honest measure: it deliberately introduces bugs (flip a comparison, drop a line, change a constant) and checks whether your test suite catches them. If a mutant survives, you have a test that runs code without actually validating it.

Mutation testing was always the gold standard and almost never run continuously, for one reason: it’s computationally expensive. You’re effectively running your whole suite many times over, once per mutation. On a real codebase that’s brutal. So it lived in research papers and the occasional heroic CI job that someone eventually disabled for being too slow.

That constraint is mostly gone — compute is cheap and parallel, and we got more comfortable spending it. And the practice arrived right when we suddenly need it most. AI-generated tests have a characteristic failure mode: they drift toward coverage metrics without meaningful assertions. The model writes a test that calls the function, exercises the path, and asserts almost nothing of substance — green checkmark, zero protection. Coverage looks great. Mutation testing is the thing that catches exactly that. It’s the verification layer for a verification layer, and it matters more now than when it was invented, because now a machine is writing the tests.

7. Living Documentation

Documentation was always supposed to be a first-class artifact. It almost never was, and the reason is by now familiar: writing docs is tedious, and the penalty for stale docs accrues slowly and lands on someone else. So docs rotted. Every team has a wiki that’s a graveyard of half-true pages from two reorgs ago.

AI changes both halves of the equation at once. It generates docs from code, so the writing cost drops. And it consumes docs as context, so the docs earn their keep immediately — a well-documented codebase is a measurably more useful codebase for an agent working in it. The ROI is immediate and bilateral, same structure as ADRs.

There’s a quiet shift hiding in there. Documentation used to be written for humans who’d mostly never read it. Now it’s also written for the agent that will read it on every task, which means stale docs don’t just confuse a future engineer — they actively degrade your tooling today. The feedback loop tightened from months to minutes. That’s the kind of change that actually moves behavior, because the cost of skipping it shows up now instead of later.

8. Runbook Generation from Incidents

On-call always leaned too hard on tribal knowledge. The person who knows why the payment service wedges at 3 a.m. is asleep, on vacation, or left the company last spring. Writing a runbook after each incident was obviously the right move and reliably the thing nobody did, because the incident was over and everyone wanted to go back to bed.

Incidents become runbooks automatically now. The agent has the incident timeline, the chat transcript, the commands that resolved it, the postmortem — and it can turn that into a structured runbook while the details are fresh, without asking an exhausted engineer to relive the night. The cost that used to fall right when motivation was lowest now falls on a system that doesn’t get tired or resentful.

I’d treat the generated runbook as a draft a human still signs off on, not gospel. But “imperfect draft, reviewed in five minutes” beats “blank page nobody ever fills in,” and that was always the real competition.

9. Property-Based Testing

Example-based tests check the cases you thought of. Property-based testing is stronger: you specify the invariants a system must always satisfy — reversing a list twice returns the original, a serialized-then-deserialized object equals the original, the account balance never goes negative — and the framework generates hundreds of adversarial inputs trying to break them. QuickCheck pioneered the approach; it finds the edge cases you’d never have written by hand.

It never went mainstream outside a few communities, and the bottleneck wasn’t tooling — good property-based libraries exist for most languages. The bottleneck was writing good property specifications. Identifying the right invariants requires deep domain reasoning: you have to understand the system well enough to state what must always be true, which is harder than writing a few example cases. Most engineers, under pressure, defaulted to the easier thing.

This is where AI helps in a way that’s less obvious than “it writes the code.” A model can generate property suites from a spec, and — more usefully — it can reason about what invariants a system should satisfy in the first place, surfacing properties you hadn’t articulated. That’s the expensive, judgment-heavy part it actually offloads. Combined with mutation testing to keep the generated properties honest, you get a testing approach that was always more powerful than example-based testing and was always too expensive in human reasoning to adopt widely.

The Catch

I’d be selling you something if I stopped there, and the data won’t let me.

In July 2025, METR ran a randomized controlled trial with experienced open-source developers working on real tasks in repositories they knew well. The developers expected AI assistance to speed them up. It slowed them down — by roughly 19%. METR’s February 2026 follow-up found that gap narrowing, and reversing on some measures, as the same kind of developers gained real experience with the tools — which is to say the 19% was a snapshot of the unfamiliar, undisciplined path, and it closes precisely as people pick up the habits this piece is about.

That finding is real and it isn’t a contradiction of everything above. It’s the missing condition. Every practice in this piece works because it imposes structure on the agent — TDD as a guardrail, the spec as input, the ADR as context, mutation testing as the check on the check. Used that way, with discipline, AI is constrained toward correctness. Used the other way — as autocomplete, as a vibe-coding partner you don’t supervise — you get more code, faster, with less correctness and a slower path to done once you account for the cleanup. The METR developers, working in code they already understood deeply, may well have been paying exactly that tax: accepting plausible suggestions that took longer to vet and fix than writing it themselves would have.

So the inversion isn’t automatic. The cost of patience dropped to zero, which makes the rigorous path finally affordable. It does not make the undisciplined path good. If anything it makes discipline more important, because a tool that produces plausible output at high volume is precisely the tool that most needs a guardrail you can’t talk your way past. A red test bar doesn’t care how confident the model sounds.

What To Do With This

The interesting question was never “what does AI make possible.” That list is enormous, mostly speculative, and not very actionable. The better question is the one this whole piece is built on: which practices did we already know were right, argue about for decades, and quietly give up on?

That list is short, specific, and yours to write. Go pull your own team’s “we should really do this but we don’t” backlog — the standing items in retros that everyone agrees with and nobody owns. I’d bet most of them have the same economic shape: high upfront cost, distant payoff, killed by human impatience under deadline. Test coverage on the legacy module. The runbooks. The ADRs for the three decisions everyone keeps re-litigating. The integration tests that would’ve caught last quarter’s outage.

Run each one through a single question: was this abandoned because it was wrong, or because it was expensive in human patience? The wrong ones, leave abandoned. The expensive-in-patience ones just got cheap. Those are the ones to pick back up first.

The impatience tax got repealed. The disciplines it used to make unaffordable are sitting right there, mostly unchanged, waiting for someone to notice the price changed.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• The Other Shoe Has Dropped — Why enterprise AI bills don’t match the per-token price collapse

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Something shifted in the last year, and it took me a while to name it.

A growing number of people are running organizations while still doing real hands-on technical work. Not as a hobby, not on weekends, not as a vanity exercise to keep their commit graph green. They are building things their teams depend on — and they are doing it as a deliberate part of the job, not in the cracks between meetings. The work has a specific shape. It is rarely a production feature. It is the layer underneath: developer productivity tooling, internal services, agentic harnesses, MCP connectors, the infrastructure that unblocks everyone else.

For most of my career this combination didn’t really hold together. You could code or you could lead, and the moment you tried to do both seriously, one of them rotted. I’ve watched plenty of technical executives keep a foot in the codebase and slowly become the bottleneck everyone routed around politely. The pattern was familiar enough to be a warning.

What changed is not that leaders suddenly got more disciplined. It’s that the time cost of meaningful technical contribution collapsed. Agentic coding made a hybrid role viable that wasn’t viable before — and the more I look at it, the more I think this isn’t a new invention at all. It’s the recovery of a very old idea that modern specialization interrupted.

I’m writing this as someone living in the middle of it. I spend roughly 30% of my time coding. And when I say coding, I mean directing a team of agents — much closer to that than hands-on work, which is probably the whole point. That’s not a full-time IC’s week, and it isn’t meant to be. It’s enough to stay close to the work that matters and to build the enabling infrastructure I think is worth my own hands on the keyboard.

TL;DR

• A distinct role is emerging: leaders who run organizations and still do deep technical work — specifically enabling/institutional work (tooling, harnesses, internal services), not critical-path product features.

• This satisfies Charity Majors’ actual advice. Her line was never “stop coding.” It was “stop writing code in the critical path.” Enabling work fits that exactly.

• The integration of strategist and practitioner has deep cross-cultural precedent — Japan’s bunbu-ryōdō, Rome’s Marcus Aurelius, China’s wen-wu, the Renaissance polymath, Mattis’s “warrior monk.” The modern role is a recovery, not a novelty.

• Agentic coding is what makes it newly viable: focused sessions now deliver output that once required sustained, uninterrupted immersion. The Anthropic 2026 data shows ~27% of AI-assisted work is work that “wouldn’t have been done otherwise.”

• Directing agents feels like delegation — the same skill leaders already use with human reports. Which is why experienced leaders adapt to it more naturally than juniors do.

The pattern, named

The difficulty with the coding executive was never philosophical. It was attentional.

Charity Majors mapped this years ago in The Engineer/Manager Pendulum, and the piece holds up because she was precise about the mechanism. Management is interruptive by design — your job is to be available, to unblock, to absorb the chaos so your team doesn’t have to. Serious engineering is the opposite. It requires blocking interruptions for long enough to hold a complex system in your head. Two incompatible attention modes. Try to run both at once and you do neither well.

But here is the part people skip when they quote her. Majors never said managers should stop coding. Her actual advice was sharper: don’t write code in the critical path. Don’t be the person others are waiting on. Stay technical, stay sharp, just don’t make yourself a dependency that blocks shipping. “The best frontline eng managers in the world,” she wrote, “are the ones that are never more than 2-3 years removed from hands-on work.”

That distinction carries the whole argument. Because there is a category of technical work that is consequential without being critical-path, and it turns out to be exactly the work senior people are best positioned to do.

Call it enabling work. Internal tooling. Developer productivity infrastructure. Agentic harnesses. MCP services that other teams plug into. Architectural prototypes that prove a direction before anyone commits to it. None of this is what blocks a release on Thursday. All of it multiplies whoever comes after. It tolerates interruption — you can pick it up Tuesday afternoon and put it down when a real fire starts — precisely because nobody is standing at your desk waiting for it.

This is also where the industry is putting its money. Gartner named platform engineering a top strategic trend for two years running and projects that 80% of large engineering organizations will run dedicated platform teams by 2026. The structural reason a leader can work here without becoming the bottleneck is built into the definition of the domain: its output unblocks others rather than blocking them.

Will Larson’s Staff Engineer archetypes circle the same territory without quite landing on it. His “Architect” sits in a permanent argument — some organizations demand the Architect stay deep in the code, others forbid it. The leader/practitioner resolves that argument by relocating it: deep in the enabling and infrastructure work, absent from production product code. Not the pendulum, not the staff IC. A real hybrid, and a newly coherent one.

This is not new

Here is where I want to slow down, because the most interesting thing about this role is how old it is.

The idea that a leader should be both a strategist and a practitioner — not separate modes to alternate between but a single integrated way of operating — shows up independently across at least five civilizations. That kind of convergence usually means a culture has found a durable answer to a real problem.

Japan gave it a name: bunbu-ryōdō (文武両道), the way of both the literary and the martial. Bun is letters, cultivation, strategy. Bu is the martial, the active, the practitioner’s hand. Ryōdō means both ways, together — not balanced, not traded off, but held at once. By the mid-fourteenth century the dual-talented warrior was already established as the model leader, and during the Edo period the Tokugawa shogunate made it official policy for the samurai class. The phrase that survives captures the stakes: culture without power is ineffective, and power without culture is barbarous.

The archetype is Miyamoto Musashi. Undefeated in more than sixty duels, often fighting with a wooden sword against live steel. He founded a two-sword school, and in the last months of his life he wrote The Book of Five Rings in a cave. He was also a recognized master of ink painting and calligraphy — his Shrike on a Withered Branch survives as a designated Important Cultural Property of Japan. The same hands that won sixty duels produced fine art a nation still protects. He didn’t oscillate between the sword and the brush. He held both, and each sharpened the other. “When I apply the principle of strategy to the ways of different arts and crafts,” he wrote, “I no longer have need for a teacher in any domain.” Mastery in one discipline illuminating all the others.

Rome had Marcus Aurelius, the philosopher-king made historical rather than theoretical. He ran the empire and commanded its armies on the Danube, and he wrote the Meditations in the war camps — fragmentary notes to himself, composed in the middle of campaigning and administration. That book was never published philosophy. It was a working journal, the most powerful man in the ancient world writing to stay grounded while doing the job.

China institutionalized the same ideal as wen and wu — civil cultivation and martial capability — and ran it for roughly thirteen centuries through the scholar-official class. Zeng Guofan is the canonical case: he rose through the imperial examinations to high Confucian office, then built and commanded an army of more than a hundred thousand, reportedly keeping a diary on Neo-Confucian ethics even as he directed the campaigns. The integration wasn’t left to personal taste. It was built into the examination and career structure.

And the archetype still lands today. James Mattis earned the nickname “Warrior Monk” — battlefield commander and devoted reader, a 7,000-book library, the Meditations carried into combat. The chain from Aurelius to Mattis is literal: the same book, eighteen centuries apart. That we still reach for “warrior monk” as a compliment for a leader tells you the integration never stopped resonating.

Across all of it, the answer is the same. The contemplative and the active were not specializations to assign to different people. They were a single discipline, each half informing the other. Modernity — with its org charts, its clean role boundaries, its professional specialization — interrupted that. The leader/practitioner is not a tech-industry novelty. It’s an old integration becoming feasible again.

Why now

So what actually changed? Not the wisdom. The economics.

The thing that made the coding executive a bad idea was the attention math. Serious technical work demanded long, unbroken stretches of focus — the exact resource a leadership schedule cannot reliably provide. You cannot design a system in the fifteen minutes between a board prep and a one-on-one. The pendulum was a real constraint, not a failure of will.

Agentic coding changes that math directly. The unit of work moved up a level. Instead of holding every implementation detail in working memory across a four-hour session, you specify intent, direct an agent, review what comes back, correct course, and direct again. A focused 30-minute session now produces what used to require an afternoon of immersion — not because the thinking got easier, but because the implementation cost collapsed.

The Anthropic 2026 Agentic Coding Trends Report puts numbers on the shift. Average session length has climbed to 23 minutes in the agentic era, up from about 4 in the autocomplete era — the work got denser, not just faster. 78% of Claude Code sessions now involve multi-file edits, up from 34% a year earlier. Teams running multi-agent workflows report 2–4x faster delivery from task creation to deployment. And the figure that matters most for this argument: roughly 27% of AI-assisted work consists of tasks that “wouldn’t have been done otherwise” — the scaling projects, the nice-to-have tools, the exploratory infrastructure that was never quite worth the manual hours.

That 27% is the enabling work. It is the category that lives or dies on time cost, and it’s the category a leader/practitioner is best placed to take on.

The arithmetic is what makes 30% credible. I documented a 6–10x multiplier on focused technical sessions in The Irreducibles earlier this year — a project I estimated at 150–200 billable hours compressed into roughly 50–70 hours of wall-clock time, most of which wasn’t coding at all. If directed work runs several times faster than hand-coding, then a day and a half a week can produce what once consumed a full-time engineer’s week. That’s not a marginal gain. It’s a change in what’s structurally possible.

There’s another dimension that doesn’t show up in the productivity numbers: the work itself is unstable. Agentic coding patterns are still shaking out. There aren’t many experienced practitioners, the field is moving fast, and we don’t yet have good consensus on which patterns are load-bearing and which are fashion. A manager who’s only reading about it can’t make that distinction on behalf of a team. You have to be in it to know.

There’s a counterintuitive wrinkle worth naming: the people best positioned to exploit this are the senior ones. A University of Chicago working paper from late 2025 found experienced developers were 5–6% more likely to succeed with AI agents for every standard deviation of work experience, largely because they worked plan-first — laying out objectives, alternatives, and steps before invoking the tool. That’s the opposite of the assumption that AI flattens the seniority curve. Expertise improves your ability to delegate to a model for the same reason it improves your ability to delegate to a person. AI doesn’t change what senior engineering is. It reveals what it always was.

Directing agents is delegation

This is the part that interested me the most, and it’s the bridge between the leadership job and the technical one.

A year or so ago, working with Claude Code felt like coding. Now it feels like delegating. I wrote about that shift when it first became undeniable — the move from being a programmer who uses AI to being something closer to a technical project manager who directs it. Anthropic’s report uses the same vocabulary, describing engineers moving “from writing code to orchestrating the systems that write it.”

What I didn’t fully appreciate at the time is how directly that maps onto the muscle leaders already have. Delegating to an agent feels, in practice, just like delegating to a human engineer. You frame the problem, set the constraints, hand it off, and come back to assess the result. Give a directive, walk away, return to completed work. That loop is the daily reality of management. Leaders developed it because they had to, and it transfers to agents almost without friction.

So the leader/practitioner doesn’t have to become a coder again in the old sense. The skill in demand is judgment plus delegation, and that’s the skill leadership has been building all along. The hands-on knowledge tells you what to ask for and whether the answer is any good. The delegation instinct does the rest.

This is why the enabling work and the agentic tools fit together so cleanly. Enabling work tends to be well-defined, non-user-facing, and long-horizon — exactly the profile agents handle well and exactly the profile that tolerates a leader’s interrupted schedule. The hands-on contribution mostly takes the form of specifying constraints and patterns, which is what I’ve called pattern mastery: when you write the pattern down, you’ve written the spec, and the spec multiplies everyone else’s output.

What it actually looks like

Let me ground this without turning it into a war story.

The concrete examples from my own work are the kind of thing I mean. I built an agentic harness — the orchestration layer I argued is the real determinant of AI coding outcomes, where the same model can swing more than a quality point depending on the scaffolding around it. I built MCP services, the org-specific connectors that replaced a handful of standalone tools in a few hours of directed work each. None of that was a production feature. All of it was infrastructure other people now depend on.

Here’s a detail that may make the point. I now have an “AI architect” on my leadership team helping maintain the very infrastructure I originally built — not just the harness, but our inference relationships, our training program, office hours, the real human work I no longer have the time, or the right, to be doing myself. And I expect to hand off more over time. The enabling work I do today partly becomes the system that does tomorrow’s enabling work. That handoff is the role in miniature: you build the thing that multiplies the team, then you put someone in place to build the next version.

The proportion matters. Around 30% hands-on keeps judgment fresh without putting me in the critical path. Even full-time senior ICs aren’t full-time coders — Bain’s Jue Wang, quoted in MIT Technology Review last December, put developer coding time at 20–40%, with the rest going to analysis, strategy, and the surrounding work. A leader at 30% isn’t doing something exotic. They’re spending their technical budget on the layer where it compounds.

The decision is not “how do I find time to code.” It’s “what enabling work is worth my own hands?” Those are different questions. The first leads to the bottleneck I watched so many executives become. The second leads somewhere useful.

The choice

I’ll resist overselling this, because it isn’t for everyone and it isn’t automatic.

This is a deliberate role, not a default. Staying technically current costs ongoing investment, and the work is often invisible — enabling infrastructure rarely shows up in a quarterly review the way a shipped feature does. The role is easy to misread, too. From the outside, a CTO who codes can look like a CTO who hasn’t let go. The defense against that reading is the discipline Majors named: stay out of the critical path. Build the multipliers, not the blockers.

The returns are real, though. Fresh judgment — the kind that lets you evaluate not just what and why but how. Trust from engineers who see you in the work rather than above it. And institutional infrastructure that makes the whole team faster, built by the person with both the technical depth and the positional authority to prioritize it.

There’s a closing note in the history worth keeping. Bunbu-ryōdō wasn’t only a personal aspiration. The Tokugawa shogunate institutionalized it — built career and class structures around the assumption that a leader should be both. China did the same with its examination system. We’re not there yet. For now, the leader/practitioner is an individual choice, made one person at a time, made viable by tools that finally collapsed the cost of staying hands-on.

But the precedent suggests where this could go. When a way of working proves durable, organizations eventually build structures around it. The era of the leader/practitioner is early. It is also, I’d argue, a return — to an integration we knew was valuable long before we had the means to make it practical again.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• From Coding with AI to Managing AI — When agentic coding starts to feel like delegation

• It’s The Harness, Stupid! — Why orchestration quality dominates AI coding outcomes

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Two stories from the last two weeks. Uber burned through its entire 2026 AI budget in four months on Claude Code, with COO Andrew Macdonald telling the Rapid Response podcast that the link between that spend and shipped consumer features “is not there yet.” The Information had the underlying numbers a few weeks earlier: engineer adoption from 32% to 84% between December and March, heavy users running $500–$2,000/month in tokens, and CTO Praveen Neppalli Naga torching $1,200 in a two-hour demo. Same week, Microsoft told thousands of engineers in its Experiences + Devices division that their Claude Code access is going away. Windows Central, summarizing The Verge’s Notepad scoop, has the cutoff at June 30 — end of fiscal year — with cost as the actual driver even though EVP Rajesh Jha framed it publicly as convergence on Copilot CLI.

Two of the most AI-forward enterprises on the planet, same tool, same week. The “AI is failing” takes were live within hours.

I don’t buy that framing.

The headlines are getting it wrong. Uber didn’t cancel anything — adoption ran ahead of the budget and the company blew its annual spend keeping up. That’s a planning failure, not a verdict on the tool. Microsoft didn’t divorce Anthropic either; they’re still consuming Claude through Azure Foundry and M365 Copilot. What they cancelled is a specific license — Claude Code at the engineer-seat level — because engineers preferred it over GitHub Copilot CLI and the division was paying for that preference.

What both stories show: AI is a new tool and we haven’t learned to use it well yet. The teams over budget pointed it at problems it wasn’t the cheapest way to solve, then let it decide for itself how much work to do per task.

I’ve made the cloud parallel here before. Early cloud was expensive and misused. Lift-and-shift workloads routinely ran two or three times their on-prem cost — I watched that play out across teams I ran, and it took years to correct through architecture. Then the industry learned: right-sizing, reserved instances, autoscaling, serverless where it fit, on-prem where it didn’t. The bills came down. Not because compute got dramatically cheaper, but because we got more careful about what we asked the cloud to do. AI is in the same phase. Cheap per-token, expensive per-task, and the gap is architectural.

A few weeks ago I ran controlled head-to-head tests on Opus 4.6 and Opus 4.7 against identical coding tasks. Both models passed every test. Opus 4.7 cost 3.6× more to do it. Same outcomes, same rate card, dramatically more tokens.

Finout’s analysis of production deployments tells the same story at scale: up to a 35% cost increase overnight, driven by tokenizer changes that don’t show up on the per-token rate card. Not one team’s bad luck — the shape of the bill across the enterprise AI buyer base right now. The second of two shoes on AI economics.

I wrote about that version-to-version cost drift in detail. Providers can collapse per-token prices in public while the per-task bill drifts upward in private. The first shoe was the per-token price collapse that made everyone optimistic. The second is the behavioral and architectural cost overhang now landing on quarterly P&Ls.

TL;DR

• Per-token costs at GPT-3.5-equivalent performance are down roughly 280× since late 2022, per Stanford’s AI Index 2025. Vendor revenue tells the opposite story: Anthropic’s annualized revenue went from $1B in January 2025 to $30B by April 2026 — a 30× move in 15 months, coming from enterprise inference, not consumer subscriptions.

• Gartner’s April 2026 survey: just 28% of AI use cases fully meet ROI expectations, 78% of IT leaders report material AI charges that didn’t show up in any procurement model.

• Gartner estimates agentic workflows consume 5–30× more tokens than equivalent chatbot interactions; Stanford’s Digital Economy Lab puts the upper bound for coding agents at 1,000×. The cost driver isn’t the model — it’s the workflow architecture wrapped around it.

• Two patterns hold the line in production. Search-first architectures put inference at the end of a deterministic pipeline. Consolidated single-shot designs replace multi-call chains.

• Inference is a power tool, not a default. Use it with specific ROI goals per call, apply it to code solutions rather than to directly solve problems, and bound it with deterministic structure on both ends.

Why the Per-Token Savings Didn’t Reach the Invoice

Per-token economics of frontier models have been collapsing for two years. Stanford’s AI Index 2025 puts the decline at roughly 280× from a late-2022 baseline at GPT-3.5-equivalent performance. Most enterprise budget conversations in 2024 started from that headline. The implicit assumption: bills should be going down.

They’re not. The clearest read comes from the vendor side. Anthropic’s annualized revenue went from $1B in January 2025 to $30B by April 2026, with roughly 80% from enterprise and API usage rather than consumer subscriptions. Anthropic now discloses 1,000+ customers spending more than $1M per year — a cohort that doubled in under two months — alongside roughly 300,000 business customers. The mid-tier ($100K–$1M/year) grew 7× year over year.

Menlo Ventures’ 2025 State of Generative AI report cross-checks at the market level: enterprise GenAI spend tripled to $37B in 2025, with LLM API consumption alone at $8.4B by mid-year. The high tier shows up in named deals: Snowflake’s $200M multi-year partnership implies a $50–70M annual run rate from one customer, and Deloitte is deploying Claude across 470,000 employees.

For a typical enterprise running multiple production AI workloads, $500K–$2M per year is now the realistic floor. Fortune 100 is running $10M–$50M+, the most AI-intensive past $100M. The Gartner numbers point the same direction: just 28% of AI use cases fully meet ROI expectations and 20% fail outright, and 78% of IT leaders report material AI charges that didn’t show up in any procurement model.

The per-token chart is real. The invoice is also real. What closes the gap is behavior. Three behaviors specifically.

Models do more work per task. Reasoning models reason. Agentic loops loop. The prompt that used to consume 4K tokens now consumes 40K because the assistant explores, plans, second-guesses, and verifies. Some of that is valuable. Much of it is the model performing thoroughness in a way that costs you money. The Opus 4.6-to-4.7 jump I documented earlier: same task, same outcome, 2.9× more output tokens and 4.8× more cache reads.

Workflows fan out. A “single” task in a modern agentic system might trigger a planner, researcher, coder, reviewer, and summarizer. Each makes its own LLM calls over overlapping context. Gartner’s March 2026 analysis puts agentic workflows at 5–30× the token consumption of an equivalent chatbot ask. Stanford Digital Economy Lab’s April 2026 arXiv paper goes further: coding agents can consume 1,000× more tokens than equivalent chat completions. The agent isn’t more expensive because it’s smarter. It’s more expensive because it’s louder.

Context windows fill themselves. Long context is a feature in marketing and a bill in practice. In our own enterprise Claude.AI usage — 82,852 messages from 329 employees over 3.5 months, audited via the Anthropic Compliance API — the average request carried 366,000 input tokens, mostly from 10-turn conversations dragging accumulated history forward into every new turn. Most production systems I’ve audited show the same fingerprint: pipelines paying for context they aren’t actually using.

None of this is fraud and none of it is mysterious. It’s the natural consequence of letting probabilistic systems decide how much work to do on every call. The savings from cheaper tokens were real. They just got consumed by an order of magnitude more tokens per task.

What I’ve Found Shipping These Systems

The teams handling this well aren’t the ones cutting AI usage. They’re changing the shape of how they use it.

The pattern I keep coming back to: treat inference as the expensive step at the end of a mostly deterministic pipeline. Do the cheap, structured work in code. Reserve the model call for the part that actually requires judgment. Then bound the call hard — context budget, output budget, quality gate on whether the call even runs.

Two examples from systems I’ve been building illustrate this from different angles.

Example 1: Code-Intelligence — Search-First Architecture

The naive version of a code review tool is obvious: dump the changed files into Claude, ask for a review. That works. It also costs roughly $0.03 per operation, scales linearly with repo size, and produces a lot of review output you didn’t need. Claude.AI offers a code review service — it ended up costing us thousands a month for just a few repos. Augment Code offers a well-regarded one as a GitHub app, but charges a platform fee (a meaningful fraction of our Anthropic spend) just to connect.

So we built our own. It leverages a multimodal search/RAG/KG engine I’d already built, so this wasn’t from scratch.

The version I actually ship uses a multi-tier search pipeline with the LLM call at the very end:

Stage 1: Vector Search    (~$0.0002 per query, semantic similarity)
Stage 2: BM25 Reranking   (~$0.0001 per query, lexical relevance)
Stage 3: Static Analysis  (~$0.0001 per query, AST + symbol resolution)
Stage 4: Quality Gate     (free, deterministic threshold check)
Stage 5: Single LLM Call  (~$0.03 per call, only if Stages 1-4 passed)

The first four stages cost about $0.0004 combined. They do the bulk of the work: deciding what code is actually relevant, ranking it, pulling structural relationships, and deciding whether the result is even worth asking an LLM about.

Hard budget controls run through the whole pipeline:

# Budget enforcement, not aspiration
MAX_CONTEXT_FILES = 6          # cap on what we send to the model
MAX_REVIEW_WORDS = 500         # cap on what the model returns
RELEVANCE_FLOOR = 0.005        # quality gate before calling the model

if combined_relevance_score < RELEVANCE_FLOOR:
    # no point spending $0.03 to get a review of weakly-related code
    return SkipReason("below relevance floor")

context = select_top_n(ranked_results, MAX_CONTEXT_FILES)
review = llm.review(context, max_output_tokens=MAX_REVIEW_WORDS * 1.4)

The RELEVANCE_FLOOR check is the part to underline. A meaningful percentage of review requests in real codebases don’t justify an LLM call at all — the changes are mechanical, the related code trivial, or the search signal weak enough that whatever the model says will be hallucinated context. Refusing to spend $0.03 on those cases is where most of the savings come from.

Rough economics across a quarter of usage:

Approach Cost per operation LLM calls per 1K operations Direct “review the diff” ~$0.030 1,000 Search-first with gates ~$0.0034 average ~430

About 80% of the workflow logic ends up deterministic: search, ranking, static analysis, gating. The model handles the last 20% — judgment on curated context. Same outcome from the user’s perspective, roughly an order of magnitude cheaper, with more predictable failure modes because most of the pipeline is debuggable code rather than prompt behavior.

The limitation: this is more work than wiring up a single LLM call, and the gates are only as good as your search infrastructure. The payoff is on the cost and determinism side, not on speed of initial implementation.

Example 2: duetto-intelligence — Context Injection Instead of Replacement

The second pattern comes from duetto-intelligence, internal tooling I’ve been building against that same enterprise Claude.AI usage — 82,852 real employee messages over 3.5 months, not a thought experiment. The problem here isn’t “should we call the LLM at all.” It’s: given that our people are already routing structured-data questions through a $0.274-per-request multi-turn Sonnet conversation, what’s a cheaper path that doesn’t degrade the answer?

The audit data made the gap concrete. Average request: 366K input tokens, ten-turn conversation, $0.274 to Anthropic. The same query answered through a Haiku single-pass against deterministically-retrieved internal data: $0.0009. A 300:1 cost ratio on the slice of traffic about structured product knowledge, CRM/account prep, JIRA, people and org lookups.

Not all traffic. Somewhere in the 35–40% range based on classified samples. About half of remaining queries genuinely need full Sonnet or Opus reasoning — writing, debugging, free-form analysis — and shouldn’t be intercepted at all.

The framing matters, because it’s easy to mis-read this as “replace Claude with a smaller model.” It isn’t. duetto-intelligence acts as a context injection layer in front of the user-facing model. When a query has structured-data intent, we route a sub-query to DI, get back a bounded structured result, and inject that into the prompt the larger model sees. The expensive model still does the reasoning — it just stops being responsible for the deterministic data retrieval it’s bad at and expensive for.

The naive design for the routing layer looks like this:

1. Classify the user's intent             → LLM call (~150 tokens)
2. Plan which subsystems to query         → LLM call (~250 tokens)
3. Call subsystem A, summarize response   → LLM call (~300 tokens)
4. Call subsystem B, summarize response   → LLM call (~300 tokens)
5. Synthesize a final answer              → LLM call (~250 tokens)
                                          Total: ~1,250 tokens, 5 calls

Each call is plausible on its own. Together they’re a tax on every user interaction. Latency stacks linearly with calls, and any one of the five can hallucinate in a way that corrupts the rest of the chain.

The consolidated design replaces three steps with deterministic code:

1. Classify intent                  → LLM call  (~80 tokens, tight tagger prompt)
2. Fan-out to subsystems            → code      (0 tokens, intent → call map)
3. Consolidated synthesis           → LLM call  (~200 tokens, structured input)
                                       Total: ~280 tokens, 2 calls

The trick is the intent classifier. It produces a tag from a fixed vocabulary of 87 tags — revenue.query.ytd, forecast.compare.year_over_year, account.lookup.contact, and so on. Each tag maps deterministically to a set of downstream calls in plain Python. No LLM in the routing step. The model isn’t asked “what should we do?” It’s asked “what is the user asking about?” — a much smaller, bounded question.

We validated against a 50-query test corpus drawn directly from the compliance data — real questions people had asked the model in production. After tuning, 100% of those queries land on the fast path with no LLM call required for routing. That’s the proof the deterministic-discipline part holds at the boundary; the routing isn’t quietly falling back to a second model call to bail itself out.

Budget enforcement is explicit in every prompt template:

SOURCE_CHAR_BUDGET = 600    # per data source pulled into context
OUTPUT_TOKEN_BUDGET = 200   # cap on synthesis response
INTENT_TAG_VOCAB = load_intent_taxonomy()  # 87 tags, versioned

def synthesize(intent_tag: str, sources: list[Source]) -> str:
    trimmed = [s.truncate(SOURCE_CHAR_BUDGET) for s in sources]
    return llm.complete(
        prompt=template(intent_tag, trimmed),
        max_tokens=OUTPUT_TOKEN_BUDGET,
    )

The economics, against measured baselines rather than estimates: current Claude.AI Chat spend across the 329-user population runs $15,264 over 3.5 months. Roughly $4,360/month, driven by that $0.274-per-request multi-turn average. If DI intercepts the 35–40% of traffic that’s structured-retrieval underneath, projected savings come in around $1,500–1,700/month, or $18–20K/year on this single user population. The leverage isn’t from picking a cheaper model. It’s from refusing to pay Sonnet rates to answer questions a deterministic system already has the data for.

The intent vocabulary is the contract. New capability means a new tag, a new downstream mapping, a new prompt template. The model never has to invent structure on the fly. This is what people mean by “use the LLM to code solutions, not to solve problems directly” — the routing logic lives in code, the tagger is a thin call, the synthesis is bounded.

The source-character budget matters more than the output budget. The compliance audit confirmed it: production overspend is on the input side. 366K input tokens against a few hundred output. Models will happily consume whatever context you hand them. Trimming at the source — 600 characters per source, no exceptions — is how you keep per-call cost from drifting upward as the system gets more capable.

The limitation: this only works on the routable slice. The pattern isn’t “eliminate inference.” It’s “stop spending $0.274 to answer questions that have a structured answer at $0.0009.”

Why “Just Use a Cheaper Model” Doesn’t Save You

A reasonable objection: aren’t the cheaper, smaller models supposed to handle this? Why not route everything to Haiku or an open model and call it solved?

The pricing math seems to support it. The behavioral math doesn’t.

Two things go wrong when you swap a cheaper model into an unstructured workflow. Cheaper models are usually less efficient per task — more turns to converge, more exploration, more hallucination, which means more retries and verification calls. A model 5× cheaper per token can run 1.5–2× more expensive per completed task if your workflow lets it spin.

And the workflow itself is where most of the cost lives. The 5–30× multiplier is structural, not modal — it exists regardless of which model you point at it. Switching from Sonnet to Haiku inside an unbounded agent loop changes the per-token cost. It doesn’t change the loop.

Model choice is a 2–5× lever. Architecture choice is closer to an order-of-magnitude lever in the systems I’ve shipped — consistently larger than what model swaps deliver. Most teams are over-tuning the model selection and under-tuning the structure around it.

The default assumption — including from vendors with strong incentives to sell you more tokens — is that the answer to AI cost is buying inference more cleverly. The actual answer is using inference less, more deliberately, with hard bounds on what each call is allowed to do.

How I Think About Inference Now

Inference is a power tool. Not a default. You don’t reach for it when a search query, a regex, or a switch statement would do. You reach for it when you need probabilistic judgment over unstructured input. Every call you don’t make is the cheapest call.

Use it to code solutions, not to solve problems. The highest-leverage use of LLMs in my workflow is generating the deterministic code that then handles the workflow without further LLM calls. A model that writes you a 50-line classifier is more valuable than a model that acts as the classifier on every request forever. The first costs tokens once. The second costs tokens every transaction for the life of the system.

Wrap every call in a budget. Context budget on the input, token budget on the output, quality gate on whether the call runs at all. Treat the LLM call as you’d treat a paid API with rate limits and SLA penalties. Because it is.

Set specific ROI targets per call. “AI-assisted code review” is too coarse to optimize. “Reviewing files with relevance score > 0.005, capped at 6 files, returning 500 words” is something you can measure cost-per-outcome on. Even loose ROI math at the call level surfaces where you’re paying for theater.

Treat behavioral cost as the primary risk. Model rate cards will keep coming down. They are not your problem. Your problem is what your pipeline asks of the model and what the model decides to do once asked. That’s the line item that grew while the unit cost dropped 280×. That’s the shoe that just dropped.

What This Means If You’re Running an AI Budget

Three things to look at, in order of how much they’ll move the line item.

Audit the call graph, not the rate card. Pull a representative day of production traffic and trace the actual LLM calls per user task. Count them. Most teams find a handful of workflows producing the majority of cost, and most of those have 2–4 LLM calls that could be replaced by deterministic code. That’s the consolidated-design pattern from the duetto-intelligence example. 50–80% reductions are common when you actually look.

Put quality gates in front of inference. For any workflow where the LLM call is expensive and the input quality is variable, add a deterministic check that decides whether the call is worth making. That’s the search-first pattern from the code-intelligence example. The savings come from the calls you don’t make, which never show up on the invoice.

Set hard context budgets and enforce them in code. Per-source character limits, per-call token caps, no “just in case” context stuffing. The output budget gets attention because it’s visible. The input budget is usually where the actual money goes.

None of this requires changing models, switching providers, or making bets on the next frontier release. It’s architectural work inside the pipeline you already have — work the per-token price chart has been letting people defer.

The teams that do it over the next two quarters will look like they got a 5–10× cost improvement from “AI getting cheaper.” The teams that don’t will look like AI got 3–4× more expensive while everyone else’s costs fell. Same providers, same models, same rate cards. Different shoe.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Opus 4.6 vs 4.7: The Real Cost of Incremental AI Improvements — The first shoe, on per-task cost drift between model versions

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

My Duetto colleague Jake Becker is sharp. He’s been ahead of AI adoption on our team — experimenting early, pushing for new tools, staying current. Last week he messaged me on Slack: “I wish I had my own CTO Assistant. Like what you have.”

I paused.

I have one. I’ve had several, in different forms, going back months. But I hadn’t said anything about it publicly.

That gap — between an AI-forward person who knows the tools and a practitioner who actually has the thing — is what this piece is about.

TL;DR

• Off-the-shelf AI tools are capable but contextually blind. You still have to leave your domain to get help.

• Serious knowledge workers — writers, researchers, historians — have always built their own knowledge systems. Never trusted a vendor to hold their material.

• Coding is shifting toward what writing has always been: directing, curating, maintaining a living body of knowledge.

• Building your own AI memory and search layer is the new rite of passage. The tool you build IS the skill being developed.

• The latest versions of my own stack: trusty-memory and trusty-search — both open source, both installable today.

Everyone Wants One. Few Have One.

Jake’s comment revealed something I’d been taking for granted.

The major vendors are actively working on this — memory features, RAG pipelines, personalization layers. But those solutions are optimized for breadth, not for one person’s actual work across functions.

ChatGPT, Claude, Copilot — these are all capable. They’re also still contextually blind to you. You have to leave your environment, paste in context, explain your situation from scratch, then interpret the output back into your actual work. Vendors are working on it. But the solutions remain generic, and every session still starts from zero.

I wrote about this in March — Everyone Blamed Clawd Bot’s Execution. The Concept Was the Problem. The structural flaw of universal assistants isn’t fixable. They require you to leave your context to get help. What actually works is the opposite: your tools get assistant capabilities, and assistance comes to where your context lives.

Off-the-shelf tools haven’t solved this. They’ve gotten more powerful — better reasoning, longer context, faster inference — but they still don’t know your codebase, your decisions, your institutional history, your current sprint. They know a lot about the world, and very little about you.

Jake wanted my assistant. But what he actually wants is his assistant. The one that knows what he knows.

That’s a different problem entirely.

The job changed first.

Coding Is Becoming Writing

Practitioners feel it before analysts name it.

A few years ago, being a strong engineer meant writing a lot of code quickly and correctly. Today, with agentic AI coders at their disposal, the best engineers I watch spend their time directing, reviewing, specifying, and curating. The unit of work has moved up a level. Implementation is increasingly delegated. Judgment — about architecture, trade-offs, what to build and why — is the differentiator.

This is not what happens when automation replaces a skill. It’s what happens when a new discipline appears.

Writers have always worked this way. A novelist doesn’t produce words per minute as a primary metric. They produce decisions — what to say, in what order, with what emphasis. The words are the output of the decisions, not the work itself. What makes a writer productive over a career isn’t typing speed. It’s having a system: notes, research, accumulated material, patterns of thought that compound over years.

Some (typically senior) engineers are arriving at the same realization. Your value isn’t the code. It’s the judgment, the accumulated context, the knowledge of what was tried and why it failed. The question is whether that accumulates in your head alone — which doesn’t scale, and doesn’t survive a context switch — or whether it lives in a system.

Good engineers who learn to use their AI tools effectively generate better code — the stack amplifies judgment and accumulated knowledge. The gap isn’t between skilled and unskilled engineers in isolation; it’s between engineers who’ve wired their knowledge into their tools and those who haven’t. The knowledge is real in both cases. Only in one case does it compound.

The writers I’ve observed who sustain serious output over decades all have the same property: they know where things are. Their research is retrievable. Their earlier thinking is available to their current thinking. The system makes the person bigger than their working memory.

That’s the gap.

Writers Don’t Trust Vendors With Their Material

Niklas Luhmann, a German sociologist working in the 1950s, produced 70 books and nearly 400 articles over his career. I haven’t written a book yet — but I have published over 200 articles at HyperDev. Worth naming the parallel. He credits his output to his Zettelkasten — a slip-box system of 90,000 interconnected index cards, each with a unique identifier linking it to related thoughts. Not a filing cabinet. A network of ideas that got richer with every addition.

And it’s worth asking: is that so different from what an AI knowledge system does today? The Zettelkasten was an analog precursor to what trusty-memory and trusty-search do programmatically — indexing ideas, linking related thoughts, surfacing connections that wouldn’t otherwise be visible. Luhmann was doing manually what these tools do automatically. Same architecture. New substrate.

He didn’t use a vendor product. He built a system that reflected how he thought. The architecture of his Zettelkasten was itself an expression of his intellectual method.

This isn’t a historical quirk. It’s a pattern. Serious knowledge workers have always built their own systems — commonplace books, research archives, private wikis. The reason is structural: the schema you design reflects how you think. That’s not something a product gives you. A product gives everyone the same schema.

Andrej Karpathy pointed at something similar last month with his LLM Wiki gist. His framing: use LLMs not just to write code, but to build and maintain a personal knowledge base. “Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase.” Three folders, structured Markdown, a large context window. He concluded: “I think there is room here for an incredible new product.”

He’s right there’s room. I wrote about his framing in What’s In Your Second Brain? The product comment is where I’d push back. You can build tooling around the pattern. You can’t productize the schema. The schema is the moat — because it reflects how you think, not how a product manager thinks you think. The ones who get it aren’t waiting for a product.

The Lightsaber Rite of Passage

In Star Wars canon, a Padawan doesn’t receive a lightsaber. They build one.

The ritual is called the Gathering. Initiates travel alone to the Crystal Caves of Ilum. They have to find their kyber crystal — the crystal that’s attuned to them through the Force. The caves are shaped by the initiate’s own fears and insecurities. The crystal doesn’t go to the strongest or the fastest. It bonds with the person who confronts what’s in the way.

Then they build it themselves, guided by Professor Huyang.

You can’t buy this. You can’t inherit it. The construction is the training. The tool reflects the builder.

I’m not the first to reach for this metaphor in tech. But I think it lands differently now. Building your own AI memory and search layer isn’t just useful. It’s diagnostic. You can’t do it without confronting what you actually know, how you actually think, what deserves to persist and what doesn’t. The schema you design for your knowledge base is a statement about your mind.

The engineers I know who are operating at the highest level right now — CTOs, senior architects, tech leads at places moving fast — they all quietly roll their own. They don’t announce it. They just have it.

My Own Lineage

I’ve been building versions of this for months.

trusty-izzie was the first — a simple wrapper. Then ai-commander, a more structured approach to context management. Then open-mpm and claude-mpm, which was where I started thinking seriously about multi-agent orchestration. Then kuzu-memory, a graph-backed memory layer. Then mcp-vector-search, semantic search over my entire codebase.

Each iteration taught me something about what I actually needed. Not what I thought I needed. What the practice revealed.

This piece was drafted with a configured writing assistant — claude-mpm loaded with my publication style guide, my voice patterns, my article archive. That’s a saber too. Not a generic chat interface. A tool shaped around how I think and write, producing work I can actually publish rather than work I have to fix. The saber list keeps growing.

The latest two are the most capable tools I’ve built.

The Latest Sabers

trusty-memory is a machine-wide AI memory daemon written in Rust. It uses what I call the Memory Palace architecture — multiple named palaces, each for a different domain. Sub-5ms baseline retrieval on Apple Silicon. It runs as an MCP server for Claude Code, which means my assistant stores and retrieves memories automatically, across sessions, without me managing any of it explicitly.

cargo install trusty-memory

Available at crates.io/crates/trusty-memory.

trusty-search is a machine-wide hybrid code search daemon, also in Rust. Always-on, one install per machine. It combines BM25 lexical search with HNSW vector search (all-MiniLM-L6-v2 INT8) and a Knowledge Graph with 1-2 hop expansion, fused via Reciprocal Rank Fusion. It exposes an MCP server with 11 tools. Stdio and HTTP/SSE transports drop straight into Claude Code.

cargo install trusty-search

Available at crates.io/crates/trusty-search.

These tools, along a set of custom reporting pythons apps along with a custom Slack Bot I use to access the data remotely, comprise my digital brain.

These aren’t products I bought. These are tools I built, iterated, and use daily. They know my codebase the way a Zettelkasten knows a scholar’s intellectual territory — not because a vendor configured them, but because I did.

To be precise: trusty-memory and trusty-search are infrastructure utilities — the memory layer and the search layer. Building the actual assistant that uses them is a separate act of customization. That’s where the lightsaber metaphor completes: the kyber crystal is only part of it. The construction — what you build with the crystal — is the saber.

When Jake said he wished he had a CTO Assistant, this is what he was gesturing at. Not a prompt template. Not a workflow. A living knowledge layer that compounds.

The Right Question

Jake asked: “Can I get a CTO Assistant?”

That’s the wrong question. It assumes the thing is available off the shelf, and the task is finding and configuring it.

The right question is: “What would it take to build one that knows what I know?”

That question is harder. It requires confronting the shape of your knowledge, what’s worth persisting, how to structure retrieval. It’s uncomfortable in the same way the Crystal Caves are uncomfortable — not because the work is technically difficult, but because you have to be honest about what you actually have.

Not everyone needs to write Rust. The specific technology isn’t the point. The point is that the engineers asking the right question are already operating differently. They’re working like writers — maintaining a living body of knowledge, building systems that compound, treating their accumulated context as an asset rather than a liability.

Writers’ discipline has been creeping into engineering for a while. AI made it urgent.

If you’re waiting for a product to hand you the thing, you’re waiting for someone to build your lightsaber. It won’t be yours.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• What’s In Your Second Brain? — Karpathy’s LLM Wiki and the case for a compounding knowledge layer

• Everyone Blamed Clawd Bot’s Execution. The Concept Was the Problem. — Why universal assistants are architecturally broken

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Six months ago, I abandoned my Rust/Tauri-based writing app. Not because Tauri was bad—it wasn’t. Obsidian simply worked better for my needs. But that experience with Rust left an impression: the compiler was unlike anything I’d encountered. It didn’t just catch errors; it made entire classes of bugs impossible.

Then I built AI Commander for managing tmux sessions. Python would have been the obvious choice—async programming, process management, plenty of libraries. But I chose Rust, partly out of curiosity, partly because I needed rock-solid thread control. The result was revelation: code that compiled cleanly worked correctly, and performance was extraordinary.

Now I’m deep into replacing both mcp-vector-search and kuzu-memory with pure Rust implementations. My new trusty-search outperforms the Python predecessor by an order of magnitude, even though that version used compiled Python extensions. This isn’t an isolated case—it’s part of a fundamental shift happening in AI development.

Rust isn’t just another systems language. It’s becoming the infrastructure language of AI, and the reason has everything to do with letting the compiler handle the heavy lifting—especially as AI systems increasingly write their own code.

The Performance Reality Check

The performance gains are measurable and significant. Hugging Face’s tokenizers library, rewritten in Rust, delivers 10-100x speedups over pure Python implementations. Polars, the Rust-based DataFrame library, consistently outperforms pandas in data processing benchmarks. In embedded systems, Rust achieves 98% of C performance while eliminating memory safety issues entirely.

This isn’t just raw speed. Memory efficiency matters significantly in production AI systems. Where Python applications might consume gigabytes for large-scale data processing, equivalent Rust implementations often require 3-5x less memory. The difference compounds when deploying AI systems at scale.

This performance advantage stems from a fundamental philosophical difference in how Rust approaches the safety-speed trade-off.

When your AI system processes millions of sensor events per second in an autonomous vehicle, or makes split-second trading decisions with millions at stake, runtime failures aren’t just inconvenient—they’re catastrophic. Python’s flexibility, its greatest strength for research and experimentation, becomes a liability in these production environments.

The Compiler Does the Heavy Lifting

Here’s where Rust’s philosophy is revolutionary. Most languages force you to choose between safety and performance, between readable code and efficient code. Rust rejects this trade-off entirely through what it calls “zero-cost abstractions.”

The principle, borrowed from C++ but perfected in Rust, is elegantly simple: “What you don’t use, you don’t pay for. What you do use, you couldn’t hand code any better.”

Consider async programming, critical for AI systems handling multiple data streams. In Python, async operations carry runtime overhead—coroutine scheduling, context switching, memory allocation for tasks. In Rust, the compiler transforms your high-level async/await code into efficient state machines. No runtime scheduler, no hidden allocations, just bare-metal performance wrapped in readable syntax.

The genius is that the compiler absorbs the complexity. You write code that looks high-level and readable:

async fn process_ai_requests(stream: &mut DataStream) -> Result<Vec<Response>, Error> {
    let mut responses = Vec::new();

    while let Some(request) = stream.next().await {
        let response = ai_model.infer(request).await?;
        responses.push(response);
    }

    Ok(responses)
}

But the compiler generates assembly that rivals hand-optimized C. The ownership system prevents data races without locks. The borrow checker eliminates memory leaks without garbage collection. The type system catches logic errors that would be runtime failures in dynamic languages.

This is the heavy lifting: converting human-friendly abstractions into machine-efficient reality, all at compile time, with zero runtime cost.

This compiler-centric philosophy becomes even more critical as we enter an era where AI systems increasingly write their own code.

The AI Code Generation Protection Paradox

As AI-powered coding tools like Copilot, Claude Code, and Cursor become mainstream, a new challenge emerges: how do you ensure correctness when the code author might not fully understand what they’ve written?

When Claude or Copilot generates Python, Java, TypeScript, or C code, human oversight becomes the primary error-checking mechanism. Code reviewers need to catch type mismatches, memory leaks, race conditions, and logic errors that might not surface until production. As AI-generated code becomes more prevalent and complex, human review processes struggle to keep pace with the volume and complexity of generated code.

Think about it: an AI might generate a perfectly logical-looking concurrent data structure in Python that works flawlessly in single-threaded tests but creates subtle race conditions under load. A human reviewer might miss the threading implications entirely. The same AI generating Rust code? The compiler would reject it outright if the data sharing wasn’t provably safe.

Rust’s compiler fundamentally changes this dynamic. It doesn’t matter if the code was written by a human, an AI, or a collaboration between both—if it compiles, entire categories of dangerous bugs simply cannot exist. Memory safety violations? Caught at compile time. Data races in concurrent code? Impossible. Use-after-free errors? Eliminated by the ownership system.

This creates a fascinating inversion: Rust may be the best language for AI-generated code precisely because it doesn’t trust the programmer. While other languages rely on developer discipline and code review to catch errors, Rust enforces correctness through the type system and ownership model. The compiler becomes an automated, exhaustive code reviewer that never gets tired, never misses subtle bugs, and never waves through “probably fine” code.

As Microsoft research demonstrates, 70% of security vulnerabilities stem from memory safety issues that could be eliminated at compile time. When AI systems generate infrastructure code, this shifts from a productivity concern to an existential safety issue.

The compiler isn’t just doing heavy lifting for performance; it’s providing safety guarantees that scale beyond human oversight.

Memory Safety in the Age of AI

Rust’s creator, Graydon Hoare, designed the language after getting stuck in a broken elevator caused by software crashes. His goal was simple: write fast, small code without memory bugs. That mission has become critical as AI systems move from labs to production infrastructure.

In traditional software, memory safety issues mean patches and updates. In AI systems controlling physical infrastructure—autonomous vehicles, medical devices, financial trading systems—they can mean catastrophic failure.

Rust’s ownership model doesn’t just prevent crashes; it eliminates entire categories of bugs that plague concurrent AI workloads. While C++ applications in multi-threaded environments regularly exhibit race conditions during testing, Rust’s compile-time guarantees make such issues structurally impossible.

But here’s the crucial part: this safety comes at zero runtime cost. Unlike garbage-collected languages that trade performance for memory safety, Rust enforces safety through compile-time checks. Your production AI system runs with the performance of C but the safety guarantees that traditional systems languages can’t provide.

The Python-Rust Symbiosis

Here’s where the narrative gets interesting. The emerging trend isn’t Rust replacing Python—it’s Rust and Python working together. The PyO3 bridge, Rust-powered Python tools like Ruff and uv, and the growing number of “Python API, Rust engine” libraries show that the two languages are becoming symbiotic.

Python remains dominant for AI research and experimentation. The ecosystem is unmatched: PyTorch, TensorFlow, scikit-learn, Hugging Face Transformers. PyTorch’s 100,000+ GitHub stars reflect an ecosystem that won’t disappear overnight. When you’re prototyping models, analyzing data, or building internal tools, Python’s flexibility and ecosystem make it irreplaceable.

But production tells a different story. When performance, memory usage, and reliability matter—when you’re building the infrastructure that powers AI systems rather than the models themselves—Rust increasingly dominates.

The pattern emerging across tech giants is clear: Python for the interface layer, Rust for the infrastructure layer. Experimentation happens in Jupyter notebooks; production happens in compiled Rust binaries.

Looking Forward: Infrastructure vs. Experimentation

This division isn’t arbitrary—it reflects the maturation of AI from research curiosity to critical infrastructure. Research requires flexibility, rapid iteration, and access to cutting-edge libraries. Production requires reliability, performance, and safety guarantees.

Rust excels at infrastructure because the compiler handles the complexity of building robust systems. Memory safety, concurrency, error handling—all the concerns that make production systems hard to build correctly—are enforced by the type system rather than relying on developer discipline.

Consider the trajectory: Microsoft aims to eliminate C and C++ from their codebase by 2030, replacing it with Rust. The Rust Foundation officially positions the language as ideal for “ultra-reliable AI systems.” Major tech companies are increasingly choosing Rust for performance-critical AI infrastructure. These aren’t experimental projects—they’re billion-dollar bets on where AI infrastructure is heading.

The Compiler Advantage

What makes Rust uniquely suited for AI isn’t just performance or safety—it’s the fundamental approach of front-loading complexity into the compiler rather than dealing with it at runtime.

In AI systems, runtime failures are exponentially more expensive than compile-time failures. A training job that crashes after days of computation. An inference system that memory-leaks during peak traffic. A edge AI device that locks up in a safety-critical situation. These failures cascade through systems in ways that research environments rarely encounter.

Rust’s compiler is exhaustive in a way that benefits AI development specifically. It catches data races in concurrent model training. It prevents buffer overflows in tensor operations. It enforces proper error handling in distributed inference systems. It does this without performance overhead, without runtime monitoring, and without hoping that comprehensive testing caught every edge case.

The compiler becomes your co-pilot in building reliable AI infrastructure.

The Personal Proof

My own experience validates this thesis. AI Commander needed complex thread management for tmux session control—exactly the kind of concurrency that’s error-prone in traditional languages. Rust’s ownership system made data sharing across threads not just safe, but intuitive. The code that compiled was correct.

With trusty-search, the performance gains over mcp-vector-search weren’t just about Rust being “faster than Python.” They were about Rust enabling architectural choices—zero-copy string processing, efficient memory layouts, fine-grained concurrency control—that would be risky or impossible in garbage-collected languages.

These aren’t micro-optimizations. They’re fundamental design differences that become critical at scale.

The Real Limitations of Rust for AI

Before declaring Rust the future, it’s essential to acknowledge where it falls short compared to Python in AI contexts.

The learning curve is steep. Rust’s ownership model requires a fundamental mental shift that can slow initial development. Developers comfortable with garbage-collected languages often struggle with borrowing rules, especially when building complex data structures. The compiler’s strict requirements, while beneficial long-term, can feel obstructionist when prototyping.

The ecosystem gap remains significant. While Rust has impressive performance libraries, Python’s AI ecosystem is vast and mature. TensorFlow, PyTorch, scikit-learn, and thousands of specialized ML libraries have no Rust equivalents. Building production ML pipelines often requires library combinations that simply don’t exist in Rust.

Compilation time can kill iteration speed. Where Python allows instant feedback during development, Rust’s compilation process—especially for complex projects—can introduce friction that slows experimentation. This matters significantly in AI research where rapid iteration drives discovery.

Talent pool challenges are real. Finding experienced Rust developers is substantially harder than finding Python developers. The language’s complexity means onboarding takes longer, potentially impacting team velocity and project timelines.

Not every AI workload benefits from Rust’s strengths. Data analysis, model training with existing frameworks, and one-off scripts often don’t require the safety and performance guarantees Rust provides. Using Rust for these tasks can be overkill that reduces productivity without meaningful benefits.

These limitations explain why the Python-Rust symbiosis model makes sense: leverage each language where it excels rather than forcing one-size-fits-all solutions.

The Future is Compiled

As AI systems become infrastructure—as they move from experimental notebooks to production systems handling millions of users—the languages that power them will need to provide the guarantees that infrastructure requires.

Python will continue to dominate the experimental and interface layers. But the backbone, the high-performance inference systems, the real-time edge deployments, the safety-critical applications—these will increasingly be built in languages where the compiler, not the runtime, ensures correctness.

Rust represents a fundamental shift in how we approach building reliable systems: moving complexity from runtime to compile time, from testing to proving, from hoping to knowing.

This shift becomes critical as we enter the era of AI-generated code. When AI systems are writing the infrastructure that runs other AI systems, traditional human oversight breaks down. We need tools that can verify correctness automatically, at compile time, without relying on human reviewers to catch subtle but catastrophic errors.

The compiler does the heavy lifting so production systems can focus on their actual job: running AI workloads reliably, safely, and at scale. In a world where AI infrastructure is becoming as critical as power grids and financial networks—and where that infrastructure is increasingly written by AI itself—that guarantee isn’t just nice to have, it’s existential.

The age of agentic AI isn’t just about better models. It’s about building the reliable, high-performance infrastructure those models need to operate in the real world, written by AI systems that can’t be trusted to write safe code in traditional languages. And increasingly, that infrastructure is being built in Rust, one compile-time guarantee at a time.

Bob Matsuoka writes about the intersection of software engineering and AI at HyperDev. His latest Rust projects replace Python infrastructure with performance gains that would make even a compiler blush.

Leonard Shelby can’t form new memories. In Memento, after the attack, every fifteen minutes or so his short-term recall resets. He has a problem most engineers would recognize: a fast working scratch space and no durable storage. So he builds a system. Polaroids with handwritten captions on the back. A wall of pinned notes. Tattoos for the load-bearing facts — the ones he can’t afford to lose, can’t afford to misfile, can’t afford to look up wrong.

What he doesn’t do is try to remember everything. He doesn’t carry a thicker notebook. He builds infrastructure to find what matters, fast, when he needs it. The pocket holds what’s relevant right now. The system decides what makes it into the pocket.

I’ve been thinking about Leonard a lot this year, because the prevailing argument in my corner of the internet is that LLMs no longer need him. The pocket got bigger. Frontier models can hold a million tokens of context. Token prices fell. So just stuff the pocket. Skip the indexing. Skip the tattoos. Read everything.

I think that’s the wrong read of where we’re going.

The real argument, fairly stated

The “RAG is dead” position isn’t crazy. It goes roughly like this: a handful of frontier models — Gemini 1.5, Claude’s extended context tiers — can now handle contexts approaching 1M tokens without choking. Token prices have fallen to around $0.60 per million for several frontier models. Standing up a real retrieval pipeline — chunking strategy, embedding model, vector store, hybrid scoring, reranker, eval harness — is 40 to 80 engineering hours, easily. For a small, static corpus you query a few thousand times a month, just dumping the whole thing into context with prompt caching is simpler and arguably cheaper.

Claude Code is the existence proof people point at. It doesn’t ship a vector database. It uses ripgrep, file globs, and reads files into context. And it works pretty well. So why bother with the rest?

I’ll concede the envelope. For a 300K-token corpus, single tenant, low query volume, mostly static — yeah, stuff it. Cache it. Read it. Don’t build a search stack to feel sophisticated. That’s a real position and I don’t want to strawman it.

But that envelope isn’t where most of the work lives. And the argument the loud version makes — that grep replaces search, that long context replaces retrieval — confuses two different things; there are cracks in that argument.

The pocket problem

Here’s the first crack. “Lost in the middle.” Liu et al., 2023, TACL: when you stuff long contexts, model performance forms a U-shape. Strong recall at the beginning, strong recall at the end, degraded recall in the middle. The 2026-gen frontier models have improved on this — early evals suggest the curve is flatter — but it’s not gone.

So the first thing the bigger pocket buys you is the ability to put a lot of important stuff exactly where the model is most likely to underweight it. You can’t reorder a 700K-token dump by relevance unless you’ve already done retrieval. Which means the question doesn’t go away — it just hides.

Second crack: cost. RAG-style queries — retrieve relevant chunks, feed 8–16K of context — run around $0.005–$0.008 per request at $0.60/million input tokens, when you count the full prompt. A naive full-context pass against a 500K-token corpus runs $0.30 in input tokens alone, before output. That’s a 40–60x ratio at list price, and it widens if you have high cache-miss rates or large output windows. At 50K daily queries — not enormous, this is a mid-sized internal tool — you’re choosing between a few hundred dollars a day and tens of thousands. That’s not a rounding error you absorb. That’s a re-platforming decision that wakes someone up.

Third crack, and this is the one that actually makes me grumpy. Grep is not search. Grep is lexical pattern matching. Searching for auth_token finds the string auth_token. It finds it in commented-out dead code from 2023. It finds it in test fixtures with hardcoded sample values. It finds it in a vendor library you don’t even own. It misses the function called refresh_session_credential that does what you’re actually looking for, because the words are different. Vector search finds that function. Grep doesn’t. They’re not the same tool. Pretending they are is how you ship the wrong fix at 2 AM.

When people say “Claude Code uses grep, so search is over,” they’re describing a tool optimized for a bounded, file-system-structured corpus where the author controls the index shape. That’s a legitimate architecture for that problem. It does not generalize to “ten million product images” or “a 30-year codebase across 800 repos” or “tenant-scoped documents under per-row access control.” Long context can’t replace a knowledge graph, either. You can serialize a graph into text — flatten the edges, encode the relationships — but then you’ve lost typed edges and efficient query-time traversal. You can’t join across it. You can’t reason about edge types. You’d need to flatten the graph to feed it in, at which point you’ve discarded the structure that made it useful.

With me so far? The argument isn’t “long context is bad.” Long context is great. The argument is that long context is a consumer of retrieval, not a replacement for it.

What good retrieval actually looks like

A modern retrieval stack, the kind that holds up under real query volume, isn’t one thing. It’s a layered pipeline.

Lexical first — BM25, TF-IDF, the unfashionable stuff. It’s still excellent at exact tokens, identifiers, error messages, version numbers. The things vector embeddings smudge.

Dense second — vector search over embeddings. all-MiniLM-L6-v2 is the workhorse at 384 dimensions. Catches semantic neighbors, paraphrases, the function-named-different-but-doing-the-same-thing case.

Fuse them — Reciprocal Rank Fusion at k=60 is the standard, and it’s standard because it works. You don’t pick BM25 or vector. You merge their rank lists.

Rerank — a cross-encoder over the top-k from RRF. Slower per-pair, but you only run it on the candidates that made it through the cheap layers.

The numbers from a 2026 benchmark pass I was reading — “From BM25 to Corrective RAG,” April 2026 — make the layered case cleanly:

• Dense vectors alone: Recall@5 of 0.587

• BM25 alone: 0.644

• Hybrid via RRF: 0.695

• Hybrid + neural reranker: 0.816

That last number is 39% better than dense-only. It’s also 17% better than RRF without the reranker. The layers compound. In systems that need both recall and precision at scale, you usually want all four layers. And — this part matters — none of them are replaced by a bigger context window. You still have to pick what goes into the pocket.

There’s a piece on top of this, too: intent. A query like “where is process_payment defined” is not the same shape as “how does the refund flow handle partial captures.” The first is a definition lookup; BM25 should dominate, you want exact identifier matching, you don’t want semantic neighbors fuzzing the result. The second is conceptual; dense embeddings should dominate, you want the chunk that explains the flow even if the words don’t match. Treating those queries identically is how you get plausible-but-wrong answers.

The EMNLP 2024 “Self-Route” paper made a related point I keep coming back to: let the model itself decide whether a query needs retrieval or full-context, based on complexity. They got better accuracy AND lower compute cost. The two approaches aren’t enemies. They’re complements with different cost/quality envelopes, and a working system uses both.

(A note on scope: I’ve been using “search” and “memory retrieval” somewhat interchangeably here, which isn’t quite right. They’re the same pipeline mechanics — retrieve, rank, inject — but different problems. Memory retrieval has a temporal model search doesn’t: recency matters, decay matters, and the source is agent history rather than a document corpus. Memory is also primarily graph-based, because what you’re trying to recover is relationships and context across time, not just chunks of text. For the purposes of this argument — context window vs. retrieval discipline — the distinction doesn’t change anything. But it’s worth naming.)

What I’m building

I have two projects in this space. I’ll be specific about what each one is and isn’t, because the field is full of demos that don’t survive contact with real corpora.

mcp-vector-search is the older one. Python, per-project, designed to live next to a codebase as an MCP server. The retrieval core is hybrid: BM25 plus HNSW over MiniLM embeddings, with knowledge-graph expansion built from tree-sitter AST parsing — function definitions, call edges, import edges, type relationships. Fifteen-plus edge types in the graph. Cross-encoder reranking on top of RRF. MMR for diversity so you don’t get five near-duplicates in the top results. Temporal decay weighted by git blame age. Cyclomatic complexity factored into ranking, because dense, gnarly code is more often what you’re looking for than the trivial wrappers around it.

trusty-search is what I’m building toward. Rust daemon, machine-wide rather than per-project, multi-tenant across all my work. Same retrieval core — RRF at k=60, MiniLM embeddings, BM25 plus HNSW — but with intent classification on the front. Queries get tagged Definition, Usage, Conceptual, or BugDebt, and each intent has pre-tuned alpha/beta weights between lexical and dense. Sub-10ms p50 on warm queries.

I’ll be honest: trusty-search is early-stage. Some of the features I just listed are scaffolding with stubs underneath. The intent classifier is rule-based at the moment and needs to become a small model. The two tools are complementary rather than competing for now — mcp-vector-search is the deeper analysis layer, trusty-search is the fast facts store you hit constantly during a session — though the long-term plan is for trusty-search to replace mcp-vector-search.

The reason I’m building both is that I keep running into the same wall: I can give an agent a million-token window and it will still ask the wrong question of the wrong file. The bottleneck is not how much I can stuff in. The bottleneck is which 8K of the 800K matters for this query. That’s a search problem. It’s been a search problem for sixty years. It’s still a search problem.

Back to Leonard

The reason Leonard works as an analogy isn’t the amnesia. It’s the discipline. He decides what makes it onto a polaroid. He decides what gets tattooed and what stays loose. He writes “DON’T BELIEVE HIS LIES” on the back of a photo because he won’t have the context to evaluate trustworthiness later — so he encodes the conclusion now, into a system he’ll find when he needs it.

The context window is the pocket. Whatever Leonard pulls out and looks at right now. It can be enormous. It can be cached. It can be cheap. None of that decides what goes in.

Retrieval is the tattoo on his wrist. The polaroid pinned to the wall. The system that decides which fact survives, which one is one query away, which one is buried. The job of that system is exactly the job that doesn’t go away when the pocket gets bigger.

The question was never “do I need RAG?” That framing turned a design discipline into a vendor category and made it easy to dismiss. The question is the one Leonard asks every morning: what do I actually need to find, and have I built the thing that finds it? At a million tokens, you’re not freed from that question. You’ve just made it more expensive to answer incorrectly.

It feels crazy that I wrote this only a year ago…

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Context Memory and Search: The Secrets to Effective Agentic Work — Why context management, memory, and retrieval are the three pillars of effective agentic systems

• What’s In Your Second Brain? — Tooling for the modern CTO: how to build external memory that actually works

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

TL;DR: I spent years avoiding AWS due to its overwhelming complexity, preferring the simplicity of Vercel and composable stacks. Then Claude MPM changed how I work with AWS. Now I’m running sophisticated multi-service AWS deployments with GPU instances, comprehensive monitoring, and infrastructure-as-code—all managed through AI-assisted tooling. The lesson: agentic approaches transform ops workflows just as profoundly as they do coding.

Remember when AWS felt like digital quicksand? Every innocent “let me just deploy this simple app” spiraled into an afternoon lost in IAM policies, VPC configurations, and security group rules that made no sense. I’d start with what should be a five-minute deployment and emerge three hours later, bleary-eyed, with a working application and absolutely no confidence I could recreate the process.

That’s why I became a Vercel evangelist. One git push, automatic deployments, zero configuration. The developer experience was everything AWS wasn’t: predictable, fast, and actually enjoyable. For most projects, this composable stack approach—Vercel for frontend, managed databases, serverless functions where needed—delivered exactly the right balance of power and simplicity.

But somewhere along the way, that changed.

The Claude Code/MPM Turning Point

Today I’m running the kind of infrastructure I would have delegated to an ops team a year ago. GPU instances for ML workloads. Multi-AZ deployments across six subnets. Sophisticated monitoring pipelines that integrate CloudWatch metrics, Cost Explorer analysis, and automated GitHub issue creation. Terraform managing multi-account infrastructure with cross-service dependencies.

The difference? I don’t even need AWS’s Q assistant. I have something better: purpose-built AWS skills in Claude MPM that handle service deployment, infrastructure analysis, and operational workflows.

This transformation illustrates something crucial about the agentic revolution: it’s not just changing how we write code. It’s fundamentally altering how we approach operational complexity.

The Infrastructure Reality Check

Let me show you what I mean with real numbers. Here’s what I’m actually running across three active projects to support internal tools:

CloudWatch Reporting Service (Serverless):

• 12 Lambda functions handling health checks, metrics aggregation, and MCP server functionality

• API Gateway HTTP API with sophisticated CORS and authentication

• DynamoDB tables for state management and external directory lookups

• SNS/SQS for alerting and dead letter queue handling

• Direct Bedrock integration with Claude 4.5 Haiku for automated analysis

• CloudWatch Events scheduling 5-minute monitoring cycles

• Secrets Manager for GitHub app credentials and API keys

Code Intelligence Platform (Compute):

• Two EC2 instances: t3.xlarge for web serving, g4dn.xlarge for GPU-accelerated indexing

• EBS volumes with gp3 storage and custom IOPS configuration

• VPC with six subnets across availability zones

• Application Load Balancer with Route53 DNS and ACM certificates

• EFS for shared file storage across instances

• CloudWatch Synthetics for endpoint monitoring

• Lambda-based Slack notifications triggered by SNS topics

Enterprise Infrastructure (Multi-Account):

• Terragrunt-managed infrastructure across production and staging accounts

• S3 backend for Terraform state with DynamoDB locking

• Cross-account IAM policies and service integration

• Integration with external providers (Sentry, GitHub, Kubernetes clusters)

A year ago, this list would have been my personal infrastructure horror story. Today, it’s Tuesday.

What Changed

The transformation wasn’t gradual. It was a step function that happened when I realized AI assistants could handle the cognitive overhead that makes AWS painful.

Before: AWS documentation as archaeological expedition. Digging through service guides, trying to understand the relationship between VPC route tables and security groups, wondering if I need an Internet Gateway or a NAT Gateway or both. Every deployment felt like solving a puzzle where half the pieces were hidden.

After: Natural language infrastructure requests. “Set up monitoring for this Lambda function with alerting to Slack” becomes a series of guided steps where the AI handles the AWS-specific implementation details while I focus on the business requirements.

The key insight is that AWS’s complexity isn’t inherently bad—it’s just cognitively expensive. When you remove that cognitive load through AI assistance, you can appreciate what all those services actually enable.

Take my monitoring setup. Previously, I would have settled for basic uptime checks because configuring comprehensive CloudWatch metrics, Cost Explorer integration, and automated issue creation felt like a weekend project. With claude-mpm AWS skills, it became an hour of guided configuration that resulted in production-grade observability.

Or consider the GPU instance management. The g4dn.xlarge for ML indexing runs sophisticated start/stop automation, monitors for runaway processes, and automatically scales EBS volumes based on data requirements. Setting this up manually would have required deep expertise in EC2 lifecycle management, CloudWatch alarms, and Lambda automation. With AI assistance, I focused on defining the business logic while the tooling handled the AWS implementation.

The DX Philosophy Still Matters

None of this means AWS wins every comparison. Vercel’s developer experience remains superior for the use cases it targets. When I need to ship a marketing site or a straightforward web application, git push deployment still beats any infrastructure-as-code workflow.

The difference is recognizing when complexity serves a purpose versus when it’s just complexity. Vercel abstracts away infrastructure concerns because most web applications don’t need granular control over compute, storage, and networking. But when you’re building systems that do need that control—ML pipelines, high-throughput data processing, complex service topologies—AWS’s granularity becomes valuable rather than burdensome.

AI assistance changes the cost-benefit calculation. When configuring VPC networking takes 20 minutes of guided conversation instead of three hours of documentation archaeology, you can choose AWS for projects where you previously would have compromised on requirements to avoid operational overhead.

But there’s an honest accounting problem buried in that logic. Claude Code isn’t free. API costs, subscription fees—if you’re running significant conversation volume to figure out your infrastructure, you’re spending real money. At some point, you’re spending more on AI assistance than a Vercel seat would cost. The “AWS saves money at scale” argument gets complicated fast when you factor in the cognitive tooling required to get there.

So let me be direct about where each wins. Pure self-service developer experience—one engineer, a web app, ship it fast? Vercel, and it’s not particularly close. The moment you need an AI co-pilot to configure your infrastructure, you’ve added a cost layer that Vercel eliminates by design. But complex multi-service deployments—ML pipelines, GPU compute alongside serverless, multi-account Terraform, monitoring infrastructure that spans six services—those don’t live in Vercel’s world. That’s where the math inverts and AWS earns its complexity premium.

The Broader Implications

This transformation reveals something important about how agentic approaches will reshape technology adoption. We’re not just making individual tasks more efficient—we’re changing which categories of tools become accessible to developers.

I see this pattern across the infrastructure stack. Database migrations and performance tuning become approachable when AI translates business requirements into specific configuration changes. Kubernetes stops being “too complex for small teams” when you can describe desired behavior in natural language and get helm charts and operators generated automatically. IAM policies, security groups, and compliance frameworks become manageable when AI can analyze your application requirements and generate least-privilege configurations.

These tools were always powerful. They were just too expensive to learn and maintain for many use cases. AI assistance changes that economics.

Where This Goes Next

We’re still early in AI-assisted infrastructure management. Today’s tooling handles deployment and configuration. Cost optimization, security posture, performance tuning—those are coming. Full system architecture from high-level requirements is probably further out than the hype suggests, but it’s not science fiction.

But the fundamental lesson remains: complexity isn’t always the enemy. Sometimes it’s just temporarily inaccessible. When AI removes the accessibility barriers, you can choose tools based on their actual capabilities rather than their learning curves.

For now, I’m running infrastructure that would have seemed impossible to manage solo a year ago. And it’s kind of fun.

AWS might still feel like quicksand sometimes. But now I have a helicopter.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

The modern CTO toolkit isn’t just apps and coding tools. The real differentiator is a custom knowledge layer — databases, search indices, memory graphs, behavioral instructions that compound over time. No product gives you this. You build it.

Andrej Karpathy gestured at something similar last month when he posted a GitHub Gist he called “LLM Wiki.” His framing: stop using LLMs just to write code, use them to build and maintain a personal knowledge base instead. “Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase.” Three folders, structured Markdown, a large context window, a few Python scripts. No RAG, no vector database. He concluded with: “I think there is room here for an incredible new product.”

He’s right that there’s room. But the product comment is where I’d push back, and I’ll get to that. What Karpathy is describing isn’t a note-taking system. It’s a personal operational knowledge layer. For CTOs specifically, that layer needs to be broader than a personal wiki — it needs live organizational data, agent-connected search, and context that persists across months of decisions. No app hands you that.

TL;DR

• Karpathy’s LLM Wiki shows the direction: LLMs as knowledge compilers, not just code generators

• A modern CTO’s “second brain” is more than PKM — it’s live databases, custom agents, and contextual search across organizational data

• When I joined Duetto as CTO, my custom toolkit let me synthesize a 150-person R&D org in weeks instead of months

• The power isn’t Obsidian. It’s what you connect to it — MCP servers, search indices, knowledge graphs

• Productizing this is theoretically possible and practically very hard, because the schema is the moat

The toolkit article got it half right

In What’s In My Toolkit: Claude Code and Family, I wrote about vanilla Claude Code’s core limitations: context evaporates, code search is keyword-based, memory doesn’t persist, execution is single-threaded. The tools I built — Claude MPM, mcp-vector-search, kuzu-memory — address each of those gaps.

But that article was about coding workflows. The real story is broader.

The same architecture that makes a coding session more effective — persistent memory, semantic search, specialized agents pulling from structured data — turns out to be extraordinarily useful for executive work. Understanding an organization, tracking decisions over time, querying data across systems, maintaining context across months of meetings and analysis. The toolkit I built for software development became the toolkit I used to onboard as a CTO.

That onboarding story is documented in detail elsewhere. Short version: I pointed a multi-agent framework at GitHub, JIRA, Slack, Confluence, and budget spreadsheets, and synthesized a 150-person R&D organization in the weeks before my start date. The difference between doing that with a chat interface versus a CLI-based orchestration layer with parallel agents and persistent memory wasn’t 2x or 5x. It was closer to 10x.

But the onboarding was just the starting gun. The second brain I assembled keeps compounding.

What’s actually in my second brain

Let me be specific. Because when people (now) hear “second brain” they usually think Obsidian vaults with color-coded tags and pretty Markdown files. That’s part of it. It’s the surface layer.

The actual power comes from what’s underneath.

The memory layer

kuzu-memory is a KuzuDB-backed knowledge graph that persists across every AI session. It stores learnings from conversations, code commits, decisions, patterns. When I start a new Claude Code session on a problem I’ve touched before, the context isn’t blank — it’s enriched with what was learned the last time.

This is the thing people underestimate. A project-specific memory that accumulates over months of work develops a kind of organizational intelligence you can’t replicate in a single conversation. It knows why a particular architectural decision was made. It knows that a vendor was evaluated and found lacking. It knows the terminology your team uses internally that differs from industry standard.

KuzuDB isn’t a product choice for its own sake — it’s graph-native, which means it handles relationships well. The connections between people, systems, decisions, and code are as important as the facts themselves.

The search layer

mcp-vector-search provides semantic search across all project files. Not keyword search — semantic search with AST parsing. When I ask “where is the analysis I did on contractor productivity last quarter,” it finds it even if the document never uses those exact words.

At Duetto, this covers everything in my CTO project: architecture records, meeting notes pulled from Granola, emails I’ve synthesized, analysis documents, planning artifacts. Months of accumulated context, all searchable in seconds. The underlying code intelligence for the engineering organization runs as a separate service — mcp-vector-search is for my working knowledge, not the codebase itself.

The databases

My CTO project has three:

• cto.db — SQLite. Work classification, people analysis, contributor data, commit history. The operational database for running analyses and reports.

• analytics.duckdb — DuckDB. OLAP queries and analytics. When I need to slice engineering output data in different ways or run something that would be painful in SQLite, it goes here.

• duetto_knowledge.db — The RAG-queryable knowledge base backing a Flask web app for interactive exploration.

These aren’t a product I bought. They’re a schema I designed, built incrementally, and own completely. The schema reflects how I think about the organization, which is precisely why it’s useful.

The connectors

gworkspace-mcp handles Drive, Docs, Sheets, Gmail, Calendar, and more. I wrote my own rather than using the off-the-shelf options — Google’s first-party integration and Anthropic’s default both have significant tool coverage gaps. Mine exposes substantially more of the Workspace API surface and integrates transparently with Claude MPM, so agents can use Google Workspace tools without any special configuration at the call site.

Beyond Workspace: Notion API for product specs and planning documents. Extraction scripts for JIRA, Confluence, Slack, Datadog, and AWS. Each system outputs to as raw data, which feeds analysis pipelines that generate reports stored in a project directory.

For company-wide memory, two more tools: duetto-memory and duetto-directory. These handle shared organizational context — information that needs to flow between tools and across team members rather than staying in a single session. Memory persists within our VPC, encrypted to individual users’ OAuth keys. Not even our own IT has access to it. Context shared from Claude Code shows up in Claude.ai, and vice versa, without any manual sync.

The entire flow is queryable. From a single Claude session, I can ask about budget trends, team velocity, specific architectural decisions, or what a particular engineer has been working on for the last three months. Because it’s all in the same context-addressable system.

Obsidian as the front door

Yes, I use Obsidian. But it’s a front door, not the building. The vault holds my personal notes, research captures, and synthesized analysis. The Obsidian Web Clipper feeds raw material into the knowledge pipeline. Templates enforce consistent structure.

Karpathy’s insight about Obsidian as IDE is right in the narrow sense: it’s the interface you use to read and organize. But the interesting work happens outside it — in the databases, the agents, the search indices, the custom scripts.

CLAUDE.md files everywhere

The context layer isn’t just data. It’s also behavioral instructions.

Every major directory in my project has a CLAUDE.md. The root CTO project one is 400 lines of conventions, routing logic, document lifecycle rules, and architectural decisions. Every subdirectory has a more focused version. Every specialized agent has its own constraints.

These files are my second brain’s schema, expressed as instructions rather than data. A single routing rule — “if the prompt mentions meeting notes, save to projects/meetings/2026-W##/“ — sounds trivial. But it means twelve months of meeting notes accumulate in consistent, queryable locations rather than wherever an agent happened to save them. Multiply that by forty routing rules across fifteen subdirectories, and the entire corpus becomes navigable. The CLAUDE.md files are what make the databases useful. Without them, the data is just data.

Karpathy put it well: “You share the schema, not the code.” The schema is the valuable part. The schema is what compounds.

My schema took months to build. It will keep getting better. No product ships with the right schema for my organization, because no product knows what I know about how Duetto’s R&D works.

The productization question

Karpathy said there’s room for an incredible product. He’s not wrong about the gap. He might be wrong about the solution.

The structural problems with productizing a second brain:

Context compounds, products don’t. My system gets smarter with every commit, meeting, and conversation. A SaaS product serves thousands of customers and maintains no one’s specific context. The more I use my system, the wider the gap between it and any off-the-shelf alternative.

The schema is the moat. My knowledge architecture reflects how I think about engineering organizations. Someone else’s knowledge architecture would be different. Products that force their schema on you — and every product does — are imposing someone else’s way of thinking on your problem. That friction is small at first and grows over time.

Privacy is structural, not incidental. My databases contain org structures, salary data, performance patterns, vendor negotiations. Routing that through third-party infrastructure creates risk that’s practically impossible to contain. When I built duetto-memory for enterprise use, the entire stack stays within our VPC, with memories encrypted to individual users’ OAuth keys. Not even IT can read them. That level of isolation is nearly impossible to provide as a multi-tenant SaaS.

Some layers could be productized — the infrastructure, not the intelligence. A well-designed memory MCP with sensible defaults. Semantic search that works without configuration. Privacy-preserving graph storage you don’t have to host yourself. The plumbing.

The schema, the decisions, and the accumulated context can’t be productized. Those are yours. That’s the point — and it’s also why the product gap Karpathy sees will remain open even after someone tries to fill it.

Can anyone do this?

There’s an access problem here, and I’d be dishonest not to acknowledge it.

Building what I’ve described requires knowing Python well enough to write extraction scripts, understanding enough about graph databases to design a schema, and being comfortable with CLI-based tooling and MCP server configuration. Not every CTO has that background. Not every technical leader wants to spend weekends building personal infrastructure.

The irony is that the people who most need better organizational intelligence — executives without deep engineering backgrounds — are least equipped to build these systems. And the people who are most capable of building them are often less interested in the executive problems the systems could solve.

Tiago Forte, who wrote Building a Second Brain, has been making this point for years. His PARA method and CODE framework are accessibility layers — ways to make the underlying ideas approachable without requiring you to build a graph database. The methodology is sound. But it was designed for knowledge workers, not for CTOs running engineering organizations who need live data pipelines, not filing systems. Well-designed for whom?

Karpathy’s LLM Wiki is explicitly a system for someone comfortable writing Python and working with file systems. His Gist has code in it. That’s a feature for his audience and a barrier for everyone else.

What I’d watch for

A few trends that will determine whether this remains a DIY space or gets productized:

MCP as infrastructure. The Model Context Protocol creates a standard interface for exactly this kind of knowledge infrastructure. Memory servers, search servers, database connectors — they all expose the same interface to any compatible AI client. The ecosystem is growing fast. As more MCP servers mature, the configuration burden drops.

Searchable context beats raw window size. Karpathy argues for plain Markdown because ~400K words fit in a modern context window. That’s true, and the window is getting larger. But the more important shift is that structured, searchable context doesn’t have a ceiling. A well-organized knowledge base that spans years of meetings, decisions, and analysis delivers more than any single context window can hold — and the value scales with the quality of the organization, not the size of the model.

Local model quality. Karpathy runs Anthropic agents via Claude Code. But local model quality is improving fast. A system that uses the cloud API for synthesis and queries but runs a local model for routine indexing tasks would be significantly cheaper and more private. Not ready yet. Getting closer.

The product Karpathy thinks exists — if it gets built — probably looks like a well-designed local MCP server with clean configuration, sensible defaults, and a plugin ecosystem for connectors. Not a SaaS. Not a cloud database. Something you install and own.

The people who need it most will have already built their own before any product ships. And in the process of building it, they’ll have accumulated the one thing no product can give them: months of their own operational context, organized the way their own mind works.

That’s not a consolation prize. That’s the whole point.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• What’s In My Toolkit: Claude Code and Family — The coding layer of the stack

• I Built a Coding Tool. Then I Used It to Onboard as CTO — Applying agent orchestration to organizational analysis

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

So if you’ve been using Claude Code and noticed it felt... off... you weren’t imagining it.

Anthropic published a full breakdown yesterday and it’s actually three separate bugs that compounded into what looked like one big degradation. The developer community was right to be concerned, and the evidence they collected was instrumental in getting this fixed.

Here’s what actually happened:

1. They silently downgraded reasoning effort (March 4)

They switched Claude Code’s default from high to medium reasoning to reduce latency. Users noticed immediately. They reverted it on April 7.

Classic “we know better than users” move that backfired. From their postmortem:

“This was the wrong tradeoff. We reverted this change on April 7 after users told us they’d prefer to default to higher intelligence and opt into lower effort for simple tasks.”

The UI was appearing frozen in high reasoning mode, so they made an executive decision to sacrifice quality for speed. Developers immediately felt the difference and pushed back hard.

2. A caching bug made Claude forget its own reasoning (March 26)

This one was particularly insidious. They tried to optimize memory for idle sessions—clear old thinking after an hour of inactivity to speed up resumption. Sounds reasonable, right?

A bug caused it to wipe Claude’s reasoning history on EVERY turn for the rest of a session, not just once. So Claude kept executing tasks while literally forgetting why it made the decisions it did.

The cascading effects were brutal:

• Every request became a cache miss

• Usage limits drained faster than expected

• Claude appeared “forgetful and repetitive”

• Sessions felt like they were constantly resetting

3. A system prompt change capped responses at 25 words between tool calls (April 16)

They added this seemingly innocent instruction: “keep text between tool calls to 25 words. Keep final responses to 100 words.”

It caused a measurable 3% drop in coding quality across both Opus 4.6 and 4.7. They caught this through ablation testing—removing the instruction and measuring the performance difference.

Reverted April 20.

The community evidence was damning

While Anthropic was investigating internally, the developer community was building their own case. Stella Laurenzo from AMD’s AI group published the most comprehensive analysis—6,852 Claude Code sessions and over 234,000 tool calls.

Her findings:

• Median visible thinking length collapsed 73% (2,200 → 600 characters)

• API calls per task spiked up to 80x from February to March

• Claude was choosing “simplest fix” over correct solutions

BridgeMind’s testing showed Opus 4.6 accuracy dropping from 83.3% to 68.3%.

The data was undeniable.

The perfect storm effect

Here’s what made this particularly hard to pin down: all three bugs affected different traffic slices on different schedules. The combined effect looked like random, inconsistent degradation.

Hard to reproduce internally. Hard for users to isolate the exact cause. It just felt... wrong.

Some sessions hit the reasoning downgrade. Others hit the caching bug. The unlucky ones hit multiple issues simultaneously. No wonder it seemed like Claude was having random bad days.

What this reveals about AI product development

This postmortem is actually refreshing in its transparency. Most AI companies would have quietly fixed the issues and moved on. Anthropic owned the mistakes publicly.

But it also highlights a fundamental tension in AI product development: users often prefer maximum capability over convenience optimizations. The reasoning effort downgrade was done for user experience (reduce perceived latency), but developers would rather wait for better output.

The lesson: don’t optimize away what users value most without asking them first.

All fixed now (v2.1.116)

As of April 20, all three issues are resolved:

• Default reasoning is now “xhigh” for Opus 4.7, “high” for others

• Caching bug squashed

• Verbosity limits removed

• Usage limits reset for all subscribers

Anthropic is also committing to more transparency going forward with a dedicated @ClaudeDevs account for deeper technical communication with developers.

The community was right to raise hell about this. And Anthropic’s response—full transparency with concrete fixes—sets a good precedent for how AI companies should handle quality regressions.

Your coding assistant is back to full strength.

Independent Validation

The technical analysis backing this story comes from multiple independent sources. Stella Laurenzo’s comprehensive audit of 6,852 sessions provided the quantitative foundation. BridgeMind’s testing offered controlled benchmark data. These weren’t isolated complaints—they were systematic investigations with reproducible findings.

When a company publishes a detailed postmortem acknowledging specific engineering decisions that degraded their product, and that postmortem aligns with community-gathered evidence, we’re seeing transparency in action. The developer community did the work to document the problems. Anthropic owned the solutions.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Opus 4.7 dropped last week. Lots of excitement. Then the other shoe dropped. I ran identical coding tasks against both Opus 4.6 and 4.7 to see if the capability improvements justify the cost increase. Both models passed all 10 tests. The quality difference is real — but Opus 4.7 consumed 3.6× more tokens and cost 3.6× more for the same outcome.

That’s not a typo. Same task, same success rate, nearly 4× the cost. I have the receipts.

Anthropic says “pricing unchanged” because the per-token rates stayed the same. What they don’t mention is that Opus 4.7 systematically burns through more tokens to complete identical work. The model writes, then revises. Opus 4.6 writes correctly the first time. Both approaches work. Only one bills you for the revision process.

TL;DR

• Both models passed 10/10 tests: Quality improvements are measurable but incremental (better typing, more thorough code)

• 3.6× cost increase for identical outcomes: $0.38 vs $1.38 for a 30-minute coding task in controlled testing

• Token consumption drives cost, not capability: Opus 4.7’s iterative working style consumes 2.9× more output tokens per task

• Agentic mode required: One-shot testing shows 4.7 fails 9/10 tests without tool access, while 4.6 passes perfectly

• Per-token rates unchanged, real bill moved anyway: 4.7 burns 4.8× more cache tokens per task — the rate card stays flat, your invoice doesn’t

The Controlled Test

Last week I ran a head-to-head benchmark using the same Level 1 coding task for both models: build a complete Python CLI tool (Markdown Table Formatter) from scratch with full test coverage.

Test setup:

• Framework: claude_agent_sdk v0.1.64, full agentic mode

• Models: claude-opus-4-6 vs claude-opus-4-7

• Success criteria: Pass all 10 provided pytest tests

• Execution: Concurrent runs with identical prompts

Both models succeeded. The difference was entirely in how they got there.

The Numbers

Metric Opus 4.6 Opus 4.7 Ratio Wall clock time 114.8s 259.1s 2.3× slower Agent turns 17 23 35% more Output tokens 6,384 18,289 2.9× more Cache read tokens 215,853 1,034,165 4.8× more Total cost $0.38 $1.38 3.6× more expensive

The Behavioral Fingerprint

The tool usage patterns reveal why 4.7 costs more:

Tool Opus 4.6 Opus 4.7 Write 6 7 Bash 6 9 Read 4 1 Edit 0 5

Opus 4.7 made 5 Edit calls to revise files after writing them. Opus 4.6 made zero — it wrote all 6 source files correctly in a single pass, ran pytest once, passed 10/10 tests.

The cache token burn (4.8× more) suggests 4.7 does extended internal reasoning between each tool call. It’s thinking harder, which shows up in better code quality — more type hints (35 vs 25 function definitions), more thorough coverage (820 vs 471 lines of code). But you pay for that thinking process.

Quality vs Cost Trade-off

The output quality difference is genuine. Opus 4.7’s code was more defensively written — better typed, more thorough on edge cases. When you’re in the middle of debugging a genuinely hairy distributed system problem or making an architectural call with real downstream implications, that extra care is worth something.

But for this Level 1 coding challenge, both approaches delivered identical functionality. The question becomes: is 40% better typing and 74% more comprehensive coverage worth 260% higher costs?

When the Premium Isn’t Optional

I tested both models in one-shot mode (no tools, single response) to see if you could avoid the iterative cost overhead.

Metric Opus 4.6 Opus 4.7 Output tokens 4,725 23,907 Cost $0.27 $0.94 Tests passed 10/10 1/10

Opus 4.7 failed catastrophically without tool access. It generated 5× more tokens but couldn’t follow output format instructions — most files were unparseable and 9 of 10 tests failed. Opus 4.6 passed perfectly on the first attempt.

This reveals a structural dependency: Opus 4.7’s quality advantages require the full agentic feedback loop. You can’t switch to a cheaper execution mode to control costs. The iterative self-correction that makes it better is also what makes it expensive — there’s no cheaper version of how this model works.

Scale It Out

Scale these numbers to something realistic:

100 equivalent coding tasks per day:

• Opus 4.6: ~$38/day → ~$13,870/year

• Opus 4.7: ~$138/day → ~$50,370/year

• Annual cost increase: +$36,500

This matches what production teams are reporting. The Finout analysis documented overnight cost jumps from $500 to $675/day after deploying 4.7. My testing provides a mechanistic explanation: the model’s working style is token-intensive by design.

The cost increase compounds with Anthropic’s separate tokenizer changes that increase consumption up to 35% for identical prompts. You get hit twice: more tokens per task, plus each token costs more to count.

This Is Anthropic’s Move, Not the Industry’s

Other providers aren’t doing this. GPT-5.4 achieves comparable benchmark performance without the tokenizer change. Anthropic can pull this off because they’re ahead on benchmarks right now — that’s the advantage, and they’re using it.

Which means this is actually a model selection problem, not a budget problem.

I’m not upgrading my agentic workflows to 4.7 by default. For complex architectural work where the reasoning depth matters — distributed systems debugging, refactoring decisions with downstream implications — yes, 4.7 earns the premium. For routine code generation, test writing, documentation? 4.6 passes the same tests at a quarter of the cost, as I just demonstrated.

Sonnet is even more aggressive on cost for work that doesn’t need Opus-level reasoning at all. I’ve been pushing more of my day-to-day agentic tasks there.

GPT-5.4 is worth keeping in rotation too. Comparable coding benchmark performance, no tokenizer games, and the competitive pressure helps if you ever need to push back on Anthropic pricing.

The Reddit community caught the tokenizer changes within hours of release while Anthropic’s communications stayed focused on “unchanged pricing.” That’s the early warning system. Watch community cost reports when a new model drops, not the vendor announcement.

Anthropic will keep doing this as long as they’re leading. The way you stay ahead of it is knowing your actual token consumption per task — not the rate card, the real burn — and routing work to the cheapest model that gets the job done.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

It’s The Harness, Stupid!

Why AI tool orchestration now matters more than foundation model quality

Author: Bob Matsuoka, CTO @ Duetto Research
April 6, 2026

TL;DR

• Same-model testing reveals 0.82-point quality spread (3.93 to 4.75) and 7x efficiency differences—orchestration dominates outcomes

• Market validation: Claude maintains 70% developer preference despite GPT-5.4 achieving model parity through superior harness quality

• Reddit analysis confirms Codex efficiency gains come from orchestration improvements, not just model upgrades

• Competitive advantage has shifted permanently from model superiority to ecosystem superiority

Bottom line: The harness era has begun. Choose tools based on workflow fit, not benchmark claims.

The $50B Model Myth

The AI industry has a fixation problem. Every week brings breathless announcements about parameter counts, training costs, and benchmark scores. “GPT-6 has 50 trillion parameters!” “Our model scored 94.7% on SWE-bench!” “We spent $2 billion on compute!”

Three converging pieces of evidence prove this approach is fundamentally wrong.

The harness matters more than the model. Choosing an AI coding tool is now primarily an engineering decision, not a model selection decision. The next competitive advantage isn’t bigger models—it’s better orchestration.

Evidence Pillar #1: The Smoking Gun Laboratory Data

The Bake-Off Setup

I designed five programming challenges ranging from 30-minute tasks to 8-hour full-stack builds:

• Level 1-2: Simple scripts and basic applications

• Level 3: API integration with Docker containerization

• Level 4: Extensible data processing pipeline (architecture test)

• Level 5: Full-stack web application with authentication

Eight agents competed: Claude Code, Claude MPM, Codex, Gemini CLI, Auggie, Qwen+Aider, DeepSeek+Aider, and Warp AI. Each received identical prompts. A panel of expert developers blind-reviewed all submissions across eight criteria: functionality, correctness, best practices, architecture, code reuse, testing, error handling, and documentation.

The Harness Advantage Data

Table 1: Same Model, Different Worlds

Four agents using identical Claude Sonnet 4.6 models. Quality scores from 3.93 to 4.75—a 0.82-point spread. claude-mpm finished in 45 minutes while warp took 313 minutes. Almost 7x longer for lower quality results.

The Scaling Pattern

The harness advantage compounds with complexity:

• Levels 1-2: All agents performed similarly. Simple tasks don’t reveal orchestration differences.

• Level 3: API integration and Docker setup separated agents that plan from those that code-and-fix. Clear gaps emerged.

• Levels 4-5: Architecture and full-stack challenges broke most agents. Only well-orchestrated systems completed the complex workflows.

The pattern is clear: as complexity increases, harness quality becomes the primary determinant of success.

Evidence Pillar #2: Market Validation — GPT-5.4 Caught Up

Model Parity Achievement

February-April 2026 benchmarks confirm GPT-5.4 has achieved parity with Claude Sonnet 4.6:

Core Benchmarks:

• SWE-bench Verified: GPT-5.4 ~80% vs Claude 79.6% (statistical tie)

• SWE-bench Pro: GPT-5.4 57.7% vs Claude 43.6% (GPT leads complex problems)

• Terminal-Bench: GPT-5.4 75.1% vs Claude ~65% (DevOps advantage)

• Context handling: Both models feature 1M token windows

Yet Claude Still Dominates Through Harness Advantages

Despite achieving model parity, the competitive landscape tells the harness story:

Market Reality:

• Developer preference: Claude 70% (superior workflow integration)

• Enterprise share: Anthropic +4.9% MoM growth, OpenAI -1.5% decline

• Revenue: Claude Code $2B ARR in 6 months

Even when models reach parity, harness quality determines adoption.

The Multi-Model Strategic Reality

Leading organizations aren’t choosing between models anymore—they’re deploying three-tier strategic architectures based on cost-performance optimization:

Tier 1: Daily Workhorse (60-70% of requests)

• Claude Sonnet 4.6: $3/$15 per million tokens

• High-volume development, routine coding tasks

• 95%+ of premium model quality at half the cost

• Default choice for most enterprise development work

Tier 2: Specialized Operations (20-30% of requests)

• GPT-5.4: $2.50/$15 per million tokens

• Terminal operations, DevOps workflows, CI/CD debugging

• 75.1% Terminal-Bench score (10-point lead over competitors)

• Inherited Codex’s terminal operation dominance

Tier 3: Premium Analysis (10-20% of requests)

• Claude Opus 4.6: $5/$25 per million tokens

• Complex reasoning, architectural decisions, high-stakes analysis

• World leader in abstract reasoning (87.4% vs GPT-5.4’s 83.9%)

• When cost justifies maximum capability

This confirms the core thesis: when models are “good enough,” teams optimize for strategic cost-performance fit, not raw capability or marketing claims.

Evidence Pillar #3: Community Validation — The Codex Orchestration Story

Reddit Confirms Orchestration Improvements

Reddit research explains Codex’s impressive efficiency results (42 minutes, 4.49 quality score). The evidence confirms improvements come from orchestration, not just model upgrades.

Architectural Evolution Evidence:

• Codex evolved from “embedded assistant” → “independent agent with multi-agent orchestration”

• GPT-5.2-Codex (Jan 2026) with 192K context + MCP tool orchestration

• “Command center for agents” interface launched Feb 2026

Workflow Efficiency Improvements:

• Developers report queuing “4-5 Codex tasks before diving into manual work”

• “2-3 completed PRs waiting for review” after a coffee break

• P99 response time 45ms vs Copilot’s 55ms through better context management

• Parallel processing capabilities that enable true background orchestration

Enterprise Orchestration Benefits:

• 70% more pull requests merged weekly at OpenAI

• 50% reduction in code review times at Cisco

• 67% reduction in median turnaround time at Duolingo

• 90% Fortune 100 adoption validates orchestration value at scale

The Community Strategic Deployment Pattern

Reddit developers now recommend different tools for different purposes:

• Claude Code: Code quality and reasoning

• Cursor: Daily coding integration

• OpenAI Codex: Complex multi-agent workflows and long-horizon autonomy

This matches exactly what the market data predicted: teams use orchestrated tools strategically rather than seeking one universal solution.

The Harness Quality Ladder

Based on all three evidence pillars, I see four tiers of orchestration quality emerging:

Tier 1: Basic Wrappers

• Simple API access, minimal context management

• Examples: Raw ChatGPT interface, basic API wrappers

• Limitation: No file coordination, poor context retention

Tier 2: Workflow Tools

• File awareness, some context management

• Examples: GitHub Copilot, basic IDE extensions

• Capability: Single-file optimization, limited cross-file understanding

Tier 3: Orchestrated Systems

• Multi-file coordination, workflow integration

• Examples: Cursor, Claude Code, well-configured aider

• Advantage: Understands project structure, handles complex tasks

Tier 4: Agentic Frameworks

• Multi-agent coordination, planning, verification

• Examples: claude-mpm, advanced orchestration systems

• Power: Full project lifecycle, quality assurance, architectural thinking

The performance cliff between tiers is exponential, not linear. Bad orchestration can make great models perform poorly; great orchestration can make good models perform excellently.

Academic and Industry Validation

This isn’t just empirical observation. Multiple 2026 research papers and industry studies support the harness thesis:

Academic Consensus:
The arXiv paper “Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems” shows that domain-tuned models with better orchestration achieve superior cost-normalized accuracy despite using smaller base models.

SWE-bench data reveals the same pattern. Cursor, Claude Code, and Auggie all use similar base models yet score between 50.2% and 55.4%, while the raw model score is only 45.9%. The 5.9-point improvement comes entirely from better context retrieval and agent design.

Business Reality Check:
Enterprise adoption surveys show a clear shift in CTO priorities. “Model performance” is dropping in tool evaluation criteria, replaced by governance, integration quality, and workflow fit. As one 2026 McKinsey report put it: “CTOs are realizing their biggest bottleneck isn’t model performance—it’s governance.”

What This Means for Engineering Leaders

Stop Optimizing for Benchmarks

The old procurement mindset was model-first: “We need access to GPT-6 for competitive advantage.” The new reality is that benchmark performance doesn’t predict practical utility. SWE-bench scores don’t tell you whether a tool will integrate with your existing workflow, handle your codebase size, or recover gracefully from errors.

Start evaluating harness quality:

• Context management: How well does it understand your project structure?

• File coordination: Can it work intelligently across multiple files?

• Error recovery: Does it handle failures gracefully or require constant babysitting?

• Workflow integration: How does it fit with your team’s existing development process?

Budget for Orchestration Quality

The three evidence pillars show that investing in better orchestration yields measurable returns:

• Quality per minute: claude-mpm achieved 4.75 quality in 45 minutes; warp achieved 3.94 in 313 minutes

• Market validation: Claude maintains dominance despite model parity through superior developer experience

• Enterprise results: 70% more PRs, 50% faster code review, 67% faster turnaround

The ROI case for harness investment is clear and quantifiable.

Team Productivity Focus

Tool choice impacts your entire development pipeline. The 7x speed difference between well and poorly orchestrated tools using the same model means tool selection is a productivity multiplier, not just a capability decision.

Better tools also reduce onboarding time and increase adoption rates. A tool that works reliably gets used; one that requires constant troubleshooting gets abandoned.

The Competitive Landscape Evolution

Codex Deserves Recognition

Codex’s performance has significantly improved. At 42 minutes for all five levels with a 4.49 quality score, it achieved by far the best efficiency in my study. GPT-5.4+ combined with the orchestration improvements OpenAI made represents a compelling package. The Reddit research confirms this wasn’t just a model upgrade—it was an architectural evolution toward multi-agent orchestration.

Claude Code’s Harness Moat

While Claude Code performed well (4.53 quality score), the market validation shows its true strength: ecosystem superiority. Despite GPT-5.4 achieving model parity, Claude maintains 70% developer preference through superior harness quality. This is exactly what sustainable competitive advantage looks like in the post-parity era.

The Multi-Model Future

All evidence points to the same conclusion: the era of picking one model is over. Leading organizations deploy three-tier cost-performance architectures, optimizing for specific strengths rather than seeking universal solutions.

Real enterprise case studies validate this pattern:

• TELUS (57,000 employees): Uses Sonnet as core engine across developer teams

• Zapier: 800+ internal agents using strategic model selection

• Financial Services: Monthly costs ~$80 at massive scale through optimized routing

The successful pattern: Sonnet for volume, GPT-5.4 for DevOps, Opus for complexity.

The Token Economics Reality

claude-mpm achieved the highest quality score (4.75) but used 87 million tokens versus codex’s 120K. This looks expensive until you consider the output: 262 comprehensive tests (vs codex’s 32), complete documentation, 100% verification rates, and multi-file coordination (note: this was also a wake-up call to me to focus on token optimization, current version is much stingier)

The 700x token multiplier isn’t overhead—it’s the cost of work a solo agent skips. Orchestration doesn’t waste tokens—it spends them on comprehensive deliverables.

The optimization question: Could you achieve 80% of the quality benefits at 30% of the token cost? The opportunity isn’t eliminating orchestration—it’s finding the minimal viable team size for maximum impact.

The Vendor Bias Problem: “Opus for Everything”

Boris Cherny, the Claude Code lead, recently advocated for using “Opus for everything.” This perfectly illustrates the disconnect between vendor recommendations and practical deployment reality.

Only someone working for Anthropic can say that.

When your employer provides unlimited access to premium models, of course you’d recommend the most expensive option for every task. But real organizations operating with P&L responsibility make strategic decisions about when premium capability justifies premium cost.

This vendor bias actually validates the multi-model thesis:

• Vendors say: “Use our premium model for everything”

• Users do: Strategic model selection based on task complexity and budget constraints

• Market reality: 70% prefer Claude for daily coding (cost/speed), GPT-5.4 for complex reasoning (quality ceiling)

Cherny’s comment inadvertently proves that cost-conscious orchestration is the real competitive battleground. Companies that figure out optimal model routing—not maximal model usage—will have sustainable advantages.

The vendors push premium. The market chooses strategically. The harness makes both possible.

The Future: Welcome to the Harness Era

What Changes for Developers

Tool selection framework:

1. Workflow fit: Does it match how your team works?

2. Integration quality: Plays well with existing tools?

3. Reliability: Can you trust it with production code?

4. Model quality: Fourth priority

What Changes for the Industry

Foundation models are becoming commodities. Differentiation shifts to integration, context management, and user experience. The next unicorns will be harness companies, not model companies.

Major funding flows to orchestration companies. Enterprise procurement evaluates integration first, model second.

The Competitive Moat Shift

The old game was: train bigger models, claim benchmark superiority. The new game is: build better orchestration, solve real workflow problems. Model access becomes a utility; workflow mastery becomes the moat.

Practical Recommendations

For CTOs and Engineering Leaders

• Audit orchestration quality: Test tools with your actual codebase for 2-week trials

• Budget 60/40: Spend more on harness development than model subscription fees

• Measure real metrics: Track pull request velocity and code review time, not benchmark scores

• Evaluate integration first: How well does it fit your existing CI/CD pipeline?

For Developers

• Test with real projects: Spend 2 days with each tool on actual work before deciding

• Learn orchestration patterns: Context management and file coordination matter more than prompts

• Invest in mastery: The 7x efficiency difference justifies significant learning time

• Ignore marketing claims: Model access means nothing without good orchestration

For the AI Industry

• Build for workflow integration: Solve real development pipeline problems

• Measure practical utility: Developer retention and task completion rates beat benchmarks

• Focus on context management: Multi-file coordination is the real competitive moat

Conclusion: The Questions That Matter Now

The old question was: “What’s the best model?”

The new question is: “What’s the best harness for my team’s workflow?”

Three evidence sources prove we’ve crossed a threshold: foundation models are “good enough,” and orchestration quality now dominates outcomes. Laboratory testing, market validation, and community confirmation point to the same reality.

The foundation model is the engine. The harness is the car. The best engine in the world won’t get you anywhere without wheels.

The harness era has begun. Drive accordingly.

Bob Matsuoka is CTO at Duetto Research and creator of Claude MPM, one of the agents evaluated in this study. All evaluation data and methodology are available at github.com/bobmatnyc/ai-coding-bake-off for reproducibility.

Appendix: Complete Results Data

Quality Scores by Criterion

GPT-5.4 vs Claude Sonnet 4.6 Market Data

SWE-bench Performance:

• SWE-bench Verified: GPT-5.4 ~80% vs Claude 79.6% (statistical tie)

• SWE-bench Pro: GPT-5.4 57.7% vs Claude 43.6% (GPT advantage on complex problems)

• Terminal-Bench: GPT-5.4 75.1% vs Claude ~65% (GPT DevOps advantage)

Market Metrics:

• Developer preference (daily coding): Claude 70%

• Enterprise market share: Anthropic +4.9% MoM, OpenAI -1.5% MoM

• Claude Code revenue: $2B ARR in 6 months

Methodology Notes

• Laboratory data: Single run evaluation with disclosed author bias

• Market data: Cross-validated across 15+ authoritative sources

• Community research: Reddit analysis across 8+ developer subreddits

• Statistical confidence: Mean inter-reviewer deviation of 0.216 points

• Reproducible: All data and prompts available in public repository

I didn’t expect to meet Mila Jovovich through a GitHub issue.

But there I was last week, deep-diving into her AI memory framework called MemPalace, when I discovered something remarkable: the “Resident Evil” and “Fifth Element” star had created one of the most talked-about AI memory systems of 2026. And she’d done it using Claude Code, the same AI-assisted development environment I use daily.

More remarkably, when I found critical bugs in her benchmark methodology, she responded directly through her Claude Code workflow, acknowledging the issues and implementing fixes. Not through a PR team or engineering intermediaries — Mila herself, using AI-assisted development to debug complex memory retrieval algorithms at 9 AM on a Thursday.

This isn’t a story about a celebrity coding stunt. It’s about something much more profound: we’ve entered an era where outcomes and features drive development, not the technical limitations of writing code.

The MemPalace Phenomenon

In April 2026, Mila Jovovich and developer Ben Sigman released MemPalace, an open-source AI memory system that immediately went viral. Within 48 hours, it had over 23,000 GitHub stars. The system claimed to achieve the first perfect score on the LongMemEval benchmark, scoring 96.6% raw recall.

The project represents something unprecedented: a free, locally-running memory system that rivals expensive cloud alternatives like Mem0 ($19-249/month) and Zep ($25+/month). It uses the “memory palace” technique — a classical memory method dating back to ancient Greece — implemented through ChromaDB and SQLite, with zero ongoing API costs.

The technical architecture includes basic Claude Code integration (save hooks every 15 messages and before context compression) and 24 tools via the Model Context Protocol (MCP), making it compatible across multiple AI platforms.

The duo had spent months building it using Claude Code’s AI-assisted development environment. As Sigman noted, he provided “the engineering chops” while Jovovich drove the architectural vision.

When Audits Meet AI-Generated Code

That’s when things got interesting.

As someone who works extensively with AI memory systems — I maintain KuzuMemory, a graph-based memory framework — I was naturally curious about MemPalace’s benchmark methodology. The claimed 96.6% recall rate was extraordinary, especially for a system running entirely locally.

So I dove in.

What I found were several methodological issues that fundamentally undermined the headline numbers. The benchmark adapter was discarding assistant turns in conversation history, causing systematic under-recall on certain question types. More critically, the benchmark wasn’t actually testing MemPalace’s core functionality — it was primarily testing ChromaDB’s raw vector search capabilities.

I filed Issue #242 documenting the assistant turn bug, and Issue #214 showing that the 96.6% score was essentially a ChromaDB score, not a MemPalace score.

Mila’s response was immediate and technically sophisticated:

“Hey @bobmatnyc — I’ve taken a look and ran it through CLI. This is a real bug and it’s urgent. You caught that benchmarks/longmemeval_bench.py at lines 189-190 builds each session’s indexed document by concatenating only user role turns... Fix priority: this must land before any public benchmark re-run.“

She didn’t deflect or dismiss. She debugged the issue herself, identified the exact lines of code causing the problem, explained the downstream impact on other benchmarks, and outlined a detailed fix plan including regression tests.

This wasn’t PR speak. This was an AI-assisted developer engaging seriously with technical criticism.

The Democratization Shift

This interaction crystallized something profound about our current moment in software development.

We’re witnessing the emergence of a new class of builders: technically-minded individuals who understand software conceptually but may not have traditional coding backgrounds. AI-assisted development tools like Claude Code, GitHub Copilot, and Cursor have lowered the implementation barrier to the point where vision and domain expertise matter more than syntax mastery.

Mila Jovovich exemplifies this shift perfectly. Without formal technical education (she left school in 7th grade for modeling), she spent months intensively learning AI-assisted development through Claude Code starting in late 2025. She understood the conceptual framework of memory palaces deeply enough to architect a sophisticated system. Her collaboration with Ben Sigman — CEO of Bitcoin lending platform Libre Labs, who provided the engineering expertise while she drove architectural vision — represents a new model of software development where domain knowledge and AI tool fluency can substitute for traditional programming backgrounds.

The fact that a movie star can release a technically competent, widely-adopted memory framework isn’t a commentary on coding getting easier (though it has). It’s about software development becoming more accessible to domain experts and visionaries who previously couldn’t bridge the implementation gap.

What MemPalace Gets Right

Despite the benchmark issues I uncovered, MemPalace demonstrates genuine technical sophistication. The memory palace metaphor isn’t just marketing — it’s a thoughtful architectural choice that makes AI memory systems more intuitive and debuggable.

The system includes elegant features like per-agent memory “wings” that prevent cross-contamination between different AI assistants. The Claude Code integration hooks are well-designed, automatically triggering memory saves at logical conversation boundaries. The MCP implementation is clean and follows established patterns.

Most importantly, the project tackles a real problem: most AI memory systems are either expensive cloud services or complex local installations. MemPalace provides a middle path that’s both free and relatively easy to deploy.

Through my testing and integration experiments, I learned techniques that improved my own KuzuMemory system. The competitive analysis forced me to think more carefully about memory organization patterns and retrieval strategies. This kind of cross-pollination benefits the entire ecosystem.

The Validation Requirement

But the benchmark controversy highlights a crucial point: democratized software development still requires traditional validation methods.

AI-assisted coding tools excel at implementation but can perpetuate subtle conceptual errors throughout a codebase. The MemPalace benchmark issues weren’t obvious bugs — they were methodological problems that required domain expertise to identify.

This creates an interesting dynamic: AI tools enable rapid development by non-traditional developers, but peer review by experienced practitioners becomes even more critical. The community response to MemPalace’s inflated benchmarks wasn’t hostile — it was collaborative debugging at scale.

Mila’s willingness to engage directly with technical criticism and implement fixes demonstrates the right approach. The democratization of software development doesn’t eliminate the need for technical rigor; it distributes that rigor across a broader community.

The Harness Thesis Validated

This story perfectly validates what I call the “harness thesis” — that we’ve entered an era where AI tool ecosystems matter more than underlying model capabilities.

MemPalace succeeded not because Mila wrote perfect code from scratch, but because she effectively orchestrated Claude Code to implement her vision. The system’s value comes from its architectural choices, integration quality, and user experience — not from novel algorithmic breakthroughs.

Similarly, my ability to audit and improve the system came not from superior coding skills, but from having developed complementary expertise with memory systems and benchmark methodology. The collaboration that emerged — distributed across GitHub issues, with contributors from multiple backgrounds — represents the new model of software development.

We’re not just building different software; we’re building software differently.

Meeting Mila Through Code

In the end, I did meet Mila Jovovich — through our AI Agents, lines of Python code, GitHub issues, and technical discussions about memory retrieval algorithms, mediated by our respective Claude Code workflows. Not the meeting I would have predicted, but somehow more meaningful than a typical celebrity encounter.

She embodies a new archetype: the technical visionary who uses AI tools to implement sophisticated ideas without traditional programming backgrounds. Her willingness to engage with criticism and continuously improve the system demonstrates the collaborative spirit that makes this new era of development possible.

The future of software isn’t just about better AI models or more powerful tools. It’s about enabling more people with domain expertise and creative vision to participate in building the systems that shape our digital world.

And sometimes, that means meeting your childhood movie star idol in a GitHub issue thread, debugging memory palace algorithms together.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• It’s The Harness Stupid — Why AI tool ecosystems matter more than model capabilities

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

Referenced Links:

• MemPalace GitHub Repository

• KuzuMemory GitHub Repository

• Issue #242: Benchmark adapter bug

• Issue #214: ChromaDB vs MemPalace scoring

Stripe engineers send Slack messages that automatically become production code. Not suggestions. Not drafts. Production code merged into their main branch, supporting over a trillion dollars in annual payment processing.

Their “Minions” system generates 1,300 pull requests per week with zero human-written code. Fire-and-forget automation from conversation to deployment. While the rest of us debate whether AI can write good code, Stripe has built a software factory that produces enterprise-grade applications at scale.

The software factory isn’t a future concept. It’s a present reality, and it represents the next fundamental challenge for engineering organizations.

What We’re Building at Duetto

I’ve been thinking about this a lot lately. At Duetto, we’re exploring what a software factory could look like for hospitality technology. Not because we want to eliminate developers, but because we’re hitting the limits of traditional development approaches for domain-specific applications.

Our challenge isn’t just writing code—it’s translating complex hotel revenue management requirements into software that works reliably across thousands of properties with different systems, data formats, and business rules. The cognitive load of keeping all these variations in mind while building features is becoming unsustainable.

What if we could describe what we need in something like our APEX specifications, and have the system generate not just code, but complete deployments? Kubernetes instances running Claude Code agents, database migrations, monitoring setup, the whole stack configured for that specific use case.

The goal isn’t replacing our engineering team. Our developers should be solving revenue optimization algorithms and building domain-specific integrations, not configuring YAML files for the hundredth deployment variation.

The Stripe Blueprint

Stripe’s Minions reveal what a production software factory actually looks like when you strip away the hype and focus on what works.

Five-Layer Pipeline: Their system transforms Slack messages into production-ready pull requests through a structured pipeline. Not magic—engineering discipline applied to automation.

Sandboxed Execution: Every agent runs in isolated containers with codebase checkouts. They can’t access production systems, can’t cause cascading failures, can’t break things outside their designated scope. The walls matter more than the model.

Surgical Tool Selection: Their Model Context Protocol provides access to hundreds of internal tools, but agents get intelligently prefetched access to only the ~15 tools relevant to their specific task. Not everything available—the right things available.

One-Shot Optimization: Instead of conversational back-and-forth, their agents are optimized for well-defined work that completes in a single execution. Better latency, lower costs, more predictable outcomes.

The results speak for themselves: 1,300 PRs weekly, zero human-written code in merged changes, supporting their entire payment infrastructure. This isn’t a pilot program. This is their production development workflow.

The Broader Software Factory Landscape

Stripe isn’t alone in building these systems, just the most public about their approach.

Netflix has their federated developer console integrating dozens of tools into a single unified experience. Spotify’s Backstage holds 89% market share among internal developer platforms, reducing time-to-tenth-pull-request by 55% for new developers.

The open source ecosystem is catching up quickly. OpenHands provides a model-agnostic platform for cloud coding agents with $18.8M in Series A funding. CodeT5 handles multi-language code generation. GitHub Copilot Enterprise is expanding beyond code completion into full workflow automation.

Major cloud providers are also building comprehensive platforms. Microsoft’s GitHub Copilot Workspace, Google’s Duet AI for developers, and Amazon’s Q Developer all represent enterprise-grade attempts at software factory capabilities.

According to Gartner, 80% of large engineering organizations now have dedicated platform teams. The question isn’t whether software factories are coming—it’s whether your organization will build one or buy one.

What a Proper Software Factory Requires

Building a software factory isn’t just about connecting AI tools to deployment pipelines. Based on what’s working at Stripe and emerging patterns across the industry, here are the essential components:

Artifact Response Systems

Your factory needs to respond to structured specifications and generate complete deployments. At Duetto, this might mean taking an APEX specification for a new revenue optimization feature and producing:

• Kubernetes deployment configurations

• Database migration scripts

• Monitoring and alerting setup

• Load testing scenarios

• Documentation

The system should handle the entire deployment lifecycle from specification to running production service, not just generate code that someone has to manually deploy.

Strategic Human Review Checkpoints

Notice I said strategic, not comprehensive. Stripe’s fire-and-forget model works because they’ve identified the specific points where human judgment adds value without blocking automation.

For enterprise applications, you need checkpoints at:

• Specification validation: Do the requirements make business sense?

• Security review: Are access patterns and data handling appropriate?

• Integration testing: Does this work with existing systems?

• Production readiness: Are monitoring and rollback capabilities sufficient?

The key is making these gates fast and decisive, not bureaucratic approval processes that defeat the purpose of automation.

Scaffolding for Error Detection

Your factory will produce broken code. That’s not a bug—that’s reality. The difference between a prototype and a production system is sophisticated error detection and recovery.

This means:

• Isolated execution environments where failures can’t cause broader damage

• Automated testing and iteration when initial attempts fail

• Multi-layer validation before anything reaches production

• Comprehensive rollback capabilities for when something gets through anyway

Stripe’s sandbox architecture is brilliant because it lets agents fail safely while learning from those failures to improve future attempts.

Success Criteria Parameters

Your factory needs to know what success looks like for each type of work. Not just “the code compiles,” but measurable business outcomes.

For a hospitality feature, success might mean:

• Performance benchmarks met under load

• Integration tests pass with five different PMS systems

• Revenue impact measurable within 30 days

• Zero customer-facing errors in the first week

Define these criteria upfront, build them into your validation pipeline, and let the factory optimize for actual business value rather than technical metrics alone.

Cost Tracking and Optimization

AI-powered development isn’t free. You need visibility into the computational costs, tool usage, and human review time for each generated system.

Stripe optimizes for this explicitly—their one-shot agents cost less than conversational approaches, their surgical tool selection reduces context costs, their automated testing prevents expensive human debugging cycles.

Track these metrics from day one. The difference between a cost-effective software factory and an expensive experiment is usually found in the operational details.

Deployment Models

Your factory needs sophisticated understanding of how to deploy different types of applications. Golden Path workflows that codify best practices, environment promotion strategies that reduce risk, and rollback procedures that restore service quickly when things go wrong.

This is where domain expertise becomes critical. A generic software factory might know how to deploy a web service, but does it understand the specific requirements for hospitality payment processing, guest data privacy, and integration with property management systems?

The Duetto Context

At Duetto, we’re thinking about how a software factory could handle the complexity of hospitality technology. Our domain has unique challenges:

Data Integration Complexity: Every hotel uses different systems with different data formats. A software factory needs to understand these variations and generate appropriate integration code.

Regulatory Requirements: Guest privacy, payment processing, accessibility compliance. The factory needs to embed these requirements into everything it produces.

Performance Characteristics: Revenue management systems need to process pricing updates in near real-time across thousands of rooms and rate plans. The factory needs to optimize for these specific performance patterns.

Operational Constraints: Hotels can’t afford downtime during peak booking periods. Deployment strategies need to account for hospitality business cycles.

We’re not trying to build a general-purpose software factory. We’re exploring how to build one that deeply understands our domain and can produce applications that work reliably in hospitality environments.

The Reality Check

Building a software factory is hard. Not because the technology doesn’t exist—Stripe proves it does—but because the organizational challenges are substantial.

ROI Demonstration: You need to show measurable productivity improvements and cost savings. “The AI is impressive” isn’t sufficient justification for the investment required.

Security and Compliance: Automated code generation that touches customer data or payment systems requires additional security layers and audit capabilities.

Developer Workflow Changes: Your engineering team needs to learn new ways of working. Some will embrace it, others will resist. Change management is as important as the technical implementation.

Quality Assurance Evolution: Your QA processes need to evolve from testing human-written code to validating AI-generated systems. Different failure modes, different testing strategies.

Integration Complexity: Your factory needs to work with existing systems, databases, APIs, and workflows. The harder the integration challenge, the longer the implementation timeline.

These aren’t reasons to avoid building a software factory. They’re reasons to approach the project with realistic expectations and proper preparation.

Looking Forward

The trajectory is clear. Software factories are moving from experimental to mainstream, with proven systems operating at enterprise scale and standardized architecture patterns emerging across the industry.

The question for engineering leaders isn’t whether this transformation will happen. It’s whether your organization will be an early adopter that shapes how software factories work in your domain, or a later adopter that implements patterns developed by others.

At Duetto, we’re betting on being early. Not because we want to be on the cutting edge for its own sake, but because the companies that figure out domain-specific software factories first will have a significant competitive advantage in application development speed and quality.

The software factory represents the next evolution of platform engineering. The organizations that master it will build better software faster than those that don’t.

The challenge isn’t technical anymore. It’s organizational, strategic, and operational.

The question is: Are you ready to build one?

About this analysis: This piece draws from comprehensive research on production software factory implementations, including detailed analysis of Stripe’s Minions architecture, enterprise platform engineering initiatives, and emerging open source solutions. The author is exploring software factory applications for hospitality technology at Duetto.

About the author: Bob Matsuoka is Chief Technology Officer at Duetto and creator of Claude MPM (Multi-agent Project Manager). He has implemented AI-assisted development workflows across enterprise engineering teams and writes about the practical realities of AI integration in software development at HyperDev.

On March 31, 2026, Anthropic accidentally shipped their entire Claude Code source—512,000 lines of TypeScript—in an npm package. What followed was perhaps the most intense technical autopsy in AI history. The verdict? Mixed, and revealing.

The criticism has been swift and pointed. A 5,594-line file with a single 3,167-line function sporting 12 levels of nesting. Regex-based frustration detection looking for “wtf” and “shit”. A quarter million wasted API calls per day from a three-line bug. As one critic put it: “A multi-billion-dollar AI company is detecting user frustration with a regex.”

But before we pile on, we need to ask: What does “good code” even mean when you’re building client-side LLM applications?

The Unprecedented Challenge

Claude Code isn’t your typical software. It’s a client-side application that orchestrates conversations with large language models, manages context across sessions, and attempts to maintain coherent state while working with fundamentally non-deterministic systems.

This creates problems that traditional software engineering practices weren’t designed for:

• Context management: Handling arbitrarily long conversations that exceed model limits

• Failure recovery: When your core computation is a 20% failure-rate API call

• State synchronization: Keeping UI, conversation history, and model context aligned

• Dynamic adaptation: Code that needs to adapt to changing model capabilities

The leaked source reveals sophisticated solutions to these problems: a three-layer memory architecture, anti-distillation mechanisms, dual parser systems for safety. The engineering is genuinely impressive, even if the implementation is sometimes ugly.

The Meta-Problem: AI Writing AI

Claude Code was partially written by Claude Code. This represents the first documented case of a large-scale AI tool generating significant portions of its own source code—not just incremental improvement, but a categorical change in development methodology that creates unprecedented quality control challenges when AI-generated code scales beyond human review capacity.

When AI generates code at scales that exceed human review capacity, traditional quality control breaks down. That 3,167-line function? Probably not written by a human. The 12 levels of nesting? Algorithmic patterns, not human design choices.

This is the real story: We’re witnessing the first major autopsy of self-bootstrapping AI tooling.

Deterministic vs. LLM Code: Different Standards Apply

I’ve been thinking about this distinction a lot lately in my work with Claude MPM, an open-source multi-agent code generation framework built on Claude Code that coordinates specialized AI agents for software development workflows. When you’re building traditional, deterministic software, all the usual rules apply. Clean functions, clear abstractions, maintainable architecture. Use your normal code analysis tools.

But when you’re building LLM-integrated systems, the rules change:

1. Failure is the default: Your core operations fail 20% of the time

2. Context is expensive: Every token counts toward limits

3. Behavior is emergent: The system does things you didn’t explicitly program

4. Adaptation is constant: Model capabilities change monthly

In this world, a 5,594-line file might be ugly, but if it successfully manages complex failure recovery across multiple conversation threads, it might also be correct.

The Code Analysis Checkpoint Strategy

This is where I’ve found success with my recent updates to the code analyzer in Claude MPM. The analyzer utilizes mcp-vector-search for comprehensive codebase analysis, providing AST-based semantic search, full-text search capabilities, and knowledge graph construction for architectural pattern detection. Instead of trying to prevent AI from generating messy code (impossible), I focus on regular refactoring and analysis checkpoints.

The analyzer has gotten very good at catching two specific issues:

1. Drift: When AI-generated code slowly diverges from intended architecture

2. Bloat: When generated solutions become unnecessarily complex over time

I make a point to run these checkpoints regularly, treating them as essential maintenance rather than optional cleanup. It’s like running cargo clippy or eslint, but for AI-generated architectural decisions.

The key insight: AI code needs different kinds of maintenance than human code.

Outcome-Based Generation: Does It Work?

Here’s my perhaps controversial take: If Claude Code successfully helps developers ship better software faster, then the messy internals might not matter as much as we think.

The leaked code reveals a system that:

• Handles millions of conversations per day

• Maintains context across arbitrarily long sessions

• Provides sophisticated memory management

• Implements multiple safety layers

• Delivers a $2.5 billion ARR product experience

Is the implementation elegant? No. Does it work? Apparently, yes. Because we can observe/measure what it’s building completely independently of what built it.

This doesn’t excuse basic engineering failures (that .npmignore mistake was embarrassing). But it does suggest we need new frameworks for evaluating AI-generated systems.

The Scaffolding Solution

Rather than trying to make AI generate perfect code, we can scaffold around the inevitable messiness:

Automated refactoring checkpoints: Regular cleanup of AI-generated bloat
Architectural constraints: Guard rails that prevent the worst patterns
Outcome validation: Testing that focuses on behavior over implementation
Human oversight: Strategic points where humans validate AI decisions

This is the approach I’ve been taking with Claude MPM, and it’s proven remarkably effective. Let the AI generate messy-but-functional code, then use tooling to clean it up systematically.

What This Means for the Industry

The Claude Code leak represents a watershed moment. It’s our first real look at what happens when AI tools build themselves at scale.

The criticism is valid—basic engineering discipline matters, even in AI systems. A missing .npmignore file is inexcusable for a billion-dollar product.

But the deeper question is whether we’re applying the right standards. Traditional code quality metrics may not capture what actually matters for AI-integrated systems.

Moving Forward

Anthropic probably is moving too quickly in some ways. The leak revealed security vulnerabilities, competitive intelligence losses, and quality control failures that suggest inadequate human oversight.

But they’re also pioneering entirely new categories of software. The problems they’re solving—context management, failure recovery, human-AI collaboration—don’t have established best practices yet.

The real lesson isn’t that AI-generated code is inherently bad. It’s that we need new practices for building, reviewing, and maintaining systems that exceed human comprehension scales.

The question isn’t whether Claude Code’s internals are messy. It’s whether we can build better scaffolding around AI-generated systems to catch the problems that matter while accepting the messiness we can’t avoid.

The Claude Code team probably needs to slow down on the basics—security, testing, deployment hygiene. But they’re moving fast on problems that genuinely require speed to solve before competitors do.

That’s a nuanced position in an industry that loves simple takes. But nuance is what the moment requires.

What do you think? Are we being too hard on AI-generated code, or not hard enough? Share your thoughts in the comments.

About this analysis: This piece draws from extensive technical analysis of the March 31, 2026 Claude Code source leak, including community responses, security assessments, and business impact analysis. The author maintains active development projects using AI-assisted coding tools and has direct experience with the challenges discussed.

About the author: Bob Matsuoka is Chief Technology Officer at Duetto and creator of Claude MPM (Multi-agent Project Manager). He has implemented AI-assisted development workflows across enterprise engineering teams and writes about the practical realities of AI integration in software development at HyperDev.

I’m looking at my (iTerm) terminal right now. Ten tmux sessions. Each session holds a different project context—one monitoring CI failures, another handling a code review, a third debugging a production issue.

This is the reality of modern agent development work.

TL;DR

• Multi-session reality: Power users average 8-12 terminal sessions; most work involves modification, bug response, and PR handling—not new code generation

• Natural workflow origination: Future systems trigger from product team actions, CI failures, and automated events rather than human prompts

• Orchestration evolution: From human-orchestrated agents to orchestration-of-orchestrators where prime coordinators are non-human

• Production examples: Stripe’s Minions (1,300 PRs/week), GitLab’s Duo Agent Platform, Meta’s REA demonstrate hierarchical agent orchestration

• Architecture shift: Claude Code’s SDK model enables workflow-driven development through persistent, context-aware agent orchestration

The 10-Tab Reality

According to recent developer workflow studies, tmux has become the standard for AI-assisted development, with persistent sessions solving the context-switching tax. The productivity advantage isn’t the multiplexing—it’s the persistence. Projects become environments you step in and out of rather than things you open and close.

But here’s what the productivity tutorials miss: most of those tabs aren’t generating software.

My current session breakdown:

• 3 sessions: non-coding -- my CTO knowledge base (currently analyzing our Sumo use), a writing assistant, and our Duetto product management framework

• 4 sessions: coding - various internal tools and MCP connectors

• 2 sessions: coding - new projects

• 1 session: code review

The 8:2 ratio holds across most senior developers I’ve observed. Most development work involves responding to existing systems, not creating new ones.

This distribution points toward something significant: the future of development orchestration isn’t human-initiated.

Beyond Prompt-Driven Development

Claude Code’s new SDK architecture reflects this reality. Instead of starting with human prompts, work originates from natural workflow events:

• Product team creates ticket → Implementation specification generated

• CI pipeline fails → Diagnostic agent analyzes failure, proposes fix

• PR submitted → Review agent examines code, suggests improvements

• Production alert triggered → Incident response agent investigates, documents findings

• Security scan detects vulnerability → Remediation agent generates patch

The pattern: Event → Agent Response → Human Review → Autonomous Resolution.

Humans remain in the loop, but as orchestrators and validators rather than initiators. The shift from “What should I build?” to “How should this system respond?”

Orchestration of Orchestrators: Production Examples

Stripe’s Blueprint Architecture

Stripe’s Minions system demonstrates mature orchestration-of-orchestrators. Their “blueprint” pattern alternates between deterministic code nodes and agentic reasoning loops, generating 1,300+ pull requests weekly.

Architecture insight: Each blueprint functions as a strict contract between orchestration and execution. Task definitions specify input requirements, output formats, constraints, and success criteria. The orchestrator manages workflow, agents handle implementation.

Security model: Every Minion execution runs in isolated VMs with no internet or production access. The system has submission authority but not merge authority—all changes require human review.

GitLab’s Intelligent Orchestration

GitLab’s Duo Agent Platform treats agents as durable actors that plan, modify code, fix pipelines, and enforce security with traceability. Multiple AI agents handle parallel tasks—code generation, testing, CI/CD fixes—while developers maintain oversight through defined rules.

Orchestration insight: GitLab positions itself as an AI orchestration plane where humans and agents share delivery responsibility. The platform coordinates multi-agent workflows across the entire software lifecycle rather than providing isolated AI tools.

Meta’s Hierarchical Agent Systems

Meta’s Ranking Engineer Agent (REA) demonstrates autonomous ML lifecycle management. REA Planner and REA Executor components, supported by shared skill and knowledge systems, autonomously evolve ads ranking models at scale.

Acquisition significance: Meta’s $2B Manus acquisition focused on orchestration infrastructure rather than foundation models. Manus’s achievement was engineering an execution layer enabling models to browse, code, manipulate files, and complete multi-step workflows autonomously.

The Architecture Implications

Beyond the Single-Agent Model

The production examples reveal a consistent pattern: successful autonomous development requires hierarchical orchestration rather than monolithic AI assistants.

Traditional approach: Human → Single Agent → Code
Emerging pattern: Event → Orchestrator → Specialized Agents → Validation → Resolution

Context Preservation at Scale

The tmux paradigm of persistent sessions maps directly to agent orchestration. Instead of recreating context for each interaction, systems maintain ongoing project understanding across multiple concurrent workflows.

Implementation insight: iTerm2’s tmux integration (-CC mode) provides the UI pattern for agent orchestration—persistent remote workspaces with native interface feel. The same architecture principles apply to agent coordination.

Where This Leads

Non-Human Prime Orchestrators

The logical endpoint isn’t humans managing multiple agents—it’s orchestrating systems that manage agent ecosystems. According to Gartner’s 2025 Agentic AI research, nearly 50% of surveyed vendors identified AI orchestration as their primary differentiator.

Pattern emergence: Meta-agents or orchestrator-generalists will control specialized agents, assign tasks, interpret results, and revise goals in real-time. Hierarchical orchestration becomes essential for enterprise-scale implementations.

The Developer Role Evolution

Instead of managing 10 terminal sessions, a framework orchestrates autonomous workflows. Each workflow maintains its own context, responds to its own triggers, and escalates to human attention when required. Some of those will be human/experimentation/new development driven, the majority will be responding to the automated lifecycle.

Skills that matter:

• Workflow boundary definition: Which autonomous streams can operate independently?

• Escalation criteria design: When do workflows require human intervention?

• Cross-workflow dependency management: How do autonomous streams coordinate?

• Quality gate enforcement: What validation must occur before autonomous resolution?

Implementation Considerations

Teams experimenting with orchestrated autonomous development should consider:

1. Event-driven architecture: Which existing workflows could trigger autonomous responses?

2. Context preservation systems: How will agent workflows maintain project understanding?

3. Isolation and security: What boundaries prevent autonomous agents from causing damage?

4. Human oversight integration: Where do human validation points occur in autonomous workflows?

5. Cross-workflow coordination: How do parallel autonomous streams avoid conflicts?

The transition from 10-tab manual orchestration to autonomous lifecycle orchestration isn’t theoretical. Stripe, GitLab, and Meta demonstrate production implementations. The question becomes implementation timeline and organizational readiness.

Early adopters are discovering that the competitive advantage comes not from having the smartest individual AI agents, but from orchestrating networks of specialized agents that collaborate effectively at scale.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• Stripe’s Minions: Inside Their Enterprise AI Coding Agent Strategy — Blueprint orchestration architecture and production metrics

• GitLab Duo Agent Platform — Intelligent orchestration across software lifecycle

• Tmux Complete Guide: AI-Powered Multi-Agent Workflows — Terminal multiplexing for autonomous development

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

TL;DR

• Users consolidate around 4 core platforms (Slack, Notion, Email+Office, AI Tool) while rejecting standalone/SASS tools

• Connectors that bring data to users beat new tools that require visits

• Infrastructure breakthrough: Slack manifests + MCP protocol + LLM services make org-specific connectors trivial

• Democratization effect: Bootcamp engineers can now build sophisticated integrations that once required senior developers

• Production evidence: 3 connectors (4-6 hours each, $1-2.5K in AI tokens) replaced 5-6 standalone tools that would cost $150-300K+ traditionally

• Tolerance for “broad-based general tools” declining — UX mindshare captures traffic even when APIs do the work

In the past two weeks, I’ve built three connectors that collectively replaced what would have been five or six standalone tools.

Engineering Search Connector: Hosted semantic and knowledge graph search service built by repurposing mcp-vector-search. Unified search across 150+ GitHub repos, 1,700+ wiki pages, and ticket systems. Accessible through Slack bot, web interface, CLI, and MCP connector for Claude.AI that brings engineering knowledge to where people already work.

CRM Data Connector: Live customer data piped directly into Claude.AI sessions via MCP. No dashboard to check, no reports to generate. Ask “What’s our pipeline this quarter?” and get live data in 1-3 seconds.

Document Workflow Connector: Artifact browser and guided PR workflow for non-technical contributors. Product managers can explore and propose changes to structured docs without touching git or learning new interfaces.

None of these required users to adopt a new primary tool. Each brings specialized functionality to platforms they already inhabit daily. And each took roughly 4-6 hours to build (plus agent time).

This isn’t a productivity humble-brag. It’s evidence of a fundamental shift in how organizations interact with their data. We’re entering the connector era — building bridges between specialized intelligence and the handful of platforms where users actually live, rather than standalone applications they have to visit.

The numbers support this pattern. Users toggle between apps 1,200 times daily, losing 40% productivity to context switching. Connector ecosystems are exploding: Slack’s marketplace hosts 2,600+ apps with 550K+ daily custom integrations. The MCP protocol went from 100K to 8M downloads in six months — unprecedented adoption for plumbing infrastructure.

The question isn’t whether Slack, Notion, and Claude.AI will survive the AI wave. It’s whether the hundreds of specialized tools competing for attention understand that the game has changed. Users have less tolerance for broad-based general tools than they once did. The platforms that capture UX mindshare will get most of the traffic, even if APIs and agents do the actual work behind the scenes.

The evidence is clear from user behavior: they don’t want to learn a new search interface, remember another login, or context-switch to yet another tab. They want the intelligence layer to meet them where they already are.

The Source of Truth Problem

Most organizations have a source-of-truth problem they haven’t fully articulated. They have Slack for real-time communication. They have Notion or Confluence for documentation. They have Google Docs for drafts that become documents that become outdated that stay around anyway. They have JIRA for tickets that may or may not reflect what was actually decided. They call this a “knowledge management system.” It’s more accurately a distributed archive of partially-intentional artifacts with no clear authority hierarchy.

The question “who owns this decision?” leads to a Slack thread from eight months ago, a Notion page that three people edited and nobody is certain is current, and a Google Doc someone linked in a comment that requires permission to access. This is the status quo. It functions, after a fashion, because humans are good at triangulating across ambiguous sources and asking colleagues to fill gaps.

AI agents are not good at this. They will confidently synthesize the eight-month-old Slack thread with the outdated Notion page and present the result as a coherent answer. The errors won’t be obvious. They’ll be subtly wrong in ways that require domain expertise to catch.

The source of truth problem was always real. It was manageable when every query ran through a human brain. It becomes actively dangerous when queries run through an inference layer first.

What you actually need — what organizations are starting to build — is a repository where the data structure enforces truth. Not a place where the right answer might be findable if you look hard enough. A place where the structure of the data makes the wrong answer harder to produce.

But here’s the connector insight: that structured repository doesn’t need to be where users spend their time. It can be the authoritative backend that feeds connectors in the platforms users already inhabit.

Where Users Actually Live

User attention has consolidated around four core platforms:

Slack: Real-time coordination, team presence, ephemeral decisions. 32.3 million daily active users with 550K+ custom integrations daily.

Email + Office Suite: Formal communication, document collaboration, external stakeholder interface. Microsoft reports 400M+ Office 365 commercial users.

Notion: Knowledge management, project tracking, collaborative documentation. 100M+ users consolidating entire productivity stacks.

Claude.AI: AI assistance, analysis, content generation. Rapidly becoming the default interface for LLM interactions across knowledge work.

Each platform serves a legitimate core function. Tool builders make the mistake of assuming they can compete for primary platform status by building something better. Users are done adopting new primary platforms. They’re consolidating around tools that already have their attention.

The pattern reveals a deeper truth: people live in transactional systems, not knowledge systems. Slack is where decisions happen. Email is where approvals flow. Claude.AI is where analysis gets done. These are transactional - work happens there daily.

Confluence is a perfectly good wiki tool. But it’s knowledge-at-rest, not transactional. People don’t live there. They visit when forced to document something, then return to their transactional workflows. The knowledge gets stale because maintenance happens in a different system than usage. (Notion manages to straddle the line between knowledge at rest and transactional)

Integration platforms like Zapier understand this - they connect 8,000+ apps with 3.4M+ business users by bringing specialized functionality to existing workflows rather than creating new destinations.

Users just want the data, dammit. They don’t want to learn your interface.

The Connector Infrastructure Moment

What changed? Three pieces of infrastructure matured simultaneously:

Slack Manifest Tool makes organization-specific bots trivial to build. The manifest.yaml format standardizes permissions, scopes, and deployment. Weeks of OAuth wrestling became hours of configuration.

MCP Protocol achieved “USB-C for AI” universal connectivity. Claude.AI, ChatGPT, and dozens of platforms support the same connector format. Build once, deploy everywhere. The 100K to 8M download growth in six months reflects pent-up demand.

LLM Services like Bedrock and OpenRouter provide natural language interfaces that make connectors intelligent rather than just data pipes. Ask questions in plain English, get structured responses, maintain conversation context.

Semantic Search Infrastructure like mcp-vector-search can be repurposed as hosted services, adding intelligence layers that understand meaning rather than just matching keywords. This transforms basic data access into contextual knowledge retrieval — a crucial enabler for connectors that need to surface relevant information rather than exact matches.

Combined, you can build a production connector in a single afternoon. Slack manifest defines the bot interface. MCP schema defines the data sources. Semantic search handles intelligent retrieval. Bedrock provides the language understanding. Deploy to AWS Lambda and you’re live.

My three connectors follow this exact pattern. The engineering search connector repurposes mcp-vector-search as a hosted service with all-MiniLM-L6-v2 embeddings for semantic and knowledge graph search, but the user interface is just Slack commands and Claude.AI MCP tools. The CRM data connector is a headless AWS service that makes customer data available through natural language queries in Claude.AI. The document workflow connector provides git workflows through a web UI that non-technical users can navigate.

Each connector took 4-6 hours to build. Each would have taken 4-6 months to build as a standalone application with user management, authentication, interface design, mobile responsiveness, and all the infrastructure a “real app” requires.

The Democratization Effect: The infrastructure shift goes beyond development speed — it’s democratizing who can build sophisticated integrations. What once required senior engineers with deep API knowledge can now be handled by bootcamp graduates following established patterns. I built these first three connectors to validate the approach, but similar projects will go to junior engineers going forward.

This changes resource allocation fundamentally. Organizations can solve integration problems without burning senior engineering cycles on “plumbing” work. Information that was once very hard to obtain is now trivial to access.

The economics are compelling. Building three production connectors cost roughly $1,000-2,500 in AI tokens over 44 days. Traditional contractor development for equivalent functionality would have run $150-300K+. The connector approach isn’t just faster — it’s 100x more cost-effective.

The adoption metrics prove the value. The CRM connector launched March 18th with 23 invocations on day one. No formal rollout, no training sessions, no onboarding docs. Just organic discovery across a 300+ person company. By week two, daily usage tripled to 95 invocations per day. Tuesday hit 152 invocations — including a 40-query analysis session in a single hour. That’s 299 queries in 7 days with zero errors, from a connector that took 4-6 hours to build.

The era isn’t about choosing between platforms. It’s about connecting specialized intelligence to the platforms users have already chosen.

Why Wikis Can’t Compete in the Connector World

Traditional knowledge management tools face a structural mismatch in connector architecture. Wikis assume users will “go to the tool” for information. Connectors flip that assumption: the tool comes to the user.

This creates specific problems:

The Authoring/Retrieval Tension: Wikis optimize for collaborative authoring — anybody can edit, flexible structure, link everything, evolve over time. This is the opposite of what retrieval needs: consistent schema, clear ownership, explicit governance. When you pipe wiki content through a connector, you inherit all the inconsistencies that collaborative authoring creates.

Search Architecture Limitations: Confluence’s search is notoriously bad because it does keyword matching on unstructured text. This was problematic before LLMs. With LLM-powered connectors, it becomes worse because the AI layer adds confidence to bad retrieval results. Users get wrong answers delivered with conviction.

Static Data Problem: Notion’s AI operates on static content snapshots, disconnected from real-time operational state. When CRM connectors query “What’s our pipeline this quarter?” through a Notion connector, it’s answering based on what someone wrote about the pipeline, not live customer data. The connector amplifies the staleness problem.

Governance at Scale: Wiki governance defaults to “community-maintained,” which means in practice nobody is responsible for accuracy. As organizations scale, wikis accumulate pages nobody knows are outdated. Connectors don’t solve this — they accelerate the distribution of stale information.

Our structured document framework represents the alternative: git-backed Markdown with schema-validated YAML frontmatter. Every document has explicit metadata: owner, status, domain, confidence, time_box. The structure is the feature. When document workflow connectors expose this, the schema ensures consistent data quality regardless of interface.

Structured document repositories outperform wikis for AI query by 35-60% in controlled tests. Clean Markdown with explicit metadata reduces token usage by 20-30% and improves retrieval accuracy significantly. This isn’t philosophical — it’s measurable.

Wikis remain useful for collaborative drafting and evolving reference material. But they’re not the right backend for connector architecture. The connector era requires structured data sources that can maintain quality across multiple interface layers.

Building Connectors vs. Standalone Tools

The strategic choice organizations face isn’t “which tool should we build?” but “should we build a tool or a connector?” My experience with three connectors illuminates the trade-offs:

Engineering search could have been a standalone search platform. Instead, it’s accessible through Slack commands, CLI tools, web interface for visualizations, and MCP tools for Claude.AI sessions. Same search capability, four different interaction models depending on user context.

CRM integration could have been a dashboard with charts and filters. Instead, it’s a headless MCP service that makes customer data available through natural language in Claude.AI. Ask “Show me deals over $100K in our target vertical” and get live results in 1-3 seconds. No dashboard to learn, no visual interface to maintain.

Document workflows could have been a product management SaaS platform. Instead, it’s a guided workflow that helps non-technical contributors interact with existing git-backed document frameworks. Browse artifacts, generate AI summaries, submit PRs — all through interfaces that match users’ technical comfort levels.

Specialized intelligence delivered through platforms users already inhabit. The connector approach wins on several fronts.

Development time: 4-6 hours vs. months. No user management, authentication, responsive design, or mobile apps to build.

Adoption friction: Zero onboarding. No new logins, training sessions, or change management overhead.

Maintenance burden: Focus on data logic and intelligence, not interface maintenance across device types and browser versions.

Integration: Connectors compose naturally with existing workflows. Slack discussions can include live Salesforce data. Claude.AI analysis can pull from engineering knowledge graphs. Standalone tools require export/import workflows.

The business case is compelling: connector development costs 10-20% of standalone application development while achieving 3-4x higher user engagement.

The implications go beyond development efficiency. Users have less tolerance for “broad-based general tools” than they once did. Managing dozens of application contexts creates unsustainable cognitive load. Platforms that capture daily attention get most of the traffic, even when APIs and agents do the computational work behind the scenes.

This creates different winner-take-all dynamics. The winners aren’t necessarily the best tools. They’re the platforms users choose to inhabit, plus the connectors that bring specialized capability to those platforms.

What This Means for Your Stack

The connector era doesn’t eliminate existing tools — it clarifies their appropriate roles and challenges their assumptions about user attention.

Slack keeps its coordination function: Real-time presence, threading, ephemeral decisions. But it becomes a command interface for structured data sources rather than a knowledge repository itself.

Notion retains collaborative authoring value: Drafting, evolving documentation, reference material. But it stops being the “source of truth” for operational decisions. That role shifts to structured backends accessible through Notion connectors.

Specialized tools survive by becoming intelligent backends: Your CRM, your monitoring system, your code repositories — these maintain their core data authority. But user interaction shifts to connector layers in platforms where users already work.

The question to ask about any tool: Is this where I want an AI agent pointing when it needs authoritative information? If the answer is no, it’s not your source of truth. It might still be valuable — as a backend, as a collaborative space, as a specialized interface for expert users. But it doesn’t earn the designation of “primary platform.”

The organizational challenge: Getting non-technical teams comfortable with structured data workflows is real change management. Document workflow connectors address this by providing guided interfaces for git-backed workflows. But someone still needs to own schema design and governance processes.

Who should build connectors first: Engineering-adjacent teams with strong PM-engineering collaboration. Organizations where AI hallucination on operational decisions creates measurable cost. Companies that have already felt the pain of distributed knowledge management.

Timing matters: Most organizations haven’t built connector strategies yet. Companies that establish structured knowledge backends with connector frontends in 2026 will have 12-18 months of advantage when AI-mediated query becomes standard practice.

The connector era isn’t about choosing between platforms. It’s about connecting intelligent backends to platforms users have already chosen. Organizations that get this right will operate with less context switching and faster access to operational data.

Users just want the data, dammit. The question is: will you bring it to them, or keep expecting them to come to you?

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders

One year ago, “50 First Dates with Claude Code“ took me several hours to write with Joanie’s help in Google Docs. This morning, I drafted two comprehensive Anthropic articles using claude-mpm agents in 45 minutes - research, generation, editing, the works.

That evolution mirrors exactly what happened across the entire AI coding industry this year (and the ascension of Claude Code, which was released shortly before I began my journey). I wasn’t just writing about the transformation. I was living it.

The Journey: From Accident to Infrastructure

HyperDev started by accident. In March 2025, I posted a LinkedIn experiment about spending 12 hours building a travel planning app with AI tools. I was a technology executive who “hadn’t coded seriously in 20 years” testing whether the productivity claims were real or hype.

The response was immediate and intense. Richard Wang left a prescient comment: “’AI allows a non-engineer to build a product without coding’ is hype... ‘AI can improve a developer’s productivity by 10x’ is true.” That experiment generated 4,000 lines of AI code in a single session and launched what became 168 articles over nine months.

April through June was chaos. I built claude-multiagent-pm, a prototype that worked well enough to be exciting and poorly enough to be frustrating. Token costs were obscene - every subprocess inherited the entire conversation context. I shipped 44 repositories. Probably a third represent false starts or abandoned approaches.

But that’s what I was learning: what not to build. Which constraints matter. Where the sharp edges live. The breakthrough insight from this period: Infrastructure beats features. The tools that demo well (flashy autocomplete, pretty interfaces) weren’t the ones that sustained daily use. The protocols, memory systems, and context management layers were what made sustained multi-agent work possible.

The Breakthrough: When Everything Changed

Mid-July 2025, Claude Code shipped context filtering. Sounds like a minor technical detail. It changed everything.

Before: my prototype burned tokens like a furnace and required constant babysitting.
After: I rebuilt everything. claude-mpm emerged with 1,545 commits over the rest of the year.

I remember the specific moment it hit me: I’d just pushed a feature that I would never have taken on without a team. Four hours of engaged time, a few days of agentic time. Twenty years away from serious coding. Four months back. Contributing production code for paying clients.

The tools weren’t just making me more productive. They were making me more ambitious. I was taking on problems that required sustained, complex thinking because I had AI teammates that could handle the execution details while I focused on architecture and strategy.

This is when my perspective shifted from “AI coding tools are interesting” to “AI coding tools are transformative.” Not because they eliminated the need for programming knowledge, but because they amplified existing knowledge into production-quality artifacts.

The writing and building formed a virtuous cycle. I documented what I learned building tools. The documentation attracted practitioners who used the tools. Their feedback improved the tools. claude-mpm gained 30+ stars and daily use across six months of client work. These weren’t GitHub tourism projects - they were tools that other practitioners adopted because they solved real problems I’d discovered through real use.

The Evidence: Predictions and Numbers

Looking back through a year of articles, my prediction accuracy was surprisingly good.

What I got right:

• Multi-agent orchestration would prove superior to monolithic assistants (claude-mpm’s adoption validated this)

• Infrastructure over features as the determining factor for tool longevity (memory systems outlasted flashy demos)

• CLI-agentic coding going mainstream (Claude Code’s 46% “most loved” rating proved the thesis)

• Pricing correction timing (18-24 months from October 2025 - signals are clear at six months)

What surprised even me:

• Speed of Claude Code’s dominance (faster than even advocates expected)

• Writing-building credibility loop (thought leadership through shipping, not just analysis)

The quantified impact tells its own story:

• 4,919 commits in nine months of sustained development

• 69.7 billion tokens processed across all tools and projects

• 198 articles published (3.2/week sustained)

• 547 production deployments across client and personal projects

• $45,000 in AI compute at rack rates (subsidized to $8,000)

But the qualitative transformation matters more. I evolved from asking “Is this real?” to asking “How do we scale this organizationally?”

The Transformation: From Observer to Practitioner to Leader

The most significant development was personal: in January 2026, I joined Duetto as CTO.

This wasn’t a career change - it was an expansion. Twenty-five years of technology leadership combined with eight months of hands-on AI development created a unique perspective. I wasn’t returning to technology leadership despite the AI work. I was taking the role because of it.

What changed wasn’t my capabilities - I’ve been leading technology teams for decades. What changed was having AI as a force multiplier that converted that knowledge into actual artifacts. I could prototype entire systems, validate approaches, and demonstrate concepts that previously would have required dedicated engineering resources.

The leadership experience informed architectural decisions. The practitioner activity created credibility. The writing documented both. Now I get to test everything I’ve been writing about at enterprise scale.

The Reality Check: Both/And, Not Either/Or

October 2025, I published “Is AI A Bubble? I Didn’t Think So Until I Heard Of SDD.” The piece synthesized something I’d been wrestling with: how can genuine transformation and bubble dynamics exist simultaneously?

The answer: they can. And do.

The AI coding revolution is real. The bubble dynamics are also real. Codeium at 70x ARR multiples (vs dot-com peak of 18x) while providing genuine value to practitioners. My $45,000 in AI compute costs exemplified both the unsustainable economics and the genuine value creation. The 82% subsidy rate can’t last, but the ROI still works at full rates for sustained professional use.

Companies with product-market fit and operational discipline will survive the correction. Those burning capital on “technical potential” without user adoption won’t. The technology remains transformative even if the valuations prove unsustainable.

What’s Next: From Individual Productivity to Organizational Transformation

The industry is transitioning from individual productivity tools to enterprise transformation frameworks. The early adopters who mastered AI-assisted development workflows now face a different challenge: scaling those practices across entire engineering organizations.

The questions have evolved:

• Year 1: “Can AI tools make me more productive?”

• Year 2: “How do we maintain code quality and security with AI-generated code?”

• Year 3: “How do we transform hiring, onboarding, and career development when AI changes what programming means?”

At Duetto, I’m working on these questions at scale. How do you migrate an enterprise engineering team from traditional development practices to AI-assisted workflows while maintaining operational excellence? How do you balance productivity gains with governance requirements? How do you rethink technical leadership when junior developers can ship senior-quality code with AI assistance?

These are the infrastructure problems that matter now. Not the tools themselves, but the organizational systems that make the tools effective at scale.

The Retrospective: What One Year Taught Me

HyperDev became valuable because it documented the transformation in real-time, from the perspective of someone living it. Not retrospective analysis of what happened, but contemporary documentation of what was happening.

The key insights:

Timing matters. Starting documentation right at the inflection point captured both the chaos and the consolidation. Personal journey paralleled industry maturation.

Practitioner perspective beats observer perspective. Direct experience with the tools, including their limitations and sharp edges, generated insights that pure analysis couldn’t match.

Building creates credibility. Shipping tools that other practitioners adopt generates more authority than analytical commentary alone.

Writing and building amplify each other. Documentation of practice creates thought leadership. Thought leadership creates opportunities. Opportunities create more practice to document.

One year ago, I was asking whether AI coding tools were genuinely transformative or just sophisticated autocomplete. The answer: they’re genuinely transformative, but in ways that none of us fully anticipated. The transformation wasn’t about eliminating the need for programming knowledge. It was about amplifying existing knowledge into production-quality artifacts faster than previously possible.

Most importantly, the transformation was about enabling individual practitioners to think and build at organizational scale. That’s what I experienced personally. That’s what I documented in nearly 200 articles. And that’s what I’m now implementing at enterprise scale.

HyperDev year one was about learning what was possible. Year two is about making it practical. Year three might be about making it inevitable.

The tools keep getting better. The workflows keep evolving. The organizational challenges keep getting more complex.

And I’m still here, still building, still documenting, now leading, still learning.

What a year it’s been. What a year it’s going to be.

HyperDev documents the real-world application of AI development tools by practitioners building production systems. For technical deep dives and business analysis of the tools behind this transformation, look for upcoming coverage of Claude Code’s competitive dominance.

The story everyone told about Clawd Bot missed the point entirely. Austrian developer Peter Steinberger built an open-source AI assistant that went viral — 145,000 GitHub stars, 2 million visitors in a week. Then Anthropic forced a trademark-based name change because “Clawd” was too similar to “Claude.” The community called it petty. DHH called Anthropic “customer hostile.” The irony: Clawd Bot users were actually buying more Claude subscriptions, providing free marketing to Anthropic, yet they still demanded the shutdown.

But everyone focused on the wrong drama. The trademark dispute was noise. The real problem was deeper: Clawd Bot was built because someone could, not because anyone needed it.

I tested Clawd Bot for about a week. The interface was clean, the onboarding smooth, the responses capable. But it required permissions I wouldn’t give to any tool — access to email, calendars, messaging, sensitive services. The execution had real problems. But even if those were fixed, it would still be solving the wrong problem.

Here’s where I should admit: I tried building a digital assistant, izzie, when I started experimenting with AI agents. I never got it to a point I found useful. Not because of technical limitations — because the entire concept of a universal assistant doesn’t match how work actually happens.

TL;DR

• Clawd Bot was successful open-source project by Peter Steinberger that Anthropic forced to rename; execution wasn’t the problem

• The real question: when do you need an “assistant”? Most execs won’t trust AI scheduling; the value is intelligent data movement between services

• Context switching is a symptom, not the root issue — the issue is what assistants should be doing at all

• The product management sessions: Granola meeting notes, calendar checks, Slack updates, Notion sync — all from within one tool, data flowing intelligently between services

• The commercial evidence: Cursor, Notion AI, Linear’s AI triage — the winners embedded AI in tools as infrastructure, not interface

• trusty-izzie’s highest value isn’t the chat interface — it’s as a local MCP service exporting personal context to every other tool

• The universal assistant category isn’t going to produce a winner. It’s going to dissolve.

What the Universal Assistant Model Gets Wrong (And It’s Not Just Execution)

Clawd Bot had serious execution problems — it’s a security nightmare requiring broad permissions across email, calendars, messaging platforms, and sensitive services. You can’t ignore that. But even if the security issues were solved, universal assistants face a deeper structural problem: they assume people need an assistant in the traditional sense.

Walk through what even a well-executed version of the same product model looks like.

Smooth onboarding. Crystal-clear use cases. High-quality AI responses. Clean interface design. Users know exactly what to ask and how to ask it.

You still have to leave whatever you’re working on to use it. And when you do, the context you were carrying — the code you were reviewing, the initiative you were drafting, the design decision you were working through — is no longer present. You’ve moved somewhere that knows nothing about any of that.

So you explain. “I’m working on the ‘YYY’ data ingestion initiative, and I need to check whether the points Mark raised in Tuesday’s meeting are addressed in the current design.” The assistant doesn’t know what ‘YYY’ is. Doesn’t have Tuesday’s meeting. Doesn’t know Mark, the current design, or the organizational context that makes “addressed” mean something specific. You load all of it by hand.

In demos, this overhead is invisible. Demo tasks are self-contained by design — the context fits in a sentence or two. In practice, your working context isn’t self-contained. It’s weeks of accumulated decisions, relationships, dependencies, and constraints that live distributed across your tools. You can’t paste it into a chat window. You can’t even fully articulate it. It’s partially tacit, partially in documents, partially in the history of the tool you’re using.

The Session That Clarified It

A few weeks into the new role at Duetto, I was doing product management work in a claude-mpm session — reviewing open initiatives, managing the PR queue, creating proposals. Standard operational work for a new CTO getting oriented.

I wanted to add an infrastructure initiative. Cloud Dev Sleds — dedicated cloud development machines for the engineering team. The context was in a meeting I’d had the day before. In the old workflow, this would mean: switch to Granola, find the right meeting, read the transcript, extract the relevant points, switch back, and then write the initiative with that context now loaded in my head rather than in the tool.

Instead I just asked: “Review my meeting with Mark yesterday in Granola to get context. I want to create the initiative as a feasibility, cost, and LOE assessment.”

The tool pulled the notes. I created the initiative. The product context — what other infrastructure work was in flight, what the team structure looked like, what the related architectural decisions were — never left. The Granola content landed inside that context rather than requiring me to carry it manually between tools.

Same session: needed to check whether I had a conflict for an upcoming demo. Calendar check, without opening Google Calendar.

Same session: the team needed a status update. Posted directly to the engineering Slack channel, with proper <@USERID> mentions so people actually got notified. The message reflected the same initiatives I’d been working on all session — not because I copy-pasted anything, but because the tool already knew what was in flight.

Later: set up a Notion sync — initiative statuses with links to the docs, updated automatically.

The efficiency argument is real but secondary. The more important thing is that the product context never left. The tool knew what initiatives existed, who owned what, what the architectural decisions were, which PRs were waiting on which engineers. When I pulled Granola notes, they arrived inside that context. When I posted to Slack, the message was informed by that context. A universal assistant would have required me to reconstruct and transport that context manually every time I needed to cross a tool boundary.

No universal assistant is going to have that work knowledge. Not because the AI isn’t capable. Because the knowledge lives in the tool, accumulated over months — PRDs, design decisions, initiative history, team assignments, the proposals that got approved and the ones that didn’t. You don’t recreate that in a chat window.

The Deep Context Problem

The thing that makes domain tools irreplaceable isn’t AI capability. It’s accumulated context.

A product management tool carries months of initiative history. The CTO knowledge base carries organizational decisions, vendor relationships, strategic context that builds over time. These aren’t things you can summarize in a system prompt. They’re queryable, interconnected, grounded in real artifacts. The tool has developed something like institutional memory — and that memory is what makes AI assistance inside the tool qualitatively different from AI assistance outside it.

Universal assistants are built for breadth. Any question, any domain, any task. That breadth is the pitch and also the structural weakness. The model that’s ready for anything is primed for nothing specifically. It has no idea that “the YYY initiative” refers to a specific ingestion redesign with a particular set of constraints, a particular set of people involved, and three months of design decisions behind it.

The inversion worth stating plainly: the tools you work in every day already have more relevant context than any assistant will. The right move is surfacing AI capabilities inside those tools, not pulling people out of those tools into a separate assistant layer.

But here’s what’s happening at the executive level. I’m finding more and more technical executives using Claude Code as knowledge assistance — not because they’re universal assistants, but because the amount of data and complexity they can manage far exceeds what standard off-the-shelf tools provide. The deep context problem can’t be solved with generic solutions.

For MPM, I built specific connectors: gworkspace-mcp, slack-mpm, notion-mpm, granola-mcp (the last from Granola, the others myself because mcp has limitations). That became as much of an “assistant” as I needed, besides izzie. No universal chat interface. Just targeted data bridges that let Claude access specific services when I’m working on something that needs their context.

The commercial evidence points the same direction. The AI tooling products with real adoption aren’t universal assistants. Cursor put AI in the editor. Notion AI put AI in the documents. Linear’s triage put AI in the issue tracker. Each works because the AI operates inside existing context. The pattern is consistent enough that it’s probably not coincidence.

What I Got Wrong About My Own Bot

I (re)built trusty-izzie as a personal assistant — natural language queries over my email and calendar history, local graph database, vector embeddings, stays on my machine. It works. But “personal assistant” was the wrong frame for where the value lies.

The thing izzie has is a grounded, real-time, locally-stored representation of my professional life — people, relationships, projects, scheduling, communications history. That’s a context store. Every tool I use should have access to it without me switching to izzie to ask.

The right version of izzie isn’t the one you talk to. It’s the one that runs as a local MCP service — always on, queryable by anything that needs personal context. The product management tool asks it about scheduling. The writing environment surfaces relevant prior conversations. The coding environment knows who owns what system before I have to explain it. None of that requires me to open izzie. It requires izzie to be infrastructure rather than interface.

If you want to try izzie yourself: izzie.bot has the details, and the full source is at github.com/bobmatnyc/trusty-izzie. I strongly recommend building from source using an agentic coder to verify the code is safe — never trust AI tooling with your personal data without auditing it first.

Not there yet. But the frame shift changes what to build next.

What the Architecture Looks Like

If you’re building a personal AI tool, the question isn’t “what will users ask the assistant?” It’s “where do users have context, and how do you bring assistance there without making them leave?”

The test is simple. Does using your tool require leaving the context where the relevant information lives? If yes, you’re fighting the architecture. Users will use it occasionally, for low-friction tasks. They won’t build their workflow around it.

The tools that pass the test: Claude Code (your codebase is the context), Cursor (you stay in the editor), Notion AI (you stay in the document), Linear AI triage (you stay in the issue tracker). The tools that fail it: every standalone AI assistant that requires opening a new interface and re-explaining what you’re working on.

For domain tools with real depth — months of accumulated decisions, relationships, history — the connectors are the product. The LLM orchestration is the interface layer. The accumulated context is what no competitor can replicate by building a better general assistant. The moat isn’t the AI. It’s what the AI is operating inside.

For personal infrastructure like izzie: build the MCP service before the chat UI. The chat UI is useful and I use it. The MCP service is what makes the tool true infrastructure rather than one more thing to switch to.

The universal assistant category isn’t going to produce a winner because the category is structured wrong. The capabilities will get absorbed by the tools where the relevant context lives — because that’s where the value is, and users will figure that out even if product teams don’t. The infrastructure driving this — entity and relationship detection, email, calendar, and task management (all built for Izzie) — will likely be delivered by the personal productivity tool providers (hello Google).

Clawd Bot wasn’t a failed product. It was wildly popular, but I suspect will have been a flash in the pan once the shininess wears off and the liabilities outweigh the usefulness. That distinction matters, because if you think it’s an execution problem, you go looking for a better universal assistant. If you understand it’s a conceptual problem — that most “assistant” work is intelligent data movement — you build infrastructure instead of interfaces.

Bob Matsuoka is CTO of Duetto and writes about AI-powered engineering at HyperDev.

Related reading:

• What Does A Pattern Master Do? — The role of expertise in AI development

• AI Power Ranking — Tool comparisons and benchmarks for AI practitioners

• LinkedIn Newsletter — Strategic AI insights for CTOs and engineering leaders