TL;DR: AI coding agents emit "I've finished" and "tests pass" the same way they emit any text — as the next likely token. That's narration, not attestation. Real review gates run independent of the agent, inspect actual artifacts, and produce verdicts bound to the specific code — so a pass can't be reused on changed code.
In This Guide
- The "model stopped talking" problem
- What transcript trust actually catches — and what it misses
- The anatomy of a real review gate
- Why the judge must be separate from the generator
- Hash-bound verdicts and why they matter for drift
- How CodeRight approaches the problem
- Common mistakes when wiring review gates into a coding pipeline
- Who this is for — and who it isn't
- Continue reading
The "model stopped talking" problem
Every AI coding agent eventually stops generating output. If you are relying on chat-style orchestration, that moment — silence — is often the only signal you have that a task is complete. The agent says it's done. You look at the output. Maybe it looks right. You ship.
This is not a verification strategy. It is optimism dressed as workflow.
The problem is structural, not adversarial. Large language models are trained to produce coherent, natural-sounding completions. "I've implemented the changes and the tests are passing" is an entirely fluent, high-probability continuation of a coding task. Whether the tests actually pass is a different question that the model's output layer does not automatically resolve. As a 2025 analysis of agent verification gaps put it: "The agent did produce code. It just didn't verify that any of it worked before declaring victory."
This is the agent version of a contractor texting "job's done" without having turned the water back on.
The failure mode compounds with orchestrated multi-agent pipelines. If the agent generating code is also the one asserting that the code is correct, you have a closed loop: the same model that made the error is evaluating whether the error exists. That is not a review gate. That is an echo chamber.
What transcript trust actually catches — and what it misses
Most agentic coding tools today use some form of transcript parsing to infer completion. The orchestrator scans for markers like "committed 3 files," "all tests pass," or "implementation complete" and treats them as ground truth. This catches catastrophic failures — if the model hallucinates an entirely fictional file path, it usually can't produce a plausible success message. But it misses a wide range of real failures:
- Code that compiles and passes tests, but doesn't solve the stated problem
- Tests written to pass rather than to verify the implementation
- Accessibility, responsiveness, or configuration issues that don't throw errors
- Documentation that's outdated the moment it's written
- Changes that exist only in the conversation context, not on disk
A follow-up analysis of what coding agents actually verify identified the core gap clearly: "'Build passes' and 'the output is production-ready' are different bars." The agent optimizes for the first bar because that's what its native tooling checks. The second bar requires gates the agent doesn't naturally run.
Warning: If your current agentic coding setup relies on the agent's self-report to decide when to stop, you are not running verification. You are running auto-complete with a completion threshold.
The anatomy of a real review gate
A functional coding agent review gate has three properties that distinguish it from a transcript check:
1. It inspects artifacts, not narration.
The gate runs git diff, checks file existence, executes the build, and runs the test suite against the actual branch. It does not pattern-match the agent's output text. The outcome-based verification approach replaces "agent said it worked" with specific checks: did the files change? Did the build exit zero? Did the tests run against real state?
2. It is independent of the generator. The agent that did the work does not score its own work. A separate model — or a deterministic checker — evaluates the output against defined criteria. Research on LLM-as-judge systems is clear on this point: "The agent doing the work should not be the sole authority deciding whether its own work is good enough." Hidden coupling — where the judge and generator share model priors or similar prompts — creates false independence while reproducing the same biases.
3. It uses observable, bounded criteria. Vague assessments ("looks good," "reasonable implementation") are not gates — they are rubber stamps. Effective review criteria are narrow and explicit: the build exits zero on the target platform, specified functions exist in the output, the diff touches only the files the task was scoped to, the implementation satisfies the acceptance criteria defined before the task ran.
Anthropic's engineering writeup on their multi-agent research system makes a similar point about completion criteria: effective orchestrators define what "done" means before work starts and evaluate against that definition, not against the agent's self-assessment.
Why the judge must be separate from the generator
The judge-generator separation is not a nice-to-have architectural detail. It's the difference between a review and a re-read.
When the same model evaluates its own output, several failure modes become likely. Style gets rewarded over substance — the model favors polished prose and coherent structure, which it already produced, over correctness it might have missed. Self-referential loops develop when multiple agents share similar training and amplify each other's errors rather than catching them. Score inflation sets in as verdicts drift toward approval, especially when no ground truth exists to anchor them.
LLM-as-judge research identifies where independent judges work reliably: pairwise comparisons between competing outputs, pass-fail decisions against explicit criteria, detection of constraint violations, and routing decisions where the judge identifies which specialist should handle a case. The pattern breaks down when judges operate on vague criteria, inspect polished prose rather than substantive evidence, or lack calibration against observable outcomes.
The practical implication for an orchestrated coding workflow: the model that writes the code should not be in the same prompt chain as the model that reviews it. They should receive the same artifact — the actual diff, the actual test output, the actual build log — and the reviewer should not have access to the generator's reasoning trace. What matters is whether the artifact meets the criteria, not whether the generator had a good reason for its choices.
Hash-bound verdicts and why they matter for drift
Here's a subtle failure mode that most review gate discussions skip: verdict recycling.
An agent completes a task. A judge reviews it, scores it as passing, and returns a verdict. The developer makes a small change — a hotfix, a refactor, a dependency bump — and the orchestrator, correctly or not, treats the previous verdict as still valid because "the feature" didn't change. The verdict travels with the ticket, not with the specific code it evaluated.
This is verdict drift. The review happened on a version that no longer exists. The passing score is now attached to code that was never scored.
The fix is straightforward in design: bind the verdict to the exact artifact it evaluated, using a content hash. A SHA-256 of the diff or the output files means the verdict is only valid for that exact state. Any change to the artifact invalidates the verdict and requires re-review. There is no way to accidentally carry a passing score forward onto changed code, because the hash won't match.
This is standard practice in CI/CD artifact integrity — OWASP's CICD-SEC-09 covers the risks of improper artifact integrity validation in pipelines — applied to the review layer of a coding agent workflow.
Tip: Before adopting any agentic coding tool, ask one question: if you change a single line after review, does the system require a new review? If the answer is "it depends on whether the reviewer notices," the verdicts aren't hash-bound, and they shouldn't be trusted.
How CodeRight approaches the problem
CodeRight is an orchestration runtime for AI coding work. The architecture is built around these problems directly.
You hand it a goal. It routes work across a roster of models you bring — any OpenAI-compatible endpoint, your own API keys, your own cost envelope. A separate jury reviews every artifact the coding agents produce. The runtime refuses to mark the goal complete until the jury has signed off, and the verdicts are hash-bound to the specific code they evaluated.
The goal is stated before work begins. The jury evaluates against that goal, independently of the model that did the coding. A verdict issued on one state of the code cannot be automatically applied to a changed state. The hash doesn't match; the review runs again.
The runtime runs async, in the background. You don't babysit a chat session. Work proceeds, gets reviewed, gets flagged for repair if the jury rejects it, and surfaces to you when it has passed — or when it has failed in a way that needs your input.
CodeRight is native: Tauri v2 + React 19 + Rust. Not Electron. Not a browser tab. Not a cloud-hosted IDE. The installer is in the tens of megabytes, not gigabytes — Tauri apps typically idle at 30–40 MB of memory vs. Electron's 150–300 MB because they use the OS's native WebView rather than bundling Chromium. On Windows and macOS, that means a quiet process that doesn't compete with your editor for RAM.
On model promotion: CodeRight is designed to log which models in your roster perform well — and which don't — so the runtime can surface evidence for promoting or rotating models over time. This behavior is built into the architecture and governed by an audit log. It is not yet fully shipped end-to-end; the intent is that the system accumulates evidence rather than running blind, and that any promotion decision is logged and visible to you.
CodeRight is pre-launch. Two-step waitlist at coderight.cc.
Common mistakes when wiring review gates into a coding pipeline
Using the same model as generator and judge. The same model, even with a different system prompt, shares the same priors and failure modes. It will tend to approve what it would have generated. The review needs a distinct model, or at minimum a distinctly different evaluation context.
Setting criteria after the fact. If you define what "done" means by looking at what the agent produced, you are reverse-engineering acceptance criteria to match the output. Define the criteria before the task runs, so the judge has a fixed target.
Trusting process signals instead of artifact signals. "The agent used the right tools" is a process signal. "The artifact meets the stated criteria" is an artifact signal. Review gates should evaluate the latter. Process signals matter for debugging, not for completion.
Running one judge instead of two. A single judge's verdict has all the failure modes of a single data point. Two independent judges evaluating the same artifact — with visible disagreement when they diverge — gives you signal about where the artifact is ambiguous or borderline. Disagreement is information. A single verdict is a coin flip with extra steps.
Letting verdicts travel with tickets instead of hashes. Any workflow where "this was already reviewed" means a verdict from a previous code state is assumed valid for the current state is running without integrity. Bind verdicts to content hashes. No hash match, no valid verdict.
Who this is for — and who it isn't
Good fit: developers who run AI coding agents regularly and have noticed that "it says it's done" is not the same as "it's done." People who want to bring their own models and API keys rather than paying per-seat for a cloud IDE. Developers on Windows who want a native process, not a browser tab. Anyone who has felt the pain of a multi-agent workflow where context bloat and self-assessment made the output untrustworthy.
Not a fit right now: teams that need shared workspaces, collaborative review, or integrations with existing project management tools — CodeRight is pre-launch and the feature set is initial. Developers who need a turnkey solution with zero configuration — BYO keys and model roster setup requires some initial work. People looking for a fully cloud-managed service where someone else handles model routing and infrastructure.
If you are in the first group: coderight.cc has the waitlist.
Continue reading
- CodeRight — the coding agent orchestration runtime — what it is, how the roster and jury work, and how to get on the waitlist
- How Model Routing Actually Works in a Multi-Agent Coding System
- BYO Keys in an Agentic Coding Workflow: What Changes
About the author
Adrian D'Souza is a US-based solo founder who builds and runs nine-plus product ventures — including Damned Designs, Rotten Hand, and the Right Suite of local-first desktop apps — on a single custom stack he built and maintains himself. He designs, engineers, and ships across hardware, fashion, and software, with no team and no VC. CodeRight came out of the same frustration this post describes: AI coding agents that narrate confidence without delivering verified output. Follow at @bogusyogi or connect on LinkedIn.
Sources
- AI coding agents lie about their work. Outcome-based verification catches it. — DEV Community (moonrunnerkc)
- AI Coding Agents Can Verify Some of Their Work Now. Here's What They Still Miss. — DEV Community (moonrunnerkc)
- LLM-as-a-Judge in Multi-Agent Systems: Where It Works, How to Build It, and Why the Flow Matters — Medium (balajibal, April 2026)
- How we built our multi-agent research system — Anthropic Engineering
- CICD-SEC-09: Improper Artifact Integrity Validation — OWASP Foundation
- Tauri vs Electron 2026: Bundle Size and Memory Footprint — Tech Insider