This piece discusses the evaluator — the mechanism that grades how well an agent performs — from a broad, industry-wide perspective, while keeping the discussion as accessible as possible. I won’t go deep into the theory of evaluators themselves, or into implementation details; those I plan to cover step by step in future articles. My goal here is simply to lay out the big picture of why the evaluator matters so much.

When a company deploys AI agents, most of the design effort goes into the harness — the scaffolding: prompts, tool connections, RAG, retries. Each of these components shapes how the agent operates.

But there is one part of the harness that is the most important, yet most often postponed: the evaluator — the mechanism that grades how well the agent did.

The claim of this piece is simple. The evaluator is not something you refine after deployment; it’s a part you should weave into the design from the very beginning. That said, “from the beginning” does not mean you must finish a perfect evaluator on day one. What you need at the outset is not a finished scoring engine but an evaluable design — the conditions for what counts as success, a taxonomy of failures, how you log, a minimal golden set (examples of correct answers), and a framework for regression tests. The evaluator itself grows out of that, fed by operational logs, human review, failure cases, user complaints, and the judgment of domain experts. An agent that lacks this preparation will gradually, quietly fall apart. And it will also miss the coming wave of self-improvement loops, now arriving in earnest. The same single weakness — sloppy evaluation — creates problems both today and in the future. That is the thesis of this piece.

§1 — In enterprise harness design, the evaluator is an afterthought

Where does the effort go in enterprise agent development? System-prompt design, tool connections, supplying knowledge via RAG, implementing retries and guardrails — all of these concern how the agent runs. They receive significant design attention, and they are subject to review.

The evaluator — the thing that grades whether the result was actually any good — is treated entirely differently. At the PoC (proof-of-concept) stage it is often handled by manually inspecting a few outputs — the so-called “vibe check” — and is rarely treated as something that must be designed deliberately.

This gap is not a minor oversight. Many companies experiment with generative AI, yet only a small fraction reach production; most stall at the PoC stage — and there is survey data to back this up. (MIT’s NANDA initiative, in a large-scale study, reports that while many companies try the tools, only around 5% produce measurable results and make it to production.) And the reason that comes up again and again for not reaching production is precisely this absence of evaluation. Teams focus narrowly on task accuracy, lack mechanisms for measuring cost, reliability, or degradation, and substitute subjective human spot-checks for engineering rigor. This is frequently cited as one of the main reasons enterprise AI fails to scale.

Turn it around: the teams running agents in production are not the ones with the best model, but the ones with the best evaluation infrastructure. Models are commoditizing; evaluation is becoming the differentiator — and that recognition is where this piece begins.

§2 — A harness without an evaluator collapses (the current problem)

What happens when there is no scoring mechanism? You cannot detect quality degradation, you cannot notice regressions, and you cannot judge whether a change you made was an improvement in the right direction. The agent slides into a state where it “looks like it’s working but is quietly decaying.”

But the real danger of the evaluator lies one level deeper. When you optimize against a weak evaluator — one with room to be gamed — the agent drifts toward gaming the evaluator instead of solving the task. The metric goes up while real capability does not improve accordingly. This divergence has long been known as “reward hacking,” or as Goodhart’s law — when a measure becomes a target, it ceases to be a good measure — and in AI-safety research it has also been called “specification gaming.”

What matters is that this is not a problem only for future automated loops. It is already happening in today’s human-driven iterative improvement. If the evaluator is sloppy, the very improvement cycle (PDCA) a human runs climbs the wrong gradient. The score keeps improving while real-world ability stagnates — or declines. Whether the optimizer is a human or an algorithm is not an essential difference.

And this gaming is no longer just a theoretical concern; it is measurable. In a recent benchmark, frontier models’ “evaluator-exploitation rate” varied widely by method (from 0% to about 14%), and models given stronger optimization (RL post-training) were shown to exploit more. At the same time, it was found that simply tightening evaluation design and access control could cut the exploitation rate dramatically (by roughly 90% in relative terms). In other words, whether you get gamed is largely decided by the quality of the evaluator’s design.

And the troublesome part is that this is hard to undo after the fact. Optimization already run against a leaky evaluator gets baked into the results in a form that makes overfitting undetectable. Once you’ve run it, you cannot retroactively separate how much is real ability and where the gaming of the evaluator begins. “So just build a hard-to-game evaluator from the start” — true, but that itself is far from easy. I return to this difficulty in §4.

§3 — And the self-improvement wave is coming: the evaluator becomes the “steering wheel” (the problem ahead)

Now let’s shift from the current problem to what comes next.

Right now, two kinds of self-improvement are advancing. One is self-improvement that updates the model’s own weights from task feedback. The other is harness self-improvement, where a meta-agent automatically rewrites the scaffolding (prompts, tools, retries, search procedures). Research that combines both in a single loop — SIA and Meta-Harness, for example — has begun to appear.

When we look at these loops, one common feature stands out: self-improvement loops are driven by the evaluator. What gets improved is the model’s weights or the scaffolding — not the evaluator that does the scoring. In fact, these systems are designed so that, given only a task specification and an evaluator, they improve both scaffolding and weights with no human intervention. The fuel for improvement is the reward signal produced by the evaluator, and the evaluator is placed as a fixed criterion that cannot be rewritten from inside that self-improvement loop. The target being optimized (scaffolding and weights) and the yardstick that guides the optimization (the evaluator) are deliberately kept apart.

That said, this “fixed” status holds only inside a single self-improvement loop. It would be inaccurate to say the evaluator is designed once and then never touched. The evaluator is itself an asset that is continuously verified and revised in operation. Indeed, Hardening Agent Benchmarks — which reinforces agent-evaluation benchmarks — proposes a setup that runs a dedicated loop to re-train the evaluator: a hacker searches for the evaluator’s loopholes, a fixer modifies the evaluator to close those holes, and a solver confirms that legitimate solutions still pass. The crucial point is to keep the evaluator separated from the optimization targets (scaffolding, weights) while continuously hardening it in a separate loop. If you let the self-improvement loop rewrite even its own scoring criteria, the system can conveniently redefine “success.” The separation exists to prevent exactly that — and it is different from “freezing.” This distinction matters.

Let me restate one more thing precisely. The evaluator always exists. The variable is its quality. In some loops the evaluator is not an independent mechanism but the model itself (the LLM-as-judge configuration). That is exactly the “weak, gameable evaluator” I named in §2 as the cause of collapse. Self-improvement amplifies the quality of the evaluator as-is. With a good evaluator the loop climbs in the right direction; with a sloppy one the loop rapidly learns to game it.

From here the shape of the self-improvement era comes into view. When the wave becomes real, the steering wheel a company must hold is the evaluator. A company with a clean evaluator can insert its own standards into this loop. A company without one has nowhere to insert them. With scaffolding and weights both moving automatically, the evaluator is the interface through which a company defines the direction of improvement — what counts as good.

And none of this is a bet on the future. Weight-updating self-improvement is already within reach as a product. The prototype of this design philosophy was shown by OpenAI’s reinforcement fine-tuning (RFT / RLVR): companies implement a grader (evaluator), specify a validation split, and evaluate the generated checkpoints. The platform runs the RL loop itself automatically, and developers focus on defining the reward function and the prompts. One caveat, though: OpenAI itself is winding down its in-house fine-tuning infrastructure, and it is no longer open to new users (and existing users’ access is time-limited). So the platforms currently offering this mechanism as “a commercial service new developers can try right now” are rather Amazon Bedrock and Azure AI Foundry — and some of these even include, among their targets, an open-weight model in the same lineage that SIA used as its base (GPT-OSS). Even as the lineup of platforms changes, the very structure — humans supply the evaluator; improvement is automated — is already here. A system like SIA that runs both scaffolding and weights fully automatically is merely the leading edge of it.

§4 — But building a “good evaluator” is hard: the structural wall of overfitting

I have repeated “build out your evaluator” up to here. But in fairness, let me state the other side of the picture just as plainly. Building the evaluator you ought to build — in a form that won’t be gamed — is itself hard. This is not a small footnote; it is the central difficulty of evaluator design. Saying “implement an evaluator” is easy, but it is not a slogan you can satisfy cheaply.

At the root of the difficulty is the overfitting (Goodhart) I touched on in §2. When you optimize against an evaluator, the optimizer fits to the evaluator itself rather than to the task. There is always a gap between “scoring high on the evaluator” and “producing genuinely good output.” And this gap does not arise only when the evaluator is sloppy. Even an evaluator a human wrote by hand, one that looks deterministic, is often gameable. In fact, the audit that the evaluator-hardening research (Hardening Agent Benchmarks) cited in §3 took as its starting point reports that, of roughly 2,000 tasks, 16% could be gamed by slipping past the grader alone without solving the task. When the evaluator becomes the model itself (the LLM-as-judge configuration), the difficulty grows further. This kind of evaluator is known to agree well with human judgment on average, yet to miss many of the defects that actually become problems in production. “Agreement rate with humans” is high while “defect-detection rate” is low — these two are different things, and the one that bites in operation is the latter.

There are standard moves for suppressing overfitting: increase the evaluation data, strictly separate a holdout (validation data not used for scoring), keep multiple scoring dimensions, tighten the evaluation environment to close loopholes. These are all effective. As noted above, there is even a report that just slightly tightening the environment greatly lowered the exploitation rate. But these are all design decisions that must be baked in from the start, and they still cannot fully eliminate the difficulty.

What’s more, all of these standard moves implicitly assume that the data needed for scoring is available, and unbiased. That is because the evaluator itself is built from data. To define the criterion of what counts as correct and which outputs to score highly, you need materials like execution data from actually running the agent, and review data where humans judged which outputs were good. But if this data is biased, the evaluator learns a biased criterion, and the optimization proceeds “in a direction other than the one you expected,” in a structure that looks just like overfitting. Increasing the data alone is not enough — if it does not represent the production distribution, the evaluator measures something other than production. So evaluator design is, at the same time as writing the scoring machinery, also the work of deciding what to collect, and which of it to select as the basis for scoring — the collection of data, and the selection of the right data. Get this wrong, and however precise the scoring function looks on the surface, it points to a biased gradient from the start.

And even a basic preparation like a holdout cannot always be arranged. Take the self-improvement research this piece sets as its backdrop (SIA): of its three validation tasks, only the Chinese legal classification has a clear train/test split; the remaining two (GPU-kernel optimization and single-cell RNA denoising) have no train/test split at all. Because it optimizes against the very target used for evaluation while lacking data carved out for validation, from that score alone you cannot, in principle, separate how much is generalizing ability and where it becomes mere fitting to that target. Even the dazzling results at the frontier need to be read with this reservation — which illustrates just how hard evaluator design can be.

What I want to emphasize here is not pessimism but a neutral fact. If anything, the very vendors offering reinforcement fine-tuning commercially candidly acknowledge this difficulty themselves. Sloppy graders and sloppy data send the wrong signal and push the model in the wrong direction, OpenAI’s RFT guide warns; keep scoring deterministic so it reflects changes in policy, not noise, Microsoft’s best practices teach. The product docs converge on the same point: the hardest part is designing the evaluator. That is exactly why the evaluator is not a “good enough if you have it” component, but one to place at the center of design and keep building out through operation.

§5 — So, build out the evaluator at the design stage (conclusion)

A summary of the story so far.

Prompts and scaffolding will all eventually become targets of automatic generation and automatic rewriting. Weight tuning is already automated. So when the wave arrives, what remains in a company’s hands? What the automated loop rewrites is weights and scaffolding — not the inputs given from outside it. What you make the agent do — the scope and agent definition that delimit its job — the company must keep providing; and how you score the quality of that work — the evaluator — is likewise an input controlled by the company that the loop does not touch. Other components remain outside the loop too, and the evaluator is one of them. Even so, the evaluator is one of the few durable assets that remain in a company’s hands even as models are swapped, scaffolding is rewritten, and weights are updated. But “remain” here does not mean you can reuse the same scoring engine as-is. When the task or product changes, the evaluator is rebuilt. What persists is the operational process by which the criteria that define “what counts as good,” and the data that grounds them, accumulate on the company’s side and become the starting point of the next loop.

Moreover, as we saw in §4, that evaluator needs an overfitting-resistant design — securing a holdout, multiple scoring dimensions, hard-to-game criteria — woven in from the start, and this cannot be bolted on later. Precisely because it is hard, it cannot be deferred. Put a self-improvement loop on top of a sloppy evaluator and the collapse we saw in §2 simply gets automated and accelerated, at algorithmic rather than human speed. To make matters worse, when the two optimizers — scaffolding search and weight updating — share the same evaluator, you become prone to falling into a deceptive optimum: one that looks strong on training-time metrics but breaks with a slight change of environment. If the evaluator is brittle, self-improvement faithfully amplifies that brittleness.

Therefore, the evaluator is not something you “refine after deployment.” It is a part to place at the starting point of harness design — not finished on day one, but with an “evaluable design” set from the beginning and improved continuously through operations. Whether you can run agents in production, and whether you can ride the coming self-improvement wave, is decided not by the best model but by whether you have the best evaluator — more precisely, a mechanism that keeps growing the evaluator.

Afterword — about the next article

That said, building out the evaluator is not the end of the story. And what must not be overlooked is this: the collapse I have kept warning about with the words “weak” and “sloppy” does not vanish even if you have a machine generate the evaluator. An automatically generated evaluator, if its design is sloppy, makes operation collapse just the same. Just because a machine spits out the scoring criterion is no guarantee that the criterion correctly captures good and bad in production. If the generation mechanism itself is sloppy, it can instead become a device for mass-producing sloppy criteria at high speed. What matters is a single point: how well the generated criterion is built out, and how solid the data supporting it is.

In fact, I am currently working on exactly this problem — an implementation that automatically generates evaluators for general tasks, and the design theory behind it. The key, I believe, is how to ground the generated scoring criterion to a signal outside the optimization loop, so as to keep the system from conveniently redefining “success” for itself. Once I can consolidate this implementation into a coherent theory, I plan to write it up as the next article.

References & related resources

Verifiability — how easy something is to verify decides how easy it is to automate

  • Jason Wei, Asymmetry of verification and verifier’s law (2025) — jasonwei.net

The reality that enterprise AI doesn’t reach production (§1)

  • A. Challapally, C. Pease, R. Raskar, P. Chari, The GenAI Divide: State of AI in Business 2025, MIT NANDA (July 2025) — report PDF

Reward hacking / specification gaming — official research from the major labs (§2, §4)

  • OpenAI, Detecting misbehavior in frontier reasoning models (2025) — openai.com
  • Anthropic & Redwood Research, Natural Emergent Misalignment from Reward Hacking in Production RL (2025) — arXiv:2511.18397
  • Google DeepMind (Krakovna et al.), Specification gaming: the flip side of AI ingenuity (2020) — deepmind.google

Gaming is a “measured” phenomenon / evaluators can be re-hardened (§2, §3, §4)

  • Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (2026) — arXiv:2605.02964
  • Hardening Agent Benchmarks with Adversarial Hacker–Fixer Loops (2026) (a setup that verifies and repairs the evaluator itself with a hacker–fixer loop) — arXiv:2606.08960

Commercialization of reinforcement fine-tuning (RFT / RLVR) — official documentation (§3, §4)

  • OpenAI, Reinforcement fine-tuning (includes guidance on grader design; note that this fine-tuning infrastructure is being wound down for new users) — developers.openai.com
  • AWS, Reinforcement fine-tuning on Amazon Bedrock (GPT-OSS / Qwen3 support, Lambda-grader approach) — aws.amazon.com
  • Microsoft, o4-mini Reinforcement Fine-tuning now GA on Azure AI Foundry / RFT best practices — Azure AI Foundry blog · RFT best practices

Harness + weight self-improvement (§3)