Scoped Meta-Harness: Design Theory for Self-Improvement Loops in Enterprise AI Agents

Meta-harness — the mechanism by which an AI agent observes its own harness and proposes modifications to it — is increasingly being implemented in the coding-agent and personal-AI domains, and its design patterns are beginning to take shape. Preprint lineages such as the Meta-Harness paper by Stanford / KRAFTON / MIT (Lee et al., 2026), HyperAgents from UBC / Meta / Edinburgh (Zhang, J. et al., 2026), and Agentic Harness Engineering (AHE; Lin, J. et al., 2026) from Fudan / Peking University are progressing in parallel, alongside open-source implementations such as auto-harness, SICA, and HALO. These approaches function well in domains where the evaluation function is clear and the modification target is closed within a single process. What I want to address here is the design question of what changes when that theory moves into the operational environment of an enterprise AI agent.

My core argument is this: the skeleton of meta-harness — a loop in which an agent observes its own harness and proposes improvements — can be maintained even when the unit of operation shifts from an individual user to an enterprise. However, several design assumptions flip. What flips, and how to structurally accommodate those changes, is what I want to work through here.

In a previous article I addressed the implementation of meta-harness in personal AI agents; in this one I focus on the design theory of meta-harness adoption in enterprises themselves.

Scope

The term "meta-harness" is used in different ways between research and operational contexts. Let me define it explicitly.

A harness is the set of surrounding structures that shape how an AI model turns input into output — prompts, tool definitions, reference data, workflows, meta-files, and guardrails. Even with the same model, a different harness produces a different agent.

A meta-harness is a mechanism through which the AI agent itself observes and evaluates its surrounding harness and returns modification proposals. The AI does not necessarily rewrite the harness directly. I will treat as the baseline for enterprise operation a design in which observation and proposal are handled by the AI, and changes are applied only after human approval.

My focus is the self-improvement of the harness layer surrounding the AI agent (prompts, rulesets, classification schemes, skill definitions, evaluation logic).

One important note: operations in which the AI generates code modification proposals and reflects them in production after human approval — the PR workflow exemplified by Claude Code and Codex CLI — are already in production use, and I am not arguing against their value. What I am placing outside scope as not yet operationally viable is the scenario — exemplified by the outer loop of the Meta-Harness paper — in which the AI continuously and autonomously modifies its own code or harness, applied to the enterprise’s core business systems themselves. This lies outside the operational range that current hallucination-mitigation techniques can support. It is less a constraint than a scoping decision about what to protect at the present time. I will come back to the rationale later.

The Engineer's Responsibility, the Meta-Harness's Responsibility

Before diving in, let me lay out the most important distinction in any discussion of enterprise meta-harness. Defining the system requirements is the engineer’s responsibility, and improving within those defined requirements is the meta-harness’s responsibility.

This is not a matter of two parties performing separate jobs in parallel. There is a single continuous design problem — "how to operate enterprise AI agents in a healthy way" — and within it I am distinguishing where the responsibility a human must take up ends, and where the responsibility that can be delegated to automation begins. Once the boundary is clear, the relationship between the two follows naturally.

This distinction is a consequence of carrying into the enterprise context the principle I articulated in my previous article on personal AI agent implementation: "The harness defines the boundaries. The agent is autonomous within them."

The engineer’s responsibility (whether the role is filled by in-house engineers, external contracted engineers, or Forward Deployed Engineer-type roles — organizations use various names, but I will refer to them collectively as "engineers") includes the following:

• Defining the AI agent’s assigned tasks, available tools, reference data, and output destinations — that is, the agent’s job description itself.
• Aggregating requests and operational feedback from multiple departments and translating them into system requirements.
• Designing the boundary between what falls within the meta-harness’s improvement scope and what does not.
• Designing the evaluation criteria, deployment conditions, and approval-authority hierarchy for improvement proposals.

The meta-harness’s responsibility is an extremely limited area confined to the boundaries the engineer has declared in advance:

• Observing structured signals (tool execution logs, quantitative metrics, A/B test results, etc.).
• Generating improvement proposals against pre-defined improvement targets (prompts, skill definitions, classification schemes, etc.) based on observed results.
• Sending improvement proposals into the approval pipeline.

By fixing this responsibility boundary up front, we can give a unified answer to the recurring questions that appear later — "who keeps requirements from sprawling?", "who decides what features to implement?" These belong to the engineer’s responsibility and should not be delegated to the meta-harness. The meta-harness is best designed as a mechanism that functions inside the boundary the engineer has set, and it should not be granted the authority to move that boundary itself.

With that boundary in place, we can now make the design requirements more concrete.

Design Requirements for an Enterprise Meta-Harness

An enterprise AI agent’s meta-harness must satisfy three requirements.

1. Bounded — Close the Improvement Scope in Advance

The agent is assigned a clear job description, and the improvement target is limited to precision improvements within that job description (accuracy, failure rate, response time, misjudgment rate). Preventing scope creep is itself a requirement.

This requirement shapes the entire direction of the design. A bounded improvement loop cannot operate unless signal sources, modification targets, and evaluation criteria are all defined in advance. The operational stance of "start operating first and adjust as we observe the situation" does not hold in an enterprise context. Declaring every boundary in advance has a design cost — one that has to be paid up front, as the threshold for beginning operations.

And as discussed in the previous section, it is the engineer who pays this design cost.

2. Signal Sources Are Limited to Observable, Structured Data

The signals that can be placed directly on the meta-harness’s automated loop are limited to observable and pre-structured signals. Specifically:

• Tool execution logs (success/failure, error codes, execution time, call patterns).
• Benchmark scores (accuracy, F1, latency, and other pre-defined quantitative metrics).
• A/B test or shadow deployment results (quantitative comparisons between versions).
• Structured user actions (good/bad button clicks, A/B preference clicks, operational logs of output adoption/rejection/editing).
• Explicit rule-violation signals (threshold breaches of internal policy or SLA, automated detection from compliance checkers).

These can be read by the AI directly to generate improvement proposals. On the other hand, problems such as how to aggregate free-text feedback or requests across multiple organizational departments are handled outside the meta-harness’s automated loop. Why this is so is discussed later in "Conditions Under Which Meta-Harness Functions."

3. The Modifiable Region Is Extremely Narrow

From the standpoint of mutability, the data surrounding an AI agent splits into two types.

Type A (append-only, tamper-resistant data): History, logs, judgment records, contracts, transaction data, meeting minutes, knowledge bases, tool execution traces, approval/rejection decision logs. The majority of audit- and compliance-relevant data — which must be preserved in a state that allows after-the-fact verification of correctness — falls here.

Type B (data that can serve as the starting point for AI modification proposals): Agent behavior rules, skill definitions, prompt templates, workflow definitions, error handling procedures, classification schemes. The meta-file group that defines the agent’s behavior.

In enterprise operations, the data overwhelmingly skews toward Type A. This is because most enterprise data must be handled in a tamper-resistant manner due to audit and operational requirements. As a result, the meta-harness’s Type B is extremely limited — confined to the restricted list above. That the improvement region is narrow means that what is designed within that narrow region, and how, must be designed with great precision.

This Type A / Type B distinction is a design concept I will come back to repeatedly. All later discussions (scope design, approval pipeline, range of modification proposals) stand on this binary classification.

Scoped Meta-Harness: A Two-Layer Scope Design

When the design requirements above are translated into implementation, the meta-harness for an enterprise AI agent is composed of a two-layer scope design.

Layer 1: The agent’s own scope. What task does this agent handle? Which tools can it use (read-only or write-capable)? Which data does it reference? Where does its output go? The definition of the agent’s "job description."

Layer 2: The improvement loop’s scope. What does the meta-harness use as signal sources? What can it propose modifications to (prompts, rulesets, classification schemes, skill definitions, etc.)? What must it absolutely not touch (system architecture, business code itself, HR information, etc.)? How are benchmarks defined?

These two layers must be aligned. This is the most important principle of Scoped Meta-Harness design.

If Layer 1 defines an "API error analysis agent" but the Layer 2 improvement loop can observe the entire enterprise system, the improvement proposals will inevitably cross boundaries. Proposals like "the root cause of this error lies in the design of the authentication flow. We recommend migration from Auth0 to Cognito" will start appearing in the improvement loop of an error log classification agent. Even when the technical reasoning is sound, the operational probability that such a proposal is accepted is essentially zero — and tokens are wasted in the meantime.

Scope Declarations Are Fixed at Design Time

Layer 1’s scope is explicitly declared before operations begin and fixed as a structured configuration file (schema-bearing definition file in TOML, YAML, etc.). Assigned tasks, permitted tools, reference data, output destinations, constraints on each I/O — these are written out completely at design time. The practice of adding new tools or new responsibilities to the agent later "because it would be useful" structurally erodes scope.

This is not a matter of aesthetics — it is a matter of performance. A line of research centered on the Berkeley Function Calling Leaderboard (Patil et al., 2025, ICML) has repeatedly shown that as the number of tools increases, the LLM’s correct tool selection accuracy drops sharply. Gan & Sun (2025)’s RAG-MCP independently demonstrated a monotonic decline in tool selection accuracy as MCP pool size grows, making explicit that the issue is not merely token consumption but selection accuracy itself. Specifically, in that paper’s MCP stress test, the baseline method (Blank Conditioning) tool selection accuracy fell to 13.62% as the number of candidate tools grew, and improved more than threefold to 43.13% once retrieval-based tool selection was introduced. In the same paper’s evaluation beyond position 100, selection failures came to exceed successes — quantitatively confirming that scaling tool count breaks not just token budgets but selection accuracy itself. Zhang, K. et al. (2026) formalized as a "tool-use tax" the phenomenon that, in the presence of semantically similar distractor tools, tool addition itself degrades accuracy — articulating the mechanism by which tool expansion interferes with other tool selections in unforeseen contexts. Anthropic’s internal measurements likewise show that 58 tool definitions consume roughly 55,000 tokens, with accuracy declining as the number of choices grows. Independent verification via LiveMCPBench (arXiv:2508.01780), which evaluated 12 frontier LLMs on real-environment MCP tasks, reported that while Claude-Sonnet-4 reached a 78.95% task success rate, the majority of models stayed within the 30–50% range, and that approximately half of all failures originated in the tool retrieval stage.

These findings impose strict requirements on Layer 1 design. The design decisions of "let’s keep some extra tools available just in case" and "let’s broaden the job description for future needs" reliably degrade the accuracy of the choices that will eventually be used, by way of choices that aren’t being used now. Breadth of scope is itself a cost.

And Layer 2 (the improvement loop) follows this Layer 1 declaration strictly. The improvement agent first reads the declaration file of the agent it is improving, and proposes improvements only within the assigned tasks, permitted tools, and reference data recorded there. Out-of-scope errors, out-of-scope improvement targets, additions of out-of-scope tools — these are structurally closed off so they can’t be proposed, via both the improvement agent’s prompt and access control.

Case Study: API Error Classification Agent

Let me make this concrete. Consider designing an agent that continuously classifies error logs from a production API and proposes root-cause hypotheses.

Layer 1 design (fixed at design time):

• Task: Classification of production API error logs into known patterns, and proposing root-cause hypotheses.
• Tools used: Log search, stack trace analysis, change history reference (all read-only).
• Reference data: Most recent 30 days of error logs, known incident DB, deployment history.
• Output destination: On-call engineers’ Slack channel.

Layer 2 design:

• Signal sources: On-call engineers’ adoption/rejection of classification results, validation adoption rate of root-cause hypotheses (both are structured quantitative signals).
• Modifiable region: Error pattern classification scheme, root-cause ruleset, input prompt.
• Non-modifiable region: API code itself, infrastructure configuration, Slack notification rules themselves, new tool additions, new data source additions.
• Benchmarks: Classification accuracy, hypothesis adoption rate, trend in misclassification rate.

What this design essentially does is intentionally align "the data range the agent sees" with "the range the meta-harness improves," closing off every other modification path structurally.

If the same agent were run without scoping, the AI would output proposals like "This error pattern occurs frequently because the authentication architecture has a design flaw. We recommend moving away from Auth0" or "The error handler design should be refactored entirely" — proposals that will be rejected immediately from an operational standpoint. This is not because the AI’s reasoning is wrong. The structural cause is that the conditions under which such proposals arise have been built into the harness design itself.

This design problem is not limited to diagnostic-type agents. The same structural failure occurs in the same place across every type — extraction agents that pull decision-maker, budget, and timeline information from sales call transcripts into CRMs; evaluation agents that summarize contracts and assess clause risk; generation agents that produce FAQ candidates from internal knowledge bases. Without scoping Layer 2, the AI returns boundary-crossing proposals like "The CRM’s field structure is itself inappropriate and should be redesigned." Although the form of the task differs, the structural failure occurs at the same location. Scoped Meta-Harness functions as a general design principle independent of agent type.

Distinguishing Improvement from Spec Change

Here’s a distinction that’s decisive when operating a Scoped Meta-Harness. Improvement and spec change are different things, and must be treated as such.

• Improvement: Work inside the Layer 1 scope declared at design time that raises accuracy, efficiency, hit rate, or false judgment rate. Modifying prompt templates, refining skill descriptions, adjusting error handling logic, subdividing classification categories — these are improvements.
• Spec change: Work that alters the Layer 1 scope itself. Adding new tools, connecting to new data sources, expanding the agent’s assigned tasks, adding output destinations — these are spec changes.

The meta-harness handles only improvements. Spec changes must not occur inside the meta-harness’s automated loop.

Why? Because if spec changes are permitted, the improvement loop becomes able to expand its scope without limit. Once a proposal like "resolving this error requires the ability to call a new API" passes at some point, that API becomes a new signal source, new error patterns become observable, and resolving them requires further new capabilities — a chain that does not stop. The improvement loop, on the surface, continues "improving," but the substance of the agent diverges entirely from the original design. The tool-count-driven accuracy decline mentioned earlier compounds with each expansion. Self-improvement that includes spec change diverges the implementation over the long term and destroys the reliability of the agent itself.

That said, the distinction cannot always be drawn cleanly in practice. Let me list three typical cases where the boundary blurs.

Classification category subdivision. Consider a proposal to split "authentication error" in the error classification scheme into "OAuth token expiry," "session timeout," and "permission denied." If the new distinction can be established within the existing signal sources (error logs) and existing reference data (error pattern DB), it is an improvement. But if establishing the new distinction requires newly importing authentication flow trace data, then a new signal source is being added — which qualifies as a spec change. Even when the surface appears to be "just finer classification," if the implementation prerequisites have changed, it is a spec change.

Replacing or adding few-shot examples. Raising the quality of examples included in the prompt is improvement, but if the examples include a new data category (a class the agent did not previously handle) and the agent comes to interpret it as part of its responsibility, then that is implicit job-description expansion — a spec change. Few-shot examples look like mere examples, but they function in part as declarations of the agent’s judgment range, and require great care.

Tool description edits. Explicitly listed as a modifiable region in the AHE paper, and one of the most actively debated modification paths in meta-harness research. Clarifying how to call the same tool is an improvement, but when the description change broadens the interpretation of "when to call it" — for example, rewriting "use for read-only queries" into "use for general data verification" — it is substantively a behavior-rule change, sitting at the boundary of spec change.

What emerges from these edge cases is that the decisive criterion is not "is the textual change small?" but "does the change expand the signal sources, reference data, or output range the agent touches?" The improvement agent’s self-question is more precisely stated as: "Can this change be implemented without moving any of the I/O boundaries declared in Layer 1?" Proposals that fail this condition are escalated outside the improvement loop.

This self-question is articulated at the improvement agent’s prompt level, with I/O boundary differences set up so they can be structurally evaluated. If a proposal does not meet this boundary condition, the improvement agent does not propose, and instead merely flags it as "this is a scope-expansion requirement and requires human spec judgment."

The spec change decision is made by humans — engineers and business owners — outside the improvement loop. Following the role division I laid out at the start, this is the natural consequence. Whether a new capability is genuinely needed, how its addition affects other agents and operations, how existing skill selection accuracy changes due to increased tool count, what the compliance impact is — these are management and operational judgments that exceed the improvement loop’s scope and should not be decided autonomously by AI.

Conditions Under Which Meta-Harness Functions

I’ve argued so far that meta-harness is a mechanism that functions "when observable structured signals exist and clear evaluation criteria are present." Conversely, where those conditions are absent, attempting to force meta-harness operation will diverge.

This precondition is also empirically backed by research on coding-agent self-improvement. Robeyns et al. (2025)’s SICA raised agent self-improvement from 17% to 53% in SWE-Bench Verified — a domain where ground truth is established. Agrawal et al. (2025)’s GEPA shows that structured text feedback (execution logs, traces, error messages, etc.) can optimize prompts more efficiently than scalar-reward-based reinforcement learning. Both results emerge under the precondition that the evaluation function is verifiable and signals are structured, and align with the three conditions I’ve organized as necessary in enterprise contexts.

To make these conditions a bit more concrete, an automated loop succeeds when the following three are aligned:

• Signals are structured: Success/failure can be captured as a binary, quantitative metrics are pre-defined, preference data can be captured discretely, and so on.
• Signal source roles are closed to a single context: The signal source is aligned with the agent’s job description, with no contamination from other contexts.
• Evaluation criteria are explicitly documented in advance: What counts as "improvement" is determined before release.

Domains where these three conditions are not aligned — free-text requests from multiple departments, cross-organizational priority adjustment, defining business requirements themselves — are not the meta-harness’s scope to begin with. These are handled outside the meta-harness as the engineer’s responsibility.

This needs to be stated plainly during enterprise adoption. Operations that claim to have "introduced enterprise meta-harness" but in substance rely on engineers interpreting and structuring free-text feedback to identify improvement targets are not meta-harnesses in the strict sense I’ve defined here. They can be included as meta-harnesses in a broader sense, but they clearly differ in level of autonomy from the kind of evaluation-function-driven outer loop autonomously circulating in Lee et al.’s paper.

In short, an enterprise AI agent’s meta-harness is a mechanism whose automation only succeeds in domains where evaluation criteria and observation signals are clearly defined in advance. Next, let me walk through specific implementation patterns that meet these preconditions.

Meta-Harness Implementation Patterns

Here are four specific implementation patterns that are either being run in enterprise environments today or can be. All satisfy the three conditions of the previous section: structured signals, contextual unity, and explicitly documented evaluation criteria.

Pattern 1: Self-Improvement on Tool Execution Errors

The most fundamental implementation pattern. It uses the execution logs of the tools (APIs, scripts, CLIs) the agent uses as its signal source, and improves the definitions and implementations of tools and skills with high error rates.

Signal sources: tool execution logs (success/failure, error codes, argument patterns); frequency and context of recurring identical errors.

Improvement agent behavior:

• On a daily/weekly cadence, aggregate execution logs and extract tools that exceed an error-rate threshold.
• Classify error patterns and form root-cause hypotheses.
• Generate improvement proposals according to the hypotheses: argument usage is incorrect → revise the skill definition file (SKILL.md); unanticipated edge case → revise the script itself or error handling logic; failure to track an API specification change → revise the relevant API call section.

Deployment flow:

• Deploy the improvement candidate to the development environment.
• Confirm there is no regression via replay testing against historical logs.
• Measure production impact via shadow deployment or A/B testing.
• If improvement is confirmed, the engineer approves → client approval → production deployment.

This pattern does not change the tool set itself permitted in Layer 1 (that would be spec-change territory). It stays within improvements to the call patterns of existing tools and their surrounding logic.

Pattern 2: Structured User Feedback for Chatbots

This is the domain where the three conditions of the previous section align most cleanly — internal FAQ bots, support chatbots, and similar areas that receive feedback directly from end users.

Signal sources: good/bad button clicks on each response; preference clicks on new vs. old versions A and B for the same question; escalation rate (proportion of conversations handed off to a human operator after the bot’s response); average response time, compliance violation detection rate.

Improvement agent behavior:

• Extract clusters of low-rated responses and identify common patterns.
• Generate improvement proposals for prompts, response templates, and reference knowledge retrieval logic.
• Generate candidate versions in the development environment.

Deployment flow:

• Run the new candidate version alongside the old version in shadow deployment.
• After accumulating a certain amount of traffic, aggregate the metrics.
• Determine whether all of multiple pre-defined metric thresholds are met (e.g., accuracy at least +3% over old version, escalation rate not worsened, response time within tolerance, compliance violation rate not increased).
• Only candidates that meet the criteria are promoted to the next version.

The key in this pattern is that all three conditions are satisfied: signals are structured (click operations are binary or discrete choice), signal source roles are unified (only operations as a user, with no contamination from opinions about business requirements), and evaluation criteria are pre-defined (release criteria are metric thresholds).

Pattern 3: Skill Updates for RAG Agents

RAG (Retrieval Augmented Generation) agents — whose reference data accumulates and changes over time — are one of the domains where meta-harness adds the most value. As the content and structure of the data shifts, the skills that retrieve and process it require updates as well.

Signal sources: accuracy of RAG retrieval results and final response (against an evaluation set); retrieval hit rate, retrieval result utilization rate (proportion of hits that went unused); structured "I couldn’t find what I was looking for"-type feedback from users; statistics on newly added data categories.

Improvement agent behavior:

• Identify regions where retrieval accuracy metrics have declined.
• Classify the cause: the query generation prompt has not kept up with new data categories → revise the prompt; the retrieval strategy in the skill definition has become outdated → revise SKILL.md; the chunking or reranking logic is inappropriate → revise the processing script.
• Generate improvement proposals.

Deployment flow: same as Pattern 2 — staged release combining automated evaluation against an evaluation set with shadow deployment.

This pattern is particularly valuable in domains where continuous fine-tuning is required as data grows and changes, and human tracking is not realistic.

Pattern 4: Accuracy Improvements for Classification/Extraction Tasks

Tasks where ground truth is obtainable (if not perfectly so) — field extraction from invoices, support ticket classification, contract clause categorization.

Signal sources: human reviewer correction logs (diffs between the agent’s extraction/classification result and the final confirmed value); per-category accuracy, misclassification patterns.

Improvement agent behavior:

• Aggregate correction logs and identify classification categories or extraction fields where error patterns cluster.
• Propose subdivision/redefinition of the classification scheme, prompt revisions, replacement of few-shot examples, etc.

Deployment flow: backtest against historical data → engineer approval → production deployment, or new-vs-old comparison in shadow mode → production deployment.

Common across these four patterns is that observable structured signals and clear evaluation criteria exist, and that the improvement target is closed within Layer 1’s scope. Conversely, where these conditions are not aligned, the meta-harness’s automated loop will not function. The practical answer to "how do we incorporate meta-harness into enterprise systems?" is to carefully select the domains where these conditions hold, and for engineers to prepare signal sources and evaluation criteria in advance so that they do.

Conditions for Applicability

That said, even in organizations that can clear the conditions for the four patterns above, building a Scoped Meta-Harness does not automatically justify its cost. Signal-source infrastructure, evaluation sets, shadow deployment infrastructure, approval pipelines, the modification-proposal capture layer — the initial labor to prepare these typically runs to several person-months per agent. After launch, evaluation set maintenance, signal source health monitoring, and the approval workflow for modification proposals remain as ongoing costs. Scale that justifies these costs is a precondition.

The economically viable cases typically fall into one of the following:

• A single agent receives a large call volume, so even small accuracy improvements compound (internal FAQ bots, production API error handling, large invoice processing pipelines, etc.).
• Multiple agents of the same kind are in operation, allowing the improvement infrastructure to be reused horizontally.
• Improvement latency directly causes compliance or SLA breaches, and human tracking is not realistic.

In organizations that don’t fit any of these — few agents, limited call frequency, improvement targets that are low-frequency edge cases — the ROI often favors regular human tuning rather than forcing a meta-harness build. The chain of reasoning that says "we introduced AI agents, so we should also build a meta-harness" tends to become over-investment when scale is absent.

There is also the matter of adoption order. The first system to build is the domain with the lowest signal-source structuring cost among the four patterns above. A good/bad button on a chatbot is a single UI addition, but capturing correction logs from contract clause evaluation as structured signals requires rebuilding the human reviewer’s workflow itself. The realistic adoption path is starting with the former, accumulating operational know-how and approval workflow norms, and then expanding to the latter. The first system establishes the governance pattern, and the second and onward reuse it — this is where the economy of horizontal expansion arises.

What Silence Means

Agent runaway does not arise from a lack of feedback. It is suppressed by a separate mechanism — the approval pipeline, modification target limitation, structural restriction of tool access. With these in place, the AI will not modify the harness on its own and collapse, even if no one returns any feedback.

But silence in enterprise operation carries another, deeper meaning. Data with improvement potential accumulates without being converted into improvements. Even when there are moments where the agent’s output feels off, that signal remains nowhere unless it is recorded as a structured signal. The agent continues to operate, but the gap between the standard the operating side imagines and the actual output quality widens.

Structuring silence is also, again, the engineer’s responsibility — not the meta-harness’s. Concretely, this involves: setting up infrastructure that automatically records adoption/rejection/edit operations on output as structured signals; running periodic review cycles with sampled evaluation and quantifying the results; surfacing agents with extremely few explicit signals as a "missed improvement opportunity risk" and proactively inserting reviews. All of this is the signal-source preparation problem that comes before the meta-harness runs.

The role division does not change here either. Humans create the signals; the AI generates the improvement proposals from them.

Approval Pipeline: Suggestion by Default, Automated Deployment Under Conditions

The final component of Scoped Meta-Harness is the flow by which modification proposals are applied to production. The default form in enterprise operation is:

• The improvement agent reads the structured signal source.
• It reads Layer 1’s scope.
• It suggests improvements that fit within scope.
• The engineer evaluates (is it in scope, does it contain no spec change, is it technically sound).
• As needed, it is escalated to the client’s approval process.
• Only approved changes are applied to the harness.

This default form is required for two reasons — technical and institutional. Technically, since the AI generating the meta-harness suggestions can itself hallucinate, unsupervised self-modification is structurally untenable. Institutionally, changes to a production agent’s behavior rules or skill definitions must be logged as the judgment of an operations-responsible party, not as an AI’s autonomous judgment.

In implementation, modification proposals from the meta-harness are always written to a capture layer and kept in a file separate from the harness file itself. Merging into the main file is performed by a separate process (the reconciler), targeting only approved proposals. Separating capture and merge into different files and different timings preserves the boundary between AI proposal and human approval as a directory structure.

The approval hierarchy is designed to follow the enterprise’s organizational structure. Trivial wording fixes to a skill definition may be approvable by an operations lead, but structural changes to an agent’s behavior rules require business-manager approval, and changes to compliance-related judgment logic require legal-side approval. The design judgment of mapping the approval hierarchy onto the organizational chart is required here.

Exception: Automated Application Under Quantitative Criteria

That said, not every modification needs to go through human approval. Where clear criteria can be defined as rules in advance, automated application is also possible.

For example, recall the chatbot pattern shown earlier. A mechanism in which a candidate new version is run alongside the old version in shadow deployment, and is automatically released when it meets all of several pre-defined metric thresholds (accuracy at least N% over the old version, escalation rate not worsened, response time within tolerance, compliance violation rate not increased, etc.) is technically and institutionally viable.

The preconditions are strict:

• All evaluation metrics are quantitatively pre-defined.
• Each metric’s threshold is pre-approved.
• The amount of traffic subject to threshold judgment is pre-decided (an amount that allows statistically significant evaluation).
• Rollback conditions and procedures are pre-prepared.

In implementing these conditions, operators face several non-trivial technical judgments. The traffic volume needed for statistically significant evaluation must be back-calculated from the effect size of the improvement target and the acceptable false detection rate, and for usage volumes typical of an internal FAQ bot, detecting a significant difference can require days to weeks. This diverges substantially from the intuition that "shadow deployment will return results in hours." Traffic splitting is also often inadequate as a simple random split — sampling that corrects for user attribute and time-of-day skew is required, and a design is needed that avoids erroneous judgments arising from behavioral differences between peak and off-peak hours. Rollback procedure presumes a design in which the new version does not absorb old-version data dependencies (cache, accumulated classification results, response history accumulated on the user side, etc.); failing to satisfy this leads to situations where "rollback was supposed to be possible, but the data has advanced and cannot be reverted." These are infrastructure requirements to be prepared before automated application becomes viable, and they fall within the engineer’s responsibility before the meta-harness runs.

A deeper problem is the risk that the evaluation metrics themselves get hacked by the improvement loop. As Goodhart’s Law states, the moment a metric becomes the target of optimization, it ceases to function as a proxy for the quality it was originally intended to represent. There are numerous reports from the reinforcement-learning literature of cases where chatbot accuracy rises while the substance becomes "mass-producing errors users don’t notice." The same mechanism operates in the improvement loop. There is no complete defense — requiring conjunctions across multiple metrics, periodically inserting human sampling audits, periodically reviewing the metric definitions themselves. The combination of these is the realistic countermeasure to buy time until collapse. If automated application judgment is built on a single metric’s threshold, improvement proposals targeting that single metric will continue to pass, and at some point the system will fail in a visible way.

And when these conditions are not in place, automated application should not be permitted.

More importantly, automated application should be limited to non-business-critical domains. Response improvements to an internal FAQ bot have limited impact even if misapplied and are easy to roll back, but logic for contract risk evaluation or credit judgment can produce business-critical impacts from incorrect changes. In the latter, human checking should remain even when quantitative criteria are met.

Who draws the line between automated application and human approval? This too is the responsibility of engineers — and business owners and the compliance department. Defining and documenting "which type of change to which agent is automatically applicable under which criteria" in advance is a precondition for automated deployment operations.

Why We Don't Address Business Systems Themselves

At the outset, I limited the scope to "the harness layer surrounding the AI agent." Let me now restate why business systems themselves are out of scope.

First, as the modification target broadens, the evaluation cost of modification proposals rises exponentially. Verifying a "classification scheme error" inferred from error logs takes minutes, but verifying "the database design of the business system is flawed" takes days. From the standpoint of token and operational cost, the practically operable scope must be kept narrow.

Second, business systems themselves have too many stakeholders outside the AI. Database design, API interfaces, authentication flow — these exist independently of AI agent use and are change-managed independently. If AI modification proposals extend to the business system itself, the organization receiving those proposals is not designed assuming AI, and operations break down.

Third, the impact of hallucinations becomes intolerable. An erroneous rewrite of a skill definition file has impact limited to that agent’s range. By contrast, when a change to the business system is hallucination-driven and incorrect, the impact range can extend across the entire business. Agent-specific hallucinations have been systematically classified and organized by Lin, X. et al. (2025), and OWASP has explicitly documented LLM06: Excessive Agency as an independent risk category in Top 10 for LLM Applications 2025 — both reflecting the industry-shared recognition that granting autonomy to agents and controlling hallucinations cannot be handled independently. The operation in which a coding agent generates a PR and a human reviews it addresses this asymmetry through a human gate, but circulating this as an autonomous loop against the enterprise’s business systems themselves currently exceeds operable risk levels. The asymmetry of risk is precisely the basis for the meta-harness’s scope setting.

Who Designs the Boundaries

All the design judgments discussed here — redrawing the Type A/B boundary, aligning Layer 1/2 scopes, distinguishing improvement from spec change, identifying the three conditions under which meta-harness functions, selecting implementation patterns, drawing the line between automated application and human approval, setting the scope of modification targets — are not purely technical judgments. They are judgments that only become possible by reference to business context, organizational structure, regulatory requirements, and operational maturity all together.

And as I’ve repeatedly stated, these judgments belong to the engineer’s responsibility, not the meta-harness’s. The meta-harness is best designed as a mechanism that functions inside boundaries, and the design of those boundaries themselves remains a human role. This is the design stance running through everything I’ve argued here.

There are currently almost no humans within organizations who can take on this design judgment. The client side does not know the internal design of AI agents, and the AI vendor side does not know the customer’s domain context. What is required is talent that understands both and can make system-boundary design judgments.

In the industry, one such role gaining attention is the Forward Deployed Engineer (FDE). The reason major AI labs are simultaneously accelerating FDE hiring in 2026 is, as far as can be observed, that the shortage of personnel able to make the kinds of design judgments discussed here has become the substantive bottleneck for enterprise AI adoption.

That said, the role title "FDE" itself remains fluid at present, and the center of gravity of the job differs by organization and context. My main point is that a role to take on the meta-harness design judgment is necessary — not that the FDE will exclusively assume it. Whether the form is an in-house engineer, business owner, consultant, or a team combining them, what I’m arguing is that someone who understands both AI agent operations and the business system needs to take on this boundary design.

With that caveat, my prediction is that the meta-harness construction I’ve described will likely become one of the core duties of FDEs over the next several years. The reason is simple: the job function that deeply understands both operational business and AI systems aligns exactly with the requirements of this boundary design. Whether the role title settles in any particular way is separate; the design problem itself is certain to persist.

References: Key Literature Cited

Meta-Harness and Harness Self-Improvement

• Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., Finn, C. (2026). Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052. — Formalization of evaluation-function-driven outer loop for harness self-improvement. I discuss the constraints and exceptions when bringing that into enterprise contexts.
• Lin, J. et al. (2026). Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. arXiv:2604.25850. — A design that turns modifications into falsifiable contracts via three-layer observability (Component / Experience / Decision), preventing the harness self-improvement loop from collapsing into trial-and-error. Improves Terminal-Bench 2 pass@1 from 69.7% → 77.0% in 10 iterations and demonstrates that frozen harnesses are transferable to SWE-bench-verified.
• Zhang, J. et al. (2026). Hyperagents. arXiv:2603.19461. — Integrates the task agent and meta agent into a single editable program and demonstrates generalization beyond coding to paper review, robotics reward design, and Olympiad math grading by making the meta-level mechanism itself editable. Serves as the design counterpoint to my claim that the meta-level should not be editable in enterprise operations.
• Robeyns, M. et al. (2025). A Self-Improving Coding Agent (SICA). ICLR 2025 Workshop SSI-FM. arXiv:2504.15228. — Demonstrates the feasibility of a self-improvement loop in SWE-Bench Verified — a domain with established ground truth — raising agent self-improvement from 17% to 53% via integration of meta-agent and target agent. Independent empirical data for my claim that meta-harness only succeeds under verifiable evaluation functions.

Bounded Scope and Tool-Count Dependence

• Patil, S. G. et al. (2025). The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. ICML 2025 (PMLR 267:48371-48392). — Systematic evaluation of function-calling accuracy degradation as tool count grows. Central paper supporting the case for fixing Layer 1 scope at design time.
• Gan, T., Sun, Q. (2025). RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation. arXiv:2505.03275. — Independently verifies the monotonic decline in tool selection accuracy with growing MCP pool size. Demonstrates retrieval-based tool selection improves selection accuracy from 13.62% to 43.13%, making explicit that "the tool-count issue is not merely token consumption but selection accuracy itself." Further reports that beyond position 100, selection failures come to exceed successes.
• Zhang, K. et al. (2026). Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents. arXiv:2605.00136. — Formalizes via a Factorized Intervention Framework the "tool-use tax" by which tool addition itself degrades accuracy in the presence of semantically similar distractor tools. Theorizes the mechanism by which tools added "just in case" in Layer 1 interfere with other tool selections in unforeseen contexts.
• LiveMCPBench (arXiv:2508.01780). Can Agents Navigate an Ocean of MCP Tools? — Evaluates 12 frontier LLMs on real-environment MCP tasks. Reports that Claude-Sonnet-4 reaches 78.95% task success rate while the majority of models stay within 30–50%, and roughly half of failures originate in the tool retrieval stage. Independent corroboration that tool selection at scale is a bottleneck.
• Anthropic internal measurements — Reports that 58 tool definitions consume roughly 55,000 tokens and that selection accuracy declines as tool count grows (anthropic.com/engineering/advanced-tool-use). Also reports that introducing the Tool Search Tool with on-demand tool loading improves MCP evaluation accuracy from 49% → 74% on Opus 4 and 79.5% → 88.1% on Opus 4.5.

Structured Signals and Prompt Optimization

• Agrawal, L. A. et al. (2025). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457. — Demonstrates that reflective prompt optimization using structured text feedback (execution logs, traces, error messages, constraint violations, etc.) can outperform scalar-reward-based reinforcement learning in rollout efficiency. Directly supports the requirement that signal sources be limited to observable structured data, and the case that this is what makes automated optimization succeed.

Agent-Specific Hallucinations and Autonomy Risk

• Lin, X. et al. (2025). LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions. arXiv:2509.18970. — The first comprehensive survey systematically classifying agent-specific hallucinations. Breaks down the agent workflow by stage and taxonomizes the types of hallucinations that can occur at each. Literature support for one of my reasons for excluding business systems themselves — the asymmetry of hallucination impact.
• OWASP (2025). OWASP Top 10 for Large Language Model Applications 2025. — Documents "Excessive Agency" as the independent risk category LLM06. Notes that as LLMs perform API calls, data manipulation, and autonomous judgment, unsupervised automation amplifies risk exponentially. An industry-standard document that supports the claim that "circulating an autonomous loop against the enterprise’s business systems themselves currently exceeds operable risk levels."