Quality Harness Engineering: The Emerging Stack for Reliable AI Systems
Quality Harness Engineering: The Emerging Stack for Reliable AI Systems
The demo worked. The team gave it a round of applause. The model handled the messy real customer email, summarised it correctly, drafted the right reply. Three months later the same system is in production and you cannot tell anyone, with a straight face, whether it is getting better or worse.
That is the moment most AI programmes I see hit a wall. Outputs drift. Behaviour changes silently. Someone tweaks a prompt to fix one workflow and accidentally breaks two others nobody had tested. Senior engineers, the same ones who were enthusiastic at the demo, start quietly routing around the AI feature in their daily work. The system is technically running. Nobody trusts it. And the question the board is now asking - "did quality actually improve this quarter?" - has no honest answer, because there is no instrument in the org that can measure it.
This is not a prompt problem. It is an infrastructure problem. And we have been treating it as a craft problem for two years.
Prompt engineering is a real discipline that has reached its operational ceiling
I want to be precise about what I am claiming, because the field has spent enough energy on cheap dunks on prompt engineering already. Prompt engineering as a craft is real. There is real skill in shaping a prompt so a model handles edge cases gracefully. Operators who are good at it produce noticeably better behaviour from the same model than those who are not. That gap is worth investing in.
The claim is narrower, and harder to argue with once you have lived it: prompt engineering as the only discipline organisations apply to AI behaviour has hit a clear operational ceiling. The ceiling shows up in a specific place. It is the moment AI behaviour stops being one craftsperson's responsibility and starts being a system property the whole organisation depends on. At that point the question changes. It is no longer "can a skilled engineer make this prompt produce a good answer on this case?" It becomes "can the organisation guarantee acceptable behaviour, across thousands of cases, as the model updates, as the prompt evolves, as the team that wrote it rotates off the project?" Prompt craft does not answer that question. Prompt craft was never trying to answer that question.
What answers it is infrastructure. A discipline of infrastructure I want to name carefully, because the name matters and the field has not yet settled on one.
Quality Harness Engineering is a specific subdiscipline within a wider harness-engineering family, not a competing rebrand
The useful name for the discipline of building reliability infrastructure around AI behaviour is Quality Harness Engineering. The word quality in front of harness engineering is load-bearing, and the reason it is load-bearing is that "harness engineering" already exists in the field, and means something slightly different.
Two pieces, both published this year, established the broader frame. In February 2026, OpenAI's engineering team published Harness engineering: leveraging Codex in an agent-first world, a case study of building an internal product almost entirely with Codex coding agents - what changes when "a software engineering team's primary job is no longer to write code, but to design environments, specify intent, and build feedback loops" for agents. In April 2026, Birgitta Böckeler at Thoughtworks published Harness engineering for coding agent users on Martin Fowler's site, formalising the mental model of feedforward and feedback loops that human engineers wrap around a coding agent so the agent can work with less supervision.
Both pieces are good, and both use harness engineering to mean the scaffolding around a coding agent specifically: the loop a software engineer or an engineering team builds around an LLM writing code. That framing is the right one for the problem they are solving. It is not the framing I am pointing at here.
Quality Harness Engineering, as I am using the term, is the scaffolding around AI behaviour in a much wider sense. It is the operational reliability stack the organisation builds around any AI system whose outputs matter: a recruiting evaluator, an architecture-review assistant, a QA review system, an SEO brief generator, a sales workflow assistant, an article humaniser. Coding agents are one instance of this family; they are not the whole family. The Fowler and OpenAI framings are upstream cousins; this one is a sibling discipline. The relationship is not competition. The relationship is that harness engineering as a field is wider than coding agents, and the production-reliability subdiscipline of it deserves its own name because the failure modes are different.
In my AI Adoption Framework for software engineers, the L1–L4 maturity ladder already lists "context, compounding, and harness engineering disciplines" as the structural shape engineers are climbing into at the senior tiers. The thing I am adding now is the name, the four-layer architecture, and the operational claim that quality harness engineering is the specific subdiscipline the field is converging toward, and that it has a discoverable shape.
The four-layer pattern is the structural shape the reliability stack is converging toward
Across AI systems running in production in delivery orgs, and across the public engineering writing that has accumulated this year, the same four layers keep appearing. They appear in different vocabularies and different tooling choices, but the layers themselves are stable. I am going to name each layer with one current illustrative tool from the public ecosystem so the architecture is concrete, but the layers are the load-bearing claim, not the tools. The tools will change. The layers will not.
The four layers are:
- Behavioural specification - defining what the AI is supposed to do, as a reusable artefact. Currently illustrated by Anthropic's Agent Skills.
- Behavioural validation - measuring whether it actually does that, repeatably. Currently illustrated by Anthropic's evaluation tooling and the broader eval-driven-development practice.
- Automatic behavioural improvement - closing the loop so the specification itself improves from validation results, without a human rewriting prompts forever. Currently illustrated by Microsoft's SkillOpt framework and similar behavioural-optimisation systems.
- Programmable backend AI systems - engineering the AI computation itself when single-skill workflows are no longer enough and the system needs structured retrieval, routing, scoring, and orchestration. Currently illustrated by Stanford's DSPy.
Each layer answers a different operational question. Each layer is doing real work the other three cannot do. The reason the field reads as confused is that vendors and writers tend to argue for one layer as if it solves the whole problem. It does not. The point of naming the four-layer pattern is to refuse that framing.
Layer One is the move from prompt as experiment to behaviour as reusable artefact
The first layer is where most teams stop, often without realising it. A team writes a prompt that produces good behaviour. The prompt lives in a notebook, in an MS Teams message, in a tool's text field, sometimes in a wiki. When a new engineer joins, they ask in the team's MS Teams channel for "the good prompt for X". This is the artisan stage. Every AI workflow goes through it.
Layer One is the move past it. An Agent Skill - the closest concrete current example, not a recommendation - is a packaged behavioural specification. It includes the instructions, the worked examples, the scripts the AI can call, the resources it can reference, the workflow rules and operational policies, the evaluation guidance, the domain conventions it should respect. It is not just a longer prompt. It is the behaviour rendered as a reusable organisational asset.
This matters because organisations do not scale through prompts. They scale through standardised workflows. The interesting Agent Skills, in any AI-enabled delivery org, are not the ones that produce clever individual outputs. They are the ones that operationalise tribal knowledge: an architecture-review workflow a principal engineer used to do in their head, a recruitment evaluation that two senior people did consistently while a third could not match it, an SEO brief generation that used to take a half-day of coordination. The skill is the artefact that captured what the experts were doing and made it reproducible by anyone authorised to invoke it. That is what Layer One produces.
The deeper architectural shift is in the word behaviour. A prompt is a request. A skill is a contract. Once the behaviour is a contract, the next two layers become possible. You can validate against a contract, and you can improve a contract, in ways you cannot do for a request that lives in an MS Teams message somewhere.
There is a small but telling signal in this transition that I have been watching. Anthropic recently extended the Skill Creator tooling - the same workflow it ships for authoring Agent Skills - so it now also helps generate the evals for the skill it just produced: test cases, scoring criteria, expected-output rubrics (Improving Skill Creator: test, measure, and refine Agent Skills). The interesting move is operational, not conceptual. Most organisations treat skill-authoring and eval-authoring as two separate disciplines staffed by two different kinds of people. The new authoring surface collapses that handoff into one workflow. It does not eliminate the case for Layer Two, which is a wider discipline than any single skill's regression suite. It does remove the most common excuse organisations use to skip Layer Two entirely: "we do not have anyone to write the evals."
Layer Two is the difference between a system that appears reliable and a system that is reliable
Most production AI systems do not have a real evaluation infrastructure. They have a vibes layer. The team remembers the good outputs more vividly than the bad ones, the people who pushed for the feature have a survivorship bias toward "it is working", and nobody runs the same test twice with a stopwatch. This is not a moral failing. It is the default state of any system whose outputs feel evaluable by reading them. The illusion is comfortable, and it is dangerous.
Layer Two breaks the illusion. The discipline is well-articulated in Anthropic's Demystifying evals for AI agents. The patterns are golden datasets, regression suites, acceptance tests, benchmark tasks, critique loops, adversarial examples, before-and-after comparisons, automated reviewers, scoring prompts. The names vary across organisations; the patterns are stable. What they share is that they turn AI outputs from emotional artefacts ("this feels right") into operational artefacts ("this passes the regression set; the deviation from yesterday's run is 1.4% on the critical subset; here are the four cases that flipped"). Where this layer lands in the build journey is mapped in the eval-driven path from AI prototype to production product.
Once Layer Two is in place, the conversation about an AI system changes shape. Specific questions become answerable. Did quality improve this week? Which examples fail repeatedly? Where are hallucinations appearing? Which workflows remain unstable? Which outputs require human escalation? Without Layer Two, those questions get debated; with Layer Two, they have answers, and the answers are reproducible by anyone who can run the suite.
This is the layer where I see the most enterprise AI initiatives quietly fail. Not by producing bad output, but by being unable to tell the difference between genuine improvement, temporary variance, accidental regression, benchmark gaming, and hallucinated quality. Demos optimise for one good case. Layer Two optimises for the distribution. You cannot scale AI behaviour reliably without it.
Layer Three is what closes the loop and stops the prompt-maintenance treadmill
Layers One and Two together are enough to operate an AI system responsibly. They are not enough to compound it. The third layer is what compounds.
Once a behaviour is a contract (Layer One) and the contract has a measurable evaluation surface (Layer Two), the next question is whether improvement requires a human to rewrite the contract every time. In the artisan phase, the answer is yes. Someone notices the system is failing on a class of examples, they edit the prompt, they hope it does not regress on cases they remembered to think about. In the Layer-Three phase, the answer is no. The system itself can generate candidate edits, validate them against the evaluation suite, accept the ones that move the score in the right direction without regressing anywhere, and reject the rest.
This is what I mean by automatic behavioural improvement. Microsoft's SkillOpt and similar systems, and the category is younger than the others so the naming is still settling, formalise this loop. They generate rollouts of the current behaviour against held-out cases. They analyse failures. They propose targeted edits to the specification. They validate the edited specification against the held-out set. They accept successful mutations and reject regressions. The human moves up a level, from rewriting prompts to setting the criteria the loop optimises against.
The reason this is the layer that compounds is that it removes the only resource that does not scale: human attention to prompt maintenance. An organisation can accumulate hundreds of skills, thousands of eval cases, complex multi-role workflows. With Layers One and Two alone, the maintenance burden of all of that grows linearly with the surface area, and senior people start getting consumed by it. Layer Three is the layer that bends that curve. It is also the layer that lets reliability improve rather than just hold steady. The same loop that catches regression can drive improvement against the eval surface directly.
This is where the analogy to other infrastructure transitions gets sharp. CI/CD was not really about the build script. It was about the moment software stopped requiring an engineer to manually validate every deploy. Layer Three is the same kind of move for AI behaviour. Without it, the organisation is doing AI maintenance manually forever.
Layer Four is for the AI systems that are no longer single workflows but small computational machines
The fourth layer is structurally different from the first three, which is why it confuses a lot of teams when they encounter it.
Layers One, Two, and Three operate on a skill: a reusable, named behaviour the organisation can invoke. Layer Four operates on a system: a graph of model calls, retrieval steps, routing decisions, scoring stages, and verification passes that together produce an output no single skill could. The clearest current example is Stanford's DSPy, which provides a programming model for these multi-stage AI systems. The relevant abstraction is not a reusable workflow. It is a programmable language-model pipeline.
A DSPy-style system looks like this: a retriever pulls candidate documents from a vector store, a router classifies the user's intent, a reasoner produces a structured intermediate answer, a verifier checks it against the retrieved evidence, and a scorer ranks alternative outputs against an evaluation metric. The whole thing is a small piece of computational machinery whose behaviour can be optimised end-to-end. This is structurally different from an Agent Skill, which operationalises a workflow a human used to do. A DSPy-style system engineers a computational AI system the organisation could not produce manually at all.
The reason this matters operationally is that organisations frequently overengineer at the wrong moment. They jump into orchestration frameworks before they have operationalised the simpler workflows that Layer One handles cleanly. That is almost always backwards. The high-leverage adoption order, in my experience leading AI-enabled delivery, is roughly: operationalise the workflows that exist (Layer One); install measurement (Layer Two); install the improvement loop on the workflows that matter most (Layer Three); and only then, where the AI system needs structured retrieval, multi-stage reasoning, automated scoring, agent routing, or backend orchestration that a single skill cannot express, introduce a programmable backend system (Layer Four).
Layer Four becomes very valuable later in the maturity curve. It does not become valuable early, and treating it as the entry point is one of the more common ways an AI programme spends a lot of money on infrastructure it cannot yet use.
The four layers complement each other; this is not a framework war
Here is the move I most want to refuse. Almost every piece of public writing about AI reliability infrastructure picks one of the four layers and argues for it as if it replaces the other three. Eval-driven-development pieces argue for Layer Two as if it makes Layer Three unnecessary. DSPy enthusiasts argue for Layer Four as if it absorbs Layer One. Skill-system advocates argue for Layer One as if measurement and optimisation will fall out as a byproduct. None of these are true.
The four layers complement each other because they optimise different things:
| Layer | Optimisation target | What it cannot do |
|---|---|---|
| Behavioural specification (Layer One) | Operational behaviour as a reusable artefact | Cannot tell you whether the behaviour is working |
| Behavioural validation (Layer Two) | Reliability of behaviour against the distribution of real cases | Cannot improve the behaviour by itself |
| Automatic behavioural improvement (Layer Three) | Continuous mutation of the specification against the validation surface | Cannot operate without Layers One and Two underneath |
| Programmable backend AI systems (Layer Four) | Compositional AI computation that no single skill expresses | Cannot replace Layer One for workflows that do fit a single skill |
This is the practical shape of the reliability stack. The layers are complementary, not competitive, and an organisation serious about AI in production needs to be able to compose all four. Not pick one and ignore the others.
Prompt engineering alone fails at the moment AI moves from craft to dependency
The reason all of this matters now, not in two years, is that AI inside organisations has crossed a quiet threshold. AI systems running in delivery orgs affect recruiting decisions, architecture reviews, software delivery, support workflows. The early phase of AI in the org, when usage was experimental, outputs were manually reviewed, scale was small, operational risk was bounded, that phase is over. The systems that are still in that phase are the ones nobody is depending on yet.
When AI behaviour becomes a dependency rather than a craft, the requirements change. Organisations need reproducibility. The same input should produce predictable behaviour, not vary with mood. They need governance. Somebody must be accountable for what the system is allowed to do and how that changes. They need regression prevention. Yesterday's wins must still be wins today, even after the prompt was edited, the model was updated, the team rotated. They need observability. When behaviour drifts, the drift must be visible before a customer notices. They need escalation policies. The system must know when to defer to a human. They need rollback. If a change degrades behaviour, the org must be able to revert quickly. They need operational consistency. The system must behave the same way whether the senior engineer who built it is in the office today or not.
Prompt engineering alone does not provide any of those guarantees. It was never designed to. Quality harnesses do. This is the gap that Quality Harness Engineering, as a named discipline, is pointing at, and it is the gap most AI programmes are currently sitting inside without a name for it.
The competitive moat in AI is shifting from model access to reliability infrastructure
There is a separate question, longer-horizon, about why this discipline matters strategically and not just operationally. It is worth a paragraph because it changes how a CTO or a Head of AI should think about where to invest the next budget cycle.
The visible industry conversation in 2026 is still dominated by frontier model capabilities. Bigger models, cheaper inference, longer context windows, more agentic behaviour. The economic value of those advances is real, and a lot of capital is chasing them. But the long-term competitive surface is moving somewhere quieter. Once frontier models converge in capability, which they are doing on a faster timeline than most leaders are budgeting for, the differentiator stops being "which model do you have access to" and starts being "what reliability infrastructure do you have around it" - the operational reliability of the workflows, the quality of the evaluation surface, the governance posture, the cost discipline (which is a sibling discipline to reliability, and which I cover separately), the organisational integration, the optimisation quality. The companies with superior quality harnesses will outperform companies with marginally better models. This mirrors every previous infrastructure transition. The operational systems eventually become more important than the raw underlying capability. AI is moving in the same direction, and the deeper layer of that transition is the operating-model change underneath AI adoption.
This is why naming the discipline matters. A C-level executive accountable for AI outcomes needs vocabulary that maps to where the value is actually moving, not vocabulary inherited from the craft era. "We need a Quality Harness Engineering capability" is a defensible budget line. "We need better prompt engineers" is no longer a defensible answer to the questions the board is asking.
The discipline does not require all four layers to start, but it does require knowing which one is next
If you read this far hoping for a buy-list, I am going to disappoint you on purpose. The four tools I named - Agent Skills, eval tooling in the Anthropic / Confident-AI / web.dev family, the SkillOpt-style optimisation systems, DSPy - are illustrative exemplars of the four layers. They are not the only implementations, and they are not necessarily the right implementations for every organisation. The layers are the load-bearing claim. The current tooling is evidence the layers exist; it is not the prescription.
The practical starting point, in my experience leading AI-enabled delivery and visible in the eval-driven-development lifecycle I cover separately in a companion piece on getting AI products from prototype to production, is to install Layer One and Layer Two first. Get one workflow operationalised as a reusable behavioural artefact. Get one evaluation suite running against it. Live in that combination long enough to learn what your distribution of real cases actually looks like, and what kinds of regression matter to your users. Only then introduce Layer Three on the workflows where prompt-maintenance burden is genuinely accumulating. And introduce Layer Four only where a single skill cannot express what the AI system needs to do.
The slowest part of this is not the tooling. It is the organisational shift, from treating AI as a craft owned by individual experts to treating AI as infrastructure owned by the same discipline that owns any other production system in the company. Quality Harness Engineering is the name for the discipline that does the owning. The companies that install it early will be the ones whose AI programmes still exist and still improve in three years. The companies that treat AI reliability as someone's prompt-craft side project will be the ones still explaining to their boards why the demo worked and the production system drifts.
The discipline has a name now. The harder question - which layer to install next, in your organisation, this quarter - is the one that matters operationally. The naming move is just the prerequisite for being able to ask it.
