> ## Content Index > Fetch the complete content index at: https://www.shiftharness.tech/llms.txt > Use this file to discover other available public pages before exploring further. # Your Eval Set Is a Depreciating Asset - URL: https://www.shiftharness.tech/eval-driven-development-renewal-loop/ - Published: 2026-07-18T00:05:00.000Z - Updated: 2026-08-19T20:44:57.000Z - Description: A passing eval suite feels like coverage. It is coverage of the world on the day you wrote it, and that world has already moved. What keeps the suite honest is a renewal loop, not a better set written once. - Author: Sergii - Tags: AI Strategy, AI Reliability, AI operating model You did the hard part. You installed evals, scored your AI feature against them, watched the suite go green, and shipped. Then a few weeks later the production incidents started creeping back. A response that should have been refused went through. A format that used to be stable came out malformed. And the strange part is that the dashboard still says pass. Nothing in your eval suite flagged any of it. That gap is the thing I keep coming back to. A green eval dashboard that no longer predicts production quality is worse than no dashboard, because it manufactures confidence at the exact moment the product is starting to drift. The team feels covered. The numbers say covered. The product is not covered. The most useful way I have found to explain why this happens is to stop thinking about the eval set as a fixed artifact and start treating it as an asset with a half-life. > **Eval driven development** is the practice of writing evaluations that define correct behavior and treating a passing eval suite as the gate for shipping AI changes. The catch most teams miss: an eval set decays. The suite written against last quarter's failures stops catching this quarter's, so the renewal loop that keeps it current matters more than the original suite. Most of what gets written about **llm evaluation** stops at construction: how to write good cases, which metrics to track, which graders to use. That is the right first step, and it is the step most teams are still on. But construction is a one-time cost and **model evaluation** is a recurring one, because the model, the inputs, and the product all keep moving after launch. Treating **ai model evaluation** as a build-once task is the quiet mistake behind the green dashboard that no longer means anything. Here is the short version before the mechanism. An unmaintained eval set loses value because the thing it measures keeps moving. Models get swapped every few months as providers ship new versions, the inputs real users send drift away from the inputs you seeded the set with, and the product behavior changes every time you ship a feature. A suite left alone after launch is an outdated map within weeks. The point of the asset framing is not that the set is doomed; it is that the value is contingent on maintenance. Individual regression cases can hold their value for a long time, and a portfolio kept current can compound it. What erodes when no one tends it is the representativeness of the coverage, the calibration of the graders, and the headroom of the capability evals. The fix, then, is not a better eval set written once. It is a renewal loop that observes production safely, adjudicates which **production traces** are real failures while keeping the graders calibrated, and maintains a versioned portfolio of failure classes, adding new ones and retiring dead ones, run by a named owner on a fixed cadence. This is the second-order problem behind [the eval-driven path](https://www.shiftharness.tech/from-ai-prototype-to-production-product-the-eval/). That earlier piece argued you should install evals to get an AI feature out of the prototype stage and into production. This one starts where that one ends: the evals you installed are already decaying, and almost nobody owns keeping them alive. ## Why an eval set loses value the moment you commit it An eval set is a snapshot of a moving target. The day you write it, every case encodes a specific model, a specific input distribution, and a specific product surface. None of those three stay still. Drift is one of the production failure modes engineers reliably underweight, and it shows up in [three distinct production failure patterns](https://www.shiftharness.tech/ai-production-failure-modes/) that each erode a different part of your coverage. Naming them separately matters, because each one needs a different part of the renewal loop to catch it. **Model drift** is the most visible source, though the word "drift" undersells how it arrives. A provider model-version upgrade is not gradual statistical drift; it is a discrete system change, a step, that lands the day you adopt the new version. The label is pedagogical, grouping it with the slower kinds, but the mechanism is a swap: when a provider ships a new model version, or you upgrade your own, the behavior your eval cases were written against changes underneath them in one move. What decays gradually is your coverage; the upgrade is the discrete event that triggers the decay. A case that caught a failure on the prior model can pass trivially on the new one, not because the underlying risk is gone but because the new model handles that specific phrasing differently. Worse, the upgrade introduces new failure modes the old cases never anticipated. So **model drift** does two things at once: it quietly retires some of your coverage and opens gaps you cannot see from a green suite. With providers shipping new versions every couple of months, a regression set written six months ago is testing a model that no longer exists. **Data drift**, or distribution drift, is slower and harder to notice. You seed an eval set with the inputs you can imagine: clean examples, a few edge cases, the failure modes you already know. Then real users arrive, and the inputs they send diverge from your seed set within the first weeks. They paste malformed data, send adversarial prompts, write in languages you did not test, and combine requests in ways you never modeled. The coverage that was representative of your traffic at launch is unrepresentative of your traffic now, and the eval set has no way to know that on its own. It keeps scoring the inputs you gave it, not the inputs you are actually getting. **Behavior drift** comes from your own side. Every shipped feature, every prompt change, every new tool you wire into the agent, every policy update changes the surface the eval set was built to protect. The set does not move when the product moves. It silently falls out of sync with what the product now does, so it ends up certifying behavior that no longer matches the system in production. This is the drift teams cause themselves and notice last, because it feels like progress while it is happening. Put the three together and the pattern is clear. The model changes, the inputs change, and the product changes, and through all of it an eval set left alone sits frozen at the state of the world on the day it was written. That is what depreciation means here, and the word is deliberate: it is the default trajectory of an asset no one maintains, not a property the set carries no matter what you do. The asset is not broken. It is dated, and a dated map of a moving territory is a liability disguised as a control. The rest of this piece is about keeping the map current. ![Editorial three-zone diagram labeling the three eval-decay sources, Model Drift, Data Drift, and Behavior Drift, each zone annotated with the coverage it erodes](https://storage.ghost.io/c/73/3e/733efc05-c397-4cf1-b19b-527c1b07dee7/content/images/2026/06/image-2-66.png) ## The half-life: treating eval coverage as a quantity that decays The reason this is dangerous is that the decay is invisible from inside the suite. Pass rate goes up and to the right, or holds steady, while real coverage falls. Teams that track pass rate as the health metric are watching the wrong number. Pass rate tells you how the current model performs against the cases you wrote. It tells you nothing about the failures happening in production that your set was never updated to catch. The number that actually matters is how much of what production is throwing at you your eval set still covers. A practical way to make the decay visible is an eval-coverage rate: of the failures you confirmed in production over the last period, what share would your current eval set have caught before they shipped? That share is your live coverage reading, and the gap is its complement, the share of confirmed failures your set missed. Watch the rate fall, or equivalently the gap grow, and you are watching coverage decay in real time. One honest limit is built into the metric: it can only count failures you detected and confirmed, so it reads the coverage you can see, not the coverage you have. Whole classes of failure that never get noticed stay invisible to it, so the measured gap is a lower bound on your true coverage loss, and the measured rate an upper bound on your true coverage. The real picture is at least this bad, never better. Treat the number as a directional floor on the problem, not a precise instrument. There is a chicken-and-egg objection here worth answering: computing this rate continuously needs the renewal machinery, the trace stream and the graders, already running. But the first reading does not. A one-time retrospective audit gets you an initial manual number, reading back over the last quarter of confirmed incidents, a sample of support cases, and a review of recent traces, and asking which your current set would have caught. That manual pass tells you whether you have a decay problem before you have built anything; the metric becomes an ongoing health signal once the loop is installed to produce it. The exact band matters less than the direction: a team that watches this number over time will see coverage erode in the weeks after a model swap, because the new model retired old cases faster than anyone wrote new ones. You need any signal that separates "the suite passes" from "the suite still covers the product," because those two statements drift apart quietly and the second one is the one that protects you. This is the distinction between AI activity and AI performance. A static eval suite that keeps passing is activity. It produces a number, it gates the pipeline, it makes the work feel disciplined. A renewed eval set whose coverage tracks the failures production is actually generating is performance. The difference is not whether you have evals. It is whether the evals you have still describe the product you are shipping. **AI evals** are not a foundation you pour once and build on top of. They are a control surface that has to be maintained at the rate the system underneath it changes, which on current model cadences is fast. ## The renewal loop: observe safely, adjudicate and calibrate, maintain the versioned portfolio If the eval set decays, the discipline is renewal. The **renewal loop** is the part of **eval driven development** that the field has mostly left implicit, and it has three steps you can install. Each one answers a question the static suite never asks: what is actually happening in production, which of it is a real failure and is the judge that decided so still trustworthy, and which failures become permanent versioned cases while which ones retire. **Observe safely.** The first step is capturing real production interactions as raw material, and the word "safely" is load-bearing: observing production means handling real user data, so the observation and the safety boundary are the same step, not a cleanup pass afterward. A trace is the structured record needed to reconstruct and evaluate an interaction: the input the user sent, the output the model returned, and the tool calls and intermediate steps in between. It is not a verbatim dump of everything that happened, it is the fields renewal actually needs to judge whether the interaction was a failure. This is the source of truth for what your product actually does, as opposed to what you assumed it would do when you wrote the seed cases. Someone has to instrument this, decide what gets logged, and make sure the trace stream is queryable rather than dumped into a log file nobody reads. Because the raw material is real user interactions, a minimum privacy boundary has to be drawn at this step and cannot be deferred: log only the fields renewal actually needs, redact sensitive content before it lands in the trace store, set a retention limit so traces do not accumulate forever, and put access control on the store so the failure-mining work does not become a quiet data-exposure surface of its own. Tools like Langfuse, Braintrust, and the tracing built into most eval frameworks exist for exactly this. The point is not the tool. The point is that without a safely captured trace stream, renewal has no input, and the loop cannot start. **Adjudicate and calibrate.** The second step has two halves, and both have to run or the step is hollow. The first half is adjudication: you apply graders to the traces, programmatic checks for the things you can verify deterministically (a malformed JSON output, a missing required field, a latency breach) and **LLM-as-judge** graders for the things that need judgment (a tone that drifted off-brand, an answer that is technically correct but unhelpful, a refusal that should not have happened). A grader does not decide that something is a failure. It surfaces a candidate, this interaction looks wrong, and a human adjudicates whether it is a real failure. This is where the volume of production gets compressed into a reviewable set, and where you decide how sensitive the graders are, since a grader that flags everything is as useless as one that flags nothing. The second half is calibration, and it is the half almost every loop skips. The graders are themselves a measuring instrument, and a measuring instrument drifts. An LLM-as-judge built on a model that gets swapped is now judging with a different model than the one it was tuned against; a programmatic check written for a prior output format silently passes everything once the format moves. So the graders need their own evaluation on a cadence: hold a small labeled set of interactions a human has already ruled on, run the graders against it, and check that they still agree with the human verdicts. An uncalibrated grader does not announce itself. It quietly passes real failures or flags clean cases, and because every downstream decision inherits its judgment, the whole loop starts renewing the wrong things while the dashboard still looks healthy. Adjudication finds the failures; calibration keeps the thing that finds them honest. **Maintain the versioned eval portfolio.** The third step is where the loop closes, and it is more than promoting one case. It is maintaining the whole set as a versioned portfolio, on both edges, so the set stays representative instead of becoming an add-only pile that grows forever and slowly tilts toward whatever failed loudest in the past. The inflow edge is promotion, and it is the part most often missing and the one that matters most, because it is the action that most directly renews the regression portion of the asset. A confirmed production failure becomes a permanent case in the eval set, but the mechanics matter: promotion generalizes the failure into a reusable case, not a verbatim replay of the exact trace. You take the trace that surfaced the problem, strip it down to the failure class it represents, and encode that class so the case catches the next instance, not just the one that already happened. Replaying the literal trace teaches the suite to pass on a single historical input while the same class of failure walks in through a slightly different door. Promotion needs three things to be real: someone with the authority to promote a case, acceptance criteria for what qualifies (a reproducible failure class, not a one-off flake), and a defined place the case lands so it runs on every future change. The outflow and balance edges are the ones an add-only loop skips, and skipping them rots the set in the opposite direction from staleness. As promotion runs, near-identical cases accumulate, so you deduplicate and reweight, because a set heavy with redundant variants of one failure overweights it while starving coverage of rarer classes. And a case written against a behavior the product no longer has, or a model version no longer in service, is dead weight that slows every run and inflates the pass rate without protecting anything, so you retire or archive it. Maintaining the portfolio means working both edges: add and generalize new failure classes on the inflow, and dedupe, reweight, and retire on the outflow. This is what "versioned" buys you, and it is why the set belongs under version control like any other asset that changes. Every add, every retirement, every reweight is a tracked change against a known prior state, so you can see exactly what the set looked like before a batch of promotions, diff the coverage one release made against the next, and roll a bad batch back when a hasty promotion turns out to be a flaky case rather than a real failure class. An unversioned eval set is an append log nobody can audit. A versioned one is a maintained asset whose history you can read, and whose mistakes you can undo. A failure that gets discussed in a standup and then forgotten renewed nothing. A failure class that becomes a tracked, versioned regression case the suite checks forever did. Two examples make the loop concrete. When a provider ships a new model version, a regression case that caught a tone failure on the prior model can pass trivially after the upgrade, because the new model phrases things differently and the specific trigger no longer fires. Meanwhile the upgrade introduces a new over-refusal pattern that no existing case covers, and it goes uncaught until a customer hits it and complains. The trace of that complaint is the raw material; the grader flags the refusal; promotion turns it into a permanent case so the next upgrade is tested against it. The second example is distribution drift in the first month: an eval set seeded with clean test inputs sails through every run while real traffic brings malformed, adversarial, and multilingual inputs the set never modeled. The renewal loop catches this because it is fed by what users actually send, not by what the team imagined they would send. ![Hand-drawn whiteboard cycle of the eval renewal loop: Observe Safely, Adjudicate and Calibrate, and Maintain the Versioned Portfolio in arrowed boxes, with a loop-back arrow into the eval set](https://storage.ghost.io/c/73/3e/733efc05-c397-4cf1-b19b-527c1b07dee7/content/images/2026/06/image-3-65.png) ## Not everything in your eval set decays at the same rate It helps to split the eval set by what each part is for, because the two kinds age differently and the renewal loop feeds them differently. In its guidance on evaluating AI agents, Anthropic draws a useful line between evaluations that lock in behaviors you already care about and evaluations that measure progress on hard, open-ended tasks. The operational reading of that distinction is the one that matters for renewal. **Regression evals** are the cases that pin down specific behaviors you have decided are correct: this input must produce this format, this category of request must be refused, this tone must hold. They are your no-backsliding guarantees, and they are the part of the set that depreciates fastest, because every model swap and every product change can quietly break a behavior you thought was settled. The renewal loop feeds primarily here. Most of what you promote from production is a regression case: a confirmed failure you never want to see again, encoded so the suite catches it forever. **Capability evals** measure how well the system does the genuinely hard part of its job, the open-ended task you are still trying to get better at. These decay more slowly because the hard task itself does not change as often, but they are also harder to renew automatically, since judging "better" requires more than a pass/fail check. They decay slower, but they do not sit still. A capability eval has its own maintenance process, on its own cadence: as the model gets better and the set's pass rate climbs toward the ceiling, the eval saturates and stops discriminating, so it has to be refreshed with harder cases and raised against the new abilities the model has acquired. A capability set that every model passes easily is no longer measuring capability; it is measuring that the bar got too low. So the renewal loop has two clocks, not one. Mixing the two and treating the whole set as one pass-rate number hides the decay, because a stable capability score can mask a regression set that is falling out of date. Separate them, and the renewal loop has a clear primary target: keep the regression set current with what production is actually breaking, and raise the capability set against what the model can newly do, each on its own clock. ## Where renewal goes wrong Most teams that fail at renewal do not fail because the loop is hard to understand. They fail on a handful of recurring mistakes, each of which lets the eval set rot in a different way. The table below names the common ones, what they do to your coverage, and the fix. | Mistake | What it does to coverage | The fix | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Treating "we have evals" as a one-time binary | Coverage freezes at launch state and decays from day one with no one watching | Make renewal a recurring practice with an owner, not a project that closes | | Tracking pass rate as the health metric | A rising pass rate hides falling coverage; the dashboard looks healthy while the product drifts. A near-100% regression pass rate means something only when the cases and their coverage are current; over a stale set, a rising pass rate is the symptom, not the all-clear | Track the eval-coverage rate (share of recent production failures the set would have caught) against a current set, not pass rate alone | | No owner for renewal | Trace-mining is everyone's job, so it is no one's job, and the loop never runs | Assign a named owner who runs trace-mining and the promotion review | | Promoting noisy or flaky cases without a confirmation step | The eval set rots from the inside as unreliable cases create false signal | Require a reproducible-failure acceptance criterion before any case is promoted | | Adding cases only after a customer-visible incident | Renewal becomes damage control instead of discipline; you are always one incident behind | Mine traces proactively on a cadence, so failures are caught before a customer hits them | | Trusting the graders to stay accurate while everything else moves | The control over the control drifts; an uncalibrated grader silently passes real failures or flags clean cases, and every downstream promotion inherits the error | Evaluate and recalibrate the graders themselves on a cadence, against a small labeled set, so the mechanism that scores production does not quietly corrupt the loop | The thread running through all six is the same. Renewal is treated as something that happens reactively, by whoever notices, when they have time. That is not a discipline. It is a hope. The bottom two rows are the subtler version of the same trap, and they map directly onto the two loop steps teams under-run. The grader-trust row is the calibration half of step two left undone: a loop that renews the eval set but never re-evaluates its own graders is maintaining the cases with an instrument that is quietly going out of calibration. And the add-only failure is the maintenance step run on the inflow edge only: promote new cases forever, never deduplicate the redundant ones or retire the dead ones, and the portfolio rots in the opposite direction from staleness, bloated with near-identical variants of old failures while rarer classes go uncovered. The loop is built to work both edges and both halves. The recurring mistake is running half of it. ## Renewal is an operating-model decision, not a backlog ticket Here is the part that determines whether any of this actually happens. The renewal loop is not a task you can drop into a backlog and trust the team to pick up. It is an operating-model decision, and it needs three things named explicitly, or it will not run. The first is an **owner**: a specific person responsible for running trace-mining and bringing candidate failures to review. Not a rotation, not a team-wide expectation, a name. The second is a **cadence**: a fixed rhythm on which the promotion review happens, weekly or per-release, so renewal is scheduled rather than triggered by incidents. The third is a **decision-right**: the explicit authority to promote a production failure into the permanent set, with the acceptance criteria attached, so promotion is a controlled act rather than an argument every time. Owner, cadence, decision-right. Without all three, the loop has no engine. This is why the eval set belongs in the operating model and not just in the test directory. Two of the components that make up an AI-product operating model are directly implicated here. The eval set is a review-and-control standard, the quality gate that decides whether a change ships. And the renewal loop is an operating cadence, the recurring rhythm that keeps that gate current. The eval set is not the whole operating model. It is one control-standard component, and like every control standard, it is worthless if no one is accountable for keeping it accurate. A control that nobody maintains is not a control. It is a label on a control. So the question to ask is not "do we have evals." Almost everyone building AI products now does. The question is "who owns renewal, on what cadence, with what right to promote a failure into a permanent test." A team that can answer that has an eval set that holds its value. A team that cannot has a depreciating asset and a green dashboard that is slowly telling it less and less. Treating the eval set as a maintained operating-model component rather than a one-time artifact is the lens Shift Harness applies to AI-product reliability: the discipline that matters is renewal, and renewal is owned, scheduled, and decided, not hoped for. ![Single-page renewal-review control checklist on a desk with the three headed fields Owner, Cadence, and Decision-Right, a calendar block for the cadence and a name in the owner field](https://storage.ghost.io/c/73/3e/733efc05-c397-4cf1-b19b-527c1b07dee7/content/images/2026/06/image-4-30.png) ## Key takeaways - An unmaintained eval set is a depreciating asset with a half-life, not a fixed artifact you build once. Maintained, it holds or compounds its value; the depreciation is the default for a set no one tends. - What erodes is coverage representativeness, grader calibration, and capability headroom, from three sources: **model drift** (a discrete version-upgrade trigger), data drift, and behavior drift. - The discipline that keeps an AI product reliable is the **renewal loop**: observe safely, adjudicate and calibrate, maintain the versioned eval portfolio, where promotion generalizes a failure into a reusable class rather than replaying the raw trace, calibration keeps the graders honest, and maintenance retires dead cases instead of only adding new ones. - The loop feeds primarily your **regression evals**; **capability evals** move on a slower clock. - Renewal needs an owner, a cadence, and a decision-right, or it does not run. - A passing static suite is false confidence. The number that matters is the eval-coverage rate (the share of recent confirmed production failures your set would have caught), not pass rate. It reads only detected failures, so the true coverage is at most what it shows, never better, never a complete measure. > **AI Transparency Notice:** This article and its accompanying images were created with the assistance of generative AI. The author directed the content, contributed the underlying ideas and analysis, and reviewed the final publication. ## Frequently Asked Questions Why does my eval suite pass while production keeps failing?▸ A pass rate measures how the current model performs against the cases you already wrote, not the failures production is now generating. An eval set is a snapshot of a moving target: the day you write it, every case encodes a specific model, a specific input distribution, and a specific product surface, and all three keep moving after launch. The model gets swapped every few months, real user inputs drift away from your seed set within weeks, and every feature you ship changes the behavior the set was built to protect. The suite keeps scoring the old world and reports green, while coverage of the actual product quietly falls. A green dashboard that no longer predicts production quality is worse than no dashboard, because it manufactures confidence at the exact moment the product is starting to drift. How often should you update an LLM eval set?▸ The useful answer is not a fixed schedule but an ownership decision. An eval set should be renewed continuously, on a fixed cadence run by a named owner, rather than refreshed quarterly or only when something breaks. Renewal needs three things named explicitly: an owner (a specific person who runs trace-mining and brings candidate failures to review), a cadence (a fixed rhythm, weekly or per-release, so renewal is scheduled rather than triggered by incidents), and a decision-right (the explicit authority to promote a production failure into the permanent set, with acceptance criteria attached). Without all three, renewal becomes whatever someone notices when they have time, which is a hope, not a discipline. A practical floor is to renew at least as fast as the system underneath the set changes, which on current model release cadences is fast. What is the difference between regression evals and capability evals?▸ Regression evals pin down specific behaviors you have decided are correct (this input must produce this format, this request must be refused, this tone must hold) and should hold near a 100% pass rate. They are your no-backsliding guarantees. Capability evals measure how well the system does the genuinely hard, open-ended part of its job, and start at low pass rates that rise as the system improves. For maintenance, the load-bearing difference is the decay rate: regression evals depreciate fastest, because every model swap and product change can quietly break a behavior you thought was settled, so the renewal loop feeds them primarily. Capability evals decay more slowly because the hard task itself does not change as often, but they are harder to renew automatically, since judging "better" requires more than a pass/fail check. How do you keep an eval set from going stale?▸ Install a renewal loop with three steps: observe safely, adjudicate and calibrate, maintain the versioned eval portfolio. Observe safely captures real interactions (input, output, tool calls, and intermediate steps) as the source of truth for what the product actually does, within a minimum privacy boundary (log only what renewal needs, redact, retention-limit, access-control the store). Adjudicate and calibrate applies graders to those traces, both programmatic checks and LLM-as-judge, to surface candidate failures a human then confirms, and on a cadence re-checks the graders themselves against a small labeled set so the instrument that scores production does not silently drift. Maintain the versioned eval portfolio closes the loop on both edges: promote confirmed, reproducible failures into permanent cases (generalized into a reusable class, not a verbatim replay), and deduplicate, reweight, and retire on the outflow, all under version control so changes are tracked and a bad batch can be rolled back. Promotion is the one most often missing and the one that matters most, because it is the only step that actually renews the asset. A failure discussed in a standup and forgotten renewed nothing; a failure encoded as a versioned regression case the suite checks forever did. What is eval decay?▸ Eval decay is the loss of an eval set's diagnostic power over time as the thing it measures keeps moving. It comes from three distinct sources, grouped under "drift" for teaching, though they are not all gradual. Model drift: a provider model-version upgrade is a discrete system change, not gradual statistical drift, and it is the trigger that decays your coverage; cases written against the old behavior can pass trivially after the swap while the upgrade introduces new failure modes the old cases never anticipated. Data drift, or distribution drift: real user inputs diverge from your seed set as people send malformed, adversarial, and multilingual inputs you never modeled. Behavior drift: every feature, prompt change, and new tool you ship changes the surface the set was built to protect, so the set certifies behavior that no longer matches the live system. Through all three, the set sits frozen at the state of the world on the day it was written. That is depreciation: the asset is not broken, it is dated, and a dated map of a moving territory is a liability disguised as a control. How do you measure whether your eval set still covers your product?▸ Track the eval-coverage rate, not the pass rate. The eval-coverage rate is the share of the failures you confirmed in production over the last period that your current eval set would have caught before they shipped; the gap is its complement, the share your set missed. Treat the rate as your live coverage reading: a set catching most recent confirmed failures is current, while one that has slid to a small fraction is measuring an increasingly thin slice of reality even as the dashboard stays green. One limit is built in: the metric can only count failures you detected and confirmed, so your true coverage is at most what the rate shows, never better, and the gap it reports is a lower bound on what you are actually missing. The exact band matters less than the direction. Watching this number over time shows coverage eroding in the weeks after a model swap, because the new model retires old cases faster than anyone writes new ones. You do not need a precise instrument, just any signal that separates "the suite passes" from "the suite still covers the product," because those two statements drift apart quietly and the second one is the one that protects you.