> ## Content Index > Fetch the complete content index at: https://www.shiftharness.tech/llms.txt > Use this file to discover other available public pages before exploring further. # How to Make Your Repository Agent-Ready: A Checklist You Can Actually Test - URL: https://www.shiftharness.tech/agent-ready-repository-checklist/ - Published: 2026-10-12T10:49:46.000Z - Updated: 2026-10-12T14:09:28.000Z - Description: Instruction files describe readiness; they don't prove it. Four probes, run on a real ticket by a worker with no context, show where your repository's path breaks, and what that same gap already costs every new engineer. - Author: Sergii - Tags: AI Enablement, Delivery Teams, AI in Software Delivery, AI Adoption, Quality Harness Engineering, AI Reliability Picture a fresh agent session opened on a repository that looks well kept. There's an AGENTS.md with build commands, a README with an architecture sketch, and CI has been green for weeks. The agent reads all of it, runs the install command, and spends the next forty minutes on a missing environment variable that three people on the team know by heart. Nothing in the repository was absent. The path through it had never been walked by anyone without that knowledge. > **Quick answer:** an **agent ready repository** is one where a worker with no prior context (a fresh agent session or a new engineer on day one) can discover its boundaries, bootstrap a safe environment, reproduce a task, and hand back evidence a reviewer accepts. You test it by running those four probes against a real task and recording each check as pass, fail or unknown. Files describe readiness; the trial gives you evidence of it, for that kind of task at that revision. ## An agent-ready repository is defined by what a stranger can do in it Most readiness advice measures what a repository contains: an instruction file, linters, a test suite, CI, documentation. The scanners that rank for **AI coding agent readiness** work the same way. Factory announced its Agent Readiness model in January 2026, scoring repositories across technical pillars and five maturity levels. Larridin's scan places a repository on levels L1 to L5 and states that checks it cannot assess are skipped and excluded from the score. Some go further: WeaveAI adds an agent's judgment on whether documented commands match the code. Those tools are useful, and they mostly tell you what is present and configured. A scan that doesn't execute a real task can't tell you whether the path works end to end. A test command can exist and take longer than the agent's command timeout, so the agent skips it or times out. A seed-data script can exist and point at a staging database the worker has no credentials for. A boundary rule can sit in a document that nobody linked from the place the agent starts reading. So the definition I use is behavioral. A repository is agent-ready for a given class of task when an unfamiliar worker can complete four things without asking a person: 1. **Discover boundaries:** what it may change, what it must not touch, and which commands are destructive. 2. **Bootstrap a safe environment:** a working local setup that can't reach production data or credentials. 3. **Reproduce the task:** observe the failing behavior or the acceptance scenario before changing code. 4. **Collect evidence:** produce proof of the change that a reviewer accepts without a walkthrough. Each check inside those probes gets one of three verdicts. **Pass** means the criterion was observed being met, **fail** means it was observed not being met, and **unknown** means there wasn't enough evidence to say either way. Record why a check was run or not separately from its verdict, and count a probe as passed only when every applicable check in it passes. Unknown is the most useful verdict on the sheet, because it marks the places where readiness lives in someone's head. Two edge cases need rules before you start. A check that doesn't apply (a task with no database has no seed-data question) is recorded as not applicable with a reason. Checks blocked by an earlier failure, such as reproduction after a setup that never ran, are recorded as blocked rather than failed. ## Run the trial with a worker who has no context The probe only measures something if the worker is genuinely unfamiliar. Four setup rules matter more than the checklist itself. > **Isolate before anything runs.** Give the worker a disposable environment with nonproduction credentials, no host mounts beyond the repository, and network access limited to what setup needs, and verify that boundary before any setup command or repository script executes. Installing dependencies already runs code: `npm ci`, for example, runs package lifecycle scripts. If you can't establish the boundary, stop the trial there. > **Use a fresh worker.** For an agent, that means a new session with no memory, no earlier chat, and no hand-written briefing beyond what the repository itself provides. For a person, it means someone who hasn't touched this codebase. If the tester already knows where the secrets file lives, the boundary probe passes on knowledge the repository doesn't hold. > **Use a real task, not a tour.** "Explore the repository" produces a summary, not a verdict. Pick a small, ordinary ticket of the kind you'd hand to an agent in production: a bug with a reproduction, or a feature with acceptance criteria. > **Set a timebox and record interventions.** Decide in advance how long each probe may take, and count every time a person has to step in. The timebox is yours to set for the repository and the task class. What matters is that it's written down before the trial, so a slow probe is recorded as slow and not explained away afterwards. ## The checklist: four probes, twelve checks Record each check with its verdict and one line of evidence: the command that was run, the file that answered the question, or the person who had to be asked. | Probe | Check | Pass looks like | Fail or unknown looks like | | ---------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- | | 1\. Boundaries | 1a. Where to start | The worker finds the entry point for the task from the repository alone | It searches for long or asks which service owns the behavior | | 1\. Boundaries | 1b. What not to touch | Generated code, existing migrations, vendored code and secrets are marked as off limits | The worker edits a generated file or an existing migration | | 1\. Boundaries | 1c. Destructive commands | Commands that reset data or deploy are named as such | A reset or deploy command sits next to the test command with no warning | | 2\. Environment | 2a. Isolation | Checked before setup runs, not assumed: the effective identity, reachable services and inherited credentials cannot touch production | Production credentials are needed, or nobody knows whether they are reachable | | 2\. Environment | 2b. One path to a running system | Inside that boundary, a reproducible documented sequence (ideally one command or container) produces a working setup | Setup depends on an undocumented variable, tool version or tribal step | | 2\. Environment | 2c. Data the task needs | Seed or fixture data exists for the task's scenario | The worker invents data or connects to a shared environment | | 3\. Reproduction | 3a. A failing signal first | The bug or acceptance scenario can be observed before implementation changes | The worker can only reason about the bug, not observe it | | 3\. Reproduction | 3b. Fast local feedback | A narrow test command runs in the timebox you set | Only the full suite or CI gives feedback | | 3\. Reproduction | 3c. Stable tests | Repeated runs under a stated protocol (same command, a set number of runs) give the same result | Flaky tests make pass and fail ambiguous | | 4\. Evidence | 4a. Before and after | Against acceptance criteria set before the work, a check fails before the change and passes after it | Tests pass both before and after, so they say nothing about the new behavior | | 4\. Evidence | 4b. Reviewer-readable proof | The worker attaches what the reviewer needs: test output, a sample, a log | The reviewer has to re-run or re-derive the result | | 4\. Evidence | 4c. Known gaps stated | The worker names what it could not verify | Unverified behavior is presented as done | Twelve checks is a working set, not a standard. If your repositories run batch jobs or mobile builds, you'll add checks to probes 2 and 3\. Keep the four probes fixed as a common structure, but compare results only between trials run under the same protocol: matched task size, worker configuration, timebox and intervention rules. A single run is evidence about that run, not about every task of its kind. ![A printed Agent-Readiness Trial sheet on a clipboard lists four probes and twelve checks, each marked P, F or U in blue ink with a short evidence note, beside a kitchen timer.](https://storage.ghost.io/c/73/3e/733efc05-c397-4cf1-b19b-527c1b07dee7/content/images/2026/10/image-2.png) ## Probe 1: boundaries are easy to miss, because nobody writes down what they never touch Boundary knowledge is the hardest to see from inside a team. Senior engineers don't think "don't edit the generated client" as a rule; they just never do it. An agent arriving cold has no such reflex, and a well-written instruction file often covers [commands and coding style](https://www.shiftharness.tech/agent-readable-coding-standards/) while leaving the no-go zones implicit. The mechanism is simple: whatever the team avoids by habit is invisible to anyone who wasn't there when the habit formed. This is where an instruction file earns its place, and it's worth keeping short. The guidance on [what belongs in operating instructions for software teams](https://www.shiftharness.tech/operating-instructions-software-teams/) applies here: list the boundaries and the commands, link the deeper documents, and leave out the codebase tour the agent will discover anyway. A boundary check passes when the worker stays inside the lines without being told in conversation. It's unknown when two senior people give different answers to "can it touch the migrations folder?" That disagreement is a finding about your repository, not about the agent. ## Probe 2: the environment probe is where unsafe work happens The environment probe matters for safety as well as speed. In ["What Breaks When LLMs Code?"](https://arxiv.org/html/2605.30777v1?ref=shiftharness.tech), researchers at Case Western Reserve analysed 547 coding-agent incidents drawn from 16,586 GitHub issues. Of those, 326 were rated high or critical, including destructive operations, and over 65% arose during bug fixing or setup and configuration work. That's issue-tracker data, so it shows what gets reported rather than a full incident rate. But it locates the risk where this probe looks: the ordinary moment when a worker is trying to get a system running. Isolation comes first in the probe because setup itself runs code. Keep two things apart. The trial environment has to be verified before setup or the trial stops. The repository's own defaults can stay unknown after a safe trial, and that unknown still blocks agent rollout. A repository passes 2a only when the honest answer to "could this setup touch production?" is no, checked rather than assumed. If the worker needs a production credential to run the task, the repository fails, however good its documentation is. If nobody can say whether the local configuration falls back to a shared database, record unknown and treat it as a stop for agent rollout on that repository until someone finds out. Check 2b can look fine on paper and still fail. Setup instructions decay without anyone noticing, because the people who could notice already have working environments. A clean CI runner can tell you whether the documented command still runs; a fresh worker is the cheap way to find out whether someone without context can find that command and follow it to a running system. ![A printed SETUP.md runbook with step 4, ask the platform channel for the .env file, circled in red and annotated undocumented: who owns this? and could this reach production?](https://storage.ghost.io/c/73/3e/733efc05-c397-4cf1-b19b-527c1b07dee7/content/images/2026/10/image-3.png) ## Probe 3: reproduce before you change anything An agent that can't observe the problem may still produce a change. It may reason from the ticket, edit plausible code, and report success. That's why reproduction gets its own probe: it separates a fix from a guess. Check 3a asks for a failing signal before implementation changes. For a bug, that's a test or a script that shows the wrong behavior. For a feature, it's the acceptance scenario expressed as something runnable, even if it fails today. Check 3b is about the length of the loop. If the only feedback comes from a full suite or from CI, the worker iterates slowly and may stop early. A narrow command that runs one area of tests keeps the loop short enough to use. Flaky tests (check 3c) deserve a hard line. If repeated runs disagree, a pass on the affected tests is weak evidence, and the agent may learn to rerun until green. Record the check as fail and fix the flakiness before measuring anything that depends on those tests. ## Probe 4: evidence is what makes the change reviewable I used to treat this probe as a nice-to-have, the tidy wrap-up after the real work. It isn't. Evidence is what lets a reviewer accept a change without reconstructing it, and review capacity is a common bottleneck once agents start producing changes. Check 4a is the one to look at first. Tests that pass both before and after a change are regression evidence for the behaviors they exercise; they don't show that the change does what the ticket asked. Asking the worker for a check that fails first and passes after, against acceptance criteria written before the work started, turns a claim into evidence. For a bug, confirm the initial failure is the defect and not a broken setup. Refactors are the exception: there's no failing baseline, so the evidence is unchanged behavior on the existing checks. Check 4c matters just as much: a worker that says "I could not verify the export on large files" is handing the reviewer the exact place to look. A worker that says nothing is handing over risk. This probe also connects to the wider loop around the agent. The [system around an AI coding agent](https://www.shiftharness.tech/harness-engineering-explained/) is mostly about where verification happens and who signs it off; this probe checks whether the repository makes that verification possible at all. ## A worked example: SHX-104, a synthetic tenant-scoped export To show the probe end to end, here's a synthetic task. It isn't drawn from any real codebase; it's built to exercise all four probes. **SHX-104:** add a CSV export of invoices for an account administrator, and the export must include only rows belonging to the requesting tenant. | Probe | Result on SHX-104 | Evidence recorded | | ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1\. Boundaries | Fail on 1b | The worker edited a generated API client to add the endpoint, because nothing marked the folder as generated | | 2\. Environment | Pass on 2a for the trial sandbox; repository default unknown | Before setup, the sandbox was checked: no production credentials, no host mounts beyond the repository, and outbound network limited to the package registry. Separately, nobody could confirm whether the repository's own local config falls back to a shared database for tenant data, which stays open as a rollout finding | | 3\. Reproduction | Unknown (3a pass; 3b, 3c unknown) | A fixture with two tenants existed, and a test calling the export endpoint failed before any change because the endpoint didn't exist yet; narrow-command timing and repeat-run stability weren't recorded | | 4\. Evidence | Fail on 4a | The worker's tests checked the CSV columns but never asserted that tenant B's rows were absent | The interesting result is the combination. The code compiled and the column tests passed, so a quick review would have approved it. Yet the property the ticket existed for, tenant isolation, was never tested, and the environment question that would have made a leak dangerous was unknown. Three remediation items fall out directly: mark generated folders, answer the shared-database question and write it down, and add tenant isolation to the evidence the reviewer expects for any export. ![A code-review diff of tests/test_invoice_export.py checks only the export columns for tenant A; checks passed, and a reviewer asks where the assertion that tenant B rows are absent is.](https://storage.ghost.io/c/73/3e/733efc05-c397-4cf1-b19b-527c1b07dee7/content/images/2026/10/image-4.png) ## Where AGENTS.md fits, and where scanners stop None of this argues against instruction files. The evidence that they matter is getting better. A Stanford and Carnegie Mellon study, ["A Few Pages of Markdown"](https://arxiv.org/html/2608.25241v1?ref=shiftharness.tech) (ASE '26), built its classifier on one sample of 441 repositories and then compared agent-first repositories in a separate sample. Those without committed AI configuration of any kind (not one particular filename) saw roughly twice the increase in cognitive complexity (+53% versus +27%). The authors call the result observational and hypothesis-generating, so it shows an association, not proof that the file caused the difference. It's still a reason to keep one. Most **AGENTS.md best practices** converge on the same advice: put commands early, state boundaries, keep it short, and don't paste a directory tour. That advice makes probe 1 cheaper to pass. It doesn't make probes 2 to 4 pass, because those depend on whether the commands work, whether the environment is isolated, and whether the tests prove anything. A file can describe all three accurately on the day it's written and be wrong a month later. Scanners sit in the same position. They read a repository at a pinned commit and mostly report what's present and configured. Run them; they're cheap and they catch real gaps. Then run the trial, because the scanner can't tell you whether a stranger gets from clone to evidence, and treat every unknown as a finding instead of something to exclude. ## A failed probe is onboarding debt you're already paying Most of what the four probes expose is friction a new engineer hits too. A boundary nobody wrote down is one a new hire learns by asking a senior engineer, and a setup command that no longer works costs each new joiner the same lost day. Agents and people differ in permissions and in how they recover from a dead end, so the overlap isn't total. Time a human run separately before quoting an onboarding cost. Where the overlap holds, the remediation list pays twice. That changes how a CTO or a delivery lead should sequence an agent rollout. Read each failure as a diagnostic hypothesis first: rule out an ambiguous ticket, a missing tool permission or a transient outage before booking it as repository debt. Then run the probe on the repositories you plan to extend agents to, sort them by result, and start with the ones that pass probes 2 and 4\. Repositories with unknowns on isolation don't get agents until the unknown is resolved. Repositories that fail on evidence get agents only with a reviewer briefed on what to check. Record the task, the commit, the agent and tool version, and the permissions used, so a later run is comparable. Then run the probe again on a schedule. Dependencies move, secrets rotate, and setup instructions drift without anyone editing them, so readiness decays the same way documentation does. A probe result is dated evidence about one repository and one class of task, and it should be read that way. > **AI Transparency Notice:** This article and its accompanying images were created with the assistance of generative AI. The author directed the content, contributed the underlying ideas and analysis, and reviewed the final publication. ## Frequently Asked Questions What does agent-ready mean for a code repository?▸ A repository is agent-ready for a class of task when a worker with no prior context can discover its boundaries, bootstrap a safe environment, reproduce the task and hand back evidence a reviewer accepts, without asking a person. It's a property you observe in a trial with a real task, not one you read off a file list, and a single trial is evidence about that task at that revision, not about every task of its kind. Is an AGENTS.md file enough to make a repository agent-ready?▸ No. An AGENTS.md file is guidance the agent reads, so it makes the boundary probe easier to pass and is worth keeping short and accurate. It can't make a broken setup command work, isolate an environment from production, or turn weak tests into evidence. Those need their own checks in the environment, reproduction and evidence probes. How do you test whether a repository is ready for AI coding agents?▸ Give a fresh worker a small, real ticket and run four probes: boundaries, environment, reproduction and evidence. Record each check as pass, fail or unknown with one line of evidence, set the timebox before you start, and count every time a person has to step in. Compare results only between trials run under the same protocol. Why record "unknown" instead of forcing a pass or fail?▸ Because unknown marks readiness that lives in someone's head or evidence nobody could reach. When two senior engineers disagree about whether a setup can reach production, that disagreement is the finding. Record the reason next to the verdict; dropping unknowns from a score hides exactly the risk the trial exists to surface. Can an automated readiness scanner replace the trial?▸ No, but it's a good first pass. Scanners such as Factory's Agent Readiness report or Larridin's level-based scan check what's present and configured at a commit, and some add an agent's judgment on whether documented commands match the code. A scan that doesn't execute a real task can't show whether an unfamiliar worker gets from clone to accepted evidence, so use the scan to find gaps and the trial to confirm the path works. How often should the readiness trial be repeated?▸ Repeat it whenever dependencies, environments or secrets change materially, and on a fixed schedule for repositories where agents work regularly. Setup instructions and tests drift without anyone editing them. Record the task, commit, agent and tool version, and permissions used so a later run is comparable.