The AI Adoption Scorecard: A Diagnostic for Profiling Operating-Model Change Through Delivery Artifacts
The survey says the team is well along. The license dashboard says every seat is active. Then you open last sprint's pull request and it reads exactly like a year ago. Here is how to profile what actually changed, role by role, from the artifacts.
There is a particular moment in an AI adoption review that I have learned to wait for. The quarterly survey is on the screen, and it says the team is well along. The license dashboard next to it says every seat is active and the token burn is up forty percent. Then someone pulls up an actual pull request from last sprint, and it reads exactly like a pull request from a year ago. Same shape, same thinness, same missing edge cases. Both pictures can be true at once, because they describe different layers. The survey captures perceived individual productivity; the pull request shows whether that change reached the shared delivery system. Only the second layer is what the team ships.
An AI adoption scorecard is a per-role diagnostic that profiles how far AI-assisted practice has changed a delivery team's shared artifacts - pull requests, test plans, project metrics reviews, requirement documents, architecture decision records, CI pipeline configs - rather than reading what the team reports about its own AI use. It does not count how much AI a team uses. It profiles whether the operating model changed and left evidence in the work.
This is a diagnostic template, not a validated instrument, and it is more useful for being honest about that. It is also not an AI readiness scorecard: a readiness scorecard asks whether a team is equipped to adopt AI before it has, and this one reads the evidence after, in the artifacts the work already produced. For a delivery team with accessible artifacts, it can usually be completed within one sprint, handing you a profile you can defend artifact by artifact and a redesign for each gap it finds. What follows is the model, the evidence rules that keep it from flattering anyone, the scorecard itself, and how to run it without turning it into theatre.
Why conventional adoption metrics fail
A self-report scorecard inflates for a mechanical reason, not because teams are dishonest. Ask a developer how often they use AI and the answer encodes intention, the one good week, and a quiet read of what leadership wants the number to be. Self-reports tend to overstate or misclassify AI use, and the drift is strongest exactly when leaders have signaled the number should rise. A scorecard whose primary input is a survey has built its measurement on the one signal that moves independently of the work.
License counts and token volumes fail in the mirror-image way. They are precise and early because the vendor and the training team export them automatically, so the dashboard gets built from the data that already exists rather than the data that would answer the question. A senior engineer who opens a tool once a week to satisfy a tracker and ignores its output counts the same as one who rebuilt their day around it. The instrument is exact about access and silent about whether the work changed.
| Signal | What it tells you | What it does not tell you |
|---|---|---|
| Licenses | Access was purchased | Whether the work changed |
| Logins | The tool was opened | Whether role behavior changed |
| Tokens | AI activity happened | Whether the artifact changed |
| Self-assessment | Perceived adoption | Whether the artifact changed |
| Artifact inspection | The work changed, or it did not | Why it changed, without more evidence |
Read the bottom row, and read it carefully. A pull request, a test plan, a metrics review, a requirement document, an ADR, a CI config: these are slower to instrument than a survey and far harder to inflate, because the artifact is a record of what the work became. The bottom row also carries the one honest limit this whole method has to manage: an artifact can change for reasons that have nothing to do with AI, so a changed artifact is a question, not yet an answer. The next two sections turn that limit into rules.
What the scorecard measures: four dimensions, not one ladder
The mistake most maturity models make is to collapse several different questions into a single climbing number. This one keeps them apart. A role is profiled on four dimensions, and they are independent, not steps on a staircase:
- Artifact and provenance change. Is there a repeatable change in the role's primary artifact, or in the provenance trail it leaves, with evidence that AI contributed to producing it?
- Workflow integration. Does that change connect to the work around it, so an upstream or downstream artifact or role consumes it?
- Governance and control. Is the AI-assisted work owned, access-controlled, and auditable, with exceptions handled?
- Outcome evidence and improvement. Did a relevant measure of quality, speed, cost, or rework move against a baseline, and does that evidence feed back into how the work is done?
These do not develop in a fixed order. A regulated organization often builds governance and access controls before it integrates AI into daily workflows or has enough use to show outcomes. A small team can integrate AI across its handoffs and even move a delivery metric without ever formalizing ownership. Governance and outcomes are the clearest case for keeping the columns apart: a team can control an AI workflow it cannot yet show results for, and another can show results it never formally governed. So you read each dimension on its own evidence and record the profile, four readings per role. You may derive a one-word label from the profile if a board update demands it, but do not pretend the underlying evidence is one linear scale, because it is not.
This profiles a different thing than individual capability. A developer can be personally sophisticated with AI and still show no change in the shared artifact, because private skill is a separate axis from the organization-level adoption stages the AI adoption maturity ladder describes. It also profiles a different thing than delivery-flow frameworks: DORA measures software-delivery performance, SPACE is a multidimensional view of developer productivity, and DX blends perceptual and operational signals. Each is real and useful, and none of them tells you whether the operating model changed. The companion 4-level AI adoption evaluation model reads adoption at the organization level; this scorecard is the role-level operational form you fill in.
The evidence rules that keep it honest
Three rules separate a real reading from a hopeful one, and they matter more than the table that follows.
Evidence of contribution is not evidence of cause. A disclosure, an execution log, or a configured workflow shows that AI was involved in producing the artifact. It does not prove AI caused the improvement you see, because a new template, a staffing change, lower task complexity, or a process tweak can move the same artifact. So the scorecard records that AI contributed to producing the artifact, and it reserves causal language for the cases where you have a controlled comparison or credible time-series evidence. Most readings will be associations, honestly labeled.
A changed artifact earns nothing without attribution. An artifact that differs from its baseline (the same role's prior sprint, a comparable past task, or a historical template for that artifact) has at least five possible causes, and only one is the one you want: AI changed it; a new template or process changed it; a mature pre-existing practice was always doing it; the change is cosmetic; or AI helped in a way that never landed in the artifact you opened. Secrets scanning, SAST, dependency checks, ADRs, and test-to-spec mapping are good engineering controls, and their presence proves none of them are AI adoption. Without attribution evidence, the entry is artifact changed; cause unknown, and you go find the trace before you score.
A trace earns no credit on its own. This is where the method can manufacture the theatre it warns against. Tell a team that disclosures move them up a dimension and you will get formulaic disclosures, reference logs, and context files nobody loads. So a trace establishes AI participation and nothing more. Each dimension has its own minimum before it counts: an artifact change has to be valid and repeatably used; an integration has to be valid and actually consumed by another workflow; governance has to be implemented and operationally used; an outcome needs a baseline, a measured trend, and a feedback action. A gate that runs but never blocks is implemented, not operationally used, and earns nothing.
The scorecard: six roles, four dimensions, one artifact to open
Here is the instrument. Each row is a delivery role and its primary artifact. Each dimension names a capability to look for, described first as a capability so the test does not depend on any particular tool. The technologies in parentheses are examples of how a team might satisfy the capability, not requirements for it, and their absence is not a failing mark.
Record each role-and-dimension cell with one rating and a separate confidence, so two reviewers reading the same artifacts land on the same profile rather than their own impressions.
| Rating | Decision rule |
|---|---|
| Not evidenced | No qualifying evidence was found |
| Emerging | Attributed evidence exists, but the use is isolated or inconsistent |
| Established | The dimension's full criterion is evidenced across the observation window |
| Unknown | Evidence was unavailable, or attribution could not be resolved |
Confidence is recorded on its own, Low / Medium / High, and an Unknown is not a failing mark, it is a flag that you could not see enough to call it. The minimum a cell needs before it reads Established is set by dimension: artifact change wants a valid, repeatably used change; integration wants that output consumed by another workflow; governance wants controls implemented and operationally used; outcomes want a baseline, a measured trend, and a feedback action. Use the same evidence grammar in every row, so a threshold reflects the dimension, not the role.
| Role (open this) | Artifact and provenance change | Workflow integration | Governance and control | Outcome evidence and improvement |
|---|---|---|---|---|
| Developer (last 10 PRs, one senior + one mid) | A repeatable, AI-attributed change in how implementation, tests, or design were produced, with a disclosure or linked trace - not just longer descriptions | The output feeds the next role: PRs draw on a shared spec and automated checks that inform the reviewer before approval (e.g. an AI-aware PR template, review gates) | A named owner, controlled access, and an audit trail for AI-assisted changes, with exceptions handled | A rework or defect delta tracked against a baseline, with per-task cost watched, feeding back into how the work is done |
| QA (last 3 test plans + defect reports) | Edge-case categories beyond the baseline plan, with a trace of how they were produced and recorded human verification; defect reports classify miss type | Tests map to a shared spec and acceptance criteria, and material coverage or traceability gaps trigger review and block merges at the team's risk threshold | An owned quality process with an audit trail for AI-generated tests, exceptions handled | Escaped defects and flake rates tracked against a baseline and fed back into the test approach |
| PM (last metrics review + quality-gates report) | A recurring, traced AI-assisted workflow changed the report itself, not the speed of status notes | PM tracking links BA, SA, developer, and QA artifacts to one work item | Delivery-impact metrics are owned and reviewed on a cadence, with exceptions handled, not a dashboard of activity counts | Those delivery-impact metrics show a baseline and trend, and the review changes what the team does next |
| BA (last requirement doc + rework log) | Structured requirements with recorded AI assistance, not just tidier formatting | Requirements feed PM, SA, and QA with traceability, and sign-off is gated on acceptance criteria and mapped tests | Requirement standards are owned and reviewed on a cadence, with exceptions handled | The recurring-ambiguity or rework pattern drops against a baseline, feeding an owned playbook |
| SA (most recent ADR + dev-environment config) | A repeatable, AI-attributed change in how options were generated, trade-offs analyzed, or risks reviewed - not just tidier formatting | The SA's gates and review boundaries are the system the other roles work inside, and they consume them | Owned standards (required gates, access and security baselines) with auditability and exceptions handled | Those gates are tuned on a measured signal, and the metrics review shows the tuning moved an outcome |
| DevOps (last 5 CI configs + deploy evidence) | An AI-attributed change in pipeline rigor (e.g. AI review or triage), traced - not faster-drafted IaC with the same stages | AI quality and security gates actually block merges or open remediation, rather than running and reporting only | An owned, auditable pipeline with a uniform security standard, exceptions handled | The pipeline is reviewed against outcome metrics that show a baseline and trend, feeding change |
Read every cell against the rules and the ratings above. Where you see the change but not the trace, the honest entry is Unknown, noted as artifact changed; cause unknown, not the higher rating. Where you see the trace but the workflow behind it is not actually used, that is Emerging, not Established.
How to run the assessment
Inspect first, then validate with practitioners. The inspection is the anchor, because an interview alone re-introduces the self-report drift the method exists to avoid. But the artifact cannot explain why it changed, and a short conversation can surface a workflow change that never landed in the document you opened. So you read the artifact, form a reading, and then check it with the people who produced it: triangulate the inspection against tool or workflow telemetry, a practitioner's explanation, outcome metrics, and a baseline comparison before you commit a dimension.
Run it transparently, with the practitioners and not on them. People should see the reading, challenge the attribution, supply missing context, and correct errors before the profile is finalized. The output redesigns workflows; it does not rank individuals, feed performance reviews, or set compensation. An instrument that became a surveillance tool would corrupt the artifacts it reads, because people would start writing for the audit instead of for the work.
Record the reading with its limits, not as a verdict. For each role and dimension, note the observation window, the sampling rationale, what you could and could not see, what you did not assess, and your confidence. Where attribution is unclear, mark it artifact changed; cause unknown and move on rather than rounding up. A second reviewer on the ambiguous cells is cheap and catches the readings where one person's prior did the scoring. The output is a profile, never a single averaged number: writing out "Developer artifact and integration Established but outcomes Not evidenced, QA artifact Emerging only, SA governed but not integrated" tells you where to act, and averaging it into one label is where the signal dies.
How to choose the next redesign
A reading is only useful if it changes what you do next, and the lever is the process, not the tool budget. Each empty dimension names a different kind of redesign, and because the dimensions are independent you can work on whichever gap is costing you most, not a mandatory next step.
- To put a change in the artifact: wire the AI step into the actual deliverable and leave a trace, so the work, not a private habit, carries the evidence.
- To integrate it: connect that artifact to the work around it through a shared spec, a gate, or a sign-off, so another role consumes the output.
- To govern it: give the AI-assisted workflow an owner, controlled access, an audit trail, and exception handling.
- To evidence outcomes: pick a measure of quality, speed, cost, or rework, baseline it, watch the trend, and feed what you learn back into the workflow.
Two cautions. These dimensions are independent, not mandatory phases: a team can sensibly govern early and evidence outcomes later, and it just should not record an Established outcome until there is a baseline and a measured trend behind it. And every redesign has to clear its dimension's minimum before the cell reads Established - the artifact actually used, the integration actually consumed, the control operationally live, the outcome baselined and trending - which is why the instrument that diagnosed the gap is the same one you re-run a quarter later, reading the artifact again to confirm the dimension actually moved rather than asking whether the change felt like progress.
Where this does not apply
A diagnostic is only honest if it names its limits. The first is the false positive that works in both directions: AI use that leaves no shared artifact, provenance, workflow configuration, telemetry, or outcome evidence is not evidenced operating-model adoption, and a changed artifact with no AI trace - a new template, a pre-existing mature practice, a cosmetic edit - is not adoption either. The question is always whether the work the team relies on came out different, in the artifact or its provenance, and whether you can connect that difference to AI.
The second limit is the boundary with private capability. The scorecard reads the shared system, so a developer with an elaborate personal setup who ships PRs that leave no shared change, provenance, or outcome evidence reads as no operating-model change, which is correct for an instrument that profiles the team rather than the individual. The third is team shape: the six-role grid assumes distinct roles, and a two-person squad should skip the rows it does not have rather than be scored against empty cells. The thresholds are written for distinct roles, and the instrument is a reading of your operating model, which differs in shape from team to team.
What you are doing when you run this is reading operating-model change through the artifacts the work produces, and that method has a name: the Shift Harness Artifact Test, a method for reading operating-model change through the artifacts teams produce, including specs, decision logs, QA plans, review patterns, governance evidence, and role-level playbooks. The scorecard is that test applied at the level of the delivery role. License counts measure procurement, token burn measures activity, and a survey measures perceived experience and reported behavior. The artifacts, and the evidence around them, are the most defensible layer for testing whether transformation reached shared delivery work. The scorecard is yours to copy, run, and argue with, which is the most a diagnostic should ask of you.
