The 4-Level AI Adoption Evaluation Model: How to Tell What Your Delivery Team Has Actually Changed

The 4-level rubric in one paragraph. L1, individual AI use: the role uses AI personally, but the artifact looks the same as a year ago. L2, role artifact change: AI is visibly shaping the role's main deliverable - sharper structure, better edge-case coverage, explicit review notes, traceability to upstream sources. L3, team workflow change: artifacts connect across roles through shared specs, conventions, and quality gates. L4, governed improvement loop: the team measures whether AI-assisted work improves quality, speed, and rework, and updates playbooks, prompts, gates, and training on evidence. The test for what level a team has reached is not the survey. It is the artifact.

Every executive AI conversation I have had over the last twelve months reaches the same uncomfortable pause. The survey said the team is at Level 3. The license dashboard says everyone is active. And yet, sitting in a delivery review, nothing about how the work gets done looks different from a year ago.

That gap is not a measurement glitch. It is the cost of evaluating AI adoption with the wrong instruments. Self-assessment surveys drift upward; people score themselves on intention, not behavior. Procurement and usage counts measure access, not transformation. None of these instruments tell you whether the role itself has changed: how a developer ships a feature, how a QA designs a test plan, how a PM runs a project metrics review.

The most reliable signal is the work itself - and the trace around it: PR comments, test evolution, rejected alternatives, clarification logs, review notes, and retrospectives. The pull requests, the test plans, the tickets, the user stories, the architecture decision records that the team produces every week. If AI is genuinely embedded in the role, the artifact shows evidence of a changed workflow: sharper structure, better traceability, broader edge-case coverage, explicit review notes, or links to upstream/downstream artifacts. If it is not, the artifact looks exactly the same as last year, regardless of what the license dashboard says.

This article gives you a four-level rubric for that inspection, per role and per artifact. It is the companion piece to a claim I made on LinkedIn recently: counting AI tool logins is not AI adoption. Real adoption is role-specific behavior change. This is how you measure it.

Before the rubric, a short orientation on which measurement instruments tell you what:

The right-hand column is where most AI-adoption programs lose track of themselves. The instrument is precise about what it measures and silent about what matters.

Why do AI adoption surveys and license dashboards both lie?

The two most common ways organizations measure AI adoption today are both broken, and they are broken in mirror-image ways.

Self-assessment surveys ask people what they do with AI. People answer with what they intend to do, what they have done once, or what they think their manager wants to hear. Recent research on workplace AI adoption shows why the measurement problem is hard: adoption rates vary depending on whether you ask workers, survey firms, or inspect actual workflow change. In my own diagnostics, the gap almost always points in the same direction: people are at L2, the survey says L3.

License counts and token volumes have the opposite failure mode. They are precise. You know exactly how many seats are active and how many tokens were consumed last month. They also measure the wrong thing. They measure whether a person opened the tool, not whether the tool changed how the work was done. A developer can burn fifty thousand tokens a week and still ship the same kind of pull request they shipped last year. A QA can have every license active and still write the same regression-testing checklist they have used for five releases.

The more defensible signal sits between the two. You go look at the artifact the role produces. A pull request, a test plan, a story, a spec, an ADR, a CI pipeline config. You ask one question: did AI change how this was produced, and is the evidence visible in the artifact itself? That is the entire rubric. The four levels below are a way to grade what you find.

A note on framing. This is the public artifact rubric. It scores observable adoption in role artifacts - what a manager can see by reading PRs, test plans, ADRs, and metrics reviews. A stricter version I use for individual capability assessment scores AI-development sophistication along a different axis: private prompting → context engineering → harness engineering → scaled agent operations, with specific thresholds on agents, skills, usage logs, and CI / AI integration. The two are tracking the same underlying transformation through different lenses. A team at public-rubric L2 may sit higher or lower on the capability axis depending on which tooling discipline is in place. The role sections below stay on the public rubric throughout.

What are the 4 levels of AI adoption (L1 to L4)?

Level 1, individual task assistance. AI improves personal productivity, but the role artifact is mostly unchanged.

Level 2, role workflow change. AI changes how the role produces its main deliverable. The artifact shows better structure, sharper reasoning, broader edge-case coverage, or explicit review notes.

Level 3, team workflow change. Artifacts connect across roles. AI-assisted work moves through shared conventions, review expectations, and human sign-off points.

Level 4, governed improvement loop. The team measures whether AI-assisted work improves quality, speed, and rework, then updates playbooks, prompts, gates, and training based on evidence.

The per-role sections below name the exact L1-to-L4 signatures by artifact and the common misread that makes the role look one level higher than it actually is. The L2 signal across roles is a primary flow that has been automated and is in routine use - feature implementation for the developer, test generation for the QA, project metrics review for the PM, requirements engineering for the BA, an agentic framework for the SA, an incident triage workflow for DevOps. The L3 signal is the harness - agent feedback loops, quality-gate hooks that block non-compliance, multi-model orchestration with implementer-and-reviewer separation, and an SA-designed / DevOps-implemented CI-and-AI pipeline with secrets detection, SAST and DAST scanning, and AI code review on PRs. The L4 signal is governance wired into delivery: multi-agent orchestration as the everyday surface, cross-project reproducibility, and a published governance layer that names approved models, security-reviewed tools, decision rights over AI-touched load-bearing changes, and audit trails for every AI-generated artifact.

Governance is wired into the delivery pipeline rather than bolted on as a quarterly review. L4 also draws a clear boundary on which models are approved for which task, which tools require security review before use, who has decision rights over agent-generated changes that touch load-bearing components, and what audit trail every AI-generated artifact must produce. The dashboard is the surface. The governance underneath is what makes the dashboard trustworthy.

How do the 4 levels show up in Developer, QA, PM, BA, DevOps, and SA artifacts?

The four levels above are abstractions. The signal lives one step deeper, in what each level looks like for the role you are inspecting. Below is the per-role rubric I use across the six delivery roles when I evaluate a team. For each role I name the level signatures by artifact, and I name the common misread that makes the role look one level higher than it actually is.

Developer

The defining frame is the move from "AI as autocomplete" to "AI as delegated engineering workforce." Private daily use → context-engineered automated flows → harness-engineered autonomous agents → scaled multi-agent teams. The developer ships pull requests. That is the artifact that tells the truth.

• L1. AI used for explanations, syntax help, occasional snippets pasted into the editor. PR descriptions, commit messages, and tests look the same as a year ago. Personal speed; no team-visible artifact change.
• L2. AI-assisted commit messages, PR descriptions, and branch naming are standard. The project has a CLAUDE.md (or equivalent context file) that the team's AI tools actually use. The developer runs at least one or two automated primary flows - feature implementation, debug, or refactor - with structured prompts. For material changes, the PR names where AI shaped implementation, tests, or design.Before, a typical PR description read "fix bug in checkout." After, it reads "fix bug in checkout per spec spec-471 §3.2; AI-assisted scaffolding for tests in commit a7f, manual edits in b21." That is the L2 signature in one line.
• L3. The PR pulls from a shared spec, uses the team's AI-aware PR template, and passes automated gates before review. Agent feedback loops are connected to tests, linting, build verification, and observability. Human review focuses on judgment: design, risk, security, and maintainability.
• L4. Multi-agent work is the everyday surface - three or more agents working concurrently on a shared codebase with coordination protocols. Cost and quality controls per task (model selection, token monitoring, success-rate tracking). The engineering manager can show the rework delta on demand.

Common misread. Heavy private AI use that never reaches the artifact. The PRs, the commits, the tests look like last year's. That is L1, not L2 - the work the team ships has not changed.

QA engineer

The defining frame is the move from "AI helps me write test cases faster" to "AI agents execute my quality strategy." Structured prompting → agent-driven test creation with LLM evaluation → self-improving quality harness → multi-agent test orchestration. The QA ships test plans, automated test suites, and defect reports.

• L1. AI used to draft test ideas in private notes. Test plans, automated suites, and defect reports look the same as last year. No team-visible artifact change.
• L2. Test plans show AI-assisted edge-case sections that go beyond what the QA would have written alone. A test-case-generator agent is in use on real features. API test automation is producing executable scripts from documentation. Requirements testability is checked before sprint commitment. Defect reports start to classify the miss type (requirement gap, weak test design, AI-generated test gap, flaky automation, missing coverage).
• L3. The QA's test plan references the shared spec, maps acceptance criteria to tests, and feeds regression impact back into the delivery workflow. Quality-gate hooks block merges on traceability, coverage, or duplicate failures. When production catches a bug the tests missed, the gap captures into rules and the next sprint's tests reflect the lesson.
• L4. Escaped defects, flaky tests, and false positives feed a measured quality improvement loop. Multi-agent test orchestration runs as the everyday surface across testing phases. Cost and quality controls are in place: model selection per task, false-positive and false-negative tracking, audit trail for AI-generated test artifacts.

Common misread. High AI usage in the QA's private workflow with unchanged test plans and unchanged defect reports. That is below the bar for L2 in this framework - the artifact has not changed.

Project manager

The defining frame is dual responsibility: use AI to improve personal delivery work and help the team adopt AI without lowering quality. Personal productivity → repeatable PM workflows plus team enablement → AI-driven delivery governance → cross-project influence and role expansion into Delivery or Program Manager. The PM is a project-operations role, not a story-writing role. Story writing belongs to the BA. The PM ships project reports, metrics reviews, risk and dependency logs, stakeholder summaries, and quality-gates evidence.

• L1. AI used for meeting prep, status notes, and stakeholder updates. Project metrics reviews, quality-gates reports, and risk logs still look like last quarter's. No team-visible change.
• L2. Repeatable PM operating workflows running on the project rhythm, not one-off experiments: meeting processing that extracts decisions and action items and risks; recurring reporting; project metrics review (cycle time, lead time, throughput trends); risk and dependency tracking that feeds the reporting; stakeholder tracking with open questions and follow-ups; quality-gates reporting with critical-path coverage and CI status. Definition of Done extended with AI quality criteria.
• L3. PM artifacts anchor the team's delivery chain: BA requirements, SA design, developer specs, QA test plans, and release-readiness evidence all connect back to the same work item via the PM's tracking. The PM tracks whether AI-assisted delivery improves flow, predictability, and quality through the metrics review. Project-level quality-gate governance is visible, monitored, and acted on across SA, QA, dev, and DevOps.
• L4. The PM can show delivery impact over time: cycle time, wait time, rework, blocked items, escaped defects, predictability, and adoption blockers. AI is no longer a side tool; it is part of project governance.

Common misread. Heavy personal-productivity AI use (meeting summaries, status drafts, Slack reformatting) with no change to the project's recurring operating cadence. That is L1 regardless of how much time the PM saves on emails - the project artifacts have not changed.

Business analyst

The defining frame is the move from "AI helps me write docs faster" to "AI agents execute my analysis strategy." Structured prompting → agent-driven requirements, discovery, and proposals → self-improving analysis with quality gates → multi-agent analysis orchestration plus AI transformation discovery and pre-sale automation. The BA ships requirement documents, discovery synthesis, gap analyses, and proposal drafts.

• L1. AI used for note-taking and quick reformatting. Requirement documents and clarification logs unchanged in structure or rigor.
• L2. A requirements engineering workflow producing structured requirements (e.g. EARS-format) with traceability back to discovery notes and conversations. Discovery synthesis from multiple transcripts. Gap analysis comparing current vs future state with prioritized recommendations. A project context file (domain glossary, business rules, deliverable standards) actually used by the BA's AI tools. Requirements quality is verified before sign-off (completeness, acceptance criteria, untestable statements).
• L3. BA requirements feed PM stories, SA design, and QA testability checks. Requirements carry source traceability to discovery, stakeholder input, business rules, and open questions. When requirement gaps cause sprint issues, the pattern feeds back into the BA's workflow and the next sprint's requirements reflect the lesson. Quality-gate hooks block sign-off without acceptance criteria, mapped tests, or domain-glossary compliance.
• L4. Requirement-related rework, sprint issues, and change requests feed back into a measured BA playbook. Multi-agent analysis orchestration runs as the everyday surface: discovery → requirements → quality validation → gap analysis → proposal, with the BA reviewing consolidated output. Cost and quality controls with escalation triggers and audit trail for AI-generated analysis artifacts. AI transformation discovery and pre-sale proposal automation are operating.

Common misread. A requirement document that looks more polished, with better formatting and fewer typos, but contains the same ambiguities that produced rework before. AI did the polish. It did not change how the requirement was elicited, synthesized, or validated. That is below the bar for L1 in this framework.

DevOps engineer

The defining frame is the move from "AI helps me write Terraform" to "I build the infrastructure and pipelines that AI agents operate in." DevOps is critical to AI-assisted development. The role builds CI and CD pipelines with AI quality gates, configures security scanning, manages environments where agents execute, and implements the infrastructure the rest of the team relies on. The SA designs the quality-gate architecture; DevOps implements and operates it. The DevOps engineer ships pipeline configs, IaC modules, deployment verification reports, post-mortems, and security policy enforcement.

• L1. AI used to brainstorm pipeline tweaks in private. CI configs, infrastructure-as-code, and security scans still look like last release's.
• L2. IaC artifacts (Terraform, CloudFormation, Kubernetes manifests) are AI-assisted with security review and idempotency verification before applying. An incident triage workflow takes logs, alerts, and metrics and produces a root-cause hypothesis with severity and remediation. A deployment verification step produces a go / no-go recommendation. Post-mortems carry structured timeline, root-cause chain, and prevention recommendations. Baseline metrics tracked: MTTR, module-creation time, manual verification steps.
• L3. The CI and AI pipeline the SA architects is actually built: AI code review on PRs, docs-on-merge bot, dependency upgrade bot, build verification gates. Security scanning infrastructure is in place - secrets detection, SAST, dependency CVE scanning, DAST against staging - with findings blocking merges or creating auto-remediation PRs. The agent runtime environment (developer workstations, sandboxed Docker, MCP server hosting) is provisioned. Infrastructure quality gates and monitoring of AI workflows are operating.
• L4. Cross-project pipeline platform with reusable templates that deploy to new projects via configuration. AIOps: anomaly detection auto-creating incident tickets, predictive failure analysis, intelligent alert routing, automated triage for known patterns. Self-healing workflows for known failure modes. Cross-project security standards applied uniformly. Platform evolution driven by cross-project data.

Common misread. Heavy private AI use for Terraform drafting with no CI and AI pipeline stages built (no AI code review on PRs, no secrets detection, no SAST in CI). The pipeline still looks like last release's. That is L1 - the infrastructure the team operates inside has not changed.

Solution architect

This is the load-bearing reframe. The defining frame is the move from "AI helps me write architecture docs" to "I architect the AI development system the team operates in." The SA is the linchpin of AI-assisted development. The role configures the agentic development framework, defines project structure, sets up quality gates and CI pipelines, and ensures the entire team can work effectively with AI agents. The SA ships ADRs, system designs, agent and hook configurations, context-file hierarchies, and organizational standards.

• L1. AI used to brainstorm trade-offs in private. ADRs are written from scratch in the same template, sometimes shorter because the SA already worked through the alternatives with AI in conversation.
• L2. The SA has configured an agentic framework for at least one project - context files, agents, hooks, MCP - as a coherent system rather than isolated pieces. A multi-layered context hierarchy is in place. Agent chain design is documented (requirements → architect → developer → tester → reviewer) with hooks preventing out-of-ownership writes. Security rules (OWASP Top 10 checks, secrets detection, dependency vulnerability scanning) are encoded as unconditional rules. Project structure is optimized for agent navigation.
• L3. ADRs and design conventions are connected to the team's AI-assisted delivery workflow. The SA defines quality gates, review boundaries, security expectations, and where AI-generated code requires extra scrutiny. The harness developers and other roles work inside - pre-commit hooks, phase-transition gates, PR gates, the security pipeline - is the SA's. The framework serves all roles, and the SA tracks framework metrics (hook trigger rates, agent success rates, false-positive rates) and tunes on measured effectiveness.
• L4. A cross-project AI development platform that deploys to new projects through configuration, not from-scratch setup. Modular framework with core, optional, and tech-stack-specific components. Cost and quality controls at scale (token budgets per project, model selection strategies, agent success rates across teams). Published organizational standards: mandatory hooks, required quality gates, model selection guidelines, security baselines applied to all projects. Platform evolution driven by cross-project metrics.

Common misread. An SA who runs every architecture decision through AI in private, writes the team's same ADR template, and never configures a single agent, hook, or context file beyond the project root. The reasoning has improved. The development environment the team operates in has not. That is the old SA role with AI on top, not L2.

Which diagnostic prompts reveal AI adoption levels?

This is a delivery-manager or AI-transformation-lead instrument. The CEO holds them to it. The point of the rubric is to make it operational. Below are inspections, not interviews. A delivery manager can run them inside a single sprint. Each one is phrased as something you go look at, not something you ask the team about.

1. Pull the last ten pull requests from your most senior developer. How many reference a spec, an issue, or a story produced with AI assistance? Now do the same with a mid-level developer. The gap, or its absence, tells you whether the seniors' AI use has propagated into shared artifacts.
2. Open the last three test plans your QA team shipped. Find the explicit edge-case sections. Then ask whether the QA has ten or more AI-generated test case sets they have reviewed and used, and whether a regression impact analysis agent ran against the last sprint's PRs. If neither, the QA role is below the bar for L1 in this framework regardless of license count.
3. Read the last project metrics review the PM shipped and the last quality-gates report they sent up the chain. The metrics review and the quality-gates report tell you whether the six required L2 PM workflows are running on the project's rhythm or are still personal-productivity experiments.
4. Take the last requirement document the BA produced. Identify every requirement that has gone through rework since it was written. Is there a pattern: same kind of ambiguity, same stakeholder, same gap? If so, the BA's elicitation has not changed. Now check whether there is a working requirements engineering agent producing EARS-format requirements with traceability to source conversations. No agent, no L2.
5. Open the most recent ADR. Does it name alternatives that were evaluated? Are any of those alternatives traceable to AI-assisted analysis? Then walk to the project's root: is there a CLAUDE.md hierarchy, a configured agent chain, and at least two MCP servers actively used by the team? If the SA has produced better ADRs but the development environment the team operates in is unchanged, the role is at the old-SA-with-AI-on-top floor, not at SA L2.
6. Open the last five CI pipeline configs the DevOps team shipped. How many include AI quality gates: secrets detection, SAST scan, AI code review on PRs, dependency CVE scanning? If none, the CI and AI pipeline is not yet built, and DevOps is at L2 at best regardless of how much Terraform the engineer generates with AI assistance.
7. Pick one production defect from the last release. Walk it backward: which role's artifact missed it, and would inspection of that artifact have shown an AI-assisted edge-case or test that should have caught it? If the trace ends at "the test plan looks like every other test plan," the team is at L1 or L2 for that role.
8. Look at the last three retrospectives. Does the team discuss AI as a part of how the role does its work, or as a separate productivity tool? L3 teams talk about AI inside the role. L1 and L2 teams talk about it as a sidebar.

If you cannot run these inspections, because the artifacts do not exist, or because they all look identical and you cannot tell the difference, that is the diagnostic.

How to score what you find

Do not average the team too early. Score each role separately based on artifact evidence.

A mixed profile is normal. A team may have Developer L2, QA L1, PM L2, BA L1, SA L3, DevOps L2. That profile is more useful than saying "the team is Level 2."

The most common false positive

The most common false positive is a team with high AI usage and unchanged artifacts. Developers move faster privately. PMs summarize meetings faster. QAs generate private test ideas. BAs polish documents. SAs brainstorm trade-offs. But the shared delivery system remains the same. That is not transformation. That is individual productivity layered on top of the old operating model.

Where the rubric stops, and where the maturity ladder picks up

This article is the rubric. The companion article is the journey. The rubric places the roles. The ladder places the organization.

This rubric does not describe the journey. It describes how you take a reading of where a team sits right now, by inspecting the work the role actually produced this sprint. This article tells you what changed inside each role. The companion article, the AI adoption maturity ladder (L0 to L4), tells you how a team moves from one rung to the next.

What the rubric leaves you with

If you walk this rubric across your delivery org honestly, you will find one of three things. Either every role is at L1 and the past year of AI investment has not changed how the work gets done. That is the most common result in the organizations I evaluate. Or one or two roles have reached L2 in isolation, which is what self-assessment surveys typically mistake for L3. Or, occasionally, the team is genuinely at L3 in some roles and L4 is the missing layer. That means the feedback loop, the governance scaffolding, and the cross-project platform, not the tooling, are the next investment.

The implication is the one I want to leave you with. If you cannot describe what L1 versus L3 looks like in artifacts for at least three of your delivery roles, you do not yet know whether your AI transformation is working. License counts measure procurement. Token-burn measures activity. Neither tells you whether the role itself has changed. The artifacts and the trace around them can.

This artifact-first reading of adoption is the lens Shift Harness applies.

Frequently Asked Questions