AI adoption: from impressive demo to operating advantage.

A support assistant drafts a convincing answer in seconds. The demo ends there. Your business still has to establish whether the answer is correct, whether it exposed another customer’s data, and whether checking it takes longer than writing it. That gap between generated output and completed work is where an AI strategy succeeds or fails.

For a CTO, the useful unit of adoption is a workflow with an owner, a baseline, and a measurable outcome. Begin with one expensive, repeatable task. Make the entire process better, including review, exceptions, and recovery. Then decide whether to expand.

1. Read the evidence at the task level

Two studies illustrate why a universal “AI productivity uplift” is a poor planning assumption. In a 2023 working paper studying 5,179 customer-support agents, Brynjolfsson, Li, and Raymond reported a 14% average increase in issues resolved per hour with a generative AI assistant. Gains were larger for less experienced and lower-skilled workers; experienced, highly skilled agents saw minimal impact. [1]

In a different setting, METR’s 2025 randomized study involved 16 experienced open-source developers completing 246 tasks in repositories they knew. Allowing early-2025 AI tools increased completion time by 19%. The developers believed the tools had sped them up. [2] Feeling faster and finishing faster are different measurements.

The support study examined a particular deployment in one company; the developer study had a small, specialized sample and a particular generation of tools. Neither estimates the return for every business, role, or current model. The actionable conclusion is to measure your workflow against a comparable baseline, including quality and review time, before budgeting the benefit.

2. Choose a workflow with recoverable mistakes

Start where inputs recur, good answers can be checked, and mistakes can be caught before they affect a customer. Internal knowledge search with citations, ticket classification, and draft support replies are useful candidates. An autonomous system issuing refunds or changing infrastructure has a different failure cost and needs stronger controls.

Compare each candidate with the simplest viable alternative: a search improvement, a template, a rules engine, or a process change. Use a language model when interpreting varied language or synthesizing context creates enough value to justify its variability. Buy a pilot before committing to a platform.

• Value: How many cases occur, what do they cost, and which business outcome can improve?
• Verifiability: Can a reviewer or deterministic check identify a bad answer?
• Data readiness: Are authoritative sources current, accessible, and permissioned?
• Failure cost: Can the system abstain, and can a person reverse the result?

Write a one-page contract: “For these ticket categories, help agents produce policy-grounded drafts; agents approve every message; success means lower time per correctly resolved ticket without worse reopen rates.” Give it a business owner and an engineering owner. Adoption counts and generated-word counts cannot substitute for that outcome.

3. Design the boundaries before adding autonomy

For the support example, begin with a read-only retrieval-augmented generation (RAG) workflow: retrieve relevant policy passages and pass them to the model with the question. This supplies current context; it does not guarantee that the answer follows the source. Validate citations and let the assistant decline when evidence is missing.

Treat retrieved documents, emails, and model output as untrusted input. A malicious instruction in a document can become a prompt injection. Enforce authorization in application code, outside the model; validate any tool call against an allowlist and its arguments against a schema. A system prompt alone is not an access-control boundary. OWASP documents prompt injection as a core application risk. [3]

Define data retention, training-use terms, regional processing, and deletion requirements before choosing a provider. Redact sensitive fields in telemetry and restrict access to sampled traces. Decide who owns source freshness and document removal. NIST’s Generative AI Profile is a useful framework for assigning and monitoring these risks. [4]

Keep model access behind a small application boundary with versioned prompts and retrieval settings. Set timeouts, bounded retries, token limits, and per-workflow spend caps. Measure end-to-end p95 latency, including retrieval and validation. Route simple tasks to cheaper models only after evaluation; use permission-aware cache keys and invalidate cached answers when their sources change. Fine-tuning can be evaluated later for stable behavior or format requirements; it is not a substitute for fresh knowledge or authorization.

4. Evaluate completed work, including the failures

Build a versioned evaluation set from permitted historical cases. Include common requests, rare expensive failures, ambiguous policy, missing evidence, stale documents, multiple languages, and injection attempts. Label expected answers and source passages with domain experts. Reserve held-out cases so prompt tuning does not simply memorize your benchmark.

Measure retrieval relevance separately from answer correctness: a grounded-looking response can cite a passage that never answered the question. Score correctness, citation support, appropriate abstention, data isolation, reviewer edits, and total completion time. Model-based grading can help triage, but calibrate it against human judgments, particularly for costly errors.

Release gates should reflect business risk. For example, a tenant isolation failure should block release even if the average quality score improves. Set acceptable quality, latency, and cost thresholds before the pilot, with enough cases to assess uncertainty. Zero failures observed in a small sample does not establish zero risk.

Then compare assisted and unassisted work on similar cases. Randomize assignment where practical, track differences in agent experience, and include review and rework time. Check downstream reopen rates and customer outcomes: a fast first reply that creates a second ticket can erase the gain. Re-run evaluations when the model, prompt, retrieval index, or source policy changes.

5. Price the accepted outcome

Token prices are an input to your economics. The denominator is correctly completed work. Count inference, retrieval, retries, human review, rejected drafts, escalation, and ongoing evaluation. Divide those costs by accepted outcomes, not by generated responses.

At 10,000 comparable cases a month, these assumptions produce about $20,667 in monthly capacity value after fixed running cost. That is not automatically cash savings: it becomes economic value when freed capacity serves demand, improves service, or changes actual spend. Include integration, procurement, training, and migration costs in the investment decision and payback calculation.

Stress-test the assumption that matters most. If assisted handling takes six minutes instead of four, contribution falls to about $1.23 per case and operating break-even rises to roughly 4,100 cases a month. A cheaper model can still cost more overall if it increases corrections. Quality and economics belong in the same review.

6. Turn 90 days into a sequence of decisions

• Days 1–15 — establish the baseline. Select one workflow, observe real handling time and failure costs, name owners, and confirm data rights. Stop if a simpler process change wins.
• Days 16–30 — build the evaluation. Label representative cases, implement the smallest read-only path, and agree on release gates. Compare with the existing workflow.
• Days 31–60 — run in shadow, then assist. First generate drafts without affecting live decisions; then pilot with a small group of consenting reviewers. Measure actual time, edits, and downstream outcomes.
• Days 61–90 — expand or stop. Increase exposure only when quality and cost gates hold. Assign an on-call owner, rehearse disabling the feature, and retain the manual queue.

Broader autonomy is a separate investment decision. Require explicit approval for consequential actions, scoped credentials, audit trails, and idempotency for retries. Expand permissions only when the evidence supports the additional failure exposure. Keep a rollback path that does not require the model to cooperate.

Your next leadership review should answer five questions: Which workflow improved? What evidence shows it? What does an accepted outcome cost? Who owns the failures? Under what conditions do we stop? Clear answers turn an impressive demo into an operating capability the business can trust.

Sources and reading notes

The research below supports the specific findings cited above. Architecture, cost assumptions, and the rollout plan are Hotpath’s proposed decision framework, not findings from those studies.

1. Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond. Generative AI at Work, NBER Working Paper 31161 (2023). The 14% figure refers to the original working-paper result; later versions may report revised estimates.
2. Joel Becker et al. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2025). A randomized study of experienced maintainers; its scope does not represent all software development.
3. OWASP GenAI Security Project. LLM01:2025 Prompt Injection. Application security guidance, not a productivity study.
4. NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (2024). A risk-management reference, not a certification.