AI operating layer and platform
From AI Experiments to Production: What’s Required
Short answer
Moving AI from pilot to production takes five things beyond the model: data access that actually works, governance with defined limits, reliability when components fail, validation that the work completed, and measurement of the outcome. Scope should widen only as evidence accumulates, so human intervention falls because the record supports it and not because controls were removed.
Why do AI pilots stall before production?
Pilots stall because production removes the conditions that made them work. Pilots run on curated data, are operated by the people who built them, and have a person checking every output. In production the data is messy and spread across systems, the users are not the builders, and nobody can read every output.
So a pilot proves a capability. It does not prove that the capability survives real access, real scale, real exceptions and real accountability, which is where the work remains.
What does production require?
Five requirements separate a deployment from a pilot. Each is a property of the system around the model, and each can be checked.
- Usable data access: connections that are reachable and populated, not merely authorized.
- Governance: defined scope and limits, with approvals for exceptions.
- Reliability: failover between models and clear handling of failure.
- Validation: a check that the work actually completed.
- Measurement: evidence of the outcome, so automation can widen on proof.
1. Usable data access
The first production failure is usually data. For a real job the whole chain must hold: authorized, reachable, fetched or synchronized where required, available to the runtime with its source recorded, and used to produce the outcome. Verify the last link, that the job succeeded. Where sources disagree, the system needs a stated rule for which one wins and should show the disagreement.
2. Governance
Governance lets a company widen AI’s authority without widening its risk. It starts with scope, since every workflow belongs to one organization, and continues with limits enforced by code, approvals for exceptions, and sending a model only the minimum data a task needs. The NIST AI Risk Management Framework is a useful external reference: it is a voluntary framework for incorporating trustworthiness into the design, development, use and evaluation of AI systems.
3. Reliability
Models become slow, rate limited or unavailable. Routing calls through one path, ordering models from measured performance on your own workload, and failing over when one is unhealthy turns an outage into a slower path instead of a stopped workflow. Reliability also covers how failure is communicated: plain language about what did not happen and what to do next.
4. Validation
An action that cannot be confirmed is an assumption. Validation checks that the outcome occurred: the record was written, the message was sent, the software ships and runs. It is easy to skip and expensive to omit, because unvalidated automation accumulates silent errors.
5. Measurement
Measurement turns an automation into something a team can widen. Define the outcome and its start and end state first, record a baseline before the change, and keep observed measurements separate from estimates. A measurement with no defined completion point cannot be reproduced by anyone else.
How do you prove an AI workflow works, in order of strength?
“Built,” “deployed” and “configured” are not evidence that a customer receives value. UbiGrowth uses a ladder of proof, from weakest to strongest, and states which rung each change reached.
| Level | What it shows | Limit |
|---|---|---|
| 1. Unit and integration tests | The logic behaves correctly | Says nothing about production data or auth |
| 2. Controlled test tenant | The main path works against production-like data and authentication | Synthetic; not real usage |
| 3. Production runtime | Deployed behavior produces the intended system result | Still not customer value |
| 4. Natural customer activity | Real use shows delivered value | Slowest; measured continuously |
Customers are not your QA team
Do not ask customers to manufacture activity so that a team can close a work item. Natural customer behavior is the ultimate evidence and should be measured continuously, while controlled tests cover the paths that can be checked without it. Be explicit about what has not yet been observed.
Configured is not operating
A lesson from UbiGrowth’s own audits: configuration read from code defaults can disagree with live runtime state, so a capability can be labelled dormant when it is running, or running when it is not. Verify live state before drawing conclusions, and track every dormant capability with a decision, an owner and a date.
What should you measure once a workflow is live?
Measure whether the outcome was delivered, how much human intervention it needed, and why it failed when it did. Activity counts, such as messages sent or prompts run, say little about whether the job got done.
- Outcome delivered: did the result the workflow exists to produce actually occur?
- Human intervention: how often a person had to step in, and for what reason, so the trend can be tracked.
- Failures by cause: data access, a rule, a model, or an external system.
- Spend against limit: whether cost stayed inside the bound set for the workflow.
How do you earn less human intervention?
The goal is to need less human intervention as evidence accumulates, not to remove people on day one. A workflow proves itself on a narrow scope, the record shows it held, and the boundary widens. If evidence shows a workflow is not improving reliability, cost or customer value, narrow or stop it.
- The outcome has been delivered repeatedly, not once, and the record shows it.
- Exceptions were routed to a person and handled, and their causes understood.
- Failures were visible and explained in plain language to the people affected.
- The approval load fell because fewer cases needed one, not because approvals were skipped.
Who owns an automated workflow?
Every automated workflow needs a named owner and a defined path for exceptions. Without them the first unusual case either blocks the workflow indefinitely or is quietly handled in a way nobody reviewed. A good escalation path says who is asked, what context they receive, and what happens if they do not respond.
Cost deserves the same discipline. Spend, state and failure limits on each workflow keep a mistake small and visible.
What is a practical starting point?
Pick one outcome with a clear start and end state. Connect the systems it depends on and confirm the connections are usable, set the boundary, run it with the record on, and review the evidence before deciding what to widen. UbiVibe and ARIA are built around this progression: governed at the start, with the evidence that supports loosening controls when they can be loosened.
Frequently asked questions
Why do most AI pilots not reach production?
Production removes the conditions that made the pilot work: curated data, builder-operators, and a person checking every output. What remains to be built is the system around the model: data access, governance, reliability, validation and measurement.
What is the first thing to fix?
Data access. Confirm that each connection is reachable and populated, and that the job succeeds end to end, before investing in anything else.
How much human oversight should remain?
Start with more and reduce it as evidence accumulates. Widen scope when the record shows the workflow delivered repeatedly and exceptions were handled, not before.
Sources
- AI Risk Management Framework — NIST — Voluntary framework published by the U.S. National Institute of Standards and Technology for incorporating trustworthiness into AI design, development, use and evaluation.
- Beyond pilots: A proven framework for scaling AI to production — AWS — Another published framework on the same problem, for readers who want a second perspective. This article does not rely on its figures.