A demonstration answers a narrow question: can a model do something useful with a prepared example? Production asks whether the whole organisation can rely on the resulting workflow on an ordinary day, including the awkward cases.
That distinction matters because a model is only one part of the system. The work begins with a trigger, crosses permissions and source systems, makes or supports a decision, changes something, records what happened and ends with a person or system responsible for the result. A deployed endpoint may sit inside that path. It is not the path.
Google Cloud makes the same engineering distinction in its MLOps architecture guidance: the difficult production problem is operating the integrated system across testing, release, deployment, infrastructure and monitoring, not merely building the model. The guidance is written mainly for predictive systems, so we use the distinction without pretending it defines every generative AI workflow.
The workflow is the unit
Name the workflow from trigger to finished outcome. A useful description can answer seven things without mentioning a model:
- What starts the work.
- Which inputs are authorised.
- Which decisions the system may make or support.
- Which systems it may read or change.
- Which exceptions leave the normal path.
- What a finished outcome looks like.
- Who owns the result.
If those answers are still broad, the production boundary is broad too. “Finance automation” has no testable edge. “Flag a likely duplicate supplier invoice for a named approver before anything posts” does.
Real path
The workflow must run through the systems and permissions it will use in real work. That includes the actual input shape, the real source of truth, the intended write effect and the human control around that effect.
Sample data can prove that a transformation is possible. It cannot expose an expired permission, an unexpected document type, a rate limit, a duplicate event or a write that the receiving system treats differently from the staging copy. Those seams belong in the production test because the organisation experiences the whole path, not the model call.
“Real” does not mean unrestricted. A workflow may begin read-only, write into a review queue, or operate on a limited class of records. The boundary should be deliberate and visible. Narrow authority is often the thing that lets useful work ship safely.
Relevant evidence
Evaluation should represent the decisions and material failure modes that matter in the workflow. NIST’s voluntary AI Risk Management Framework calls for testing in conditions similar to deployment and for behaviour to be measured during operation. OpenAI’s current evaluation guidance recommends task-specific cases drawn from the real distribution, human calibration where judgement is involved, and evaluation on every change.
Neither source says a passed test proves future correctness. Evidence is bounded by its cases, criteria, graders and similarity to production. A useful eval record therefore says what it covers and what it does not. When a new failure appears in operation, the represented case set should grow.
The practical question is not “what is the score?” It is “which decision does this evidence support?” A release threshold, a human-review rule and a rollback decision may each need different evidence.
Operable failure
An operated workflow has a response when it refuses, degrades, errors or exceeds its boundary. That response may be to stop, queue, retry, revert, ask for approval or send the work to a person. “Unknown” is an honest state when a response has not yet been designed.
NIST’s framework treats monitoring, incident response, recovery, change management, override and deactivation as connected operating concerns. It does not promise that every failure can be anticipated. The production test is whether the known paths have named responses and the unknown path has a safe place to land.
This is where a runbook becomes part of the system rather than a document written after it. The operator should be able to move from a visible signal to containment, escalation and recovery without finding the original builder first.
Owned change
Deployment, telemetry, access and change control need accountable owners. One person does not have to perform every task, but the business and technical responsibilities cannot be implied.
The owner should know which version is running, what changed, which tests covered it, where the production signals appear and who can stop or roll it back. A dashboard without an owner is decoration. A release process without a refusal path is a suggestion.
The delivery-assurance mechanism in our work register illustrates the distinction. Its receipts are produced inside the delivery path and bound to the change they cover. A missing, expired or mismatched stage can deny the merge. The mechanism proves that the recorded process ran. It does not prove that the process was good, that no defect exists or that an authorised person could never disable the gate.
Durable ownership
Production responsibility has to survive the delivery team. The receiving organisation needs the repository, deployment route, controlled access, telemetry, evaluation harness, runbook and people who have practised using them.
Google’s account of SRE onboarding and responsibility transfer combines production-readiness review, documentation, hands-on training, access rights and progressive transfer of operational responsibility. That is Google’s service-operating practice, not a universal handover standard. It supports the underlying point: source code alone is not an operating capability.
At Esya, durable ownership is an acceptance condition. The work is not complete because a repository changed hands. It is complete when the receiving team can run the workflow, interpret its evidence, respond to failure and make the next change.
A production board for one meeting
Use this board against one candidate. Mark evidence that exists, work that is promised and gaps that are still unknown. The first missing condition is the next planning decision.
| Condition | Evidence to inspect | A gap sounds like |
|---|---|---|
| Real path | Authorised inputs, actual integrations, bounded write effect | “It only runs on sample data.” |
| Relevant evidence | Task cases, failure cases, criteria, reviewer and known gaps | “It looks right in the demo.” |
| Operable failure | Refusal, retry, fallback, escalation and recovery paths | “Call the builder if it breaks.” |
| Owned change | Deployment, telemetry, access and release authority | “Engineering will watch it.” |
| Durable ownership | Repo, eval harness, runbook, controlled access and trained team | “The code is in your account.” |
Five present conditions do not certify a workflow as safe or reliable. They give the owner enough structure to decide whether real work should cross the boundary, which risks remain and who is accepting them.
Sources and limits
This is Esya’s practical production test, not a definition issued by a standards body. It draws on NIST’s voluntary AI RMF 1.0, Google Cloud’s integrated-system view of MLOps, Google’s primary research on testing production ML systems and OpenAI’s task-specific evaluation guidance. None guarantees that a compliant page, process or workflow will perform correctly in the future.
The five-condition plate is an Esya explanatory reconstruction. It contains no client artefact or client result.