The Architecture Checks Before an AI Pilot Goes to Production

Why AI Pilots Fail to Reach Production is about the organisation: who owns the system, what an error costs, and whether anyone is allowed to say no. This piece is the architecture side of the same gap: the checks on the system itself.

A pilot is usually judged on whether the model gives good answers to good inputs. Production asks something else. What happens with inputs nobody planned for, at a volume nobody tested, with data the pilot never touched? Those answers sit in the architecture around the model, and they are cheaper to settle before an AI pilot goes to production than after it does.

Seven architecture checks, in the order they tend to come up.

1. Use the model only where judgement is needed

Many steps in a workflow are lookups or fixed classifications. A rule or a database query handles them faster, cheaper and the same way every time. Route those first, and send the model only the cases that genuinely need interpretation. A system that sends everything to the largest available model pays for judgement on work that needed none.

2. Check every output before anything acts on it

If a model's output feeds another system, it should be checked against a defined format before it is used: the fields that must be present, the values that are allowed. Free text passed straight into an application fails quietly, and the failure shows up somewhere downstream, far from its cause.

3. Keep writes behind a person until the system has earned trust

Reading data is recoverable. Changing it is not always. For any step that writes to a system of record, sends something to a customer or moves money, put a human approval in front of it at launch. Remove the approval step by step, once the logs show the error rate is one the business accepts.

4. Know where the data goes, and for how long

Draw the path: what goes into the prompt, what the provider retains, what ends up in logs and embeddings, and for how long. An enterprise buyer's security team will ask for exactly this, and a vague answer can stall a deal the product had already won.

5. Log enough to reconstruct any answer

For any output, you should be able to say which prompt version produced it, what context was retrieved, what it cost, and who saw or approved it. Without that, a complaint cannot be investigated and a fix cannot be verified.

6. Decide what happens when it is wrong

Every system needs a defined path for a bad answer: block it, fall back to a rule, or route it to a person. Decide that before launch. Deciding it after the first incident means deciding it under pressure, with a customer waiting.

7. Cost it at ten times today's volume

Token and compute costs that look trivial in a pilot scale with usage, and sometimes faster, as prompts grow and context gets longer. Model the cost at current volume and at ten times it. If the second number breaks the business case, the architecture has to change before the traffic arrives.

Where these fit

None of these is exotic. Each one is a decision someone has to make and write down. A pilot runs on sample data, at low volume, with nobody waiting on the answer, so nothing forces the decision. Production does.

The AI Pilot to Production checklist puts these checks next to the ownership, money and security questions, as one gate you can run yourself before funding the next phase.