An AI prototype is impressive because you only ever see it succeed. You type the input you had in mind, the model answers well, and the demo lands. Production is the opposite: thousands of inputs you did not imagine, a provider that rate limits you at the worst moment, and a model that quietly changes behaviour the week after you ship.

This is a list of the ten things that actually have to be settled before an AI application carries real users. It is not about whether your code compiles. It is about the decisions that are expensive to reverse once you have traffic.

One thing to clear up first, because the two problems get confused. If your app was built by an AI tool like Lovable or Bolt and you need it hardened, that is a different job, and we wrote the longer AI prototype to production guide for it. This page is about an app that calls a model at runtime, whatever wrote the surrounding code.

Key takeaways

  • Write evals before you ship. Without them you cannot tell an improvement from a regression, and you will be changing prompts constantly.
  • Pin the model version. Providers update models, and "the same prompt" stops meaning the same thing.
  • Put a hard cost ceiling per user and per day in front of every model call. Token spend is the failure mode nobody budgets for.
  • Decide what happens when the provider is slow, rate limited or down. That path runs more often than you think.
  • Know exactly what user data leaves your infrastructure, and be able to answer that in one sentence.

1. Evals, before anything else

The single biggest difference between a prototype and a production AI application is that the production one can be measured. If you cannot say whether a prompt change made things better or worse, you are not iterating, you are guessing.

You do not need a framework to start. You need a file of real inputs and the output you would accept.

  • Collect 30 to 50 real inputs, including the ugly ones: empty, enormous, off topic, hostile, in the wrong language.
  • For each, write down what a good answer looks like, or at least what would be unacceptable.
  • Run the set on every prompt or model change and compare. Even a manual pass over 40 cases beats shipping blind.
  • Keep every case a user complained about. Your eval set should grow every time something goes wrong in production.

Automate the grading later, with string checks for the mechanical cases and a model-graded rubric for the subjective ones. Start by just having the set.

2. Pin the model version

If you call gpt-4o or claude-sonnet-4 without a dated version, you have given a third party permission to change your product's behaviour without telling you. Providers do deprecate and update models.

  • Call a pinned, dated model identifier in production.
  • Keep the model id in configuration, not scattered through the code, so you can change it in one place.
  • When you move to a new version, run your eval set against it first and treat it like any other dependency upgrade.
  • Subscribe to the provider's deprecation notices and diary the end-of-life dates.

3. A cost ceiling, enforced in code

Token spend is unlike any other infrastructure cost: a single user with a loop can cost more in an hour than your hosting does in a month. Budget alerts tell you after the money is gone.

  • Cap the tokens per request, in the request itself, not by convention.
  • Rate limit model calls per user and per IP, separately from your general API limits.
  • Set a daily spend ceiling that degrades the feature rather than letting it run. A disabled feature with a clear message beats a surprise invoice.
  • Log the token count and cost of every call with the user id attached, so you can see who and what is expensive.
  • Cache aggressively. Identical or near-identical prompts are more common than you expect, and a cache hit is free and instant.

4. Latency, and what the user looks at while waiting

Model calls are slow by the standards of everything else in your stack. A request that takes eight seconds is normal and is also long enough that a user assumes it is broken.

  • Stream the response if the output is text a human reads. Perceived latency matters more than total latency.
  • Put a timeout on every call. Decide the number deliberately and make it shorter than your users' patience.
  • Move anything that does not need to be synchronous into a job queue, and tell the user where it is.
  • Measure the slow end, not the average. The 95th percentile is the experience people complain about.

5. The failure path

In a prototype, a failed model call is a stack trace you shrug at. In production it is a user watching a spinner. Providers return 429s, time out and have outages, and your application has to have an answer for each.

  • Retry with exponential backoff and jitter, on the right errors only. Never retry a call that is failing because the input is invalid.
  • Have a fallback: a smaller or different model, a cached answer, a simpler non-AI code path, or an honest message. Choose on purpose rather than defaulting to a 500.
  • Make retries idempotent so one user action cannot produce three charges or three records.
  • Consider a second provider for anything business critical, and keep the integration behind one interface so swapping is a config change.

6. Wrong answers, and what they are allowed to touch

Models produce confident wrong answers. That is not a bug you will fix, it is a property you design around. The question is what a wrong answer can reach.

  • Ground answers in your own data where facts matter, and show the source so the user can check.
  • Give the model an explicit way to say it does not know, and reward that path in your evals.
  • Never let a model output reach an irreversible action, money moving, email sending, data deleting, without either a deterministic validation or a human confirming.
  • Validate structure: if you expect JSON matching a schema, parse and validate it, and have a defined behaviour for when it does not match.
  • Put the stakes in writing. For a draft suggestion, a wrong answer is friction. For anything medical, legal or financial, it is a liability, and the design has to reflect that.

7. What data leaves your infrastructure

You should be able to say, in one sentence, what user data is sent to which model provider and how long they keep it. If you cannot, that is the task.

  • Write down exactly which fields go into a prompt. Strip or tokenise what does not need to be there.
  • Check the provider's retention and training terms for the specific tier you are on, and get a data processing agreement if you have business customers.
  • If you handle anything regulated or special category, confirm the legal basis before launch, not after a customer's security review.
  • Update your privacy policy to name the use of a third-party model provider. This is routinely forgotten, and it is the easy part.
  • Be careful with logs. Prompt and output logging is the most useful debugging tool you have and also the place sensitive data quietly accumulates.

8. Prompt injection and tool permissions

If your application puts anything a user or a web page wrote into a prompt, that content can try to issue instructions. If the model can also call tools, the instruction can have consequences.

  • Treat all retrieved and user-supplied content as data, never as instructions, and say so in the system prompt.
  • Give tools the narrowest permissions that work. A read-only key is not a compromise if reading is all the feature needs.
  • Put the authorisation check on the tool, in your own code, based on the session identity. Never let the model decide who it is acting for.
  • Keep a human confirmation step in front of anything destructive or outbound.

9. Observability built for non-determinism

Normal application monitoring tells you a request returned 200 in 300ms. For an AI feature that tells you almost nothing, because a 200 can contain a useless answer.

  • Trace each call with its prompt, model, version, token counts, latency, cost and the output.
  • Give every generation an id and surface it in the UI, so a support conversation can start with the exact exchange.
  • Add a thumbs up or down, and actually read what comes back. It is the cheapest source of eval cases you will ever get.
  • Alert on the things that move before users complain: refusal rate, empty outputs, schema validation failures, average cost per request, timeout rate.

10. The rollout, and the way back

Prompt and model changes are code changes with unusually wide blast radius, because one edit can alter every answer the product gives.

  • Put prompts in version control, reviewed like code. A prompt in a database field that anyone can edit live is an outage waiting to happen.
  • Ship behind a flag and roll out to a slice of traffic first.
  • Keep the previous prompt and model one config change away, and make sure someone who is not you knows how to do it.
  • Write down what "worse" looks like before you roll out, so the decision to revert is not a debate.

Where teams actually get caught

If you only have time for three of these, do evals, the cost ceiling and the failure path. Those three are what separate a demo from something you can leave running over a weekend.

The ones that cause the most expensive surprises later are the data questions. Changing a prompt is an afternoon. Discovering during an enterprise security review that you have been sending customer records to a provider on a consumer tier, with no agreement in place and no record of what was sent, is a quarter.

FAQ

How is this different from normal production readiness?

Everything you already know still applies: auth, input validation, backups, monitoring, a deploy pipeline. An AI application adds four properties that ordinary software does not have. It is non-deterministic, so the same input can give different output. It has a variable per-request cost. It depends on a third party you cannot debug. And it can be wrong while looking completely confident. The ten items above exist because of those four properties.

Do I need an eval framework before launch?

No. You need an eval set. A file of real inputs and what you would accept as an answer, run by hand before each change, gets you most of the value. Reach for a framework when running the set by hand becomes the thing stopping you from shipping, which usually happens somewhere past a hundred cases.

Should I fine-tune before going to production?

Usually not. Prompting, retrieval over your own data and a better model are faster to try and far easier to reverse. Fine-tuning makes sense once you have a stable task, a real dataset and evals good enough to prove the tuned model is better. Doing it before you have evals means you cannot tell whether it helped.

How much should I budget for model calls?

Measure instead of guessing. Take your eval set, run it against the model you intend to ship on, and record the token counts. That gives you a real cost per request, which you multiply by your expected volume. Then set the ceiling above that number and alert well below it. The point of the exercise is not the forecast, it is knowing what one user costs.

What if the prototype was built by an AI tool as well?

Then you have both problems at once: the generated code needs the usual hardening, and the model calls inside it need everything on this page. Start with the code, because an authorisation gap hurts you faster than a missing eval set. The AI prototype to production guide covers that side, and auditing a vibe coded app before launch is the hands-on version.