Your AI system needs an evaluation suite before it needs a better prompt
Prompt engineering without measurement is guessing with extra steps. The suite is what turns a demo into something you can safely change.
Most AI systems that fail in production do not fail because the model was not good enough. They fail because nobody could tell whether a change made things better or worse, so the system drifted — one plausible-sounding tweak at a time — until it was quietly worse than the version that shipped.
The fix is unglamorous. Before improving a prompt, build the thing that tells you whether the improvement is real.
The suite comes from your cases, not a benchmark
Public benchmarks measure something adjacent to your problem at best. What you need is thirty to a hundred real cases from your own domain, with known correct answers, including the ones that went badly. On the incident-response work, ninety past incidents with confirmed root causes became the regression suite. On the document pipeline, every correction a human reviewer made was stored against the original document — the reviewers were building the test set as a side effect of doing their jobs.
That second pattern is worth stealing. If your system has a human checkpoint, the corrections made there are the highest-quality evaluation data you will ever get, and you are probably throwing them away.
Structured output makes grading possible
You cannot reliably grade prose. If the system returns free text and something downstream parses it, you have two failure modes tangled together and no clean way to score either. A strict schema, validated on the way out, gives you a field-by-field comparison against known-correct values.
It also removes the worst failure mode in these systems: the confident invention. A schema violation is an exception you can route to a human. A plausible-looking wrong number in a paragraph is a liability that ships.
Score the things that actually cost you
- Exact-match accuracy per field, not an aggregate over the whole document.
- Calibration — when it says it is confident, is it? Overconfidence is more expensive than error.
- Refusal quality — does it say "I do not know" on the cases where it should?
- Cost and latency per case, because a 3% accuracy gain for 4× the spend is a decision, not an improvement.
A confident wrong answer during an incident is worse than silence. The suite is how you find out which one you are shipping.
Run it on every change, automatically
The suite only works if it runs without anyone deciding to run it. Wire it into CI. Any change to a prompt, a tool definition, a model version or a schema fails the build if it regresses. This is the same discipline you already apply to code, and it is startling how often teams that would never merge an untested function will happily edit a prompt in production.
On the document pipeline, straight-through processing climbed from 81% to 94% without a single prompt change — the gains came from three schema revisions the suite pointed at. That is the actual return: not a better prompt, but knowing where the problem was.