A traditional function returns the same output for the same input, so you write a test and move on. A language model feature gives slightly different answers each time, and a small prompt change can improve one case while breaking five others. Teams without a testing process end up shipping on vibes: someone tries a few questions, the answers look fine, it goes live.
We treat LLM features like any other code that needs a test suite. The suite just looks different.
Build an evaluation set from real inputs
We collect examples of what users will send: real support questions, real documents, real search queries. Synthetic examples fill gaps, but real ones catch the odd phrasing and messy formatting that break things. For each example we write down what a good response must contain or must not contain.
A starting set of 100 to 300 examples covers most features. It grows every time someone finds a failure in production.
Score with checks, not only opinions
Some checks are plain code:
- Does the output parse as the JSON schema we expect?
- Does it cite at least one source, and does that source exist?
- Does it stay under the length limit?
- Does it avoid banned content, such as prices we don't offer or competitor names?
Others need judgement: is the answer correct and does the cited passage support it? For those we use a second model as a grader with a strict rubric, and we check the grader against a sample of human ratings. If the grader disagrees with people too often, we fix the rubric before trusting its scores.
Run it on every change
The evaluation runs in CI whenever someone changes a prompt, swaps a model, changes retrieval settings or updates a tool. The report shows the overall score and, more usefully, which examples got worse. A change that raises the average while breaking a critical case doesn't ship.
Model providers update models too. We pin model versions where possible and run the full set before moving to a new version.
Test the unhappy paths
Our evaluation sets include cases designed to fail gracefully:
- Questions the system has no information about
- Attempts to make it ignore its instructions
- Inputs in other languages, or full of typos
- Requests for things it shouldn't do, like giving legal advice
The expected answer for many of these is a polite refusal or a handoff. Those are easy to break when you tune the prompt for the happy path.
Keep watching after launch
Evaluation doesn't stop at release. We log inputs and outputs (with personal data handled according to the client's policy), sample them each week for human review, and add every failure to the test set. After a few months, the set reflects what users do rather than what we guessed they'd do.
Why it's worth the effort
The first evaluation set takes a few days to build. After that, each change takes minutes to verify instead of an afternoon of manual checks, and you can switch models or prompts with evidence instead of hope.
