How We Test LLM Features: Golden Sets, LLM-as-Judge, and Eval Gates in CI

Key takeaways

An eval is a test that runs a prompt against fixed inputs and scores the output on properties rather than on exact text. Evals replace exact-equality assertions for any code path whose output comes from a model.

A golden set is a version-controlled file of real inputs paired with the properties their outputs must satisfy. On our team, every production bug in an AI feature becomes one more row in that file.

Deterministic assertions — Zod schema validation, required substrings, forbidden strings, numeric ranges — catch most regressions, run in milliseconds, and cost nothing. We run them before any judge call.

LLM-as-judge is a second model call that scores the first model's output against a written rubric. A judge is only trustworthy when its model ID is pinned, its rubric asks binary questions, and its verdicts have been compared against human labels.

We gate CI on regression, not perfection: the build fails when the pass rate drops below a baseline committed to the repository, and raising that baseline is a reviewed commit.

At Devya we ship AI features into client products, and the failure that taught us this lesson was mundane. A route turned a free-text support message into a structured ticket and worked well. Weeks later an engineer edited the system prompt for an unrelated reason — a wording tweak in the tone instructions — and extraction started inventing order IDs for messages that contained none. Every test in the repository still passed, because the only tests covered the code around the model. Nothing in CI was reading what the model said.

Why can't we unit-test an LLM feature the normal way?

A large language model is a non-deterministic function, so asserting exact string equality fails for reasons unrelated to correctness. Setting temperature to 0 makes output far more stable but not byte-identical: providers batch requests and floating-point accumulation differs between batches, so no major provider guarantees reproducible tokens. A suite that asserts on exact strings is red on Tuesday and green on Wednesday, and teams learn to ignore it within a sprint.

Our approach is to split the code at the model boundary. Prompt assembly, retry logic, output parsing, tool dispatch, and every branch consuming the parsed result are ordinary deterministic functions that deserve ordinary unit tests. Only the single call crossing into the model needs an eval. Once that seam exists, the eval asserts on properties true of any correct answer: does it parse against the schema, does it stay in the requested language, does it cite only identifiers present in the input, does it refuse when the context contains no answer, does it ever echo the system prompt.

What is a golden set and where do the cases come from?

A golden set is a version-controlled file of real inputs paired with the properties their outputs must satisfy. Ours typically starts at roughly a dozen cases — one per input shape we can name — and grows every time production surprises us. We do not write synthetic cases when real ones exist, because the cases that catch regressions are the awkward real ones: the empty message, the message written in Arabic, the message that is only an order number, the message that is a prompt-injection attempt pasted out of an email.

Two storage rules have paid for themselves. First, keep fixtures as JSON or JSONL next to the prompt they test, so the case list is reviewable in a pull request. Second, keep the prompt in its own file rather than in a template literal inside a route handler, so a prompt change and its eval results appear in the same diff. That single move turns "someone edited the prompt" from an invisible event into a reviewable one.

In code, a Vitest suite loops over the fixture file and calls generateObject from the Vercel AI SDK with a pinned model ID, temperature 0, the committed system prompt, and a Zod schema describing the ticket. generateObject validates model output against the schema before returning, so a shape regression throws rather than flowing into an assertion. A second assertion loops over every returned order ID and requires that the ID appears in the original message — the grounding check that would have caught our invented-identifier bug.

When should we use a deterministic check instead of LLM-as-judge?

Use a deterministic check whenever the failure can be expressed as a predicate over the string, and reach for a judge only for properties requiring reading comprehension. Deterministic checks are free, instant, and never disagree with themselves; a judge is a second model call with its own error rate and its own bill.

Output shape and JSON validity: deterministic, via Zod schema validation. Schema validation is exact, so a judge could only be worse and more expensive.

Grounding — every ID, price, or quote appears in the input: deterministic, via substring or set membership. Hallucinated identifiers are string-detectable and are the most damaging class of error.

Forbidden content — system-prompt leakage, internal URLs, competitor names: deterministic, via a regex denylist. Safety properties must be exact, never probabilistic.

Language, length, and format constraints: deterministic. Word counts and Unicode script ranges are trivially computable.

Did the answer actually address the question: LLM-as-judge. This requires comprehension and no predicate expresses it.

Is the answer semantically equivalent to a reference answer: LLM-as-judge or embedding similarity. Many correct phrasings exist, so exact matching under-reports success.

How do we keep the LLM judge from drifting?

Treat the judge as production code with a version, not as an oracle. Pin the exact model ID rather than a floating alias, because a silent provider-side model change reads as a quality regression in your own product. Ask binary questions instead of scores from one to ten: "does the answer state a refund deadline, yes or no" is reproducible, while a helpfulness rating drifts and turns the threshold into an arbitrary number. Keep the rubric in a committed file reviewed like any other source.

Calibration is the step teams skip. Hand-label twenty to thirty outputs, run the judge over the same outputs, and inspect the disagreements. A judge never compared against a human is a number, not a measurement. Where budget allows we use a different model family for the judge than for the generator, because models tend to rate their own phrasing style favourably.

The judge returns a small object — a boolean for grounded, a boolean for answersQuestion, and a short reason string. The reason field is not decoration: when an eval fails overnight, a one-line explanation is the difference between a triageable failure and a red square nobody can interpret.

How do we run evals in CI without paying for every pull request?

Run evals in two tiers. Tier one is a fast subset on every pull request: all deterministic checks plus a handful of representative cases, with no judge calls. Tier two is the full set with judging, triggered nightly and on any pull request touching the prompt or fixture directories — expressed as a paths filter in the workflow rather than a human convention. A documentation-only pull request should cost nothing.

Three details keep tier one honest. Model calls are IO-bound, so cases run concurrently with a concurrency cap set below the provider rate limit. CI gets its own API key routed through a single gateway, so eval spend is a legible line item instead of a mystery inside the product budget. And genuine flakiness is expected: a case that fails once and passes on retry is a signal about output variance, so we retry a failed case once and fail the build only when it fails twice.

What pass rate should fail the build?

Gate on regression rather than on a fixed absolute score. A rule requiring 100% of evals to pass fails constantly on borderline cases and gets disabled within a month. We commit a baseline file holding the current pass rate per suite, and CI fails when the measured rate drops below that baseline by more than a small tolerance. Raising the baseline after a genuine improvement is a normal reviewed commit, which makes quality a ratcheting number living in git history rather than a claim in a standup.

Two categories are exempt from the statistics. Schema validity and safety checks — prompt leakage, invented identifiers, denylisted content — hard-fail at zero tolerance, because a single occurrence is a bug rather than a sample. Keeping those separate is what lets the fuzzy score stay fuzzy without anyone arguing that a leak is within tolerance.

None of this produces a perfect measurement of whether a feature is good. What it produces is a file of the ways the feature has been wrong, a build that turns red when it gets worse in one of those ways, and permission to refactor a prompt without holding your breath. For a non-deterministic feature, that is the whole win.

FAQ

Q: Do we need a dedicated eval framework to start?

A: No. Vitest or the built-in node:test runner plus a JSON fixture file covers the first several months. Tools such as Promptfoo, Evalite, and Braintrust become worth adopting once you want dataset management, run-over-run tracing, and a UI for reviewing judge disagreements.

Q: How many cases should a golden set contain?

A: Start with one case per input shape you can name, usually ten to twenty, and grow it from production failures. Coverage of distinct failure modes matters far more than the case count.

Q: Does temperature 0 make model output deterministic?

A: No. It sharply reduces variance, but no major provider guarantees byte-identical output because request batching changes floating-point accumulation. Assert on properties of the output regardless of temperature.

Q: Should the judge model be the same model that generated the answer?

A: Prefer a different model family when budget allows, since models tend to favour their own phrasing style. If the same model must be reused, calibrate the judge against human labels before trusting its scores.

Q: How do we stop eval costs from growing with the team?

A: Run deterministic checks first and skip judging any case that already failed one, restrict full runs to prompt-touching pull requests and a nightly schedule, and cache model responses keyed by a hash of prompt file, model ID, and input.