Build an eval gate for an LLM application

An eval gate runs your quality checks on a fixed set of known cases before every release, and stops the release when a number falls below its threshold.

A change to one prompt can make an LLM application worse in ways no unit test notices. An eval gate is how you find out before your users do.

What an eval gate is, and what it protects

LLM evaluation, often shortened to evals, means measuring what a system built on a language model produces against cases whose right answers you already know. The gate is the step that turns those measurements into a decision: the release goes out, or it does not.

The problem it solves is quiet regression. Model output is not fixed. A new model version, a reworded prompt or a refactor of the code around the model can each change the results, and the change rarely announces itself. If your application also has a safety check, such as a filter that catches invented content, that check can regress too: it starts missing things, or it starts flagging things that are fine. Both kinds of failure look like a normal day until someone complains.

A useful gate measures two sides of every check. Precision asks whether it stays quiet on good output. Recall asks whether it catches bad output. A checker that flags nothing has perfect precision and is useless, and one that flags everything has perfect recall and is just as useless.

Precision: a golden set you checked by hand

A golden set is a small collection of real outputs from your own system whose facts a person has checked by hand. It is the reference the gate compares against.

Three habits make one trustworthy:

• Use real output. Take results from live runs on real inputs, not examples you wrote to look tidy.
• Write the checked facts down. Next to each output, keep a list of facts it must contain and a list of wrong things it must never say. The wrong things are often the most useful part: they record mistakes you have already seen.
• Freeze the input. If the input can change, the scores cannot be reproduced. Pin it by content hash, not by a branch name or "latest".

With that in place, every case can be rerun offline and must give the same verdict, and the same score, as the day it was recorded.

Recall: plant the errors yourself

Precision on good output is half the picture. To measure recall you need bad output whose mistakes you know exactly, and the simplest way to get it is to make it: take golden outputs and plant errors in them. For a documentation tool, that means invented file names, invented methods on real classes, invented members of standard-library modules.

Add a guard that every planted error really is wrong. If a planted function name turns out to exist in the repository, a test can pass by accident. Then require each plant to be caught and to lower the score.

Be honest about one limit. Errors planted by the people who wrote the checker tend to be the errors the checker already handles. That number is useful for catching regressions, but it is not a measure of how the checker will do on mistakes nobody has seen yet.

Honest recall: blind, held-out rounds

For that, ask someone who has not read the checker to write the errors. This is a held-out set: data kept away from the people tuning the system, so the result is not flattered by what they know. Ask the same tester for real names that must not be flagged, so false alarms are measured on the same footing.

Expect the first blind round to hurt. It will find bugs your own plants never touched. Fix them, keep the misses visible as known failures, and when the number stops moving, run a second round written by someone new. The fresh round will usually score lower than the old one. That lower number is the honest one, because it is closest to what the next model's mistakes will meet, and it is the one to publish.

Turn the numbers into a gate

The gate itself is a short script. It runs the production checker over the golden set, the planted errors and the blind rounds, compares the results with a file of thresholds, writes a scorecard and exits non-zero on any regression. A thresholds file can be as small as this:

{
  "golden_false_alarms_max": 0,
  "planted_recall_min": 1.0,
  "heldout2_recall_min": 0.8326,
  "heldout2_false_alarms_max": 0
}

Four details make the result something you can trust:

• Measure behaviour, not test markers. Count what the checker actually caught, so editing a skip marker cannot hide a regression.
• Ratchet the thresholds. When a metric beats its threshold, raise the threshold in the same change. Never lower one without writing down why.
• Test the gate. Build a deliberately broken checker, one that stops checking a whole kind of name, and require the gate to fail on it.
• Keep it fast. A gate that takes minutes stops being run by hand. Offline checks with no model calls can run in seconds.

A worked example: one checker, three numbers

Nehemvix writes documentation guides from Git repositories and checks every file path and code name a guide mentions against the code. How that checker works is in catching hallucinated names. Here is how its gate is built.

The golden set is 12 guides from live runs in October 2026: five repositories in four languages, three models and three audiences. One fixture records must_contain facts such as "HMAC" for the signing library itsdangerous, and must_not_contain mistakes such as "SHA-256 by default", because its default digest is SHA-1. Each repository is stored so its files are byte-identical to what the model read. Today all 12 reproduce their scores exactly, with no false alarms.

The authors' own plants are all caught. The first blind round was a harder lesson: recall on names written blind started at 80.2%, and the round found real bugs, including invented root files such as main.py that were never checked at all. After two rounds of fixes it reached 94.3%. A second blind round, written by a reviewer forbidden to read the checker or the first round's misses, came in lower at 83.3%, with none of 376 real names wrongly flagged. That second number is the one Nehemvix publishes, and the full method and results are on how every guide is checked.

The gate once proved its own speed rule. An early version of one scan backtracked on nested patterns, and the eval suite took 65 seconds. A linear check brought it down to 1.5 seconds.

Where it runs

The same checks run at several points, and only some of them block anything:

What to copy: a checklist

• Freeze inputs by content hash.
• Keep hand-checked facts, and known mistakes, next to each golden output.
• Plant errors and check that each plant really is wrong.
• Separate your own plants from blind rounds, and publish the blind number.
• Count false alarms, with a threshold of zero.
• Raise thresholds in the same change that beats them.
• Test the gate with a deliberately broken checker.

One thing Nehemvix leaves out is an LLM judge, a second model asked to grade the first. Run on every guide it would bill every user, and its verdict would be another model's opinion. That is why the score a reader sees is labelled "structure and grounding, not factual correctness". If you add a judge, calibrate it against your hand-checked facts first.

We run this gate as one stage of AELSHA™ ASTRA, our framework for taking agents to production.

Related reading: testing for prompt injection in repositories covers the live side of the same discipline, and agentic AI examples shows where these numbers sit in one whole system.

Learning this to build your own agent? AELSHA™ can help you design it.