How we check every guide
Measured, not assumed. Every guide is checked against the code it describes, and we publish how well the checks work.
- 12/12
- reference guides, with hand-checked facts, reproduce exactly
- 94.3%
- of planted names caught in a blind round
- 83.3%
- caught in a second, harder blind round
The grounding score
Each guide gets a score from 0 to 100 for how well it sticks to what is really in the code: whether the files, functions and classes it names exist, whether its citations resolve, and whether its required sections are present. It checks names, paths and citations, not whether every explanation is right, and the guide says so.
Names we couldn't find
When a guide mentions something that isn't in the repository, Nehemvix rewrites that section once with the list of unverified names. The rewrite is kept only if it has fewer of them. Whatever remains is listed at the end of the guide, so you know exactly what to double-check.
How well the checks work
We test the checker itself, the way you would test any safety system:
- Reference guides: real guides from 5 repositories in 4 languages, with facts a person checked by hand. Each one reproduces its score exactly, with no false alarms.
- Names we plant ourselves: every invented name we plant into a real guide is caught.
- Blind rounds: a separate reviewer, who never saw the checker, planted the names. These rounds are harder, and they are where the two percentages above come from. The second covered all five repositories, and not one real name was flagged as invented.
- Prompt injection: in our live tests, repositories containing instructions aimed at AI tools were documented without the model obeying any of them, and the guide lists that text for you.
These checks run before every release, and a regression in any of them stops the release.
Numbers like these are only as good as the method behind them, which is why we wrote about what to ask of any agent score, ours included.
How they are built, step by step: catching hallucinated function and file names, building an eval gate for an LLM application and testing for prompt injection.
Answers to your questions
Answers in the question panel are checked the same way: paths and names in an answer are looked up in the code, and anything unverified is shown with the answer.