# From 116 to 118 of 120: why I threw away the fix that scored perfectly > A company knowledge assistant that only quotes its documents reached 118 of 120 (98.3%) on its live test. The first fix repaired all four misses on replay and was discarded anyway, because it only worked for this set of documents. - Author: Wasim Jalali - Published: 2026-09-25 - URL: https://wasimjalali.com/writings/from-116-to-118-of-120-the-fix-i-threw-away/ [Useful Brain](https://github.com/wasimjalali/useful-brain) is a question-answering assistant I built for company documents: handbooks, policies, plans. An employee asks a question and gets an answer made only of sentences that actually appear in those documents, each with a citation, or a plain "not enough evidence" when the documents do not say. After the [repair work in August](/writings/from-72-to-95-repairing-a-grounded-rag-agent/) it scored 114 of 120 on its live test. By September the misses that kept coming back had one shape: the assistant found the right document and then quoted a lookalike sentence from the document next to it. This post is about the day I fixed that twice. A fresh run on the morning of 6 September scored 116 of 120. The first fix I wrote repaired all four misses on replay, and I deleted it, because it worked by matching this corpus's wording and would not have survived a new set of documents. The fix I kept is smaller and more general, and took the run to 118 of 120 (98.3%). The honest framing: runs of this system land anywhere between 116 and 120, so treat the two-point move as the category totals moving, not a pinned number.

One morning, two fixesNorthwind battery, 120 locked questions, GLM 5.3 Flash

118/12098.3% with the general fix, the strongest run on this battery

The fix that scored 4 of 4 on the misses was the one thrown away. Strength over score.
## What was tested The test, an eval in the jargon, is a fixed set of 120 questions with known right answers and known right documents, scored automatically: 70 factual questions, 17 traps where a plausible wrong document sits next to the right one, 13 permission cases where the answer lives in a document the asking person cannot read, 10 unanswerable questions and 10 multi-hop questions whose answer needs two documents. The corpus is 65 synthetic company documents with access rules. Every scoring rule is the same as in August and none was loosened: every sentence must be a verbatim span of a retrieved document, a multi-hop answer must cite both documents, and a permission or unanswerable question must come back as `insufficient_evidence`. The model was GLM 5.3 Flash on Cloudflare Workers AI, the one chosen in [the model bake-off](/writings/picking-a-model-for-a-grounded-rag-agent/). Retrieval, the step that finds candidate passages, was locked for the whole of this work. Only the answer layer, the code between retrieval and the validator that checks every sentence, was allowed to change. ## Four misses, one shape The fresh run's four misses were one problem wearing four outfits. In each, retrieval delivered the gold documents, the ones the answer should cite. The draft then cited a neighbouring document instead: the handbook restating a rule the dedicated policy owns, or a page that mentions a programme it does not define. Company documents do this constantly. A retention section says "request earlier deletion through the Customer Data Access Requests process". A recruiting page says "the hiring bonus is a different program". The sentence in front of the model names the document that owns the answer, and the model, having found a sentence that sounds right, stops there. I started calling these pointer cases: the evidence itself points somewhere else.

A pointer caseWhat the model saw on each of the four misses

Both documents were in front of the model. The cited one only points at the answer; the uncited one holds it.
## The fix that worked and got deleted My first fix was host-side sentence grafting. When a draft cited a neighbour and left an owner document uncited, the host itself picked a sentence from the owner, choosing by how many words it shared with the question, and grafted it into the answer before validation. I replayed the four failing questions and it repaired all four. On the scoreboard it was finished. It was also wrong, and I knew it the moment I read the selection rule back. Word overlap picks the right sentence in this corpus because these 65 documents were generated with consistent phrasing: the question and the owning sentence tend to share vocabulary. Point the same code at a real company's documents, where the question says "days off" and the policy says "annual leave entitlement", and the graft picks a wrong sentence with the same confidence. Worse, a wrong graft would arrive already stitched into an answer that reads fluently. The whole reason this assistant exists is that a confident wrong answer about a leave policy is worse than no answer. So the grafting went in the bin, with its 4 of 4. The report records it under "deliberately not done". The lesson I keep relearning on this project is that a fix that fits the test is a fix that fits the test, and the number it produces is not evidence of anything except that. ## The fix I kept The kept design splits the job in two. The host does the part that can be done deterministically and without any knowledge of the corpus: noticing that the evidence points at an uncited document. The model does the part that needs judgement: deciding whether that document actually answers the question, and in which sentence. Three changes, all in the answer layer, shipped as prompt version `grounded-answer.v9`. **Pointer detection.** A small module reads the question, the draft and the sentences the draft cited, and looks for uncited evidence documents named by their title words, plus documents whose text explicitly contrasts itself with a cited one ("is a different program", "separate from", "not interchangeable", "does not replace"). It contains no question ids and no corpus vocabulary. It selects no sentences. Its only output is a hint: these documents were named, take a second look. **A wider coverage pass.** The assistant already had a coverage pass, a second model call that asks, for multi-part questions, for the exact evidence sentence answering each still-open part. It now also runs for single-part drafts that carry a pointer hint, with a stronger instruction about telling similar programmes apart. That is one extra model call, only when the evidence itself points elsewhere. As before, the host keeps an addition only if it re-validates against the evidence ledger, so the model cannot smuggle in a sentence that is not there. **Two prompt rules, and a bug.** The chat prompt now says that when the evidence calls two programmes different, each must be answered from its own document. And while wiring the hints I found that single-word titles (`privacy-policy.md` reduces to one distinctive word once "policy" is dropped as a stop word) could never trigger a coverage hint under the old rule, which demanded two matching title words. The threshold is now the smaller of two and the number of words the title has.

The answer pipelineWhere the September change landed

  1. Hybrid retrieval and access filterLockedKeyword, vector, rerank. Not touched in this work
  2. Model draft with search toolPrompt v9Programmes the evidence calls different are answered each from its own document
  3. Pointer detectionNewUncited documents named by title, or contrasting themselves with a cited one. Detection only; no sentences chosen
  4. Coverage passWidenedWas multi-part questions only. Now also single-part drafts with a pointer hint. Additions must re-validate
  5. Refusal guard, then verbatim salvageUnchangedAs left after the August review
  6. Grounding validatorLockedEvery sentence verbatim and cited, or insufficient_evidence
The host gained one deterministic step that points; the model kept the step that decides.
## 116 to 118 The full battery ran again with the kept fix. Factual, trap and permission questions did not move, because they were already at or within one of their ceilings. Two categories moved: unanswerable from 9 to 10 of 10, and multi-hop from 8 to 9 of 10.

Before and after, by categoryHollow is the fresh 116 run, filled is the run with the kept fix

+2questions, both in the categories that held the pointer misses

Three categories were already at or one short of their ceiling and stayed there. The gain is in the two that moved.
Two misses remain, and neither has the pointer shape. `q093` is an identifier lookup where the assistant refused with the right document retrieved. `q120` is an expanded multi-hop question whose answer cited the complaint escalation document and dropped the service-level credit policy. I re-asked both, and both passed. That is run-to-run variance in a reasoning model, the same churn the August post warned about, and it is why the honest claim is a band of 116 to 120 rather than 118.

The kept fix, one cell per question118 of 120 passed

2misses remain, and both pass when re-asked

The recorded run. Two misses remain: q093 and q120. Compare the six in the August post, four of which failed for every model tested.
## What stayed constant Retrieval was locked, and the numbers say so. The deterministic retrieval suite, which runs beside every live pass as a regression guard, held at exactly its August values. On the live stack, retrieval delivered the gold documents for 99.5% of questions (97.4% in the August run), with zero forbidden-document retrievals and zero degraded turns.

Retrieval, unchangedDeterministic suite, identical to the August values

The same four numbers as the August post, to three decimals. Every gain in this post came from the answer layer.
## What this does not show - The 116 is a fresh run, not a measured change from the August 114. Between the two, the answer layer and retrieval went through another round of changes, recorded in the repo's execution tracker rather than in an eval report, and none of those runs is frozen. The 6 September morning run of that system scored 116; that is the number this fix was measured against. It has no frozen snapshot either; its category counts are recorded in the report's table. Only the 118 run is frozen. - Both residual misses pass on replay, which is evidence of variance, not of a fix. Expect 116 to 120 across runs and read category totals, not single questions. - Complete-answer latency was p50 22.9s and p95 58.9s. The report records that as comparable to August (p50 15.6s, p95 56.3s, measured beside four other model runs; a solo August segment measured p50 about 19.6s). Still above the 15s p95 production target, still an open item. - Citation broadening is still visible: 10 of 120 grounded answers cited a document outside the gold set, one more than August's 9. Gold documents retrieved but uncited fell from 16 rows to 15. - The grafting attempt's 4 of 4 was a smoke replay of the four failing questions, not a full run, and it has no frozen snapshot. That is deliberate: it was never a candidate for the record. - The pointer detector was written against one synthetic corpus. Its title matching and contrast phrases are general in design, but they have only been exercised on the 65 Northwind documents. ## Evidence The frozen findings for the 118 run are [`findings.glm-5.3-flash.json`](https://github.com/wasimjalali/useful-brain/blob/main/evals/results/2026-09-06/findings.glm-5.3-flash.json) under `evals/results/2026-09-06/`. The source report is [Pointer-triggered coverage: 116 to 118 without fitting the corpus](https://github.com/wasimjalali/useful-brain/blob/main/evals/system-evals/2026-09-06-northwind-pointer-coverage.md), the detector is [`src/lib/agent/pointer-completion.ts`](https://github.com/wasimjalali/useful-brain/blob/main/src/lib/agent/pointer-completion.ts), and the reproduce commands are in the [evals index](https://github.com/wasimjalali/useful-brain/tree/main/evals). This is the third post about the same assistant. The first covers the repair from 72% to 95%: [From 72% to 95%: fixing an assistant that only quotes its sources](/writings/from-72-to-95-repairing-a-grounded-rag-agent/). The second ran five models through the same test and kept the cheapest of the two best: [Same score, one ninth the price: choosing a knowledge assistant's model](/writings/picking-a-model-for-a-grounded-rag-agent/).