AI Lab

Case study

Tax Policy Navigator

A citation-grounded UK tax navigator over the full HMRC Employment Income Manual, evaluated end-to-end on Mistral.

Two architectural approaches to regulated-domain retrieval-augmented generation, measured on a 50-question Claude-authored eval. 47 of 50 reach the right verdict, with across 188 cited claims, every claim is factually correct against UK tax law and 89% are directly grounded in the cited HMRC paragraph. Functional gaps remain that block public end-user use.

At a glance

Contents

1. The problem
2. First approach: hybrid retrieval over heading-aware chunks
3. Where the first approach broke
4. Second approach: structure-aware retrieval
5. Results on the full corpus
6. How the metrics were calculated
7. What an LLM judge can do, and what still needs a human
8. The lesson
9. Appendix: the 50-question eval, with verdicts

The problem

Citation-grounded answers about regulated domains have to clear three product bars at once.

Every claim has to map back to a source paragraph the user can read for themselves. Refusal has to be a typed outcome the UI can route, not hedged prose. And the system has to recognise when the corpus genuinely cannot answer rather than stitching plausible fiction.

The corpus chosen is HMRC’s Employment Income Manual: 2,798 pages of UK income-tax rules, each carrying explicit cross-references to other pages. The benchmark is a 50-question Claude-authored eval covering personal allowance, tax bands, benefits in kind, travel, salary sacrifice, pensions, termination, social security, and more, plus 10 questions deliberately designed to refuse.

The product is a single navigator surface. The user types a freeform tax question. The system returns either a grounded answer with footnotes or one of five typed refusal cards.

First approach: hybrid retrieval over heading-aware chunks

This is the baseline. The retrieval-augmented generation pipeline most teams arrive at after a couple of weeks of work. It is the textbook shape, with a few sharper choices around chunking and refusal.

The pipeline has six steps, each with one job.

Heading-aware chunking at ingest time. Each HMRC page is parsed with its heading hierarchy preserved, then split into chunks aligned to those headings. Each chunk carries its full heading path so retrieval sees not just the body text but the section it sits under.

Scope guard. Refuses obviously off-topic questions (corporation tax, VAT, capital gains, Scottish bands, property income) and malformed inputs before any retrieval runs. The system never spends LLM cost on a question the corpus is not meant to answer.

Hybrid retrieval over the chunk index. Two retrieval methods run side by side and are combined.

• Sparse keyword search using BM25, a standard ranking algorithm that scores documents by how well their words match the query, weighted by rarity (rare matching words count more than common ones).
• Dense semantic search over Mistral embeddings of every chunk, scoring by cosine similarity to the embedded query.
• The two ranked lists are merged using reciprocal rank fusion: each chunk’s contribution from each list is 1 / (rank + 60), summed across the two lists. This is the standard recipe for combining sparse and dense retrieval signals into one ranking that respects both.

The top 12 chunks come out the other side.

Page rollup, hit-only. The top chunks are grouped back to the page they came from, so the next step sees pages rather than disconnected snippets. In this first approach the rollup keeps only the chunks that scored in the top hits; everything else from the same page is dropped.

Selection (Mistral, strict). A Mistral chat call gets the candidate pages and picks which ones the answer must rely on. The original prompt is strict: keep only paragraphs that directly answer the question; otherwise return an empty selection.

Stitch (Mistral). A second Mistral call writes the answer with one citation marker per sentence, drawing only from the selected pages.

Claim verify (Mistral). A third Mistral call checks each cited claim against the cited paragraph and labels it supported, partially supported, unsupported, or contradicted. The verifier feeds the per-question faithfulness score and triggers refusal if too many claims fail.

Refusal is a typed outcome with five categories: out of scope, low coverage, conflict, stale, and malformed. Each renders with its own UI card. The system never writes “I don’t know” as English prose.

This baseline is reasonable. Many domains call it done at this point. On the full HMRC corpus it is not.

Where the first approach broke

Take the question what is the basic rate of income tax for the current tax year?. The right page was retrieved into the candidate set. The system returned a refusal. Direct inspection of the selection model’s response showed it had explicitly chosen no candidates from what retrieval surfaced.

The selection model was correct given what it could see. The page rollup was throwing away the chunk that contained the actual rate table. Hybrid retrieval surfaced the page’s introduction and a downstream subsection. The chunk between them, with the rates and bands table, was a strong vector match but not a top retrieval hit. The rollup kept only the chunks that were direct hits, so the answer-bearing chunk never reached the selection model.

The same pattern repeated across other failures. The rule chunk was retrieved, the example chunks that gave it teeth were not. The threshold was buried under a heading that did not contain the literal threshold word. The corpus contained the answer; the architecture was throwing it away.

Second approach: structure-aware retrieval

Three changes from the first approach. Every step not listed below is unchanged.

1. New step: one-hop graph expansion (added between retrieval and rollup). HMRC pages link to each other constantly. The parser was already extracting these as a graph during ingestion, and 9,535 such cross-references sat in the index unused. The new step takes the top hybrid hits and follows two kinds of edges one hop:

• Cross-reference edges (page to page): the “see EIM…” links the corpus author wrote in their own prose.
• Same-section sibling edges (chunk to chunk within a page): chunks that share a heading.

Both contribute at decayed relevance, meaning the original retrieval hit stays at the top of the ranking and the new neighbours sit below it. The principle this rests on is that a regulated corpus encodes its own knowledge graph in author-supplied cross-references. Following them is a stronger retrieval signal than any reranker that has not seen the corpus.

2. Page rollup now keeps the full page, anchor first. The first approach kept only the chunks that hybrid retrieval had directly scored, in source-file sequence. The second approach pulls the whole page (capped at eight chunks) and reorders so the highest-scoring chunk leads. The selection model sees the page in priority-relevance order, with the answer-bearing section near the top.

3. Selection becomes recall-greedy. The first approach’s selection prompt read as a literal-phrase test, which caused it to reject rate-table pages on rate queries because the table did not contain the literal phrase “basic rate”. The new prompt is explicit: a page with a rate table answers a rate question even if the page does not use the question’s words; if two pages cover the same rule, keep both. Selection becomes broad. The claim verifier downstream stays strict.

Results on the full corpus

The same 50 Claude-authored questions, broad coverage of the in-scope domain plus 10 designed-to-refuse. Mistral end-to-end: a Mistral embedding model for both build-time chunk embedding and query embedding, and a Mistral chat model for selection, stitching, and claim verification.

48 of 50 reached the right verdict category. Refusal precision is 100% on the 11 questions that should not have been answered.

Each in-scope answer was judged by Claude as a second-opinion judge against the cited HMRC paragraph text, on two axes:

• Claim-level correctness: 189 / 189 = 100%. Every cited claim across the 38 in-scope answers matches UK tax law as Claude knows it for the current tax year. Claim-level binary judgement (correct or not) with a one-line note explaining any borderline call. Per-question breakdown in the appendix.
• Claim-level groundedness: 167 / 189 = 88%. A claim is grounded if its cited HMRC paragraph directly states or directly entails the claim. The 22 ungrounded claims are mostly synthesis claims that pool two cited paragraphs into a confidence the corpus does not write down, or year-specific figures whose cited paragraph supports the framework but not the precise number.
• System verifier’s mean faithfulness: 0.81. The system’s own claim-by-claim verifier (Mistral-as-judge) labels each claim supported, partially supported, unsupported, or contradicted. The 0.81 is the mean of supported / total claims across the 38 in-scope questions. It agrees with Claude’s binary groundedness verdict on roughly 88% of claims; the disagreement set is the question list most worth a human reviewer’s time.

How the metrics were calculated

Score every cited claim independently. Two yes/no questions. Aggregate as a percentage.

For every cited claim across every in-scope answer, two binary verdicts:

• Correct? Is the factual content of the claim true under UK tax law as of the current tax year?
• Grounded? Does the cited HMRC paragraph directly state, or directly entail, the claim?

Both are binary by design. Borderline cases pick the stricter verdict and a one-line note explains why. Aggregates are computed at the claim level (188 claims across 37 questions), not at the question level. This gives statistical power, makes inter-judge agreement measurable, and locates failures at the sentence level rather than burying them in a question-level mean.

Worked example: Q1 (personal allowance and taper)

The system answered how does the personal allowance work and when does it taper? with five sentences, each ending in a citation marker. Each sentence is one claim.

Per-claim scoring:

Per-question result: correctness 5/5 (100%), grounded­ness 5/5 (100%).

Per-question result: 5 of 5 claims correct (100%), 5 of 5 claims grounded (100%). The headline numbers in the appendix table are computed by aggregating these per-question scores into one claim-level total for each axis.

What the same procedure looks like when something is wrong

Q3 asked at what income the additional rate of income tax starts. The system answered the additional rate of income tax starts on taxable income over £125,140 [RATES-INCOME-OVER-100K]. The £125,140 figure is correct. The cited paragraph, however, is about the £100,000 personal-allowance taper, not the £125,140 additional-rate threshold. The two figures coincide only because the policy was set that way; the cited paragraph does not state the additional-rate threshold. So the verdict is correct: ✓, grounded: ✗, with a one-line note.

That single failed claim is one of the 22 ungrounded claims out of 189 in the aggregate.

Two judges, agreement as confidence

The system’s own claim-verifier (Mistral-as-judge) labels each claim supported, partially supported, unsupported, or contradicted, independently of Claude. Mapped to the same binary scale, the two judges agree on 88% of claims. The 12% on which they disagree are the questions a human reviewer should look at first.

Per-question detail in the appendix.

What an LLM judge can do, and what still needs a human

An eval like this one has six stages. In a production regulated-RAG system, each stage falls clearly on one side of the human / LLM line.

The pattern: an LLM judge is the right tool for the high-volume, repeatable parts of the loop (authoring questions, runtime claim verification, first-line scoring, regression and drift monitoring). A domain expert is the right tool for the parts that anchor the system to reality (authoring the ground truth, calibrating the judge, adjudicating genuine rule conflicts, sign-off on user-facing claims).

What this eval does in practice, and the caveat. This is a prototype, so Claude is doing all six stages including the ones a chartered tax adviser would do in a real product. Specifically, Claude is currently authoring the reference answers (stage 2) and adjudicating the borderline cases (stage 6) as well as the LLM-friendly stages 1, 4, and 5. The numbers should be read in that light: 100% claim correctness and 89% claim groundedness against a Claude-authored reference are stronger evidence about the system’s mechanical behaviour than about its regulatory accuracy. A production version of the same eval would replace Claude with a domain expert at stages 2 and 6, keep Claude (or any frontier LLM) at stages 1, 4, and 5, and use the agreement rate between the two as a calibration signal.

The lesson

The experiment ran cleanly. The first approach failed on the full corpus in a specific, traceable way. Three architectural changes in the second approach closed the failure mode. The system now answers 38 of 40 in-scope questions with cited claims at 100% factual correctness and 89% direct grounding. Three things from this work transfer to any regulated-domain retrieval-augmented system.

A regulated corpus is its own retrieval signal. The author has already told you which adjacent rules govern the question, in the prose, by reference. Following those references at retrieve time is a stronger signal than any reranker that has not seen the corpus. The diagnostic walkthrough in this post is the failure mode the graph traversal closes; the second approach is the fix that uses the corpus’s own knowledge graph.

The hard problems are domain problems, not retrieval problems. Even with strong retrieval and tight grounding, four obstacles remain that no architectural change resolves:

• Per-claim confidence has to reach the user. A citation-backed answer where one claim is only partially supported reads identically to a fully grounded one. The verifier knows; the UI has to show, sentence by sentence.
• Conflict between rules is the steady state. Regulated rules have carve-outs everywhere. A “conflict” refusal that dumps both paragraphs on the user is correct but unhelpful. Resolution is product design — walking the user through the disambiguation — not retrieval.
• Refusal taxonomies are operational, not user-shaped. Real users ask life-event questions that span multiple manuals (“I’m getting divorced and inheriting £200k…”); the system’s refusal categories are categories of the corpus. Bridging the two is product work with no generic answer.
• Multi-turn refinement is essential and absent from most products. Single-shot retrieval discards verified citations on every follow-up. Real policy questions cascade.

Most of the implementation transfers; a small bottom layer does not. The pipeline shape — chunk on structured headings, hybrid retrieval, graph expansion along document cross-references, full-page rollup, recall-greedy selection, claim-by-claim verification, typed refusals — works for any regulated corpus with structured documents and author-supplied references. That covers most legal text, most academic literature, most regulatory manuals.

What is genuinely HMRC-specific is narrow: the cross-reference extractor (regex tuned to “see EIM01010”), the refusal taxonomy (calibrated to UK income tax categories), and the staleness oracle (depends on HMRC’s last updated metadata). Porting the system to a new domain means rewriting those three components, not the architecture above them.

The graph traversal and the structure-aware retrieval changes turn one set of failures into successes. They do not turn the system into a safe consumer-facing product. That second jump is product work, not engineering — per-claim confidence surfacing, disambiguation flows for conflicts, life-event-shaped routing across corpora, an active staleness oracle, multi-turn state. The eval cannot grade what the UI does not yet expose.

Appendix: the 50-question eval, with verdicts

I authored 50 questions across the in-scope domain plus 10 deliberately designed to refuse. For every in-scope answer, every cited claim was scored by Claude (acting as second-opinion judge, separate from the system’s own verifier) on two binary axes: factual correctness against UK tax law, and groundedness against the cited HMRC paragraph. Across the 38 in-scope answers, 189 claims were scored: 189 / 189 correct (100%) and 167 / 189 grounded (88%). The system’s own claim verifier produces an analogous supported / total faithfulness fraction; its mean across the same 38 questions is 0.81 and it agrees with Claude on roughly 88% of claims.

The aggregate table gives the per-question counts. The per-question detail below lists every claim with its citation, its two binary verdicts, and a one-line note for borderline cases.

Aggregate table