AI Lab

Prototype

The reasoning was defensible. The answer was wrong.

A runnable wealth-management demonstration of why agent reliability depends on governed data and deterministic, firm-owned evals.

On a regulated-finance task, the same model produces two different answers because only one dataset encodes the firm's record of truth. A deterministic eval tied to that truth catches the difference, and no external benchmark can substitute for it.

The short version

The model returned £2,512,500. The firm-defensible answer was £1,512,500. It had double-counted a superseded account, not because it reasoned badly, but because the firm’s raw systems never recorded which account was the record of truth.

That gap is the argument. An agent’s answer on a regulated-finance task is bounded by the facts encoded in the firm’s data, and most of those facts, which account supersedes which, who is entitled to what, what each field means, are decisions the firm makes rather than anything a model can read off the records or recover from training. Encoding them is what I will call the arrangement: the governed transformation from operational records into an authorised, reconciled, semantically defined view. Arranging the data well makes the right answer possible. It does not, on its own, make the agent reliable.

I built a small, runnable demonstration on one wealth-management task: total a client’s assets under management. The same model is run first on the raw systems, then on the arranged view.

The task

A ledger of holdings, and a CRM that maps clients to accounts. A real firm has dozens more systems, but two is enough to make the point. The question: what is the client’s total AUM, in pounds?

What the model did on raw data

Handed the raw tables, the model worked carefully. It listed the client’s accounts, summed each one, noticed a holding was in euros and converted it at a sensible rate, even noting that “the exact rate would be provided in a real scenario.” It returned £2,512,500.

That figure is wrong by exactly one million pounds. The CRM links the client to ACC-100-OLD, a superseded copy of ACC-100, and nothing in the raw tables, or in the model’s training, says which is the record of truth. So it counted both.

Why the model could not help it

Then the same model, at the same settings, got the same question over arranged data: the canonical accounts only, values already normalised to pounds, the entitlement rule applied. It returned £1,512,500. Correct.

The arrangement carried the firm-specific facts the model could not infer. That unglamorous work appears on no leaderboard, but it is where the answer came from.

The control that caught it

One part of the demonstration matters as much as the arrangement. The error was caught by a deterministic check against a known-correct figure, not by a score and not by a second model grading the first.

A second model might catch an arithmetic slip or a misread instruction, but given the same evidence it could not know that ACC-100-OLD was superseded. The decisive control has to encode firm-owned truth, outside the model and outside the evidence the model was handed. In a regulated firm that is what an eval is for: a test you own, tied to an answer you can defend. Benchmarks and model graders still test real things; what they cannot do is stand in for a firm-specific eval bound to governed truth.

The arrangement makes the correct answer available. The eval determines whether the agent produced it.

The mechanics

The raw data is two tables. The ledger holds the positions:

The CRM links the client to three accounts: ACC-100, ACC-101 and ACC-100-OLD. That is everything the model gets in the raw run. A naive face-value sum of those tables produces £2,550,000, though it improperly adds euros to pounds. The model did better, converting the euro holding itself, which is how it reached £2,512,500. Either way it has no way to resolve the duplicate. That the model’s rate matched the 0.85 the arrangement later authorises is convenient, not essential: it isolates the version-of-record failure from any FX disagreement, so the remaining one million pounds has a single, unambiguous cause.

The arrangement adds three pieces of governed context absent from the two operational tables:

1. Version-of-record. ACC-100-OLD is a superseded snapshot of ACC-100. Only ACC-100 and ACC-101 are canonical.
2. Entitlement. Count only the client’s canonical accounts.
3. Semantic layer and reference data. Each market value is denominated in its own currency; normalise to pounds using the authorised FX rate (the euro line at 0.85).

Apply those and the reconciled view is ACC-100 at £1,000,000 plus ACC-101 at £512,500 (the €250,000 line becomes £212,500), for a correct total of £1,512,500. That is the only number the eval treats as right.

The eval is deliberately dull. It holds the golden figure and compares it to the typed total_aum_gbp field the model is required to return as JSON, passing only if the two match within a pound. A number scraped from prose would invite parser ambiguity; a typed field does not. The model, settings, question and output contract stay fixed. Only the evidence changes: operational tables in one run, the governed view in the other.

Run it

$ python run.py
  RAW data       model said £2,512,500   [FAIL]  off by +1,000,000 vs golden
  ARRANGED data  model said £1,512,500   [PASS]  matches golden

The model returns its working and a final {"total_aum_gbp": 1512500}; the eval reads that field and checks it against the golden total. The same command runs offline against canned responses, so the result reproduces without an API key.

What this is, and is not

It is a toy, deliberately. The numbers are synthetic and the task is small, but every rule in it is a real reconciliation concern: version-of-record, de-duplication, currency base.

The uncomfortable version, for anyone planning to put agents on regulated financial data: on any task whose answer depends on firm-specific state, model capability cannot compensate for facts the firm has not encoded. A better model next year still will not know which of your accounts is the record of truth. You have to tell it, and being able to tell it is an asset a firm either owns or does not.

Run in June 2026 at temperature zero, through Mistral’s mistral-large-latest alias (a mutable pointer, so the date fixes the run). Synthetic data only.